How-to template — a keyword scraper

Side post for "How to use analyst-server" (pua-4-side-posts, criterion pua-4-how-to-templates). This is a template: the six stops from part 2, filled in for one use case so you can copy it. Swap the words in ‹angle brackets› for your own. The screens are the same ones part 2 describes. Kind of work: Research (understand first). Drafted 2026-10-08.

Why use the path for a scraper? A scraper produces a lot of text quickly, and most of it is noise. Running it as an initiative makes you say up front what question the keywords answer. Then the briefing reports what was found that matters, not how many pages were fetched.

① say it ──► ② read it back ──► ③ place it ──► ④ watch it build ──► ⑤ judge it ──► ⑥ change it
  "watch ‹sites›   "does it name      under Research   lane "scrape-‹name›"  "did it surface    "drop a noisy
   for ‹words›"     the question?"                                          something new?"   site, add a word"

1 · Builder session — state the intent

Intent text to copy:

"I want to know when ‹any of these five county sites› posts anything that mentions ‹'bid', 'RFP' or 'handyman'›, so I can respond within ‹two days›. Check ‹daily›, keep a copy of each hit, and only bring me the new ones."

Width check. Ask the session: is there already a watch list or standing question that covers this? Is this a one-off look or a standing watch? Are the sites ones you are allowed to fetch on a schedule? A standing watch belongs under Research, beside the box's other standing questions. A one-off look is a question for the session, not an initiative.

The tree you accept:

big-init-scrape-‹name›         "New ‹bid› notices on ‹sites› reach me within ‹two days›"
├── 1 · Sources        the site list, each with how it is fetched and how often
├── 2 · Keywords       the words and phrases, and what a false hit looks like
├── 3 · Hits           where each hit is kept, deduplicated, with its source and date
├── 4 · Delivery       only NEW hits, in the briefing, inside the response window
└── 9 · Operations     derived: what is owed, by whom, in what order

Example criterion, 3 · Hits: "A notice is stored once, however often it is seen — REACH-TEST: running the scrape twice on the same day adds zero new hit records the second time."

Briefing you describe: "Daily. First line: how many new hits since yesterday, and the oldest one I haven't answered. Then each new hit with its site, date and the sentence that matched. Then any site that failed to fetch."

2 · Briefing draft — read it back

What the draft should say, first two lines:

Intent: new ‹bid› notices on ‹five county sites› reach me within ‹two days›. Verdict: Started — 2 of 5 sites fetched by hand; keywords not yet tested for false hits.

A draft that leads with "312 pages fetched" is reporting effort. Ask for the count of new hits.

3 · Workbench — place it and give it a lane

Open the workbench at Research (?root=big-init-nature-research) and check:

• it does not duplicate a standing question or watch list already there;
• Sources says how each site is fetched: a plain request for most sites, and the scraping tools only for a site that blocks plain requests. A paid proxy spends money, so that choice is yours;
• the schedule it will run on is named under Sources or Delivery, not left implicit.

Lane: scrape-‹name›.

4 · Lane activity — watch it build

Filtered to scrape-‹name›, expect: each site fetched once by hand, the keyword list tested against a known notice, the dedupe test, then the first scheduled day. A site that keeps failing shows up here before it shows up as a missed bid.

5 · Initiative briefing report — is the intent fulfilled?

What the filled-in report should show:

The judge's question: did it bring you a notice you would otherwise have missed, in time to answer it? A month of zero hits might mean the sites are quiet, or it might mean the keywords are wrong. The report should let you tell the difference by showing the sites were fetched.

6 · Builder session — request changes

Typical ask to copy:

"Half the hits are 'handyman' in job ads, not bids. Drop that word unless it is next to 'bid' or 'quote', and add the city of ‹name›'s procurement page."

The session files it verbatim as an upgrade proposal. The amended Keywords and Sources criteria appear on the workbench, and the loop runs again from stop ③.

Back to the side-posts list: 04-side-posts.md · The six stops in general: 02-screen-path.md