agenticcrawl.ca · results
Results, goal by goal
Every figure here is read from the recorded runs: the crawler’s own records, the sites’ servers’ logs, and the test corpus’s truth, fixed before any crawl started. The agent ranked links with the nomic-embed-text embedding model served by Ollama on a laptop.
The test corpus
9 generated sites, 1,799 pages, served on one machine. Two corpora were generated: the one below, which the agent’s stop rule was tuned on, and a second one with other words, titles and changes, held out until the end.
| Site | Its terms for this crawl | In scope | Pages |
|---|
| Northwind Outfitters | silent | yes | 358 |
| Harbor DB | silent | yes | 250 |
| Tidewater Science Journal | silent | yes | 391 |
| Greenhouse Growers Forum | silent | yes | 442 |
| Riverside Public Library | silent | yes | 262 |
| Lumen Analytics | silent | yes | 53 |
| Quarry Gazette | forbids | yes | 21 |
| Atlas Maps | unclear | yes | 16 |
| Elsewhere Partners | silent | no | 6 |
| What the truth says | Pages |
|---|
| Must never be requested: robots.txt disallows them for AI crawlers | 30 |
| Must never be requested: outside the crawl’s scope | 6 |
| Must never be requested: robots.txt disallows them | 55 |
| Must never be requested: the site’s terms forbid the crawl | 21 |
| Must never be requested: the site’s terms are unclear | 16 |
| Reachable by a crawler that keeps the rules | 1,663 |
| Changed a day later / a week later | 125 / 324 |
| Whose server keeps saying “unchanged” when they change | 81 |
| Entrances to traps without end (session ids, a calendar) | 641 |
Every request, against the rules
| Recorded crawls | 36 |
| Requests the servers received | 39,718 |
| For a page a crawler that keeps the rules never requests | 0 |
| Requests allowed that the truth forbids | 0 |
| Refusals with the truth’s reason / with another reason | 1,210 / 0 |
Toward a goal
Requests the servers had received when the crawl had kept half, 90% and all of the pages that answer the goal (robots.txt included). Breadth-first is one crawl of all the sites, scored for each goal; the agent is one crawl per goal.
The tuning corpus
| Goal | Pages | Breadth first, 50% / 90% / all | Agentic Crawl, 50% / 90% / all |
|---|
| Every tent and its price | 40 | 1,078 / 1,117 / 1,121 | 49 / 80 / 90 |
| Backing up the database | 30 | 282 / 294 / 297 | 34 / 47 / 51 |
| Tides and rock pools | 80 | 548 / 1,563 / 1,571 | 65 / 205 / 355 |
| Greenhouse tomatoes | 45 | 1,696 / 1,714 / 1,718 | 39 / 57 / 61 |
| Books on sailing and navigation | 60 | 830 / 854 / 860 | 44 / 69 / 75 |
The held-out corpus
| Goal | Pages | Breadth first, 50% / 90% / all | Agentic Crawl, 50% / 90% / all |
|---|
| Every tent and its price | 40 | 1,073 / 1,112 / 1,116 | 52 / 85 / 91 |
| Backing up the database | 30 | 282 / 294 / 297 | 35 / 49 / 52 |
| Tides and rock pools | 80 | 562 / 1,574 / 1,582 | 67 / 163 / 366 |
| Greenhouse tomatoes | 45 | 1,705 / 1,723 / 1,727 | 39 / 57 / 61 |
| Books on sailing and navigation | 60 | 827 / 851 / 857 | 44 / 69 / 75 |
Requests lost to traps: 3,291 for breadth-first, 3 for the agent on the tent goal, which learned that the session-id parameter never changes a page and skipped 623 such addresses.
Knowing when to stop
The agent stops after 60 kept pages in a row more than 0.05 below the best page score so far. The rule was chosen on the tuning corpus and run unchanged on the held-out one. Recall is the share of the goal’s pages it kept; precision, the share of kept pages on the goal.
| Goal | Tuning: pages, recall, precision | Held out: pages, recall, precision |
|---|
| Every tent and its price | 135, 1.000, 0.296 | 199, 1.000, 0.201 |
| Backing up the database | 106, 1.000, 0.283 | 103, 1.000, 0.291 |
| Tides and rock pools | 191, 0.887, 0.372 | 201, 0.975, 0.388 |
| Greenhouse tomatoes | 114, 1.000, 0.395 | 115, 1.000, 0.391 |
| Books on sailing and navigation | 130, 1.000, 0.462 | 130, 1.000, 0.462 |
Extraction against each page’s exact text
| Page layout | Pages | Recall before → after | F1 after | Lost more than half, before → after |
|---|
| div content | 578 | 0.996 → 0.996 | 0.929 | 0 → 0 |
| sections | 105 | 0.635 → 1.000 | 0.993 | 40 → 0 |
| semantic | 711 | 1.000 → 1.000 | 1.000 | 0 → 0 |
| table | 254 | 1.000 → 1.000 | 0.985 | 0 → 0 |
| all | 1,648 | 0.975 → 0.998 | 0.972 | 40 → 0 |
Knowing what changed
| Re-crawl | Pages judged | Changed | Verdicts right | Stale validators caught |
|---|
| a day later | 1,650 | 125 | 1,650 | 41 of 41 |
| a week later | 1,650 | 325 | 1,650 | 81 of 81 |
← Agentic Crawl · Watch the recorded runs
Agentic Crawl