A full pass is ~6,500 requests over a quarter of an hour, and it was killed three
times by system memory pressure - twice at around 2,500 and once at 1,000. Results
lived only in memory until the final write, so each kill threw away every request it
had made. The file on disk survived, which is correct, but the pass never landed and
the bandwidth was spent for nothing.
Every record now carries `at`, the moment it was checked, and the table is written
after each batch. A killed run loses at most one batch, and rerunning with
VERIFY_MAX_AGE_H skips anything already stamped. Writing mid-pass is safe because a
checkpoint only ADDS to or REFRESHES records - it can never shrink the table, which is
the failure the snapshot guard exists for elsewhere.
Proven rather than asserted: killed mid-pass, 300 stamped records survived on disk;
the restart reported "resuming - 600 URL(s) already checked within 24h, 5876 left to
do" and skipped them. The fourth memory kill then arrived unplanned during the real
run and cost 500 records instead of 1,900.
With the pass finally complete, the point of it: bible.is verification switched from a
body-size heuristic to their API two changes ago, but only the URLs added since had
been re-checked. 2,609 of 2,868 records still asserted "verified" on the older basis.
All 2,708 now come from /api/bibles, and none from the heuristic.
The last 160 were neither shipped nor broken - leftovers from filesets no build has
referenced since the FCBH ranking changed, quietly carrying the old method. A full
pass now prunes any record for a URL that nothing ships and nothing proved dead, so
the table means exactly one thing: every URL this map currently ships, plus the
history of everything ever proven dead. Broken records are never pruned; forgetting a
dead link is how it gets shipped again.
That prune sat behind an early exit that fired whenever nothing was stale, so the
first run to reach it reported success and pruned nothing. Removed.
Table: 6,476 records, 6,397 verified, 74 broken and dropped, 5 uncheckable, 0
unstamped, 0 on the old method.
270fae1a02