270fae1a02
A full pass is ~6,500 requests over a quarter of an hour, and it was killed three times by system memory pressure - twice at around 2,500 and once at 1,000. Results lived only in memory until the final write, so each kill threw away every request it had made. The file on disk survived, which is correct, but the pass never landed and the bandwidth was spent for nothing. Every record now carries `at`, the moment it was checked, and the table is written after each batch. A killed run loses at most one batch, and rerunning with VERIFY_MAX_AGE_H skips anything already stamped. Writing mid-pass is safe because a checkpoint only ADDS to or REFRESHES records - it can never shrink the table, which is the failure the snapshot guard exists for elsewhere. Proven rather than asserted: killed mid-pass, 300 stamped records survived on disk; the restart reported "resuming - 600 URL(s) already checked within 24h, 5876 left to do" and skipped them. The fourth memory kill then arrived unplanned during the real run and cost 500 records instead of 1,900. With the pass finally complete, the point of it: bible.is verification switched from a body-size heuristic to their API two changes ago, but only the URLs added since had been re-checked. 2,609 of 2,868 records still asserted "verified" on the older basis. All 2,708 now come from /api/bibles, and none from the heuristic. The last 160 were neither shipped nor broken - leftovers from filesets no build has referenced since the FCBH ranking changed, quietly carrying the old method. A full pass now prunes any record for a URL that nothing ships and nothing proved dead, so the table means exactly one thing: every URL this map currently ships, plus the history of everything ever proven dead. Broken records are never pruned; forgetting a dead link is how it gets shipped again. That prune sat behind an early exit that fired whenever nothing was stale, so the first run to reach it reported success and pruned nothing. Removed. Table: 6,476 records, 6,397 verified, 74 broken and dropped, 5 uncheckable, 0 unstamped, 0 on the old method. |
||
|---|---|---|
| .. | ||
| external-bundles | ||
| people-groups | ||
| posters | ||
| tables | ||
| vendor | ||
| README.md | ||
data/ — the bundler's source-of-truth data
The law: data lives here; public/ is generated. Every folder in data/ is copied into the
served web-root by the build (internal/generate-tree.mjs), exactly like the per-profile data in
plugin-designs/*/data/. You should be able to rm -rf public/ and get it all back with bun run build.
| Folder | Served at | What it is |
|---|---|---|
people-groups/ |
/assets/people-groups/ |
People-group datasets (CSV/JSON) the maps read at runtime |
vendor/ |
/assets/vendor/ |
Third-party runtime assets (e.g. table libraries) |
posters/ |
/assets/posters/ |
Rendered poster webp set (heavy pipeline output; poster walls fetch these) |
tables/ |
/assets/tables/ |
The MERGED served table set (its index.json is the merge; per-profile plugin-designs/*/data/tables are build-time sources) |
external-bundles/ |
not served | Prebuilt map IIFEs vendored from sibling projects (not built here); only the optional layouts/ shell mounted them, so the build no longer copies them into public/ |
Don't edit anything under public/ by hand — change it here (or in plugin-designs/*/data/,
layouts/*/assets/) and rebuild.