0e0b142719
The FCBH fetcher read per-fileset video scope for 1,666 languages and that file was
committed. A later run, while live.bible.is was rate-limiting, produced 1,103 and
overwrote the good file with the degraded one. Nothing failed, nothing warned, and the
artefact silently lost a third of its content. It was caught only because git status
happened to be read, and the fix was a git checkout.
That is the third time this shape has appeared here:
- a VERIFY_LIMIT smoke run replaced the full verification table with 30 rows;
- dropping a dead link removed it from the corpus, so the next run forgot it was
dead and the build put it straight back;
- this one.
Every snapshot in this bundle is built from a network read, and a network read has a bad
day. The dangerous case is never the run that throws - that leaves the old file alone -
it is the run that half-succeeds and writes anyway.
scripts/_snapshot-guard.mjs now stands in front of every catalogue write. A snapshot may
grow freely and may shrink a little, but losing more than a tenth of the previous run is
treated as a failed fetch: the existing file is left untouched and the script exits 1.
It is deliberately not an equality check, because catalogues do genuinely lose entries
when a publisher retires something, and a gate nobody can satisfy gets switched off.
SNAPSHOT_FORCE=1 is the one loud way past it.
Proven by replaying the exact failure: feeding it the 1,103-entry version refuses the
write, names the real numbers (563 fewer, 13.8%, below the 90% floor) and leaves the
file byte-identical to the good one. SNAPSHOT_FORCE=1 writes it. An equal write passes.
All six fetchers are wired: ebible, arclight, arclight-share-urls, door43, zume, fcbh.
Re-running zume, door43 and ebible for real passes the guard at 52, 208 and 1,237 codes
and produces byte-identical data, so the guard perturbs nothing.
The one exception is the checkpoint write in the resumable share-urls fetcher, which
starts from the existing file as a cache and can only grow. It must now declare itself
with a CHECKPOINT-WRITE-OK marker, so the exemption is a statement rather than an
oversight.
New gate check:snapshot-guard fails if any fetch-*.mjs writes OUT directly without that
marker, or never calls writeSnapshot at all - a guard that is easy to bypass is a guard
that will be bypassed. It refuses to run against an empty file list rather than passing
vacuously, and --poison injects a raw write and requires the check to catch it. Chained
into build:unlocked and its doc copy kept in sync.
68 lines
3.5 KiB
JavaScript
68 lines
3.5 KiB
JavaScript
/**
|
|
* _snapshot-guard.mjs — refuse to replace a good catalogue snapshot with a worse one.
|
|
*
|
|
* THE FAILURE THIS EXISTS FOR, MEASURED, NOT IMAGINED. The FCBH fetcher read per-fileset
|
|
* video scope for 1,666 languages and that file was committed. A later run of the same
|
|
* script, while live.bible.is was rate-limiting, got 1,103 — and overwrote the good file
|
|
* with the degraded one. Nothing failed, nothing warned, and the artefact silently lost a
|
|
* third of its content. It was caught only because `git status` happened to be read.
|
|
*
|
|
* It is the third time this shape has appeared in this bundle:
|
|
* - a VERIFY_LIMIT smoke run replaced the full verification table with 30 rows;
|
|
* - dropping a dead link removed it from the corpus, so the next run forgot it was dead
|
|
* and the build put it straight back;
|
|
* - this one.
|
|
* Every snapshot here is built from a NETWORK read, and a network read has a bad day. The
|
|
* dangerous case is not the run that throws — that leaves the old file alone — it is the
|
|
* run that half-succeeds and writes anyway.
|
|
*
|
|
* THE RULE: a snapshot may grow freely, and may shrink a little, but a large drop is
|
|
* treated as a failed run rather than as news. The count compared is whatever the caller
|
|
* says identifies the snapshot's content (languages, codes, entries).
|
|
*
|
|
* Deliberately NOT a hard equality check: catalogues genuinely do lose entries when a
|
|
* publisher retires something, and a gate nobody can satisfy gets switched off. The
|
|
* threshold is a floor on the fraction retained, and it is overridable in one obvious,
|
|
* loud way for the case where the drop is real.
|
|
*/
|
|
import { readFileSync, writeFileSync } from 'node:fs'
|
|
|
|
/** Default: a run that lost more than 10% of the previous snapshot is not trusted. */
|
|
const FLOOR = Number(process.env.SNAPSHOT_FLOOR || 0.9)
|
|
const FORCE = process.env.SNAPSHOT_FORCE === '1'
|
|
|
|
/**
|
|
* @param {string} out absolute path to write
|
|
* @param {string} body the serialised snapshot
|
|
* @param {(json:any)=>number} countOf how to count what matters in this snapshot
|
|
* @param {string} label what the count is, for the message
|
|
*/
|
|
export function writeSnapshot(out, body, countOf, label) {
|
|
const next = countOf(JSON.parse(body))
|
|
let prev = null
|
|
try { prev = countOf(JSON.parse(readFileSync(out, 'utf8'))) } catch { prev = null }
|
|
|
|
if (prev !== null && prev > 0 && next < prev * FLOOR) {
|
|
const lost = prev - next
|
|
const pct = ((100 * lost) / prev).toFixed(1)
|
|
if (!FORCE) {
|
|
console.error(
|
|
`SNAPSHOT GUARD: REFUSING TO WRITE ${out}\n` +
|
|
` this run produced ${next} ${label}; the file on disk has ${prev}.\n` +
|
|
` That is ${lost} fewer (${pct}%), below the ${(FLOOR * 100).toFixed(0)}% floor.\n` +
|
|
` A half-finished network read looks exactly like this, and overwriting would\n` +
|
|
` destroy good data silently. The existing file is UNCHANGED.\n` +
|
|
` If the drop is real (a publisher retired entries), rerun with SNAPSHOT_FORCE=1.`,
|
|
)
|
|
process.exit(1)
|
|
}
|
|
console.warn(`SNAPSHOT GUARD: ${next} ${label} vs ${prev} on disk (${pct}% lost) — writing anyway because SNAPSHOT_FORCE=1.`)
|
|
writeFileSync(out, body)
|
|
return
|
|
}
|
|
|
|
writeFileSync(out, body)
|
|
if (prev === null) console.log(`SNAPSHOT GUARD: no previous ${out} to compare; wrote ${next} ${label}.`)
|
|
else if (next < prev) console.log(`SNAPSHOT GUARD: ${next} ${label} (was ${prev}) — a small drop, within the floor.`)
|
|
else console.log(`SNAPSHOT GUARD: ${next} ${label} (was ${prev}).`)
|
|
}
|