From 0e0b142719a2c973f5fb26e2d984fcf44e28b438 Mon Sep 17 00:00:00 2001 From: doxa-maps-88655c70 <88627970+doxa-maps-88655c70@users.noreply.github.com> Date: Thu, 24 Sep 2026 11:59:48 +0000 Subject: [PATCH] A half-finished catalogue fetch can no longer overwrite a good one The FCBH fetcher read per-fileset video scope for 1,666 languages and that file was committed. A later run, while live.bible.is was rate-limiting, produced 1,103 and overwrote the good file with the degraded one. Nothing failed, nothing warned, and the artefact silently lost a third of its content. It was caught only because git status happened to be read, and the fix was a git checkout. That is the third time this shape has appeared here: - a VERIFY_LIMIT smoke run replaced the full verification table with 30 rows; - dropping a dead link removed it from the corpus, so the next run forgot it was dead and the build put it straight back; - this one. Every snapshot in this bundle is built from a network read, and a network read has a bad day. The dangerous case is never the run that throws - that leaves the old file alone - it is the run that half-succeeds and writes anyway. scripts/_snapshot-guard.mjs now stands in front of every catalogue write. A snapshot may grow freely and may shrink a little, but losing more than a tenth of the previous run is treated as a failed fetch: the existing file is left untouched and the script exits 1. It is deliberately not an equality check, because catalogues do genuinely lose entries when a publisher retires something, and a gate nobody can satisfy gets switched off. SNAPSHOT_FORCE=1 is the one loud way past it. Proven by replaying the exact failure: feeding it the 1,103-entry version refuses the write, names the real numbers (563 fewer, 13.8%, below the 90% floor) and leaves the file byte-identical to the good one. SNAPSHOT_FORCE=1 writes it. An equal write passes. All six fetchers are wired: ebible, arclight, arclight-share-urls, door43, zume, fcbh. Re-running zume, door43 and ebible for real passes the guard at 52, 208 and 1,237 codes and produces byte-identical data, so the guard perturbs nothing. The one exception is the checkpoint write in the resumable share-urls fetcher, which starts from the existing file as a cache and can only grow. It must now declare itself with a CHECKPOINT-WRITE-OK marker, so the exemption is a statement rather than an oversight. New gate check:snapshot-guard fails if any fetch-*.mjs writes OUT directly without that marker, or never calls writeSnapshot at all - a guard that is easy to bypass is a guard that will be bypassed. It refuses to run against an empty file list rather than passing vacuously, and --poison injects a raw write and requires the check to catch it. Chained into build:unlocked and its doc copy kept in sync. --- .../README-SECTION-3-cli-commands.md | 2 +- internal/check-snapshot-guard.mjs | 69 +++++++++++++++++++ package.json | 3 +- .../scripts/_snapshot-guard.mjs | 68 ++++++++++++++++++ .../scripts/fetch-arclight-index.mjs | 6 +- .../scripts/fetch-arclight-share-urls.mjs | 10 ++- .../scripts/fetch-door43-index.mjs | 4 +- .../scripts/fetch-ebible-index.mjs | 4 +- .../scripts/fetch-fcbh-index.mjs | 7 +- .../scripts/fetch-zume-index.mjs | 6 +- 10 files changed, 164 insertions(+), 15 deletions(-) create mode 100644 internal/check-snapshot-guard.mjs create mode 100644 plugin-designs/gospel-resources/scripts/_snapshot-guard.mjs diff --git a/documentation/README-SECTION-3-cli-commands.md b/documentation/README-SECTION-3-cli-commands.md index 5bc7a69a8..120099a3f 100644 --- a/documentation/README-SECTION-3-cli-commands.md +++ b/documentation/README-SECTION-3-cli-commands.md @@ -25,7 +25,7 @@ The scripts these commands are grounded in (from this branch's `package.json`): // claiming to be ground truth, and this one had silently stopped being it. "dev": "bun x vite", "build": "bash internal/build-lock.sh", -"build:unlocked": "bun run check:template-literal && bun run check:mjs-parse && bun run check:coverage-consistent && bun run check:sfc-parse && bun run check:tab-radius && bun run check:synced-css && bun run check:wp-embed && bun run check:doc-scripts && bun run library-index && bun run check:library-index && bun run clean && bun run token && for d in plugin-designs/*/; do BUNDLE=$(basename \"$d\") bun x vite build || exit 1; done && bun run tree && bun run check:drop && bun run check:parse && bun run verify && bun run check:sri && bun run check:public-hygiene && bun run check:label-joins && bun run check:facet-cascade && bun run check:facet-one-field && bun run check:heartbeat && bun run check:kpi-unit && bun run check:composite-reg && bun run check:descriptions && bun run check:no-abs-paths && bun run check:no-codename && bun run check:dep-names && bun run check:composite-srcs && bun run check:embed-targets && bun run check:nested-token && bun run gallery", +"build:unlocked": "bun run check:template-literal && bun run check:mjs-parse && bun run check:coverage-consistent && bun run check:snapshot-guard && bun run check:sfc-parse && bun run check:tab-radius && bun run check:synced-css && bun run check:wp-embed && bun run check:doc-scripts && bun run library-index && bun run check:library-index && bun run clean && bun run token && for d in plugin-designs/*/; do BUNDLE=$(basename \"$d\") bun x vite build || exit 1; done && bun run tree && bun run check:drop && bun run check:parse && bun run verify && bun run check:sri && bun run check:public-hygiene && bun run check:label-joins && bun run check:facet-cascade && bun run check:facet-one-field && bun run check:heartbeat && bun run check:kpi-unit && bun run check:composite-reg && bun run check:descriptions && bun run check:no-abs-paths && bun run check:no-codename && bun run check:dep-names && bun run check:composite-srcs && bun run check:embed-targets && bun run check:nested-token && bun run gallery", "preview": "bun x vite preview", "serve": "node internal/serve-strict.mjs", // 404s a directory, like production "cap:sync": "./node_modules/.bin/cap sync android", diff --git a/internal/check-snapshot-guard.mjs b/internal/check-snapshot-guard.mjs new file mode 100644 index 000000000..b7d06eb80 --- /dev/null +++ b/internal/check-snapshot-guard.mjs @@ -0,0 +1,69 @@ +#!/usr/bin/env node +/** + * check-snapshot-guard.mjs — every catalogue fetcher must write through the snapshot guard. + * + * WHY. A fetcher writes a snapshot built from a NETWORK read, and a network read has a bad + * day. The dangerous case is not the run that throws — that leaves the old file alone — it + * is the run that half-succeeds and writes anyway. Measured in this bundle: the FCBH fetcher + * read video scope for 1,666 languages, was committed, then a later run during a rate-limit + * produced 1,103 and silently overwrote the good file. Nothing failed and nothing warned. + * + * scripts/_snapshot-guard.mjs refuses a write that loses more than a tenth of the previous + * snapshot. This check exists so the next fetcher someone adds cannot quietly skip it: a + * guard that is easy to bypass is a guard that will be bypassed. + * + * THE ONE ALLOWED EXCEPTION is a checkpoint write in a resumable fetcher, which starts from + * the existing file as a cache and can only grow. It must say so on the line above, with the + * marker below, so the exemption is a deliberate statement rather than an oversight. + * + * Self-test: --poison injects a raw write and confirms this check fails on it. + */ +import { readFileSync, readdirSync } from 'node:fs' +import { resolve, dirname } from 'node:path' +import { fileURLToPath } from 'node:url' + +const HERE = dirname(fileURLToPath(import.meta.url)) +const DIR = resolve(HERE, '..', 'plugin-designs/gospel-resources/scripts') +const POISON = process.argv.includes('--poison') +const MARKER = 'CHECKPOINT-WRITE-OK' + +const files = readdirSync(DIR).filter((f) => /^fetch-.*\.mjs$/.test(f)) +if (!files.length) { + console.error('SNAPSHOT-GUARD-CHECK: FAIL — no fetch-*.mjs found; this check would pass vacuously.') + process.exit(1) +} + +const bad = [] +let guarded = 0 +let exempt = 0 +for (const f of files) { + let src = readFileSync(resolve(DIR, f), 'utf8') + if (POISON && f === files[0]) src += '\nwriteFileSync(OUT, JSON.stringify(out))\n' + const lines = src.split('\n') + let usesGuard = false + lines.forEach((line, i) => { + if (/writeSnapshot\s*\(/.test(line)) usesGuard = true + if (!/writeFileSync\s*\(\s*OUT\b/.test(line)) return + // a checkpoint write must be declared on one of the three lines above it + const above = lines.slice(Math.max(0, i - 4), i).join('\n') + if (above.includes(MARKER)) { exempt++; return } + bad.push(`${f}:${i + 1} writes OUT directly with writeFileSync and carries no ${MARKER} marker`) + }) + if (usesGuard) guarded++ + else bad.push(`${f} never calls writeSnapshot() — its snapshot is unguarded`) +} + +if (bad.length) { + console.error(`SNAPSHOT-GUARD-CHECK: FAIL — ${bad.length} problem(s):`) + for (const b of bad) console.error(` ${b}`) + if (POISON) { + console.log('SNAPSHOT-GUARD-CHECK: SELF-TEST PASS — the injected raw write was caught.') + process.exit(0) + } + process.exit(1) +} +if (POISON) { + console.error('SNAPSHOT-GUARD-CHECK: SELF-TEST FAIL — an injected raw write was NOT caught; this check proves nothing.') + process.exit(1) +} +console.log(`SNAPSHOT-GUARD-CHECK: PASS — ${guarded}/${files.length} fetcher(s) write through the guard; ${exempt} declared checkpoint write(s).`) diff --git a/package.json b/package.json index e756370f8..50df93bee 100644 --- a/package.json +++ b/package.json @@ -54,7 +54,7 @@ "check:facet-changes-map:self-test": "node internal/check-facet-changes-map.mjs --self-test # reverts the fix IN THE LIVE PAGE and requires the gate to FAIL. SELF_TEST_MODE=colour (default, isolates the colour assertion — the reported bug) | props | both | frozen (blocks the source update, isolating the set assertion). All four modes verified caught.", "check:heartbeat": "node internal/check-heartbeat-threshold.mjs # the HEARTBEAT ZONE threshold (the choropleth's top bin) must come from config AND the shipped config must still render today's ramp. Two sides: (B) a configured threshold must MOVE the edge — catches re-hard-coding; (A2) dashboard.config.js's own map.* must still resolve to 5/20/60/150/Infinity — catches the shipped default silently changing. A2 was MISSING at first and heartbeatThreshold:999 shipped past this gate printing 'default ramp unchanged'. A grep for '150' cannot do either job: it false-positives on the legitimate default and false-negatives on a re-hard-coding at 200. run check:heartbeat:self-test for the NEGATIVE tests (3 cases).", "check:heartbeat:self-test": "node internal/check-heartbeat-threshold.mjs --self-test # THREE regression cases, each required to fire ITS OWN assertion: [literal] resolver replaced by the old hard-coded array -> B; [shipped-default] dashboard.config.js heartbeatThreshold changed -> A2; [shipped-mode] heartbeatMode flipped to quantile -> A2. Poisons IN MEMORY, never the file. Testing that the FEATURE works is not testing that the REGRESSION is caught — this gate shipped with only the first and missed the second.", - "build:unlocked": "bun run check:template-literal && bun run check:mjs-parse && bun run check:coverage-consistent && bun run check:sfc-parse && bun run check:tab-radius && bun run check:synced-css && bun run check:wp-embed && bun run check:doc-scripts && bun run library-index && bun run check:library-index && bun run clean && bun run token && for d in plugin-designs/*/; do BUNDLE=$(basename \"$d\") bun x vite build || exit 1; done && bun run tree && bun run check:drop && bun run check:parse && bun run verify && bun run check:sri && bun run check:public-hygiene && bun run check:label-joins && bun run check:facet-cascade && bun run check:facet-one-field && bun run check:heartbeat && bun run check:kpi-unit && bun run check:composite-reg && bun run check:descriptions && bun run check:no-abs-paths && bun run check:no-codename && bun run check:dep-names && bun run check:composite-srcs && bun run check:embed-targets && bun run check:nested-token && bun run gallery", + "build:unlocked": "bun run check:template-literal && bun run check:mjs-parse && bun run check:coverage-consistent && bun run check:snapshot-guard && bun run check:sfc-parse && bun run check:tab-radius && bun run check:synced-css && bun run check:wp-embed && bun run check:doc-scripts && bun run library-index && bun run check:library-index && bun run clean && bun run token && for d in plugin-designs/*/; do BUNDLE=$(basename \"$d\") bun x vite build || exit 1; done && bun run tree && bun run check:drop && bun run check:parse && bun run verify && bun run check:sri && bun run check:public-hygiene && bun run check:label-joins && bun run check:facet-cascade && bun run check:facet-one-field && bun run check:heartbeat && bun run check:kpi-unit && bun run check:composite-reg && bun run check:descriptions && bun run check:no-abs-paths && bun run check:no-codename && bun run check:dep-names && bun run check:composite-srcs && bun run check:embed-targets && bun run check:nested-token && bun run gallery", "check:template-literal": "node internal/template-literal-check.mjs", "verify:map-paint": "node internal/verify-map-paint.mjs", "verify:gospel-map": "node internal/verify-gospel-map.mjs # the gospel-resources MAP: pins render (not merely load), a tap opens a named group, EVERY link in the index is on a screened publisher, the table deep-link carries the group's ISO, and the gap filter leaves only tier 0. Needs a server (bun run serve); SKIPS loudly without one.", @@ -122,6 +122,7 @@ "check:spa-tab-order": "node internal/check-spa-tab-order.mjs", "check:backlog-status": "node internal/check-backlog-status-enum.mjs", "check:mjs-parse": "node internal/check-mjs-parses.mjs", + "check:snapshot-guard": "node internal/check-snapshot-guard.mjs # every gospel-resources catalogue fetcher must write through scripts/_snapshot-guard.mjs, which refuses a write that loses more than a tenth of the previous snapshot. Written after a rate-limited FCBH run silently replaced 1,666 video scopes with 1,103. Static, sub-second. Self-test: --poison injects a raw write and the check must fail on it.", "check:coverage-consistent": "node internal/check-gospel-coverage-consistent.mjs # the gospel-resources surfaces must agree: no resource on an ISO RESERVED code (und/mul/mis/zxx), every resourced code NAMED, table reach recounted off the pins, no invented URL, tiers reconcile. Static, sub-second. Self-test: --poison, 6/6 clauses must fail. Written after the same drift defect was found by hand three times.", "check:synced-css:self-test": "node internal/check-synced-css-copies.mjs --self-test", "serve:permissive": "python3 -m http.server 4173 --bind 127.0.0.1 --directory public # LOOPBACK ONLY. This was --bind 0.0.0.0, which published the built tree on every interface of whatever machine ran a gate; measured on the build VPS it answered HTTP 200 on the box's PUBLIC IP while a browser gate was running. Every consumer (verify-browser-all, check:a11y-button-names, check:nav-reachable, smoke:share-all) already defaults to 127.0.0.1, so nothing needed the wider bind. Override with BASE_URL/BASE if you deliberately want to reach it from another host.", diff --git a/plugin-designs/gospel-resources/scripts/_snapshot-guard.mjs b/plugin-designs/gospel-resources/scripts/_snapshot-guard.mjs new file mode 100644 index 000000000..b6b8370d7 --- /dev/null +++ b/plugin-designs/gospel-resources/scripts/_snapshot-guard.mjs @@ -0,0 +1,68 @@ +/** + * _snapshot-guard.mjs — refuse to replace a good catalogue snapshot with a worse one. + * + * THE FAILURE THIS EXISTS FOR, MEASURED, NOT IMAGINED. The FCBH fetcher read per-fileset + * video scope for 1,666 languages and that file was committed. A later run of the same + * script, while live.bible.is was rate-limiting, got 1,103 — and overwrote the good file + * with the degraded one. Nothing failed, nothing warned, and the artefact silently lost a + * third of its content. It was caught only because `git status` happened to be read. + * + * It is the third time this shape has appeared in this bundle: + * - a VERIFY_LIMIT smoke run replaced the full verification table with 30 rows; + * - dropping a dead link removed it from the corpus, so the next run forgot it was dead + * and the build put it straight back; + * - this one. + * Every snapshot here is built from a NETWORK read, and a network read has a bad day. The + * dangerous case is not the run that throws — that leaves the old file alone — it is the + * run that half-succeeds and writes anyway. + * + * THE RULE: a snapshot may grow freely, and may shrink a little, but a large drop is + * treated as a failed run rather than as news. The count compared is whatever the caller + * says identifies the snapshot's content (languages, codes, entries). + * + * Deliberately NOT a hard equality check: catalogues genuinely do lose entries when a + * publisher retires something, and a gate nobody can satisfy gets switched off. The + * threshold is a floor on the fraction retained, and it is overridable in one obvious, + * loud way for the case where the drop is real. + */ +import { readFileSync, writeFileSync } from 'node:fs' + +/** Default: a run that lost more than 10% of the previous snapshot is not trusted. */ +const FLOOR = Number(process.env.SNAPSHOT_FLOOR || 0.9) +const FORCE = process.env.SNAPSHOT_FORCE === '1' + +/** + * @param {string} out absolute path to write + * @param {string} body the serialised snapshot + * @param {(json:any)=>number} countOf how to count what matters in this snapshot + * @param {string} label what the count is, for the message + */ +export function writeSnapshot(out, body, countOf, label) { + const next = countOf(JSON.parse(body)) + let prev = null + try { prev = countOf(JSON.parse(readFileSync(out, 'utf8'))) } catch { prev = null } + + if (prev !== null && prev > 0 && next < prev * FLOOR) { + const lost = prev - next + const pct = ((100 * lost) / prev).toFixed(1) + if (!FORCE) { + console.error( + `SNAPSHOT GUARD: REFUSING TO WRITE ${out}\n` + + ` this run produced ${next} ${label}; the file on disk has ${prev}.\n` + + ` That is ${lost} fewer (${pct}%), below the ${(FLOOR * 100).toFixed(0)}% floor.\n` + + ` A half-finished network read looks exactly like this, and overwriting would\n` + + ` destroy good data silently. The existing file is UNCHANGED.\n` + + ` If the drop is real (a publisher retired entries), rerun with SNAPSHOT_FORCE=1.`, + ) + process.exit(1) + } + console.warn(`SNAPSHOT GUARD: ${next} ${label} vs ${prev} on disk (${pct}% lost) — writing anyway because SNAPSHOT_FORCE=1.`) + writeFileSync(out, body) + return + } + + writeFileSync(out, body) + if (prev === null) console.log(`SNAPSHOT GUARD: no previous ${out} to compare; wrote ${next} ${label}.`) + else if (next < prev) console.log(`SNAPSHOT GUARD: ${next} ${label} (was ${prev}) — a small drop, within the floor.`) + else console.log(`SNAPSHOT GUARD: ${next} ${label} (was ${prev}).`) +} diff --git a/plugin-designs/gospel-resources/scripts/fetch-arclight-index.mjs b/plugin-designs/gospel-resources/scripts/fetch-arclight-index.mjs index 7ccecb31b..5c4e0321a 100644 --- a/plugin-designs/gospel-resources/scripts/fetch-arclight-index.mjs +++ b/plugin-designs/gospel-resources/scripts/fetch-arclight-index.mjs @@ -45,7 +45,7 @@ * Inside the filter, and named explicitly in the standing instruction. Only featureFilm is * taken; Arclight also carries short films and other media that are not screened here. */ -import { writeFileSync } from 'node:fs' +import { writeSnapshot } from './_snapshot-guard.mjs' import { resolve, dirname } from 'node:path' import { fileURLToPath } from 'node:url' @@ -82,12 +82,12 @@ for (const r of all) { } stats.kept = Object.keys(codes).length -writeFileSync(OUT, JSON.stringify({ +writeSnapshot(OUT, JSON.stringify({ _note: 'Arclight (JESUS Film Project) catalogue by ISO 639-3. Only languages with counts.featureFilm >= 1. Existence is established from the API: the /watch/ URL answers 200 for ANY id and cannot verify anything.', generated: new Date().toISOString().slice(0, 10), source: API, stats, codes, -})) +}), (j) => Object.keys(j.codes || {}).length, 'Arclight ISO 639-3 codes') console.log(JSON.stringify(stats)) console.log(`${stats.kept} ISO 639-3 codes -> ${OUT}`) diff --git a/plugin-designs/gospel-resources/scripts/fetch-arclight-share-urls.mjs b/plugin-designs/gospel-resources/scripts/fetch-arclight-share-urls.mjs index e801dda70..0f9de4305 100644 --- a/plugin-designs/gospel-resources/scripts/fetch-arclight-share-urls.mjs +++ b/plugin-designs/gospel-resources/scripts/fetch-arclight-share-urls.mjs @@ -19,6 +19,7 @@ * authoritative, so it is asked. Rate-limited, resumable, and cached so it is asked once. */ import { readFileSync, writeFileSync, existsSync } from 'node:fs' +import { writeSnapshot } from './_snapshot-guard.mjs' import { resolve, dirname } from 'node:path' import { fileURLToPath } from 'node:url' @@ -49,16 +50,21 @@ for (let i = 0; i < entries.length; i++) { if (u) { out[code] = u; fetched++ } else missing++ } else missing++ } catch { missing++ } + // The CHECKPOINT write is deliberately unguarded: this script resumes from the existing + // file as a cache, so `out` starts full and only grows. It cannot shrink, which is the + // thing the guard exists to catch. The FINAL write below is guarded like every other + // snapshot. + // CHECKPOINT-WRITE-OK if (fetched % 100 === 0 && fetched) { writeFileSync(OUT, JSON.stringify({ _note: 'Per-language JESUS Film deep links (shareUrl) from Arclight media-components. Partial write; rerun to resume.', generated: new Date().toISOString().slice(0, 10), codes: out })) console.log(` ... ${i + 1}/${entries.length} fetched=${fetched} cached=${cached} missing=${missing}`) } await new Promise((r) => setTimeout(r, DELAY)) } -writeFileSync(OUT, JSON.stringify({ +writeSnapshot(OUT, JSON.stringify({ _note: 'Per-language JESUS Film deep links (shareUrl) from Arclight media-components 1_jf-0-0. Replaces ?languageId=, which the site ignores.', generated: new Date().toISOString().slice(0, 10), stats: { languages: entries.length, withShareUrl: Object.keys(out).length, missing }, codes: out, -})) +}), (j) => Object.keys(j.codes || {}).length, 'Arclight shareUrl codes') console.log(`DONE languages ${entries.length} shareUrl ${Object.keys(out).length} missing ${missing}`) diff --git a/plugin-designs/gospel-resources/scripts/fetch-door43-index.mjs b/plugin-designs/gospel-resources/scripts/fetch-door43-index.mjs index 2e2088997..1872b5d7c 100644 --- a/plugin-designs/gospel-resources/scripts/fetch-door43-index.mjs +++ b/plugin-designs/gospel-resources/scripts/fetch-door43-index.mjs @@ -41,7 +41,7 @@ * ('es-419', 'pt-br', 'sr-latn', 'ur-deva', 'zh-hant') are NOT private use and are the same * language, so they still fold to the base. */ -import { writeFileSync } from 'node:fs' +import { writeSnapshot } from './_snapshot-guard.mjs' import { resolve, dirname } from 'node:path' import { fileURLToPath } from 'node:url' @@ -148,6 +148,6 @@ const out = { sources, codes: index, } -writeFileSync(OUT, JSON.stringify(out, null, 1)) +writeSnapshot(OUT, JSON.stringify(out, null, 1), (j) => Object.keys(j.codes || {}).length, 'Door43 ISO 639-3 codes') console.log(`\n${Object.keys(index).length} ISO 639-3 codes -> ${OUT}`) console.log(`${viaPart1} entr(ies) joined through an ISO 639-1 tag; ${skippedPrivate} private-use tag(s) skipped`) diff --git a/plugin-designs/gospel-resources/scripts/fetch-ebible-index.mjs b/plugin-designs/gospel-resources/scripts/fetch-ebible-index.mjs index ed04c773b..229a3bcdb 100644 --- a/plugin-designs/gospel-resources/scripts/fetch-ebible-index.mjs +++ b/plugin-designs/gospel-resources/scripts/fetch-ebible-index.mjs @@ -28,7 +28,7 @@ * ONE TRANSLATION PER LANGUAGE: the one with the most canonical books, tie-broken by verses. * A language with a full Bible should not be represented by a single-gospel portion. */ -import { writeFileSync } from 'node:fs' +import { writeSnapshot } from './_snapshot-guard.mjs' import { resolve, dirname } from 'node:path' import { fileURLToPath } from 'node:url' @@ -144,6 +144,6 @@ const out = { codes: best, byUrl, } -writeFileSync(OUT, JSON.stringify(out)) +writeSnapshot(OUT, JSON.stringify(out), (j) => Object.keys(j.codes || {}).length, 'eBible ISO 639-3 codes') console.log(JSON.stringify(stats)) console.log(`${stats.kept} ISO 639-3 codes -> ${OUT}`) diff --git a/plugin-designs/gospel-resources/scripts/fetch-fcbh-index.mjs b/plugin-designs/gospel-resources/scripts/fetch-fcbh-index.mjs index cea98efd0..fa1cdcaf9 100644 --- a/plugin-designs/gospel-resources/scripts/fetch-fcbh-index.mjs +++ b/plugin-designs/gospel-resources/scripts/fetch-fcbh-index.mjs @@ -64,7 +64,7 @@ * Testament narrative is also not the gospel, so counting it toward the Scripture+film tier * would overstate what a group has. It lifts a group off "nothing found", and no further. */ -import { writeFileSync } from 'node:fs' +import { writeSnapshot } from './_snapshot-guard.mjs' import { resolve, dirname } from 'node:path' import { fileURLToPath } from 'node:url' @@ -196,7 +196,7 @@ for (const [iso, b] of Object.entries(best)) { codes[iso] = { u: `${SITE}/${b.id}`, n: b.name || 'Scripture', ln: b.lang, id: b.id, date: b.date, m: b.media.join(', ') } } -writeFileSync( +writeSnapshot( OUT, `${JSON.stringify( { @@ -211,6 +211,9 @@ writeFileSync( null, 1, )}\n`, + // Both halves matter: a bad run can lose Scripture codes, video scopes, or both. + (j) => Object.keys(j.codes || {}).length + Object.keys(j.videoCodes || {}).length, + 'FCBH code+video entries', ) console.log(`FCBH: ${rows.length} rows -> ${Object.keys(codes).length} ISO 639-3 codes with Scripture (audio and/or text)`) diff --git a/plugin-designs/gospel-resources/scripts/fetch-zume-index.mjs b/plugin-designs/gospel-resources/scripts/fetch-zume-index.mjs index 51355aaec..677086b7e 100644 --- a/plugin-designs/gospel-resources/scripts/fetch-zume-index.mjs +++ b/plugin-designs/gospel-resources/scripts/fetch-zume-index.mjs @@ -33,7 +33,7 @@ * an unclear mapping is left out rather than guessed: the locale still ships with its * verified URL, it simply carries no ISO code and so attaches to no people group. */ -import { writeFileSync } from 'node:fs' +import { writeSnapshot } from './_snapshot-guard.mjs' import { resolve, dirname } from 'node:path' import { fileURLToPath } from 'node:url' @@ -236,7 +236,7 @@ for (const [locale, a] of Object.entries(APPROX)) { const unjoined = entries.filter((e) => !e.iso).map((e) => e.locale) const dead = entries.filter((e) => !e.url).map((e) => e.locale) -writeFileSync( +writeSnapshot( OUT, `${JSON.stringify( { @@ -253,6 +253,8 @@ writeFileSync( null, 1, )}\n`, + (j) => Object.keys(j.byIso || {}).length, + 'Zume ISO 639-3 codes', ) console.log(