0e0b142719
The FCBH fetcher read per-fileset video scope for 1,666 languages and that file was
committed. A later run, while live.bible.is was rate-limiting, produced 1,103 and
overwrote the good file with the degraded one. Nothing failed, nothing warned, and the
artefact silently lost a third of its content. It was caught only because git status
happened to be read, and the fix was a git checkout.
That is the third time this shape has appeared here:
- a VERIFY_LIMIT smoke run replaced the full verification table with 30 rows;
- dropping a dead link removed it from the corpus, so the next run forgot it was
dead and the build put it straight back;
- this one.
Every snapshot in this bundle is built from a network read, and a network read has a bad
day. The dangerous case is never the run that throws - that leaves the old file alone -
it is the run that half-succeeds and writes anyway.
scripts/_snapshot-guard.mjs now stands in front of every catalogue write. A snapshot may
grow freely and may shrink a little, but losing more than a tenth of the previous run is
treated as a failed fetch: the existing file is left untouched and the script exits 1.
It is deliberately not an equality check, because catalogues do genuinely lose entries
when a publisher retires something, and a gate nobody can satisfy gets switched off.
SNAPSHOT_FORCE=1 is the one loud way past it.
Proven by replaying the exact failure: feeding it the 1,103-entry version refuses the
write, names the real numbers (563 fewer, 13.8%, below the 90% floor) and leaves the
file byte-identical to the good one. SNAPSHOT_FORCE=1 writes it. An equal write passes.
All six fetchers are wired: ebible, arclight, arclight-share-urls, door43, zume, fcbh.
Re-running zume, door43 and ebible for real passes the guard at 52, 208 and 1,237 codes
and produces byte-identical data, so the guard perturbs nothing.
The one exception is the checkpoint write in the resumable share-urls fetcher, which
starts from the existing file as a cache and can only grow. It must now declare itself
with a CHECKPOINT-WRITE-OK marker, so the exemption is a statement rather than an
oversight.
New gate check:snapshot-guard fails if any fetch-*.mjs writes OUT directly without that
marker, or never calls writeSnapshot at all - a guard that is easy to bypass is a guard
that will be bypassed. It refuses to run against an empty file list rather than passing
vacuously, and --poison injects a raw write and requires the check to catch it. Chained
into build:unlocked and its doc copy kept in sync.
70 lines
3.8 KiB
JavaScript
70 lines
3.8 KiB
JavaScript
#!/usr/bin/env node
|
|
/**
|
|
* fetch-arclight-share-urls.mjs — get the REAL per-language JESUS Film deep link.
|
|
*
|
|
* THE DEFECT THIS FIXES. We shipped https://www.jesusfilm.org/watch/?languageId=<id> for
|
|
* 2,087 languages. MEASURED IN A BROWSER: the current site ignores that query parameter —
|
|
* a real id and a bogus id render the same generic library page. The film was real and
|
|
* the link did not carry anyone to it.
|
|
*
|
|
* Arclight's media-components endpoint returns a shareUrl per language, and that IS a
|
|
* deep link: /watch/jesus/german-standard. It discriminates, which the old URL never did —
|
|
* a real slug returns ~900KB carrying the media component id `1_jf-0-0` 369 times, a
|
|
* bogus slug returns a 33KB page carrying it 0 times.
|
|
*
|
|
* WHY FETCHED PER LANGUAGE AND NOT DERIVED FROM THE NAME. The slug looks derivable —
|
|
* "Pashto, Southern" -> pashto-southern — and 8 of 8 sampled names transformed correctly.
|
|
* Eight is not 2,087, and a wrong slug is not a broken link but a link to SOMEONE ELSE'S
|
|
* language, which is the failure this repo guards hardest against. The API is
|
|
* authoritative, so it is asked. Rate-limited, resumable, and cached so it is asked once.
|
|
*/
|
|
import { readFileSync, writeFileSync, existsSync } from 'node:fs'
|
|
import { writeSnapshot } from './_snapshot-guard.mjs'
|
|
import { resolve, dirname } from 'node:path'
|
|
import { fileURLToPath } from 'node:url'
|
|
|
|
const HERE = dirname(fileURLToPath(import.meta.url))
|
|
const REPO = resolve(HERE, '../../..')
|
|
const ARC = resolve(REPO, 'data/tables/arclight-index.json')
|
|
const OUT = resolve(REPO, 'data/tables/arclight-share-urls.json')
|
|
const UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'
|
|
const JF = '1_jf-0-0'
|
|
const DELAY = Number(process.env.DELAY_MS || 250)
|
|
|
|
const arc = JSON.parse(readFileSync(ARC, 'utf8')).codes
|
|
const prior = existsSync(OUT) ? JSON.parse(readFileSync(OUT, 'utf8')) : { codes: {} }
|
|
const out = { ...prior.codes }
|
|
const entries = Object.entries(arc)
|
|
let fetched = 0, cached = 0, missing = 0
|
|
|
|
for (let i = 0; i < entries.length; i++) {
|
|
const [code, v] = entries[i]
|
|
if (out[code]) { cached++; continue }
|
|
try {
|
|
const r = await fetch(`https://api.arclight.org/v2/media-components/${JF}/languages/${v.id}`, { headers: { 'User-Agent': UA } })
|
|
if (r.ok) {
|
|
const d = await r.json()
|
|
// https, always: the API returns http:// and a plain-http link in a field context
|
|
// is a downgrade nobody asked for.
|
|
const u = String(d.shareUrl || '').replace(/^http:\/\//, 'https://')
|
|
if (u) { out[code] = u; fetched++ } else missing++
|
|
} else missing++
|
|
} catch { missing++ }
|
|
// The CHECKPOINT write is deliberately unguarded: this script resumes from the existing
|
|
// file as a cache, so `out` starts full and only grows. It cannot shrink, which is the
|
|
// thing the guard exists to catch. The FINAL write below is guarded like every other
|
|
// snapshot.
|
|
// CHECKPOINT-WRITE-OK
|
|
if (fetched % 100 === 0 && fetched) {
|
|
writeFileSync(OUT, JSON.stringify({ _note: 'Per-language JESUS Film deep links (shareUrl) from Arclight media-components. Partial write; rerun to resume.', generated: new Date().toISOString().slice(0, 10), codes: out }))
|
|
console.log(` ... ${i + 1}/${entries.length} fetched=${fetched} cached=${cached} missing=${missing}`)
|
|
}
|
|
await new Promise((r) => setTimeout(r, DELAY))
|
|
}
|
|
writeSnapshot(OUT, JSON.stringify({
|
|
_note: 'Per-language JESUS Film deep links (shareUrl) from Arclight media-components 1_jf-0-0. Replaces ?languageId=, which the site ignores.',
|
|
generated: new Date().toISOString().slice(0, 10),
|
|
stats: { languages: entries.length, withShareUrl: Object.keys(out).length, missing },
|
|
codes: out,
|
|
}), (j) => Object.keys(j.codes || {}).length, 'Arclight shareUrl codes')
|
|
console.log(`DONE languages ${entries.length} shareUrl ${Object.keys(out).length} missing ${missing}`)
|