Say which pools were really searched for THIS language

The sheet's "checked" line is what tells a reader whether "nothing
found" means anything, and it was wrong in both directions. 5,131
codes omitted Scripture Earth, which had in fact been swept for every
one of them. And 110 codes named GRN as checked when the sweep has
never looked at them — GRN is crawled a batch at a time at the delay
their robots.txt asks for, so 2,100 codes are still unread.

The cause was ordering, the same lesson this file already carries for
the dead-link drop: the pass that normalises provenance ran before the
Scripture Earth and GRN fetchers, so the codes those fetchers create
kept the raw constant. A tidying pass that runs before the thing it
tidies reports a confident answer about a room still filling up. It
now runs after every source attaches, and the count is exact — 282
codes name GRN, 282 have been decided, and none that was decided goes
unnamed.

Scripture Earth is claimed unconditionally because that sweep really
did cover every code this map uses: a code absent from their index is
a checked negative. GRN is claimed per code, and the rest carry
"grn (not yet swept)" in the unchecked list instead.

Clause M of the coverage gate now fails if any code names a pool the
sweep never decided for it. Poison is 13/13.
This commit is contained in:
doxa-maps-88655c70 2026-09-24 19:55:12 +00:00
parent e888a2e5ba
commit 65e959a08f
8 changed files with 73 additions and 19 deletions

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View file

@ -90,6 +90,7 @@ if (POISON) {
codes.zzj = { l: 'Both at once', z: 'https://zume.training/en', za: { u: 'https://zume.training/pus', l: 'Pashto', n: 'x' } } // J
codes.zzk = { l: 'Ships a known-dead link', f: 'https://jesusfilm.org/watch/jesus/4101' } // K
poisonStale = true // L
codes.zzm = { l: 'Claims a pool nobody swept for it', ck: 'ebible, grn', s: 'https://ebible.org/zzm/' } // M
}
const SPECIAL = (c) => c === 'und' || c === 'mul' || c === 'mis' || c === 'zxx' || (c >= 'qaa' && c <= 'qtz')
@ -266,13 +267,37 @@ if (verif && verif.links) {
}
}
/* ── M: a named pool was really searched FOR THAT CODE ───────────────────────────────────
The `checked` line on a group's sheet is what tells a reader whether "nothing found"
means anything. GRN is swept a batch at a time at the crawl delay their robots.txt asks
for, so most codes have not been reached yet — and the build briefly claimed grn for 110
codes it had never looked at, because the pass that normalises provenance ran before the
fetchers that create those codes. A pool named here must be one this code was actually
searched in. */
{
let grnSeen = null
try {
const j = read('data/tables/grn-index.json')
grnSeen = new Set([...Object.keys(j.byIso || {}), ...(j.none || [])])
} catch { /* no snapshot yet: the claim cannot be made either, checked below */ }
const liars = []
for (const [c, v] of Object.entries(codes)) {
if (!v || v.sg === 1) continue
if (!/\bgrn\b/.test(String(v.ck || ''))) continue
if (!grnSeen || !grnSeen.has(c)) liars.push(c)
}
if (liars.length) {
add('M', `${liars.length} code(s) name grn as checked though the sweep never decided them (e.g. ${liars.slice(0, 3).join(', ')}) — the provenance pass must run AFTER every fetcher attaches`)
}
}
const clauses = new Set(fails.map((f) => f.slice(3, 4)))
if (POISON) {
// D is included deliberately: it is the clause that would have caught the 506
// resources dropped from a table titled "every link", and the most valuable clause is
// the worst one to leave unproven. The injected codes above (und, zzq) are absent from
// the table, so they poison D as well as A and B.
const want = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L']
const want = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M']
const caught = want.filter((c) => clauses.has(c))
console.log(`COVERAGE-CONSISTENT --poison: ${caught.length}/${want.length} clause(s) caught their injected defect (${caught.join(',')})`)
if (caught.length !== want.length) {
@ -298,3 +323,4 @@ console.log(' G no plain-http URL H no story set typed as S
console.log(' I row type agrees with its host J approximate never reads as confirmed')
console.log(' K every shipped link is verified, and none known-broken ships')
console.log(' L the verification record is current, not merely consistent')
console.log(' M a pool named as checked was really searched for that code')

View file

@ -159,7 +159,14 @@ try { scriptureEarth = JSON.parse(readFileSync(SCRIPTUREEARTH, 'utf8')).byIso ||
// Global Recordings Network via 5fish (scripts/fetch-grn-index.mjs). Optional like the rest.
let grn = {}
try { grn = JSON.parse(readFileSync(GRN, 'utf8')).byIso || {} } catch { grn = {} }
let grnSeen = new Set()
try {
const j = JSON.parse(readFileSync(GRN, 'utf8'))
grn = j.byIso || {}
// Every code the sweep has DECIDED — found or not found. A code it has not reached yet is
// neither, and must not be described as searched.
grnSeen = new Set([...Object.keys(grn), ...(j.none || [])])
} catch { grn = {}; grnSeen = new Set() }
let zume = {}
let zumeApprox = {}
@ -340,17 +347,6 @@ const NOT_SCANNED = 'api_bible, dbl, progress_bible'
// have nothing, and the honest statement is which pool could still have it.
const SIGN_SCANNED = 'arclight (14 sign languages, all captured), fcbh (none)'
const SIGN_NOT_SCANNED = 'deaf_bible_society, door_international — both key-gated'
for (const v of Object.values(codes)) {
if (v.sg === 1) {
v.ck = SIGN_SCANNED
v.un = SIGN_NOT_SCANNED
continue
}
const had = String(v.ck || '').split(',').map((x) => x.trim()).filter(Boolean)
const all = new Set([...had, 'ebible', 'arclight', 'door43', 'fcbh'])
v.ck = [...all].join(', ')
v.un = NOT_SCANNED
}
// Attach Zume where the language matches. A code with no other resource still gets an
// entry, because "training exists in your language" is a real answer to "what is there".
@ -459,6 +455,38 @@ for (const [c, a] of Object.entries(zumeApprox)) {
zumeApproxCodes++
}
// PROVENANCE, AFTER EVERY SOURCE HAS ATTACHED. This loop used to run before the Scripture
// Earth and GRN attaches, so the 110 codes those fetchers CREATE kept the raw constant and
// claimed grn had been checked for them when it had not. It is the same ordering lesson as
// the dead-link drop below: a normalising pass that runs before the thing it normalises
// reports a tidy answer about a room that is still filling up.
// entries(), not values(): the GRN sweep is decided PER CODE, so this loop needs the code.
for (const [c, v] of Object.entries(codes)) {
if (v.sg === 1) {
v.ck = SIGN_SCANNED
v.un = SIGN_NOT_SCANNED
continue
}
// WHAT WAS ACTUALLY CHECKED FOR THIS CODE, not what the pipeline can check in general.
// This line is read by a person deciding whether "nothing found" means anything, so a
// pool named here must really have been searched FOR THEM.
//
// scriptureearth is unconditional: that sweep covered every code this map uses, and a
// code absent from their index is a checked negative, not an unchecked one.
//
// grn is NOT. It is swept a batch at a time at the ten-second crawl delay their robots.txt
// asks for, so most codes have not been looked at yet. Claiming it for all of them would
// have told 2,100 groups their gap was searched in the largest gospel-audio pool there is
// when it was not — the exact stale-caveat error this file has had to correct twice before.
const had = String(v.ck || '').split(',').map((x) => x.trim()).filter(Boolean)
const all = new Set([...had, 'ebible', 'arclight', 'door43', 'fcbh', 'scriptureearth'])
const grnDecided = grnSeen.has(c)
if (grnDecided) all.add('grn')
else all.delete('grn')
v.ck = [...all].sort().join(', ')
v.un = grnDecided ? NOT_SCANNED : `${NOT_SCANNED}, grn (not yet swept)`
}
// Drop anything verification proved dead. THIS MUST RUN AFTER EVERY SOURCE HAS ATTACHED:
// it used to sit before the audio merge, so 71 dead bible.is filesets were attached after
// the drop had already happened and shipped anyway, while the pass reported "dropped 0".

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long