Judge a bible.is link the way the verifier already does, by its API
The band shipped an hour ago compared the page against an empty shell measured at run time. It separates cleanly today, but verify-gospel-links.mjs had already dropped that rule for this host and recorded why: their WAF began answering HEAD with a 403 of the same size for real and bogus filesets, and the rule agreed that everything was dead. /api/bibles/<id> answers 200 or a 75-byte 404. Also corrects two claims in the same file. jesusfilm.org was excluded on the strength of the retired ?languageId= shape, not the shareUrl path it ships now; and knowgod.com is not unverifiable — Cru's API proves those pairs, the verifier checks them that way, and this file simply has not reproduced it yet.
This commit is contained in:
doxa-maps-88655c70
parent
98903cb18b
commit
32a6b8476f
1 changed files with 61 additions and 58 deletions
|
|
@ -16,26 +16,31 @@
|
|||
* WHAT THIS FILE USED TO SAY, AND WHY IT WAS WRONG. The first version excluded four hosts on
|
||||
* the grounds that they "NEVER 404" and so could not be re-checked by status. That belief was
|
||||
* inherited, not measured, and it quietly exempted 5,760 of 8,394 shipped links — 69% of the
|
||||
* corpus, including the two largest hosts on the map. Measured on 2026-09-25, on the exact
|
||||
* URL SHAPE the map actually ships:
|
||||
* corpus, including the two largest hosts on the map — while the bundle's own verifier had
|
||||
* ALREADY solved three of the four. Measured on 2026-09-25, on the exact URL SHAPE the map
|
||||
* actually ships, and reconciled against scripts/verify-gospel-links.mjs, which is the
|
||||
* authority on per-host method and should be read before adding a host here:
|
||||
*
|
||||
* jesusfilm.org /watch/jesus/<slug> real 200 · three different bogus slugs ALL 404,
|
||||
* by HEAD as well as GET. It does 404. The exclusion was simply false, and
|
||||
* 2,049 links sat outside every check because of it.
|
||||
* live.bible.is /bible/<filesetId> always 200, but a bogus fileset returns an empty
|
||||
* SPA shell of 7,860–7,865 bytes while real filesets return 13.5–85 KB.
|
||||
* Status is useless; SIZE separates them cleanly, at better than 1.7×.
|
||||
* This is not a new trick — scripts/fetch-fcbh-index.mjs already builds the
|
||||
* index this way and records `shellBytes` for it. Only this check had not
|
||||
* learned it. 2,636 links.
|
||||
* by HEAD as well as GET. It does 404. The exclusion here was inherited
|
||||
* from the OLD /watch/?languageId=N shape, which really was byte-identical
|
||||
* for a valid and a bogus id — scripts/verify-gospel-links.mjs migrated to
|
||||
* the Arclight shareUrl path and has been checking these for some time.
|
||||
* Only this file never learned. 2,049 links.
|
||||
* live.bible.is /bible/<filesetId> the page always answers 200, so it cannot be judged
|
||||
* at all. Its API can: /api/bibles/<filesetId> returns 200 with the fileset
|
||||
* for a real id and a 75-byte 404 for a bogus one. Same discriminator the
|
||||
* verifier uses, same in-band abort. 2,636 links.
|
||||
* fivefish.org /<iso639-3> always 200; found and not-found differ by page
|
||||
* TITLE, which is how scripts/fetch-grn-index.mjs classifies them. Real,
|
||||
* but their robots.txt asks Crawl-delay: 10, so a 25-link sample is four
|
||||
* minutes of polite waiting and belongs in its own slower band, not here.
|
||||
* 982 links, still unchecked, deliberately and on the record.
|
||||
* knowgod.com /<path> byte-identical 3,864-byte "Know God" page for a real
|
||||
* path and a nonsense one. This one really is unverifiable by fetching it.
|
||||
* 94 links, and saying so is the honest answer.
|
||||
* 982 links, still unchecked here, deliberately and on the record.
|
||||
* knowgod.com /<path> byte-identical 3,864-byte page for a real path and
|
||||
* a nonsense one, so the PAGE proves nothing. Cru's own API does prove it —
|
||||
* a (language, tool) pair exists when the resource's latest translations
|
||||
* intersect that language's — and the verifier already checks them that
|
||||
* way. Not reproduced here yet. 94 links.
|
||||
*
|
||||
* The lesson is the one this repo keeps relearning: a claim about the outside world decays,
|
||||
* and an exclusion is a claim. Each host below carries the measurement that justifies its
|
||||
|
|
@ -45,9 +50,8 @@
|
|||
* IT IS SHOWN TO BE ABLE TO FAIL. Every band runs its controls in band, every run:
|
||||
* - status band: URLs that must not exist, one per host shape. Any 2xx and the check
|
||||
* reports that it proved nothing and exits non-zero rather than reporting a pass.
|
||||
* - shell band: a bogus fileset must read as shell (or the check cannot see death) AND a
|
||||
* fixed real fileset must read as substantive (or the check would call the whole host
|
||||
* dead the day its page shape changes).
|
||||
* - mapped band: a real fileset must answer 200 from the API AND an impossible one must not
|
||||
* (or the check is either a false green across the host or a libel on every link in it).
|
||||
*/
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { resolve, dirname } from 'node:path'
|
||||
|
|
@ -67,22 +71,32 @@ const STATUS_MEANINGFUL = new Set([
|
|||
'jesusfilm.org', 'www.jesusfilm.org',
|
||||
])
|
||||
|
||||
/** Hosts that answer 200 for anything but whose EMPTY page has a measurable size. */
|
||||
const SHELL_MEASURED = {
|
||||
/**
|
||||
* Hosts where the PAGE cannot be judged but a sibling URL carries an honest status.
|
||||
*
|
||||
* This is not a heuristic on a number the site controls. An earlier draft of this band
|
||||
* compared the page's body size against an empty shell measured at run time; it separated
|
||||
* cleanly the day it was written (7,860b shell against 13.5–85 KB of real page) and it is
|
||||
* still the technique scripts/fetch-fcbh-index.mjs used to BUILD the index. But the verifier
|
||||
* had already abandoned it for this host, and says why: their WAF started answering HEAD with
|
||||
* a 403 whose body was 520 bytes for real and bogus alike, and the size rule silently agreed
|
||||
* that everything was dead. A rule that survives only while the front door behaves is not a
|
||||
* rule. The API 404s properly, costs a tenth of the page, and describes exactly the fileset
|
||||
* the shipped page renders.
|
||||
*/
|
||||
const URL_MAPPED = {
|
||||
'live.bible.is': {
|
||||
/* Measured at run time, never hard-coded: the shell grows when they ship a bundle. */
|
||||
shellProbe: 'https://live.bible.is/bible/ZZZQQQ',
|
||||
/* Must come back substantive, or the size rule has stopped separating anything. */
|
||||
livingControl: 'https://live.bible.is/bible/ENGKJV',
|
||||
/* 7,860 shell vs 13,547 smallest real page on 2026-09-25 — 1.72×. 1.25 is slack. */
|
||||
aliveRatio: 1.25,
|
||||
/* /bible/ENGKJV -> /api/bibles/ENGKJV */
|
||||
map: (u) => u.replace('/bible/', '/api/bibles/'),
|
||||
mustExist: 'https://live.bible.is/api/bibles/ENGKJV',
|
||||
mustNotExist: 'https://live.bible.is/api/bibles/ZZZZZZ',
|
||||
},
|
||||
}
|
||||
|
||||
/** Hosts left out, with the reason, so the exclusion stays a measurement and not a habit. */
|
||||
const UNCHECKABLE = {
|
||||
'fivefish.org': 'found/not-found differ by title only, and Crawl-delay: 10 needs its own band',
|
||||
'knowgod.com': 'byte-identical page for a real and a nonsense path — unverifiable by fetch',
|
||||
const NOT_CHECKED_HERE = {
|
||||
'fivefish.org': 'title-only signal behind Crawl-delay: 10 — needs its own slower band',
|
||||
'knowgod.com': "page bytes are identical; the verifier proves these via Cru's API, not reproduced here",
|
||||
}
|
||||
|
||||
/** Must not exist. If one of these passes, the check cannot see death and says so. */
|
||||
|
|
@ -113,7 +127,7 @@ async function get(url, method = 'GET') {
|
|||
|
||||
/* A 808 KB film page times 25 is 20 MB of somebody else's bandwidth to learn one number that
|
||||
HEAD already carries. Status hosts are asked with HEAD; the shell band needs the body. */
|
||||
const status = (url) => get(url, 'HEAD').then((r) => r.status)
|
||||
const status = (url, method = 'HEAD') => get(url, method).then((r) => r.status)
|
||||
|
||||
let table
|
||||
try { table = JSON.parse(readFileSync(TABLE, 'utf8')) } catch {
|
||||
|
|
@ -130,7 +144,7 @@ let skipped = 0
|
|||
for (const u of verified) {
|
||||
let h
|
||||
try { h = new URL(u).host } catch { continue }
|
||||
if (!STATUS_MEANINGFUL.has(h) && !SHELL_MEASURED[h]) { skipped++; continue }
|
||||
if (!STATUS_MEANINGFUL.has(h) && !URL_MAPPED[h]) { skipped++; continue }
|
||||
if (!byHost.has(h)) byHost.set(h, [])
|
||||
byHost.get(h).push(u)
|
||||
}
|
||||
|
|
@ -159,49 +173,38 @@ if (blind.length) {
|
|||
}
|
||||
console.log(`LINK-ROT: status controls pass — ${ctl.length} impossible URL(s), none answered 2xx.`)
|
||||
|
||||
/* ── controls: the shell band, both directions ─────────────────────────────────────────── */
|
||||
const shells = {}
|
||||
for (const [host, cfg] of Object.entries(SHELL_MEASURED)) {
|
||||
/* ── controls: the mapped band, both directions ────────────────────────────────────────── */
|
||||
for (const [host, cfg] of Object.entries(URL_MAPPED)) {
|
||||
if (!byHost.has(host)) continue
|
||||
const empty = await get(cfg.shellProbe)
|
||||
const living = await get(cfg.livingControl)
|
||||
const floor = empty.len * cfg.aliveRatio
|
||||
if (!empty.len) {
|
||||
console.error(`LINK-ROT: PROVED NOTHING — ${host} shell probe returned nothing to measure.`)
|
||||
process.exit(1)
|
||||
}
|
||||
if (living.len <= floor) {
|
||||
const yes = await status(cfg.mustExist, 'GET')
|
||||
const no = await status(cfg.mustNotExist, 'GET')
|
||||
if (!(yes >= 200 && yes < 300) || no === 200) {
|
||||
console.error(
|
||||
`LINK-ROT: PROVED NOTHING — ${host} answered its KNOWN-GOOD control with ${living.len}b, ` +
|
||||
`no bigger than the ${empty.len}b empty shell.`,
|
||||
`LINK-ROT: PROVED NOTHING — ${host} answered ${yes} for a fileset that exists and ` +
|
||||
`${no} for one that cannot. It is not discriminating.`,
|
||||
)
|
||||
console.error(' The size rule has stopped separating a real page from an empty one, so a')
|
||||
console.error(' pass here would mean nothing and a fail would libel every link on the host.')
|
||||
console.error(' A pass here would be a false green across every link on the host, and a')
|
||||
console.error(' fail would condemn links that are fine. Neither is worth reporting.')
|
||||
process.exit(1)
|
||||
}
|
||||
shells[host] = floor
|
||||
console.log(
|
||||
`LINK-ROT: ${host} shell measured at ${empty.len}b; known-good page ${living.len}b ` +
|
||||
`(${(living.len / empty.len).toFixed(1)}×). Alive means over ${Math.round(floor)}b.`,
|
||||
)
|
||||
console.log(`LINK-ROT: ${host} controls pass — real fileset ${yes}, impossible fileset ${no}.`)
|
||||
}
|
||||
|
||||
/* ── the sample ─────────────────────────────────────────────────────────────────────────── */
|
||||
const dead = []
|
||||
for (const { host, url } of sample) {
|
||||
if (shells[host] !== undefined) {
|
||||
const r = await get(url)
|
||||
if (r.len <= shells[host]) dead.push([url, `${r.len}b — empty shell`])
|
||||
continue
|
||||
const cfg = URL_MAPPED[host]
|
||||
/* GET on the mapped band: their WAF blocks HEAD, which is how the old size rule broke. */
|
||||
const st = cfg ? await status(cfg.map(url), 'GET') : await status(url)
|
||||
if (!(st >= 200 && st < 300)) {
|
||||
dead.push([url, cfg ? `${st} from ${cfg.map(url)}` : st || 'no response'])
|
||||
}
|
||||
const s = await status(url)
|
||||
if (!(s >= 200 && s < 300)) dead.push([url, s || 'no response'])
|
||||
}
|
||||
|
||||
console.log(
|
||||
`LINK-ROT: re-checked ${sample.length} verified link(s) across ${byHost.size} host(s) ` +
|
||||
`(${skipped} link(s) on hosts nothing can check: ` +
|
||||
`${Object.keys(UNCHECKABLE).join(', ')}).`,
|
||||
`(${skipped} link(s) not sampled here: ` +
|
||||
`${Object.entries(NOT_CHECKED_HERE).map(([h, why]) => `${h} — ${why}`).join('; ')}).`,
|
||||
)
|
||||
if (dead.length) {
|
||||
console.error(`LINK-ROT: FAIL — ${dead.length} link(s) we call "checked" no longer resolve:`)
|
||||
|
|
|
|||
Loading…
Reference in a new issue