Every shipped link has evidence, and 1,412 of them did not
Three findings, one thread: rule 4 says verify a URL before shipping it, and the check I wrote for it yesterday was measuring one file. 1. THE TABLES PAGE WAS SHIPPING DEAD FORM. people-group-resources.json — the "Resources by People Group" tab, which a reader clicks — carried 2,294 URL cells, of which 1,412 were jesusfilm.org/watch/?languageId=<id>. That form is IGNORED by the site: a real id and a bogus one render the same generic library page, which is why the map migrated onto Arclight shareUrls. The migration never reached this table, so the booleans beside those links were being refreshed while the links themselves went stale for a year of builds. 1,417 film and 882 scripture URLs re-derived from the index, which is authoritative for both and whose URLs are the verified ones. 2. TWO ARCLIGHT LANGUAGES, ONE ISO CODE — resolved, with evidence rather than a ruling. jos was SKIPPED here, and the note said resolving it meant deciding whether the code or the publisher's name was authoritative, "a data question for someone who can check it". Checked: media-languages?iso3=jos returns 7138 "Jordanian Sign Language" and 139080 "Lebanese Sign Language", and 7138's shareUrl redirects to .../jordanian-sign-language.html while a bogus arc.gt code 404s. So the code stays authoritative for the join and the NAME breaks the tie the code cannot. The old note also had a fact wrong — it called Lebanese "a different language (lbs)", but lbs is LIBYAN Sign Language and Lebanese has no code of its own, so this is two Levantine varieties sharing one code, not a mis-filing. Three Deaf groups now reach the JESUS Film in their own sign language through a link that resolves; 0 retired-form links remain in the index. 3. THE GATE'S SCOPE WAS THE BUG. It walked the pin index alone, which is how item 1 hid from it. It now reads every table the SURFACES read, taken from the surfaces, and resolves each URL against four records: the verification table, the org contacts, the SUN pages (proved by content against a catch-all control) and doxa.life's own listing for prayer pages, which cannot be proved by fetch because pray.doxa.life answers 200 for any slug. Provenance fields are excluded by path — a citation is not an offer. 8,445 links measured yesterday; 12,292 shipped links today, 0 unrecorded, 0 broken, 0 that no status code can prove. Reading every STAGED file instead reported 6,115 unrecorded and 53 broken, all of them correct behaviour: API endpoints in source fields, and build caches that still hold the 53 share URLs the verifier proved dead — which is exactly why the pin builder drops them. A gate that goes red on correct behaviour gets switched off.
This commit is contained in:
Doxa Maps
parent
3179c24561
commit
c790b39840
21 changed files with 4780 additions and 8866 deletions
File diff suppressed because one or more lines are too long
|
|
@ -59,7 +59,7 @@
|
|||
"meta": {
|
||||
"title": "The Deaf Gap",
|
||||
"grain": "one row per Deaf people group",
|
||||
"generated": "2026-09-25",
|
||||
"generated": "2026-09-27",
|
||||
"measured": "150 sign languages · 234 people groups · 84,159,050 people. 136 languages / 183 groups / 55,199,840 people with NOTHING found.",
|
||||
"bridge_rule": "A sign language is bridged ONLY by another sign language. Text and audio in a spoken language do not reach a Deaf community.",
|
||||
"compounding_problem": "40 of these languages carry PRIVATE-USE (qaa-qtz) codes outside ISO 639-3. No resource pool indexes a private-use code, so those groups cannot be matched to a resource even if one exists.",
|
||||
|
|
@ -2621,7 +2621,7 @@
|
|||
"iso639_3": "jos",
|
||||
"countries": "SYR",
|
||||
"population": 221000,
|
||||
"signed_film": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"signed_film": "https://arc.gt/g8wmw",
|
||||
"has_signed_film": true,
|
||||
"has_any_resource": true,
|
||||
"code_unmatchable": false,
|
||||
|
|
@ -2660,7 +2660,7 @@
|
|||
"iso639_3": "jos",
|
||||
"countries": "JOR",
|
||||
"population": 116000,
|
||||
"signed_film": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"signed_film": "https://arc.gt/g8wmw",
|
||||
"has_signed_film": true,
|
||||
"has_any_resource": true,
|
||||
"code_unmatchable": false,
|
||||
|
|
@ -2738,7 +2738,7 @@
|
|||
"iso639_3": "jos",
|
||||
"countries": "LBN",
|
||||
"population": 51000,
|
||||
"signed_film": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"signed_film": "https://arc.gt/g8wmw",
|
||||
"has_signed_film": true,
|
||||
"has_any_resource": true,
|
||||
"code_unmatchable": false,
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
"title": "Doxa people groups — what the gospel is available in, and what is missing",
|
||||
"grain": "one row per people group on Doxa's roster",
|
||||
"source": "pray.doxa.life roster joined to the resource index by ISO 639-3",
|
||||
"generated": "2026-09-26",
|
||||
"generated": "2026-09-27",
|
||||
"groups": 2107,
|
||||
"by_access": {
|
||||
"nothing found": 369,
|
||||
|
|
@ -19768,7 +19768,7 @@
|
|||
"translated_by": "",
|
||||
"translated_year": "",
|
||||
"scripture_url": "",
|
||||
"film_url": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"film_url": "https://arc.gt/g8wmw",
|
||||
"audio_url": "",
|
||||
"bible_video_url": "",
|
||||
"stories_url": "",
|
||||
|
|
@ -20048,7 +20048,7 @@
|
|||
"translated_by": "",
|
||||
"translated_year": "",
|
||||
"scripture_url": "",
|
||||
"film_url": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"film_url": "https://arc.gt/g8wmw",
|
||||
"audio_url": "",
|
||||
"bible_video_url": "",
|
||||
"stories_url": "",
|
||||
|
|
@ -21928,7 +21928,7 @@
|
|||
"translated_by": "",
|
||||
"translated_year": "",
|
||||
"scripture_url": "",
|
||||
"film_url": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"film_url": "https://arc.gt/g8wmw",
|
||||
"audio_url": "",
|
||||
"bible_video_url": "",
|
||||
"stories_url": "",
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
"bridge_strength_note": "MEASURED: 24% of aligned rows rest on a bridge shared by only 2-3 groups. bridge_strength grades this (strong >=8 groups, moderate >=4, weak <4). A WEAK bridge is a hypothesis, not a route — e.g. \"Gawri\" was offered as the bridge for 10.3M Pakistanis where the real lingua franca is Urdu, which no unengaged group here claims as primary and which this method therefore cannot see. Filter on bridge_strength before acting.",
|
||||
"bridge_caveat": "A bridge candidate is measured from THIS corpus (a language primary to >1 DX group in the country). It is NOT proof the group speaks it, and it is NOT a national trade language — a lingua franca no unengaged group claims as primary is invisible here.",
|
||||
"pools_caveat": "Resource status is from eBible + JESUS Film. GRN and FCBH audio are key-blocked and unchecked — absence is never proof.",
|
||||
"resource_flags_refreshed": "2026-09-25",
|
||||
"resource_flags_refreshed": "2026-09-27",
|
||||
"resource_flags_source": "data/tables/people-group-pins.json — the same index the map draws. Curated bridge, internet and channel columns are untouched."
|
||||
},
|
||||
"columns": [
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"meta": {
|
||||
"title": "Gospel resource link verification",
|
||||
"generated": "2026-09-25T09:56:59.860Z",
|
||||
"generated": "2026-09-27T01:21:18.374Z",
|
||||
"method": "HEAD, redirects followed. verified = HTTP 200. broken = 404/410. unknown = unreachable, an ambiguous status, or a URL shape that cannot be checked by fetch.",
|
||||
"scope": "Every URL in data/tables/people-group-pins.json, deduplicated.",
|
||||
"states": {
|
||||
|
|
@ -96,12 +96,12 @@
|
|||
]
|
||||
},
|
||||
"summary": {
|
||||
"records": 8524,
|
||||
"checkedThisPass": 8,
|
||||
"records": 8525,
|
||||
"checkedThisPass": 1,
|
||||
"carriedForward": 74,
|
||||
"pruned": 0,
|
||||
"broken": 74,
|
||||
"verified": 8445,
|
||||
"verified": 8446,
|
||||
"unknown": 5
|
||||
},
|
||||
"byHost": {
|
||||
|
|
@ -123,7 +123,7 @@
|
|||
"broken": 1
|
||||
},
|
||||
"arc.gt": {
|
||||
"verified": 33,
|
||||
"verified": 34,
|
||||
"broken": 1
|
||||
},
|
||||
"zume.training": {
|
||||
|
|
@ -42766,6 +42766,11 @@
|
|||
"state": "verified",
|
||||
"detail": "5fish redirected to its language page https://fivefish.org/15537?language=Pashai%2C%20Southeast (the site answers 200 for any path)",
|
||||
"at": "2026-09-25T09:56:59.503Z"
|
||||
},
|
||||
"https://arc.gt/g8wmw": {
|
||||
"state": "verified",
|
||||
"detail": "HTTP 200 (via https://www.jesusfilm.org/watch/jesus.html/jordanian-sign-language.html)",
|
||||
"at": "2026-09-27T01:21:17.988Z"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -11,7 +11,7 @@
|
|||
"honest_limit": "This is NOT the same as a national trade language. It finds languages widely used AMONG THE UNENGAGED GROUPS TRACKED HERE, and will MISS a lingua franca that no unengaged group claims as primary (e.g. Hindi in India). Named bridge_candidates, not trade_languages, for that reason.",
|
||||
"why_not_a_list": "No open machine-readable list of national trade languages was available; bridge candidates are derived from this corpus."
|
||||
},
|
||||
"resource_flags_refreshed": "2026-09-25",
|
||||
"resource_flags_refreshed": "2026-09-27",
|
||||
"resource_flags_source": "data/tables/people-group-pins.json. prevalence_rank, bridge_candidate, groups_speaking and full_bible are curated and untouched."
|
||||
},
|
||||
"columns": [
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load diff
|
|
@ -25,9 +25,10 @@
|
|||
* Self-test: `--poison` adds a fabricated URL to the corpus, marks a shipped one broken, and moves
|
||||
* an `unknown` onto a host that IS checkable. All three arms must fire.
|
||||
*/
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { readFileSync, readdirSync } from 'node:fs'
|
||||
import { resolve, dirname } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { TREE_NAME } from './_tree-name.mjs'
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url))
|
||||
const REPO = resolve(HERE, '..')
|
||||
|
|
@ -64,12 +65,33 @@ try {
|
|||
CONTENT against a catch-all control (a bogus path returns 904 characters; the New Testament page
|
||||
returns 22,043). The recursive walk found these three the same way it found the contacts — by not
|
||||
knowing which fields were supposed to hold links. */
|
||||
/* PRAYER PAGES ARE PROVED BY A LISTING, NOT A FETCH — the fourth record, and the reason is in
|
||||
fetch-pray-index.mjs: pray.doxa.life answers 200 with identical bytes for ANY slug, so a status
|
||||
check there is a false green. Existence comes from doxa.life's own /api/people-groups/list, and
|
||||
every URL this build ships is built from a slug that listing returned. A pray.doxa.life URL that
|
||||
is NOT in that index is therefore a CONSTRUCTED one, and must fail here. */
|
||||
let prayListed = new Set()
|
||||
try {
|
||||
const pray = JSON.parse(readFileSync(resolve(REPO, 'data/tables/pray-index.json'), 'utf8')).byKey || {}
|
||||
for (const e of Object.values(pray)) if (e && e.u) prayListed.add(e.u.split('?')[0])
|
||||
} catch { /* no prayer index: those URLs then have no evidence, and must fail below */ }
|
||||
/* ⚠ THE INDEX IS A SUBSET AND THE ROSTER IS THE LISTING. pray-index.json holds only the 1,723
|
||||
groups whose (name, country) matched a pin; the access table ships a prayer link for all 2,107,
|
||||
so 809 of them were reported unrecorded here. They are not unproved — doxa-people-groups.json IS
|
||||
the API listing, one row per group with the slug doxa.life returned, and the index is that
|
||||
listing narrowed by a join this check has no business requiring. */
|
||||
try {
|
||||
const roster = JSON.parse(readFileSync(resolve(REPO, 'data/tables/doxa-people-groups.json'), 'utf8')).rows || []
|
||||
for (const r of roster) if (r && r.pray_url) prayListed.add(String(r.pray_url).split('?')[0])
|
||||
} catch { /* no roster: the subset above still covers what it covers */ }
|
||||
|
||||
let sunVerified = new Set()
|
||||
try {
|
||||
const sun = JSON.parse(readFileSync(resolve(REPO, 'data/tables/sun-deaf-index.json'), 'utf8'))
|
||||
if ((sun.meta || {}).catch_all_control) sunVerified = new Set((sun.resources || []).map((r) => r.url))
|
||||
} catch { /* no SUN record: its links then have no evidence, and must fail below */ }
|
||||
|
||||
const fails = []
|
||||
const idx = JSON.parse(readFileSync(resolve(REPO, 'data/tables/people-group-pins.json'), 'utf8'))
|
||||
const verif = JSON.parse(readFileSync(resolve(REPO, 'data/tables/gospel-link-verification.json'), 'utf8')).links || {}
|
||||
|
||||
|
|
@ -78,6 +100,9 @@ const found = new Map()
|
|||
const walk = (node, path) => {
|
||||
if (node == null) return
|
||||
if (typeof node === 'string') {
|
||||
/* A citation is not an offer. `source`, `_note` and everything under `meta` record WHERE the
|
||||
data came from — an API endpoint, a licence page — and no surface renders them as links. */
|
||||
if (/(^|\.)(meta|_note|source|sources|site|fetchedFrom|licenceSource|boundarySource|why|note)(\.|$|\[)/.test(path)) return
|
||||
if (/^https?:\/\//i.test(node) && !found.has(node)) found.set(node, path)
|
||||
return
|
||||
}
|
||||
|
|
@ -89,11 +114,46 @@ walk(idx.sun, 'sun')
|
|||
walk(idx.bridges, 'bridges')
|
||||
walk(idx.bridgeCandidates, 'bridgeCandidates')
|
||||
|
||||
/* ⚠ THE PIN INDEX IS NOT THE ONLY THING SHIPPED, and scoping this check to it repeated the very
|
||||
blind spot it was written to close: people-group-resources.json — the "Resources by People
|
||||
Group" tab — was serving 2,294 URL cells of which 1,412 were the retired jesusfilm
|
||||
?languageId= form, links the site ignores, while the map beside it had been migrated onto
|
||||
Arclight shareUrls. Nothing saw it, because every check looked at one file.
|
||||
*
|
||||
* ⚠ AND THE SCOPE IS "WHAT A READER CAN CLICK", which is narrower than "what is staged". Reading
|
||||
* every staged file reported 6,115 unrecorded links and 53 broken ones, and NONE of them is
|
||||
* offered to anybody: they are API endpoints in `source` fields, and the raw upstream catalogues
|
||||
* (arclight-share-urls and friends) which are build CACHES that happen to sit in the same folder —
|
||||
* they still hold the 53 share URLs the verifier proved dead, which is exactly why the pin builder
|
||||
* drops them. Judging those as shipped would make this check red on correct behaviour, and a gate
|
||||
* that is red on correct behaviour gets switched off. So the scope is the tables the SURFACES
|
||||
* read, taken from the surfaces, plus provenance fields excluded by path. */
|
||||
const SURFACES = [
|
||||
resolve(REPO, 'plugin-designs/gospel-resources/data-tables.vue'),
|
||||
resolve(REPO, 'plugin-designs/gospel-resources/resources-map.vue'),
|
||||
resolve(REPO, 'plugin-designs/gospel-resources/resources-list.vue'),
|
||||
resolve(REPO, 'plugin-designs/gospel-resources/charts.vue'),
|
||||
]
|
||||
const wanted = new Set()
|
||||
for (const f of SURFACES) {
|
||||
let src = ''
|
||||
try { src = readFileSync(f, 'utf8') } catch { continue }
|
||||
for (const m of src.matchAll(/file:\s*'([a-z0-9-]+)'/gi)) wanted.add(`${m[1]}.json`)
|
||||
for (const m of src.matchAll(/tables\/([a-z0-9-]+)\.json/gi)) wanted.add(`${m[1]}.json`)
|
||||
}
|
||||
if (wanted.size < 5) fails.push(`only ${wanted.size} reader-facing table(s) found in the surfaces — the scope is not being derived, so this check would under-measure`)
|
||||
const POOL = resolve(REPO, 'public', TREE_NAME, 'assets/tables')
|
||||
for (const f of wanted) {
|
||||
/* The verification record is a LEDGER, deliberately keeping URLs proved dead; walking it would
|
||||
ask the record to vouch for itself. */
|
||||
if (f === 'gospel-link-verification.json') continue
|
||||
try { walk(JSON.parse(readFileSync(resolve(POOL, f), 'utf8')), `tables/${f}`) } catch (e) { fails.push(`${f} is read by a surface but could not be read here: ${e.message}`) }
|
||||
}
|
||||
|
||||
if (POISON) found.set('https://ebible.org/this-was-never-verified/', 'codes.__poison.s')
|
||||
|
||||
const fails = []
|
||||
const unrecorded = [], broken = [], unknownOnCheckable = []
|
||||
let unknownBlind = 0, verified = 0, contacts = 0, byContent = 0
|
||||
let unknownBlind = 0, verified = 0, contacts = 0, byContent = 0, listed = 0
|
||||
let poisonedBroken = false, poisonedUnknown = false
|
||||
for (const [url, where] of found) {
|
||||
let e = verif[url]
|
||||
|
|
@ -101,6 +161,11 @@ for (const [url, where] of found) {
|
|||
else if (POISON && !poisonedUnknown && e && e.state === 'verified' && !statusBlind(url)) { e = { ...e, state: 'unknown' }; poisonedUnknown = true }
|
||||
if (!e && orgVerified.has(url)) { contacts++; continue }
|
||||
if (!e && sunVerified.has(url)) { byContent++; continue }
|
||||
/* The shipped URL may carry a tracking suffix the index does not (or vice versa); the SLUG is
|
||||
what the listing proved, so compare on origin+path. */
|
||||
if (!e && /pray\.doxa\.life/i.test(url)) {
|
||||
if (prayListed.has(url.split('?')[0])) { listed++; continue }
|
||||
}
|
||||
if (!e) { unrecorded.push(`${url} (${where})`); continue }
|
||||
if (e.state === 'broken') { broken.push(`${url} (${where})`); continue }
|
||||
if (e.state === 'unknown') {
|
||||
|
|
@ -131,5 +196,6 @@ if (fails.length) {
|
|||
}
|
||||
console.log(`LINK-COVERAGE: PASS — ${found.size} distinct shipped link(s): ${verified} verified by fetch, ` +
|
||||
`${contacts} translator contact(s) and ${byContent} SUN page(s) verified in their own records, ` +
|
||||
`${listed} prayer page(s) proved by doxa.life's own listing, ` +
|
||||
`${unknownBlind} on targets no status code can prove; ` +
|
||||
'0 unrecorded, 0 broken.')
|
||||
|
|
|
|||
|
|
@ -31,19 +31,32 @@ const UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Geck
|
|||
const JF = '1_jf-0-0'
|
||||
|
||||
/**
|
||||
* `jos` IS SKIPPED, AND THE REASON IS UPSTREAM. ISO 639-3 jos is Jordanian Sign Language.
|
||||
* Arclight's roster files iso3 "jos" under languageId 139080, whose own name field reads
|
||||
* "Lebanese Sign Language" — a different language (lbs). Meanwhile the JESUS Film DOES exist
|
||||
* at languageId 7138, named Jordanian Sign Language, which is the id the pin index carries.
|
||||
* ═══ TWO ARCLIGHT LANGUAGES, ONE ISO CODE — RESOLVED, WITH THE EVIDENCE ═════════════════════
|
||||
*
|
||||
* So the ISO join — the thing this whole map is built on — disagrees with the publisher's own
|
||||
* label for this one code, and the fallback below resolved 139080 to Magdalena: the wrong film
|
||||
* AND the wrong language, shown to three people groups. Left exactly as it was rather than
|
||||
* guessed at. Resolving it means deciding whether the code or the name is authoritative here,
|
||||
* and that is a data question for someone who can check it against Ethnologue, not something
|
||||
* to settle inside a fetcher.
|
||||
* This code was SKIPPED, and the note here said resolving it meant deciding whether the ISO code
|
||||
* or the publisher's name was authoritative — "a data question for someone who can check it".
|
||||
* It has now been checked, against Arclight's own API rather than against an opinion:
|
||||
*
|
||||
* media-languages?iso3=jos returns TWO ids
|
||||
* 7138 name "Jordanian Sign Language" JESUS film -> arc.gt/g8wmw
|
||||
* 139080 name "Lebanese Sign Language" (the id the roster file lists under jos)
|
||||
* arc.gt/g8wmw redirects to .../watch/jesus.html/jordanian-sign-language.html — the publisher's
|
||||
* own slug names the language, and a bogus arc.gt code 404s, so it discriminates.
|
||||
*
|
||||
* ⚠ AND THE OLD NOTE WAS WRONG ON A FACT. It called Lebanese Sign Language "a different language
|
||||
* (lbs)". ISO 639-3 lbs is LIBYAN Sign Language; Lebanese Sign Language has no code of its own.
|
||||
* So this is not a mis-filing to be corrected upstream — it is two Levantine varieties sharing one
|
||||
* code, which ISO does not separate. Nothing here can be "given back" to a Lebanese code, because
|
||||
* there is none, and joining 139080 to our lbs would be a NAME match on the wrong country.
|
||||
*
|
||||
* THE RULE, therefore: where several Arclight ids share an ISO code, take the one whose own NAME
|
||||
* agrees with the language this map names for that code, and record the other. The code stays
|
||||
* authoritative for the JOIN; the name only breaks a tie the code cannot. Our three jos groups are
|
||||
* Jordanian, so they get the Jordanian film and not the Lebanese variety.
|
||||
*/
|
||||
const SKIP = new Set(['jos'])
|
||||
const OVERRIDE = {
|
||||
jos: { id: 7138, why: 'two Arclight ids carry iso3 jos (7138 Jordanian, 139080 Lebanese); 7138 is the one whose name agrees with this map\'s language for the code, and its shareUrl slug says jordanian-sign-language' },
|
||||
}
|
||||
const DELAY = Number(process.env.DELAY_MS || 250)
|
||||
|
||||
const arc = JSON.parse(readFileSync(ARC, 'utf8')).codes
|
||||
|
|
@ -54,10 +67,12 @@ let fetched = 0, cached = 0, missing = 0
|
|||
|
||||
for (let i = 0; i < entries.length; i++) {
|
||||
const [code, v] = entries[i]
|
||||
if (SKIP.has(code)) { cached++; continue }
|
||||
if (out[code] !== undefined) { cached++; continue }
|
||||
/* An override replaces the ROSTER's id, not the result: the shareUrl is still fetched from the
|
||||
API for that id, so a wrong override cannot invent a link. */
|
||||
const id = OVERRIDE[code] ? OVERRIDE[code].id : v.id
|
||||
try {
|
||||
const r = await fetch(`https://api.arclight.org/v2/media-components/${JF}/languages/${v.id}`, { headers: { 'User-Agent': UA } })
|
||||
const r = await fetch(`https://api.arclight.org/v2/media-components/${JF}/languages/${id}`, { headers: { 'User-Agent': UA } })
|
||||
if (r.ok) {
|
||||
const d = await r.json()
|
||||
// https, always: the API returns http:// and a plain-http link in a field context
|
||||
|
|
@ -98,12 +113,12 @@ for (let i = 0; i < entries.length; i++) {
|
|||
* than falling back to a URL nobody can check. */
|
||||
let alt = null
|
||||
try {
|
||||
const c = await fetch(`https://api.arclight.org/v2/media-components?languageIds=${v.id}&limit=25`, { headers: { 'User-Agent': UA } })
|
||||
const c = await fetch(`https://api.arclight.org/v2/media-components?languageIds=${id}&limit=25`, { headers: { 'User-Agent': UA } })
|
||||
if (c.ok) {
|
||||
const cj = await c.json()
|
||||
const feature = (cj._embedded?.mediaComponents || []).find((m) => m.subType === 'featureFilm')
|
||||
if (feature) {
|
||||
const fr = await fetch(`https://api.arclight.org/v2/media-components/${feature.mediaComponentId}/languages/${v.id}`, { headers: { 'User-Agent': UA } })
|
||||
const fr = await fetch(`https://api.arclight.org/v2/media-components/${feature.mediaComponentId}/languages/${id}`, { headers: { 'User-Agent': UA } })
|
||||
if (fr.ok) {
|
||||
const fd = await fr.json()
|
||||
const fu = String(fd.shareUrl || '').replace(/^http:\/\//, 'https://')
|
||||
|
|
@ -130,7 +145,18 @@ for (let i = 0; i < entries.length; i++) {
|
|||
writeSnapshot(OUT, JSON.stringify({
|
||||
_note: 'Per-language JESUS Film Project deep links (shareUrl) from Arclight media-components. A STRING is the JESUS Film (component 1_jf-0-0). An OBJECT {u,t} is a different JESUS Film Project feature in that language — the JESUS component has no entry for it — and t is the real title, which the surfaces must show instead of "JESUS Film". null means no feature exists and the film is dropped rather than linked to the retired ?languageId= URL, which nothing can verify.',
|
||||
generated: new Date().toISOString().slice(0, 10),
|
||||
stats: { languages: entries.length, withShareUrl: Object.keys(out).length, missing },
|
||||
/* COUNTED, NOT NAMED. withShareUrl used to be Object.keys(out).length, which counts the nulls
|
||||
and the alternative-feature objects too — so a file holding one null reported every language
|
||||
as having a shareUrl, and the previous run reported 2,087 of them while jos was absent
|
||||
entirely. Each state is now counted as what it is. */
|
||||
stats: {
|
||||
languages: entries.length,
|
||||
recorded: Object.keys(out).length,
|
||||
jesusFilm: Object.values(out).filter((v) => typeof v === 'string').length,
|
||||
otherFeature: Object.values(out).filter((v) => v && typeof v === 'object').length,
|
||||
noFeature: Object.values(out).filter((v) => v === null).length,
|
||||
notAnswered: missing,
|
||||
},
|
||||
codes: out,
|
||||
}), (j) => Object.keys(j.codes || {}).length, 'Arclight shareUrl codes')
|
||||
console.log(`DONE languages ${entries.length} shareUrl ${Object.keys(out).length} missing ${missing}`)
|
||||
console.log(`DONE languages ${entries.length} JESUS film ${Object.values(out).filter((v) => typeof v === 'string').length} other feature ${Object.values(out).filter((v) => v && typeof v === 'object').length} no feature ${Object.values(out).filter((v) => v === null).length} not answered ${missing}`)
|
||||
|
|
|
|||
|
|
@ -118,6 +118,28 @@ const bump = (k) => { changed[k] = (changed[k] || 0) + 1 }
|
|||
const any = hasAny(c)
|
||||
if (r.any_resource_found !== any) { r.any_resource_found = any; bump('any_resource_found') }
|
||||
if (r.no_resource_found !== !any) { r.no_resource_found = !any; bump('no_resource_found') }
|
||||
|
||||
/* ═══ THE URLS, AND THIS TABLE IS SHIPPED ═════════════════════════════════════════════
|
||||
* The booleans were being refreshed while the LINKS beside them were not, and this table
|
||||
* is the map's "Resources by People Group" tab — a reader clicks these. Measured before
|
||||
* this pass: 2,294 URL cells, of which 1,412 were the retired
|
||||
* jesusfilm.org/watch/?languageId=<id> form. That form is IGNORED by the site — a real id
|
||||
* and a bogus one render the same generic library page — which is why the map migrated off
|
||||
* it onto Arclight shareUrls. The migration never reached here, so the tables page kept
|
||||
* offering links that go nowhere in particular, for a year's worth of builds.
|
||||
*
|
||||
* The index is authoritative for these two fields and its URLs are the verified ones, so
|
||||
* they are copied across. Only where the index HAS one: a curated URL is never blanked by
|
||||
* a pool that happens to have no entry. */
|
||||
const sUrl = (codes[c] || {}).s, fUrl = (codes[c] || {}).f
|
||||
if ('scripture_url' in r && sUrl && r.scripture_url !== sUrl) {
|
||||
if (/[?&]languageId=/.test(r.scripture_url || '')) bump('retired_form_replaced')
|
||||
r.scripture_url = sUrl; bump('scripture_url')
|
||||
}
|
||||
if ('film_url' in r && fUrl && r.film_url !== fUrl) {
|
||||
if (/[?&]languageId=/.test(r.film_url || '')) bump('retired_form_replaced')
|
||||
r.film_url = fUrl; bump('film_url')
|
||||
}
|
||||
}
|
||||
// THE POOL LIST IS A CLAIM TOO. This table still told readers GRN was "not included:
|
||||
// requires an API key" — false on both halves. GRN is included, 583 codes of it, and it
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -59,7 +59,7 @@
|
|||
"meta": {
|
||||
"title": "The Deaf Gap",
|
||||
"grain": "one row per Deaf people group",
|
||||
"generated": "2026-09-25",
|
||||
"generated": "2026-09-27",
|
||||
"measured": "150 sign languages · 234 people groups · 84,159,050 people. 136 languages / 183 groups / 55,199,840 people with NOTHING found.",
|
||||
"bridge_rule": "A sign language is bridged ONLY by another sign language. Text and audio in a spoken language do not reach a Deaf community.",
|
||||
"compounding_problem": "40 of these languages carry PRIVATE-USE (qaa-qtz) codes outside ISO 639-3. No resource pool indexes a private-use code, so those groups cannot be matched to a resource even if one exists.",
|
||||
|
|
@ -2621,7 +2621,7 @@
|
|||
"iso639_3": "jos",
|
||||
"countries": "SYR",
|
||||
"population": 221000,
|
||||
"signed_film": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"signed_film": "https://arc.gt/g8wmw",
|
||||
"has_signed_film": true,
|
||||
"has_any_resource": true,
|
||||
"code_unmatchable": false,
|
||||
|
|
@ -2660,7 +2660,7 @@
|
|||
"iso639_3": "jos",
|
||||
"countries": "JOR",
|
||||
"population": 116000,
|
||||
"signed_film": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"signed_film": "https://arc.gt/g8wmw",
|
||||
"has_signed_film": true,
|
||||
"has_any_resource": true,
|
||||
"code_unmatchable": false,
|
||||
|
|
@ -2738,7 +2738,7 @@
|
|||
"iso639_3": "jos",
|
||||
"countries": "LBN",
|
||||
"population": 51000,
|
||||
"signed_film": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"signed_film": "https://arc.gt/g8wmw",
|
||||
"has_signed_film": true,
|
||||
"has_any_resource": true,
|
||||
"code_unmatchable": false,
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
"title": "Doxa people groups — what the gospel is available in, and what is missing",
|
||||
"grain": "one row per people group on Doxa's roster",
|
||||
"source": "pray.doxa.life roster joined to the resource index by ISO 639-3",
|
||||
"generated": "2026-09-26",
|
||||
"generated": "2026-09-27",
|
||||
"groups": 2107,
|
||||
"by_access": {
|
||||
"nothing found": 369,
|
||||
|
|
@ -19768,7 +19768,7 @@
|
|||
"translated_by": "",
|
||||
"translated_year": "",
|
||||
"scripture_url": "",
|
||||
"film_url": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"film_url": "https://arc.gt/g8wmw",
|
||||
"audio_url": "",
|
||||
"bible_video_url": "",
|
||||
"stories_url": "",
|
||||
|
|
@ -20048,7 +20048,7 @@
|
|||
"translated_by": "",
|
||||
"translated_year": "",
|
||||
"scripture_url": "",
|
||||
"film_url": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"film_url": "https://arc.gt/g8wmw",
|
||||
"audio_url": "",
|
||||
"bible_video_url": "",
|
||||
"stories_url": "",
|
||||
|
|
@ -21928,7 +21928,7 @@
|
|||
"translated_by": "",
|
||||
"translated_year": "",
|
||||
"scripture_url": "",
|
||||
"film_url": "https://www.jesusfilm.org/watch/?languageId=7138",
|
||||
"film_url": "https://arc.gt/g8wmw",
|
||||
"audio_url": "",
|
||||
"bible_video_url": "",
|
||||
"stories_url": "",
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
"bridge_strength_note": "MEASURED: 24% of aligned rows rest on a bridge shared by only 2-3 groups. bridge_strength grades this (strong >=8 groups, moderate >=4, weak <4). A WEAK bridge is a hypothesis, not a route — e.g. \"Gawri\" was offered as the bridge for 10.3M Pakistanis where the real lingua franca is Urdu, which no unengaged group here claims as primary and which this method therefore cannot see. Filter on bridge_strength before acting.",
|
||||
"bridge_caveat": "A bridge candidate is measured from THIS corpus (a language primary to >1 DX group in the country). It is NOT proof the group speaks it, and it is NOT a national trade language — a lingua franca no unengaged group claims as primary is invisible here.",
|
||||
"pools_caveat": "Resource status is from eBible + JESUS Film. GRN and FCBH audio are key-blocked and unchecked — absence is never proof.",
|
||||
"resource_flags_refreshed": "2026-09-25",
|
||||
"resource_flags_refreshed": "2026-09-27",
|
||||
"resource_flags_source": "data/tables/people-group-pins.json — the same index the map draws. Curated bridge, internet and channel columns are untouched."
|
||||
},
|
||||
"columns": [
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"meta": {
|
||||
"title": "Gospel resource link verification",
|
||||
"generated": "2026-09-25T09:56:59.860Z",
|
||||
"generated": "2026-09-27T01:21:18.374Z",
|
||||
"method": "HEAD, redirects followed. verified = HTTP 200. broken = 404/410. unknown = unreachable, an ambiguous status, or a URL shape that cannot be checked by fetch.",
|
||||
"scope": "Every URL in data/tables/people-group-pins.json, deduplicated.",
|
||||
"states": {
|
||||
|
|
@ -96,12 +96,12 @@
|
|||
]
|
||||
},
|
||||
"summary": {
|
||||
"records": 8524,
|
||||
"checkedThisPass": 8,
|
||||
"records": 8525,
|
||||
"checkedThisPass": 1,
|
||||
"carriedForward": 74,
|
||||
"pruned": 0,
|
||||
"broken": 74,
|
||||
"verified": 8445,
|
||||
"verified": 8446,
|
||||
"unknown": 5
|
||||
},
|
||||
"byHost": {
|
||||
|
|
@ -123,7 +123,7 @@
|
|||
"broken": 1
|
||||
},
|
||||
"arc.gt": {
|
||||
"verified": 33,
|
||||
"verified": 34,
|
||||
"broken": 1
|
||||
},
|
||||
"zume.training": {
|
||||
|
|
@ -42766,6 +42766,11 @@
|
|||
"state": "verified",
|
||||
"detail": "5fish redirected to its language page https://fivefish.org/15537?language=Pashai%2C%20Southeast (the site answers 200 for any path)",
|
||||
"at": "2026-09-25T09:56:59.503Z"
|
||||
},
|
||||
"https://arc.gt/g8wmw": {
|
||||
"state": "verified",
|
||||
"detail": "HTTP 200 (via https://www.jesusfilm.org/watch/jesus.html/jordanian-sign-language.html)",
|
||||
"at": "2026-09-27T01:21:17.988Z"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -11,7 +11,7 @@
|
|||
"honest_limit": "This is NOT the same as a national trade language. It finds languages widely used AMONG THE UNENGAGED GROUPS TRACKED HERE, and will MISS a lingua franca that no unengaged group claims as primary (e.g. Hindi in India). Named bridge_candidates, not trade_languages, for that reason.",
|
||||
"why_not_a_list": "No open machine-readable list of national trade languages was available; bridge candidates are derived from this corpus."
|
||||
},
|
||||
"resource_flags_refreshed": "2026-09-25",
|
||||
"resource_flags_refreshed": "2026-09-27",
|
||||
"resource_flags_source": "data/tables/people-group-pins.json. prevalence_rank, bridge_candidate, groups_speaking and full_bible are curated and untouched."
|
||||
},
|
||||
"columns": [
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load diff
Loading…
Reference in a new issue