f9808d0c8b
Same shape as the eBible finding, same cause. Arclight was ALREADY a checked pool and only 306 language codes carried a film link, while the catalogue lists 2,330 media languages with 2,087 carrying a feature film. Being inside a pool is not the same as having been read. COVERAGE: pins with a real URL 6,985 -> 8,935, 56.4% -> 72.2%. Language codes 2,068 -> 3,196. 1,950 pins lift off "nothing found" and 1,214 move to Scripture AND film. The resource table grows 1,497 -> 3,043 rows across 2,307 languages. ★ THE VERIFICATION HAD TO BE REDONE, and that is the part worth keeping. The obvious check is to sample watch URLs and confirm 200. Ten of ten passed. The check was worthless: https://www.jesusfilm.org/watch/?languageId=<id> is a single-page app and answers 200 for EVERY id — measured, `0` and `abcdef` both return 200. A check that cannot fail proves nothing, and that one would have certified 1,781 links on a status code that is always 200. The API discriminates where the page does not: media-languages/<id> returns a real counts.featureFilm for a genuine language and 404 for a bogus one. So existence is established from the API and the watch URL is the deep link for a language already confirmed. Re-verified that way: 10/10 sampled languages carry featureFilm >= 1, and the negative control — a bogus id — returns featureFilm 0, so the check can fail and does, on the case that should fail. featureFilm >= 1 IS THE COVERAGE TEST, not roster membership: 21 of the 2,330 listed media languages have no feature film and are dropped. Only featureFilm is taken; Arclight also carries short films and other media that were not screened. FILL, NEVER OVERWRITE, as with eBible — a curated film_url stands, and the catalogue only supplies a link where there was none. 1,695 of the codes had no row in the resources table at all. Verified: all browser probes pass — pins, table, deep link — 12,379 pins rendered, and the map now reads green across Papua New Guinea, Indonesia, Brazil and Central Africa where it was red, leaving the remaining red concentrated in the Sahel, the Horn, north India and Central Asia. The pin table is 1.1MB but 281KB gzipped; the four repeated pool strings compress 4:1, so interning them would buy nothing on the wire.
74 lines
3.6 KiB
JavaScript
74 lines
3.6 KiB
JavaScript
#!/usr/bin/env node
|
|
/**
|
|
* fetch-arclight-index.mjs — cache the JESUS Film Project's Arclight catalogue by ISO 639-3.
|
|
*
|
|
* THE SAME SHAPE THAT PAID OFF WITH EBIBLE. Arclight was ALREADY a checked pool, and only
|
|
* 306 language codes carried a film link. The catalogue lists 2,330 media languages, 2,087
|
|
* of them with a feature film. Being inside a pool is not the same as having been read.
|
|
*
|
|
* ★ featureFilm IS THE COVERAGE SIGNAL, NOT ROSTER MEMBERSHIP. A language appearing in
|
|
* media-languages does not mean the film exists in it; counts.featureFilm >= 1 does. 21 of
|
|
* the 2,330 are listed with no feature film and are dropped here.
|
|
*
|
|
* ★ THE WATCH URL CANNOT BE VERIFIED BY STATUS CODE, and this is the part worth reading.
|
|
* https://www.jesusfilm.org/watch/?languageId=<id> is a single-page app: it answers 200 for
|
|
* EVERY id, including `0` and `abcdef`. MEASURED — all three return 200. So a sample of
|
|
* watch-URLs "resolving 200" is a check that cannot fail and proves nothing at all about
|
|
* whether a film exists.
|
|
*
|
|
* The API is what discriminates: media-languages/<id> returns 200 with a real
|
|
* counts.featureFilm for a genuine language and 404 for a bogus one. So existence is
|
|
* established from the API, and the watch URL is the deep link for a language the API has
|
|
* already confirmed. Verifying it the other way round would have shipped 1,781 links on the
|
|
* strength of a status code that is always 200.
|
|
*
|
|
* DOCTRINE: the JESUS Film is a dramatisation of Luke's Gospel by the JESUS Film Project.
|
|
* Inside the filter, and named explicitly in the standing instruction. Only featureFilm is
|
|
* taken; Arclight also carries short films and other media that are not screened here.
|
|
*/
|
|
import { writeFileSync } from 'node:fs'
|
|
import { resolve, dirname } from 'node:path'
|
|
import { fileURLToPath } from 'node:url'
|
|
|
|
const HERE = dirname(fileURLToPath(import.meta.url))
|
|
const OUT = resolve(HERE, '../../..', 'data/tables/arclight-index.json')
|
|
const API = 'https://api.arclight.org/v2/media-languages'
|
|
const UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'
|
|
const WATCH = (id) => `https://www.jesusfilm.org/watch/?languageId=${id}`
|
|
|
|
const all = []
|
|
let page = 1, pages = 1
|
|
do {
|
|
const res = await fetch(`${API}?limit=1000&page=${page}`, { headers: { 'User-Agent': UA } })
|
|
if (!res.ok) throw new Error(`arclight page ${page}: HTTP ${res.status}`)
|
|
const d = await res.json()
|
|
pages = d.pages && d.limit ? Math.ceil(d.total / d.limit) : 1
|
|
all.push(...(d._embedded?.mediaLanguages || []))
|
|
page++
|
|
} while (page <= pages && page < 20)
|
|
|
|
const stats = { fetched: all.length, noFeatureFilm: 0, noIso: 0, kept: 0 }
|
|
const codes = {}
|
|
for (const r of all) {
|
|
const ff = Number(r?.counts?.featureFilm?.value || 0)
|
|
if (ff < 1) { stats.noFeatureFilm++; continue }
|
|
const iso = (r.iso3 || '').trim()
|
|
if (iso.length !== 3) { stats.noIso++; continue }
|
|
const prev = codes[iso]
|
|
// more speakers wins when one ISO code maps to several Arclight languages
|
|
const spk = Number(r?.counts?.speakerCount?.value || 0)
|
|
if (!prev || spk > prev.spk) {
|
|
codes[iso] = { u: WATCH(r.languageId), id: r.languageId, n: r.name || '', spk }
|
|
}
|
|
}
|
|
stats.kept = Object.keys(codes).length
|
|
|
|
writeFileSync(OUT, JSON.stringify({
|
|
_note: 'Arclight (JESUS Film Project) catalogue by ISO 639-3. Only languages with counts.featureFilm >= 1. Existence is established from the API: the /watch/ URL answers 200 for ANY id and cannot verify anything.',
|
|
generated: new Date().toISOString().slice(0, 10),
|
|
source: API,
|
|
stats,
|
|
codes,
|
|
}))
|
|
console.log(JSON.stringify(stats))
|
|
console.log(`${stats.kept} ISO 639-3 codes -> ${OUT}`)
|