Harvest the whole Arclight catalogue: film coverage 306 -> 2,087 languages

Same shape as the eBible finding, same cause. Arclight was ALREADY a checked pool and
only 306 language codes carried a film link, while the catalogue lists 2,330 media
languages with 2,087 carrying a feature film. Being inside a pool is not the same as
having been read.

COVERAGE: pins with a real URL 6,985 -> 8,935, 56.4% -> 72.2%. Language codes 2,068 ->
3,196. 1,950 pins lift off "nothing found" and 1,214 move to Scripture AND film. The
resource table grows 1,497 -> 3,043 rows across 2,307 languages.

★ THE VERIFICATION HAD TO BE REDONE, and that is the part worth keeping. The obvious
check is to sample watch URLs and confirm 200. Ten of ten passed. The check was
worthless: https://www.jesusfilm.org/watch/?languageId=<id> is a single-page app and
answers 200 for EVERY id — measured, `0` and `abcdef` both return 200. A check that
cannot fail proves nothing, and that one would have certified 1,781 links on a status
code that is always 200.

The API discriminates where the page does not: media-languages/<id> returns a real
counts.featureFilm for a genuine language and 404 for a bogus one. So existence is
established from the API and the watch URL is the deep link for a language already
confirmed. Re-verified that way: 10/10 sampled languages carry featureFilm >= 1, and
the negative control — a bogus id — returns featureFilm 0, so the check can fail and
does, on the case that should fail.

featureFilm >= 1 IS THE COVERAGE TEST, not roster membership: 21 of the 2,330 listed
media languages have no feature film and are dropped. Only featureFilm is taken;
Arclight also carries short films and other media that were not screened.

FILL, NEVER OVERWRITE, as with eBible — a curated film_url stands, and the catalogue
only supplies a link where there was none. 1,695 of the codes had no row in the
resources table at all.

Verified: all browser probes pass — pins, table, deep link — 12,379 pins rendered, and
the map now reads green across Papua New Guinea, Indonesia, Brazil and Central Africa
where it was red, leaving the remaining red concentrated in the Sahel, the Horn, north
India and Central Asia. The pin table is 1.1MB but 281KB gzipped; the four repeated
pool strings compress 4:1, so interning them would buy nothing on the wire.

data/tables/index.json

@@ -68,7 +68,7 @@
   },
   {
    "name": "gospel-resource-links",
-   "rows": 1497
+   "rows": 3043
   }
  ]
 }

plugin-designs/gospel-resources/scripts/build-people-group-pins.mjs

@@ -40,6 +40,7 @@ const OUT = resolve(REPO, 'data/tables/people-group-pins.json')
 const INDEX = resolve(REPO, 'data/tables/index.json')
 const D43 = resolve(REPO, 'data/tables/door43-index.json')
 const EBIBLE = resolve(REPO, 'data/tables/ebible-index.json')
+const ARC = resolve(REPO, 'data/tables/arclight-index.json')
 
 const rowsOf = (p) => {
   const d = JSON.parse(readFileSync(p, 'utf8'))
@@ -58,6 +59,9 @@ try { d43 = JSON.parse(readFileSync(D43, 'utf8')).codes || {} } catch { d43 = {}
 // eBible catalogue snapshot (scripts/fetch-ebible-index.mjs). Same optional contract.
 let eb = {}
 try { eb = JSON.parse(readFileSync(EBIBLE, 'utf8')).codes || {} } catch { eb = {} }
+// Arclight / JESUS Film snapshot (scripts/fetch-arclight-index.mjs).
+let arc = {}
+try { arc = JSON.parse(readFileSync(ARC, 'utf8')).codes || {} } catch { arc = {} }
 
 // codes: one entry per ISO 639-3 that actually carries something worth linking to.
 const codes = {}
@@ -72,6 +76,8 @@ for (const r of res) {
   // earlier pass; the catalogue is broad but not hand-checked, so it only supplies a
   // link where there was none. Curation beats coverage when the two disagree.
   const ebUrl = !s0 && ee ? ee.u : ''
+  const aa = arc[c]
+  const arcUrl = !f && aa ? aa.u : ''
   codes[c] = {
     // Published Door43 resources, each a real browsable URL. `subject` travels with the
     // link so the UI can say "Stories" rather than implying a Bible translation.
@@ -79,7 +85,7 @@ for (const r of res) {
     l: r.language || '',
     ...((s0 || ebUrl) && { s: s0 || ebUrl }),
     ...(!s0 && ebUrl ? { sv: ee.t || '', sf: ee.full ? 1 : 0 } : {}),
-    ...(f && { f }),
+    ...((f || arcUrl) && { f: f || arcUrl }),
     // has_audio carries a meaningful third state: 'unknown' means a pool we could not
     // query (key-blocked), NOT a no. Collapsing it to a boolean would turn "we did not
     // look" into "there is nothing" — the exact dishonesty the sheet's note guards.
@@ -111,6 +117,12 @@ for (const [c, dd] of Object.entries(d43)) {
 // Same pass for eBible: 1,237 catalogued codes against 862 in the resources table, so most
 // of the catalogue describes languages that have no row to attach a link to. Without this
 // their pins keep saying "nothing found" while a published translation exists.
+let arcOnly = 0
+for (const [c, aa] of Object.entries(arc)) {
+  if (codes[c]) { if (!codes[c].f) codes[c].f = aa.u; continue }
+  codes[c] = { l: aa.n || '', f: aa.u, ck: 'arclight', un: 'ebible, door43, grn, api_bible, dbl, progress_bible' }
+  arcOnly++
+}
 for (const [c, ee] of Object.entries(eb)) {
   if (codes[c]) {
     // code exists but came from the Door43-only pass above and has no scripture yet
@@ -172,6 +184,7 @@ console.log(`REAL URL        ${withUrl} (${pct(withUrl)})`)
 console.log(`language codes  ${Object.keys(codes).length}`)
 console.log(`door43 codes    ${Object.keys(d43).length}  (${d43Only} new)`)
 console.log(`ebible codes    ${Object.keys(eb).length}  (${ebOnly} new)`)
+console.log(`arclight codes  ${Object.keys(arc).length}  (${arcOnly} new)`)
 console.log(`lifted off 0    ${liftedByD43}  (had nothing; now have a published Door43 resource)`)
 console.log(`dropped nocoord ${noCoord}`)
 console.log(`wrote ${OUT} (${(readFileSync(OUT).length / 1024).toFixed(0)} KB)`)

plugin-designs/gospel-resources/scripts/fetch-arclight-index.mjs

@@ -0,0 +1,74 @@
+#!/usr/bin/env node
+/**
+ * fetch-arclight-index.mjs — cache the JESUS Film Project's Arclight catalogue by ISO 639-3.
+ *
+ * THE SAME SHAPE THAT PAID OFF WITH EBIBLE. Arclight was ALREADY a checked pool, and only
+ * 306 language codes carried a film link. The catalogue lists 2,330 media languages, 2,087
+ * of them with a feature film. Being inside a pool is not the same as having been read.
+ *
+ * ★ featureFilm IS THE COVERAGE SIGNAL, NOT ROSTER MEMBERSHIP. A language appearing in
+ * media-languages does not mean the film exists in it; counts.featureFilm >= 1 does. 21 of
+ * the 2,330 are listed with no feature film and are dropped here.
+ *
+ * ★ THE WATCH URL CANNOT BE VERIFIED BY STATUS CODE, and this is the part worth reading.
+ * https://www.jesusfilm.org/watch/?languageId=<id> is a single-page app: it answers 200 for
+ * EVERY id, including `0` and `abcdef`. MEASURED — all three return 200. So a sample of
+ * watch-URLs "resolving 200" is a check that cannot fail and proves nothing at all about
+ * whether a film exists.
+ *
+ * The API is what discriminates: media-languages/<id> returns 200 with a real
+ * counts.featureFilm for a genuine language and 404 for a bogus one. So existence is
+ * established from the API, and the watch URL is the deep link for a language the API has
+ * already confirmed. Verifying it the other way round would have shipped 1,781 links on the
+ * strength of a status code that is always 200.
+ *
+ * DOCTRINE: the JESUS Film is a dramatisation of Luke's Gospel by the JESUS Film Project.
+ * Inside the filter, and named explicitly in the standing instruction. Only featureFilm is
+ * taken; Arclight also carries short films and other media that are not screened here.
+ */
+import { writeFileSync } from 'node:fs'
+import { resolve, dirname } from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+const HERE = dirname(fileURLToPath(import.meta.url))
+const OUT = resolve(HERE, '../../..', 'data/tables/arclight-index.json')
+const API = 'https://api.arclight.org/v2/media-languages'
+const UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'
+const WATCH = (id) => `https://www.jesusfilm.org/watch/?languageId=${id}`
+
+const all = []
+let page = 1, pages = 1
+do {
+  const res = await fetch(`${API}?limit=1000&page=${page}`, { headers: { 'User-Agent': UA } })
+  if (!res.ok) throw new Error(`arclight page ${page}: HTTP ${res.status}`)
+  const d = await res.json()
+  pages = d.pages && d.limit ? Math.ceil(d.total / d.limit) : 1
+  all.push(...(d._embedded?.mediaLanguages || []))
+  page++
+} while (page <= pages && page < 20)
+
+const stats = { fetched: all.length, noFeatureFilm: 0, noIso: 0, kept: 0 }
+const codes = {}
+for (const r of all) {
+  const ff = Number(r?.counts?.featureFilm?.value || 0)
+  if (ff < 1) { stats.noFeatureFilm++; continue }
+  const iso = (r.iso3 || '').trim()
+  if (iso.length !== 3) { stats.noIso++; continue }
+  const prev = codes[iso]
+  // more speakers wins when one ISO code maps to several Arclight languages
+  const spk = Number(r?.counts?.speakerCount?.value || 0)
+  if (!prev || spk > prev.spk) {
+    codes[iso] = { u: WATCH(r.languageId), id: r.languageId, n: r.name || '', spk }
+  }
+}
+stats.kept = Object.keys(codes).length
+
+writeFileSync(OUT, JSON.stringify({
+  _note: 'Arclight (JESUS Film Project) catalogue by ISO 639-3. Only languages with counts.featureFilm >= 1. Existence is established from the API: the /watch/ URL answers 200 for ANY id and cannot verify anything.',
+  generated: new Date().toISOString().slice(0, 10),
+  source: API,
+  stats,
+  codes,
+}))
+console.log(JSON.stringify(stats))
+console.log(`${stats.kept} ISO 639-3 codes -> ${OUT}`)

public/assets/tables/index.json

@@ -68,7 +68,7 @@
   },
   {
    "name": "gospel-resource-links",
-   "rows": 1497
+   "rows": 3043
   }
  ]
 }