Harvest the whole eBible catalogue, not a tenth of it

eBible was ALREADY one of the two pools this repo checked, and that is what made the
gap invisible: 120 language codes carried an eBible link while the catalogue lists
1,550 translations across 1,239 codes with canonical books. The pool had been read at
roughly a tenth of its content, and ~1,600 people groups were being told "nothing
found" about a published translation in their own language.

COVERAGE: pins carrying a real URL 5,349 -> 6,985, 43.2% -> 56.4%. Language codes
978 -> 2,068. The resource table grows 581 -> 1,497 rows across 1,314 languages. 1,090
of the new codes had no row in the resources table at all, which is why filling gaps
in the existing 862 would have missed most of it.

FILL, NEVER OVERWRITE. An existing scripture_url was curated and verified by an earlier
pass; the catalogue is broad but not hand-checked, so it only supplies a link where
there was none. Where the two disagree, curation wins.

CANON TEST: OTbooks + NTbooks > 0. A deuterocanonical-only publication is not the
66-book canon and is dropped; DC books ALONGSIDE OT/NT are not a reason to drop, since
the 66 are present either way. Measured on this snapshot: 3 rows carry no books at all.

DOMAINS ARE ALLOW-LISTED AFTER A SCREEN, because publicationURL does not always point
at ebible.org. Four domains appear and each was fetched and read: ebible.org (1,242),
png.bible (302, "Papua New Guinea Scriptures"), alkitab.pw (3, an eBible mirror), and
inscript.org (1) — EXCLUDED, because its page establishes no publisher or doctrinal
provenance and an unclear source is left out. A domain not on the list is dropped, so a
future catalogue update cannot introduce a publisher nobody screened.

TWO BROKEN LINKS CAUGHT STRUCTURALLY: nko and nlx publish 'http:///nkoNT/' — a
malformed URL with an EMPTY host, broken in eBible's own data. Requiring a non-empty
host rejects them; a looser check for the 'http://' prefix passes them happily, which
is how a "real" link that resolves nowhere gets shipped. badUrl and badDomain are
counted separately for the same reason — folded together, this hid.

Verified: the CSV parser was diffed against a reference parser across all 1,550 rows,
0 mismatches, so no URL is attached to the wrong language. A seeded random sample of 20
catalogue URLs resolved 20/20 at 200. All five browser probes still pass — pins, table,
deep link, Door43 lift, and 12,379 pins rendered — and the map now shows Papua New
Guinea, Indonesia and South America carrying resource colour where they were solid red.

data/tables/index.json

@@ -68,7 +68,7 @@
   },
   {
    "name": "gospel-resource-links",
-   "rows": 581
+   "rows": 1497
   }
  ]
 }

plugin-designs/gospel-resources/scripts/build-people-group-pins.mjs

@@ -39,6 +39,7 @@ const SRC_RES = resolve(REPO, 'data/tables/people-group-resources.json')
 const OUT = resolve(REPO, 'data/tables/people-group-pins.json')
 const INDEX = resolve(REPO, 'data/tables/index.json')
 const D43 = resolve(REPO, 'data/tables/door43-index.json')
+const EBIBLE = resolve(REPO, 'data/tables/ebible-index.json')
 
 const rowsOf = (p) => {
   const d = JSON.parse(readFileSync(p, 'utf8'))
@@ -54,21 +55,30 @@ const res = rowsOf(SRC_RES)
 // build still runs and simply adds no Door43 links, rather than failing the whole table.
 let d43 = {}
 try { d43 = JSON.parse(readFileSync(D43, 'utf8')).codes || {} } catch { d43 = {} }
+// eBible catalogue snapshot (scripts/fetch-ebible-index.mjs). Same optional contract.
+let eb = {}
+try { eb = JSON.parse(readFileSync(EBIBLE, 'utf8')).codes || {} } catch { eb = {} }
 
 // codes: one entry per ISO 639-3 that actually carries something worth linking to.
 const codes = {}
 for (const r of res) {
   const c = r.iso639_3
   if (!c) continue
-  const s = r.scripture_url || ''
+  const s0 = r.scripture_url || ''
   const f = r.film_url || ''
   const dd = d43[c]
+  const ee = eb[c]
+  // FILL, NEVER OVERWRITE. An existing scripture_url was curated and verified by an
+  // earlier pass; the catalogue is broad but not hand-checked, so it only supplies a
+  // link where there was none. Curation beats coverage when the two disagree.
+  const ebUrl = !s0 && ee ? ee.u : ''
   codes[c] = {
     // Published Door43 resources, each a real browsable URL. `subject` travels with the
     // link so the UI can say "Stories" rather than implying a Bible translation.
     ...(dd && dd.items.length ? { d: dd.items.map((i) => ({ s: i.subject, u: i.url })) } : {}),
     l: r.language || '',
-    ...(s && { s }),
+    ...((s0 || ebUrl) && { s: s0 || ebUrl }),
+    ...(!s0 && ebUrl ? { sv: ee.t || '', sf: ee.full ? 1 : 0 } : {}),
     ...(f && { f }),
     // has_audio carries a meaningful third state: 'unknown' means a pool we could not
     // query (key-blocked), NOT a no. Collapsing it to a boolean would turn "we did not
@@ -87,7 +97,7 @@ for (const r of res) {
 // languages outside that set. Without this pass their pins stay tier 0 — the map telling a
 // reader "nothing found" for a group that has a published resource, which is the one error
 // this map must never make. MEASURED: 213 pins across ~64 languages were in exactly that state.
-let d43Only = 0
+let d43Only = 0, ebOnly = 0
 for (const [c, dd] of Object.entries(d43)) {
   if (codes[c] || !dd.items?.length) continue
   codes[c] = {
@@ -98,6 +108,18 @@ for (const [c, dd] of Object.entries(d43)) {
   }
   d43Only++
 }
+// Same pass for eBible: 1,237 catalogued codes against 862 in the resources table, so most
+// of the catalogue describes languages that have no row to attach a link to. Without this
+// their pins keep saying "nothing found" while a published translation exists.
+for (const [c, ee] of Object.entries(eb)) {
+  if (codes[c]) {
+    // code exists but came from the Door43-only pass above and has no scripture yet
+    if (!codes[c].s) { codes[c].s = ee.u; codes[c].sv = ee.t || ''; codes[c].sf = ee.full ? 1 : 0 }
+    continue
+  }
+  codes[c] = { l: '', s: ee.u, sv: ee.t || '', sf: ee.full ? 1 : 0, ck: 'ebible', un: 'arclight, grn, api_bible, dbl, progress_bible' }
+  ebOnly++
+}
 
 const num = (v) => { const n = Number(v); return Number.isFinite(n) ? n : null }
 const pins = []
@@ -148,7 +170,8 @@ console.log(`pins            ${pins.length}`)
 console.log(`joined a record ${joined} (${pct(joined)})`)
 console.log(`REAL URL        ${withUrl} (${pct(withUrl)})`)
 console.log(`language codes  ${Object.keys(codes).length}`)
-console.log(`door43 codes    ${Object.keys(d43).length}  (${d43Only} not present in the resources table at all)`)
+console.log(`door43 codes    ${Object.keys(d43).length}  (${d43Only} new)`)
+console.log(`ebible codes    ${Object.keys(eb).length}  (${ebOnly} new)`)
 console.log(`lifted off 0    ${liftedByD43}  (had nothing; now have a published Door43 resource)`)
 console.log(`dropped nocoord ${noCoord}`)
 console.log(`wrote ${OUT} (${(readFileSync(OUT).length / 1024).toFixed(0)} KB)`)

plugin-designs/gospel-resources/scripts/build-resource-links-table.mjs

@@ -42,8 +42,13 @@ for (const [name, , , iso3, code, pop] of rows) {
   if (r.sample.length < 3) r.sample.push(name)
 }
 
+// Every publication domain that can appear, named. 'other' is a fallback that should stay
+// empty: 258 rows landed in it when png.bible was missing here, which tells a reader nothing
+// about where the link goes and hides an unscreened domain if one ever slips into the index.
 const SOURCE = (u) =>
   /ebible\.org/i.test(u) ? 'eBible.org'
+  : /png\.bible/i.test(u) ? 'PNG.Bible'
+  : /alkitab\.pw/i.test(u) ? 'eBible.org (mirror)'
   : /jesusfilm|arc\.gt/i.test(u) ? 'JESUS Film'
   : /door43/i.test(u) ? 'Door43 (unfoldingWord)'
   : 'other'

plugin-designs/gospel-resources/scripts/fetch-ebible-index.mjs

@@ -0,0 +1,113 @@
+#!/usr/bin/env node
+/**
+ * fetch-ebible-index.mjs — cache eBible.org's translation catalogue as an ISO 639-3 index.
+ *
+ * THE FINDING THIS EXISTS FOR. eBible was ALREADY one of the two pools this repo checked,
+ * but only 120 language codes carried an eBible link. The catalogue lists 1,550 translations
+ * across 1,239 codes with canonical books — the pool had been harvested at roughly a tenth
+ * of its content, and ~1,600 people groups were reading "nothing found" because of it.
+ *
+ * CANON TEST: a row is kept only if OTbooks + NTbooks > 0. A publication that is
+ * deuterocanonical ONLY is not the 66-book canon and is dropped. (Measured on this
+ * snapshot: 1,547 of 1,550 rows pass, 0 are DC-only — the 3 dropped carry no books at all.)
+ * DCbooks alongside OT/NT is NOT a reason to drop: many canon publications append them, and
+ * the 66 books are present either way. The type is what matters, not the appendix.
+ *
+ * DOCTRINE SCREEN ON THE PUBLICATION DOMAIN, not just on eBible's reputation — publicationURL
+ * does not always point at ebible.org. Four domains appear in this catalogue and each was
+ * checked by fetching it:
+ *   ebible.org   1242  — the pool already in use, public-domain and openly-licensed Bibles
+ *   png.bible     302  — "Papua New Guinea Scriptures", Holy Bible in PNG languages
+ *   alkitab.pw      3  — serves eBible.org's own site; a mirror
+ *   inscript.org    1  — EXCLUDED. "inScript Bible Browser" and nothing else; its page
+ *                        establishes no publisher or doctrinal provenance, and the standing
+ *                        rule is that an unclear source is left out. Cost: one row.
+ * Any domain not on the allow-list below is dropped, so a future catalogue update cannot
+ * silently introduce a publisher nobody screened.
+ *
+ * ONE TRANSLATION PER LANGUAGE: the one with the most canonical books, tie-broken by verses.
+ * A language with a full Bible should not be represented by a single-gospel portion.
+ */
+import { writeFileSync } from 'node:fs'
+import { resolve, dirname } from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+const HERE = dirname(fileURLToPath(import.meta.url))
+const OUT = resolve(HERE, '../../..', 'data/tables/ebible-index.json')
+const SRC = 'https://ebible.org/Scriptures/translations.csv'
+const UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'
+const ALLOWED = new Set(['ebible.org', 'png.bible', 'alkitab.pw'])
+
+const res = await fetch(SRC, { headers: { 'User-Agent': UA } })
+if (!res.ok) throw new Error(`ebible catalogue: HTTP ${res.status}`)
+const text = await res.text()
+
+// minimal RFC4180 parse — titles contain commas and quoted quotes
+function parseCsv(s) {
+  const rows = []; let row = [], f = '', q = false
+  for (let i = 0; i < s.length; i++) {
+    const c = s[i]
+    if (q) {
+      if (c === '"' && s[i + 1] === '"') { f += '"'; i++ }
+      else if (c === '"') q = false
+      else f += c
+    } else if (c === '"') q = true
+    else if (c === ',') { row.push(f); f = '' }
+    else if (c === '\n') { row.push(f); f = ''; rows.push(row); row = [] }
+    else if (c !== '\r') f += c
+  }
+  if (f || row.length) { row.push(f); rows.push(row) }
+  const head = rows.shift().map((h) => h.replace(/^/, '').trim())
+  return rows.filter((r) => r.length > 1).map((r) => Object.fromEntries(head.map((h, i) => [h, r[i] ?? ''])))
+}
+
+const num = (v) => (String(v).trim().match(/^\d+$/) ? Number(v) : 0)
+const rows = parseCsv(text)
+// badUrl is kept SEPARATE from badDomain: they are different defects and folding them
+// together hid one. MEASURED in this snapshot: nko and nlx publish 'http:///nkoNT/' — a
+// malformed URL with an EMPTY host, broken in eBible's own catalogue. A loose check for
+// the 'http://' prefix passes it happily; requiring a non-empty host is what rejects it.
+// Two broken links out of 1,550 is the difference between shipping real resources and
+// shipping a 'real' link that resolves nowhere.
+const stats = { total: rows.length, noCanon: 0, badUrl: 0, badDomain: 0, kept: 0 }
+const best = {}
+for (const r of rows) {
+  const code = (r.languageCode || '').trim()
+  const url = (r.publicationURL || '').trim()
+  if (!code || code.length !== 3 || !url) continue
+  const books = num(r.OTbooks) + num(r.NTbooks)
+  if (books === 0) { stats.noCanon++; continue }
+  const host = (url.match(/^https?:\/\/([^/]+)/) || [])[1]?.toLowerCase()
+  if (!host) { stats.badUrl++; (stats.malformed ||= []).push(`${code} ${url}`); continue }
+  if (!ALLOWED.has(host)) {
+    stats.badDomain++
+    ;(stats.excluded ||= []).push(`${code} ${host} ${(r.shortTitle || '').slice(0, 30)}`)
+    continue
+  }
+  const verses = num(r.OTverses) + num(r.NTverses)
+  const cur = best[code]
+  if (!cur || books > cur.books || (books === cur.books && verses > cur.verses)) {
+    best[code] = {
+      u: url,
+      t: (r.shortTitle || r.title || '').trim(),
+      books,
+      verses,
+      nt: num(r.NTbooks) > 0,
+      ot: num(r.OTbooks) > 0,
+      full: num(r.OTbooks) >= 39 && num(r.NTbooks) >= 27,
+    }
+  }
+}
+stats.kept = Object.keys(best).length
+
+const out = {
+  _note: 'eBible.org translation catalogue, keyed by ISO 639-3. One translation per language (most canonical books). Rows with no OT/NT books are dropped; publication domains are allow-listed after a doctrine screen. inscript.org excluded: no established provenance.',
+  generated: new Date().toISOString().slice(0, 10),
+  source: SRC,
+  allowed_domains: [...ALLOWED],
+  stats,
+  codes: best,
+}
+writeFileSync(OUT, JSON.stringify(out))
+console.log(JSON.stringify(stats))
+console.log(`${stats.kept} ISO 639-3 codes -> ${OUT}`)

public/assets/tables/index.json

@@ -68,7 +68,7 @@
   },
   {
    "name": "gospel-resource-links",
-   "rows": 581
+   "rows": 1497
   }
  ]
 }