Two curated tables stopped disagreeing with the map about what exists

gaps-and-opportunities and people-group-resources predate everything added this flight.
Their curated columns — the bridge-language analysis, internet access, messaging
channels — are the reason they exist and are untouched. Their resource booleans had
drifted into saying things the map disproves:

  80 rows     heart_has_any=false while the index has a resource   (6M people)
  317 rows    "no scripture" while the index has TEXT Scripture
  277 rows    the same, for the bridge language
  5 rows      "no film" while the index has one
  2,106 rows  has_audio = the string "unknown", written when no audio pool had been
              searched. Faith Comes By Hearing has been searched since.

Only the facts the pin index is authoritative for are rewritten. Rebuilding these tables
would have destroyed curated work to fix a few columns.

I GOT THE DERIVED FIELDS WRONG FIRST AND CAUGHT IT BEFORE COMMITTING. `gap` reads
naturally as "no resource in the heart language", and defining it that way changed 416
rows. The table's own meta defines it as "nothing found in the heart language AND no
in-country bridge language carries anything" — my version would have silently discarded
half its meaning while looking like a correction. Every formula is now taken from the
table's documented meaning and was verified to reproduce the existing column on all
2,106 rows BEFORE anything was rewritten: gap, opportunity (which also requires the
bridge to be spoken by more than one group in the country), aligned, and status. With
the right formula, gap moves 12 rows, not 416.

has_audio stays THREE-STATE and never false: true where FCBH publishes audio, "unknown"
otherwise. GRN is the main audio pool and answers 403 to everything, so "no audio
exists" is a claim this build cannot make. 1,297 true, 809 unknown.

Verified after the fact, not assumed: zero remaining contradictions between either
table and the pin index, and all four derived formulas still reproduce their columns
2,106/2,106. Both tables written through the snapshot guard. Tables tab still renders
real cells.
This commit is contained in:
doxa-maps-88655c70 2026-09-24 13:25:14 +00:00
parent 78e9f3cc2f
commit 85a3d61780
7 changed files with 7782 additions and 7656 deletions

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,114 @@
#!/usr/bin/env node
/**
* refresh-legacy-resource-flags.mjs — re-derive the resource booleans in the two curated
* tables from the pin index, so they stop contradicting the map beside them.
*
* THE PROBLEM. gaps-and-opportunities and people-group-resources predate everything this
* flight added. Their curated columns — the bridge-language analysis, internet access,
* messaging channels — are the reason they exist and are NOT touched here. But they also
* carry resource booleans, and those had drifted into saying things the map disproves:
*
* 80 rows heart_has_any=false while the index has a resource (6M people)
* 317 rows "no scripture" while the index has TEXT Scripture
* 5 rows "no film" while the index has a film
* 2,106 rows has_audio = the string "unknown", written when no audio pool had been
* searched. Faith Comes By Hearing has been searched since.
*
* WHY REFRESH RATHER THAN REBUILD. The bridge analysis cannot be regenerated from the pins —
* it is curated work over a narrower, hand-checked set. Rebuilding would destroy it to fix a
* few columns. So only the facts the index is authoritative for are rewritten, and the
* fields DERIVED from them are recomputed in the same pass, because leaving `gap` or
* `opportunity` stale would just move the contradiction one column to the right.
*
* has_audio STAYS THREE-STATE. true where FCBH publishes audio; "unknown" otherwise — never
* false. GRN is the main audio pool and still answers 403 to everything, so "no audio exists"
* is a claim this build cannot make.
*/
import { readFileSync } from 'node:fs'
import { resolve, dirname } from 'node:path'
import { fileURLToPath } from 'node:url'
import { writeSnapshot } from './_snapshot-guard.mjs'
const HERE = dirname(fileURLToPath(import.meta.url))
const REPO = resolve(HERE, '../../..')
const PINS = resolve(REPO, 'data/tables/people-group-pins.json')
const { codes } = JSON.parse(readFileSync(PINS, 'utf8'))
const v = (c) => codes[c] || null
const hasText = (c) => { const x = v(c); return !!(x && (x.s || (x.au && /text/.test(x.au.m || '')))) }
const hasFilm = (c) => { const x = v(c); return !!(x && x.f) }
const hasAudio = (c) => { const x = v(c); return !!(x && x.au && /audio/.test(x.au.m || '')) }
const hasAny = (c) => { const x = v(c); return !!(x && (x.s || x.f || x.au || x.bv || (x.d && x.d.length))) }
const load = (name) => JSON.parse(readFileSync(resolve(REPO, `data/tables/${name}.json`), 'utf8'))
const changed = {}
const bump = (k) => { changed[k] = (changed[k] || 0) + 1 }
// ── gaps-and-opportunities ────────────────────────────────────────────────────────────
{
const j = load('gaps-and-opportunities')
for (const r of j.rows || []) {
const c = r.heart_iso639_3
if (!c) continue
const s = hasText(c), f = hasFilm(c), any = hasAny(c)
if (r.heart_has_scripture !== s) { r.heart_has_scripture = s; bump('heart_has_scripture') }
if (r.heart_has_film !== f) { r.heart_has_film = f; bump('heart_has_film') }
if (r.heart_has_any !== any) { r.heart_has_any = any; bump('heart_has_any') }
const b = r.bridge_iso639_3
if (b) {
const bs = hasText(b), bf = hasFilm(b)
if (r.bridge_has_scripture !== bs) { r.bridge_has_scripture = bs; bump('bridge_has_scripture') }
if (r.bridge_has_film !== bf) { r.bridge_has_film = bf; bump('bridge_has_film') }
}
// DERIVED FIELDS, RECOMPUTED FROM THE TABLE'S OWN DOCUMENTED DEFINITIONS — not from a
// definition invented here. Each formula below was verified to reproduce the existing
// column on all 2,106 rows before anything was rewritten, because a refresh that
// silently redefines a curated column is worse than the drift it set out to fix. A
// first attempt defined `gap` as simply !heart_has_any, which would have changed 416
// rows and quietly discarded the "and no in-country bridge either" half of its meaning.
//
// gap nothing in the heart language AND no in-country bridge carries anything
// opportunity nothing in the heart language, but a bridge spoken by MORE THAN ONE
// group in the same country does have something
// aligned an opportunity where the environment lines up too: internet yes/partial
// and a known channel
const bridgeHas = !!(r.bridge_has_scripture || r.bridge_has_film)
const gap = !any && !bridgeHas
if (r.gap !== gap) { r.gap = gap; bump('gap') }
const opp = !any && bridgeHas && (r.bridge_groups_in_country || 0) > 1
if (r.opportunity !== opp) { r.opportunity = opp; bump('opportunity') }
const net = r.internet_access === 'yes' || r.internet_access === 'partial'
const aligned = opp && net && !!r.top_channel
if (r.aligned !== aligned) { r.aligned = aligned; bump('aligned') }
const status = any ? 'reached in heart language'
: opp ? 'opportunity — bridge language has a resource'
: 'gap — nothing found in heart or bridge language'
if (r.status !== status) { r.status = status; bump('status') }
}
j.meta = { ...(j.meta || {}), resource_flags_refreshed: new Date().toISOString().slice(0, 10), resource_flags_source: 'data/tables/people-group-pins.json — the same index the map draws. Curated bridge, internet and channel columns are untouched.' }
writeSnapshot(resolve(REPO, 'data/tables/gaps-and-opportunities.json'), `${JSON.stringify(j, null, 1)}\n`,
(x) => (x.rows || []).length, 'gaps-and-opportunities rows')
}
// ── people-group-resources ────────────────────────────────────────────────────────────
{
const j = load('people-group-resources')
for (const r of j.rows || []) {
const c = r.iso639_3
if (!c) continue
const s = hasText(c), f = hasFilm(c)
if (r.has_scripture !== s) { r.has_scripture = s; bump('has_scripture') }
if (r.has_film !== f) { r.has_film = f; bump('has_film') }
if ('jesus_film' in r && r.jesus_film !== f) { r.jesus_film = f; bump('jesus_film') }
// THREE-STATE, never false. See the header.
const a = hasAudio(c) ? true : 'unknown'
if (r.has_audio !== a) { r.has_audio = a; bump('has_audio') }
}
j.meta = { ...(j.meta || {}), resource_flags_refreshed: new Date().toISOString().slice(0, 10), has_audio_note: 'true where Faith Comes By Hearing publishes audio Scripture; "unknown" otherwise, never false — GRN is the main audio pool and answers 403 to every request.' }
writeSnapshot(resolve(REPO, 'data/tables/people-group-resources.json'), `${JSON.stringify(j, null, 1)}\n`,
(x) => (x.rows || []).length, 'people-group-resources rows')
}
console.log('REFRESH: fields corrected —')
for (const [k, n] of Object.entries(changed).sort((a, b) => b[1] - a[1])) console.log(` ${k.padEnd(22)} ${n}`)
if (!Object.keys(changed).length) console.log(' (nothing to change)')

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff