From a60eb3b5acab4a7953710a14c89d0bcdeb337e02 Mon Sep 17 00:00:00 2001 From: Doxa Maps Date: Mon, 28 Sep 2026 23:33:37 +0000 Subject: [PATCH] Write down how a project list becomes ~100 zones, and gate the numbers Project 7000, 4700, 2000 and 700 each need their own clusters of about 100 groups, and the next one should be built by following a pattern rather than by copying this plugin. The document separates two halves that turn out to be independent: which GROUPS are in the list (library/lists/, shared, already built and gated by check-list-prop-unit) and which ZONES they are cut into (this plugin's data/clusters.json). A new list needs the second half only if it wants its own territories. Measured rather than described: 28 leaf zones from 29 clusters.json entries -- the two deaf buckets collapse into one leaf. Never count zones from the buckets; ask leafZones(buildLegendNodes(...)), which is the one answer poster, packet and legend all agree on. Counting by hand produced 27 once (forgetting deaf) and a delete of 30 files another time. 5 registry lists, all present, with the membership counts they actually have (unreached 2082, unengaged 1929, frontier 1745, frontier-500k 71). doxa has no membership file, correctly: it is the unfiltered default. 7 selector types, and which type a zone uses IS the statement of how it was zoned -- from wagf_block (pure geography) through to an explicit pg_slugs roster. The 12 pg_slugs zones are the honest case, not the lazy one: below country level there is no ADM1 field to select by, so a rule would be a fiction that looks principled. The deaf poster stays recorded as the known exception. It is global and non-geographic by construction, the Driver has not solved it and said so, and the pattern says that rather than inventing a zoning. check-list-pattern-doc.mjs re-derives every number from clusters.json, lists.json and the membership files and compares them to the document's own text. Neither side is hardcoded twice, so it fails in both directions: a doc that drifts, and data that moves while the doc sits still. Shown capable of failing, five ways: leaf count 28->27, a membership count, a selector count, and a real new zone (which trips three claims at once). The fifth is the useful one -- malformed data now exits 75, NOT MEASURED, because an uncaught throw exits 1 and would read exactly like the document being wrong. Found by mutation-testing; the first draft crashed with a stack trace I would have mistaken for a pass at catching drift. Also fixed in the harness: the registry-id parser matched any backticked lowercase table cell and read the selector table's `union` as an unknown list -- a false failure against a correct doc. Each table is now parsed from its own section. --- internal/check-list-pattern-doc.mjs | 163 ++++++++++++++++++ .../documentation/PATTERN-A-LIST-OF-ZONES.md | 122 +++++++++++++ 2 files changed, 285 insertions(+) create mode 100644 internal/check-list-pattern-doc.mjs create mode 100644 plugin-designs/my-upg-100-list/documentation/PATTERN-A-LIST-OF-ZONES.md diff --git a/internal/check-list-pattern-doc.mjs b/internal/check-list-pattern-doc.mjs new file mode 100644 index 000000000..e79839650 --- /dev/null +++ b/internal/check-list-pattern-doc.mjs @@ -0,0 +1,163 @@ +#!/usr/bin/env node +/** + * check-list-pattern-doc — the numbers in PATTERN-A-LIST-OF-ZONES.md must still be true. + * + * That document tells the next agent how to turn a project list (7000/4700/2000/700) into ~100 + * zones, and it does it by QUOTING MEASUREMENTS: 28 leaf zones, 5 registry lists, 7 selector types, + * four membership files with specific slug counts. Prose like that is the first thing to rot — the + * data moves, the sentence does not, and a confident wrong number in a how-to is worse than no + * how-to, because the reader has no reason to doubt it. + * + * ★ IT RE-DERIVES EVERY NUMBER FROM THE SOURCE AND COMPARES TO THE DOC'S OWN TEXT. It does not + * assert the numbers are 28/5/7 — hardcoding the expected value on both sides is the tautology this + * whole class of gate falls into, where the check restates the doc instead of testing it. One side + * is parsed out of the markdown, the other is computed from clusters.json / lists.json / the + * membership files. They must agree. + * + * So this gate fails in BOTH useful directions: + * · someone adds a zone or a list and does not update the doc → mismatch + * · someone edits the doc's numbers without the data changing → mismatch + * + * ⚠ The deaf collapse (29 cluster entries → 28 leaves) is asserted explicitly, because it is the + * exact discrepancy that has already caused two real errors: a zone count of 27 that forgot `deaf`, + * and a delete of 30 files in the belief there were 28. + * + * Exit 0 pass · 1 fail · 75 NOT MEASURED. + */ +import { readFileSync, existsSync, readdirSync } from 'node:fs' +import { resolve, dirname, join } from 'node:path' +import { fileURLToPath } from 'node:url' + +const HERE = dirname(fileURLToPath(import.meta.url)) +const REPO = resolve(HERE, '..') +const PLUGIN = join(REPO, 'plugin-designs/my-upg-100-list') +const DOC = join(PLUGIN, 'documentation/PATTERN-A-LIST-OF-ZONES.md') + +const fails = [] +const notes = [] +const ok = (label, got) => notes.push(` ok ${label.padEnd(58)} ${got}`) +const bad = (label, expected, got) => fails.push(`${label}\n doc says: ${expected}\n source is: ${got}`) + +if (!existsSync(DOC)) { + console.error('LIST-PATTERN: NOT MEASURED (exit 75) — the document is absent; nothing to check.') + process.exit(75) +} +const doc = readFileSync(DOC, 'utf8') + +// ── 1. leaf zones. The doc's headline count, and the one with a history of being wrong. ────────── +const clusters = JSON.parse(readFileSync(join(PLUGIN, 'data/clusters.json'), 'utf8')) +const tree = await import(join(PLUGIN, 'data/legend-tree.js')) + +// ⚠ buildLegendNodes THROWS on a malformed zone (assertUniformLeafShape guards the consumer shape), +// and an uncaught throw here would exit 1 — the same code as a real FAIL, so "the doc is wrong" and +// "the data will not even load" would look identical to a reader and to CI. They need different +// answers: one is a doc to fix, the other is a broken clusters.json that this gate cannot speak to +// at all. Found by mutation-testing: adding a zone without its `count` field crashed the run with a +// stack trace and exit 1, which I would otherwise have read as the gate correctly failing. +let leaves +try { + leaves = tree.leafZones(tree.buildLegendNodes(clusters)) +} catch (e) { + console.error('LIST-PATTERN: NOT MEASURED (exit 75) — clusters.json does not build a legend tree,') + console.error(' so nothing can be said about whether the document matches it. Fix the data first;') + console.error(' the zone-shape gate (assertUniformLeafShape) is the one describing the problem:') + console.error(` ${e.message.split('\n').slice(0, 3).join('\n ')}`) + process.exit(75) +} + +const docLeaves = Number((doc.match(/⇒\s*(\d+)/) || [])[1]) +if (docLeaves !== leaves.length) bad('leaf-zone count', docLeaves, leaves.length) +else ok('leaf zones, re-derived through leafZones()', leaves.length) + +// The entry count, and that the difference is exactly the deaf collapse. +const entries = Object.values(clusters).filter(Array.isArray).flat().length +const docEntries = Number((doc.match(/(\d+)\s+entries,\s+\d+\s+zones/) || [])[1]) +if (docEntries !== entries) bad('clusters.json entry count', docEntries, entries) +else ok('clusters.json list entries', entries) + +const deafLeaves = leaves.filter((z) => /deaf/i.test(z.id || z.key || '')) +const deafEntries = (clusters.deaf_lists || []).length +if (deafLeaves.length !== 1) bad('the deaf collapse yields ONE leaf', 1, deafLeaves.length) +else if (entries - leaves.length !== deafEntries - 1) { + bad('entries minus leaves is explained by the deaf collapse alone', `${deafEntries - 1}`, `${entries - leaves.length}`) +} else ok('the 29→28 collapse is exactly the deaf buckets merging', `${deafEntries} entries → 1 leaf`) + +// ── 2. the registry. Every list the doc tables must exist, with those aliases. ──────────────────── +const registry = JSON.parse(readFileSync(join(REPO, 'library/lists/lists.json'), 'utf8')) + +/** + * Slice the markdown to ONE table, by the heading it sits under. + * ⚠ NOT OPTIONAL. The first version of this gate matched `^| \`id\` |` across the WHOLE document + * and swept up the SELECTOR table too, reporting `union` as an unknown registry list — a false + * failure against a correct doc. Two tables in one file share a row shape; the only thing that + * distinguishes them is where they are. + */ +function section(fromHeading, toHeading) { + const a = doc.indexOf(fromHeading) + if (a < 0) return '' + const b = toHeading ? doc.indexOf(toHeading, a + 1) : -1 + return doc.slice(a, b < 0 ? undefined : b) +} +const listSection = section('## Half 1', '## Half 2') +const zoneSection = section('## Half 2', '## What you get') + +const docIds = [...listSection.matchAll(/^\|\s*`([a-z0-9-]+)`\s*\|/gm)].map((m) => m[1]) +const realIds = registry.lists.map((l) => l.id) +const missing = realIds.filter((id) => !docIds.includes(id)) +const extra = docIds.filter((id) => !realIds.includes(id)) +if (missing.length) bad('every registry list appears in the doc table', `missing ${missing.join(', ')}`, realIds.join(', ')) +else if (extra.length) bad('the doc table lists only real registry ids', `unknown ${extra.join(', ')}`, realIds.join(', ')) +else ok('registry lists, all present in the doc table', realIds.length) + +if (registry.default !== 'doxa') bad('the default list', 'doxa', registry.default) +else ok('the default list is the unfiltered one', registry.default) + +// ── 3. selector types. The doc's table is the vocabulary the pattern hands the next author. ─────── +const selCounts = {} +for (const v of Object.values(clusters)) { + if (!Array.isArray(v)) continue + for (const l of v) { + const t = l?.selector?.type + if (t) selCounts[t] = (selCounts[t] || 0) + 1 + } +} +for (const [type, n] of Object.entries(selCounts)) { + const row = zoneSection.match(new RegExp(`^\\|\\s*\`${type}\`\\s*\\|\\s*(\\d+)\\s*\\|`, 'm')) + if (!row) bad(`selector type "${type}" has a row in the doc`, 'a row', `${n} zones use it, undocumented`) + else if (Number(row[1]) !== n) bad(`selector "${type}" zone count`, row[1], n) +} +const docSelTypes = [...zoneSection.matchAll(/^\|\s*`(\w+)`\s*\|\s*\d+\s*\|/gm)].map((m) => m[1]) + .filter((t) => t in selCounts || /^(wagf_|country_|union|pg_slugs|deaf_)/.test(t)) +for (const t of docSelTypes) { + if (!(t in selCounts)) bad(`doc documents selector "${t}"`, 'in use', 'no zone uses it') +} +if (!fails.length) ok('selector types and their zone counts', Object.keys(selCounts).length + ' types') + +// ── 4. membership files. The doc quotes four slug counts; doxa must have none. ──────────────────── +const MEM = join(REPO, 'public/dx-maps/assets/people-groups/lists') +if (!existsSync(MEM)) { + console.error('LIST-PATTERN: NOT MEASURED (exit 75) — membership directory absent, so the counts') + console.error(` in the doc cannot be checked either way: ${MEM}`) + process.exit(75) +} +for (const f of readdirSync(MEM).filter((f) => f.endsWith('.json'))) { + const id = f.replace(/\.json$/, '') + const n = (JSON.parse(readFileSync(join(MEM, f), 'utf8')).slugs || []).length + const row = listSection.match(new RegExp(`^${id}\\s+(\\d+)`, 'm')) + if (!row) bad(`membership "${id}" is quoted in the doc`, 'a row', `${n} slugs, undocumented`) + else if (Number(row[1]) !== n) bad(`membership "${id}" slug count`, row[1], n) +} +if (existsSync(join(MEM, 'doxa.json'))) { + bad('doxa has NO membership file (it is the unfiltered default)', 'absent', 'present') +} else ok('doxa has no membership file, as the default requires', 'absent') + +// ── report ─────────────────────────────────────────────────────────────────────────────────────── +for (const n of notes) console.log(n) +if (fails.length) { + console.error('LIST-PATTERN: FAIL — PATTERN-A-LIST-OF-ZONES.md disagrees with the data it describes.') + for (const f of fails) console.error(` ✗ ${f}`) + console.error('\n Fix whichever is wrong. If the DATA changed, update the doc; if the DOC drifted,') + console.error(' correct the number. Do not delete the claim to make this pass.') + process.exit(1) +} +console.log(`LIST-PATTERN: PASS — ${notes.length} claims in PATTERN-A-LIST-OF-ZONES.md re-derived from clusters.json, lists.json and the membership files.`) diff --git a/plugin-designs/my-upg-100-list/documentation/PATTERN-A-LIST-OF-ZONES.md b/plugin-designs/my-upg-100-list/documentation/PATTERN-A-LIST-OF-ZONES.md new file mode 100644 index 000000000..5975aae25 --- /dev/null +++ b/plugin-designs/my-upg-100-list/documentation/PATTERN-A-LIST-OF-ZONES.md @@ -0,0 +1,122 @@ +# PATTERN — a project list becomes ~100 zones (what UPG-100 actually does) + +Project 7000, 4700, 2000 and 700 each need their own clusters of roughly 100 groups. This document +is so the next one is built by FOLLOWING a pattern rather than by copying this plugin. + +Two halves, and they are independent — that separation is the pattern's main structural claim: + +| half | what it decides | where it lives | gated by | +|---|---|---|---| +| **which groups** | which people groups are in the list at all | `library/lists/` (shared) | `internal/check-list-prop-unit.mjs` | +| **which zones** | how those groups are cut into ~100-group territories | your plugin's `data/clusters.json` | `internal/check-group-set-standard.mjs` | + +A new list needs the SECOND half only if it wants its own territories. The first half already +carries all five lists. + +--- + +## Half 1 — the list prop. Already built; do not rebuild it. + +`library/lists/peopleGroupList.js` is shared machinery, not UPG-100's. The registry +(`library/lists/lists.json`) already defines every list the Driver named: + +| id | aliases | label | +|---|---|---| +| `doxa` | `2000`, `project-2000` | Project 2000: DOXA's people groups (the default) | +| `unreached` | `7000`, `project-7000` | Project 7000: unreached people groups | +| `frontier` | `4700`, `project-4700` | Project 4700: frontier people groups | +| `frontier-500k` | `700`, `project-700` | Project 700: frontier people groups of 500,000 or more | +| `unengaged` | — | Unengaged unreached people groups | + +`resolvePeopleGroupList(profileConfig)` picks one, **first answer wins**: profile-config `list` → +`?list=` → `window.MAP_LIST` → `doxa`. An unknown value falls back to the default and warns once, +naming every known id. + +MEASURED, so you can trust the counts rather than re-derive them — membership files under +`public/dx-maps/assets/people-groups/lists/`: + +``` +unreached 2082 slugs +unengaged 1929 +frontier 1745 +frontier-500k 71 +doxa (no file, and correctly so — the default is the unfiltered API result) +``` + +★ **The filter FAILS CLOSED.** If a non-default list's membership cannot be loaded, the map shows +NO rows and the console names the list and the URL. A map that silently showed all 2,107 while the +embed asked for Project 700 would be showing the wrong list with no sign of it — the failure mode +this design exists to prevent. Do not "improve" this into a fall-back-to-everything. + +**To add a list:** a registry entry + a membership file from +`library/scripts/people-groups/build-lists.mjs` + label/definition keys in every +`library/i18n/locales//lists.json`. The gate checks all three (24 cases) and will tell you which +locale you missed. + +--- + +## Half 2 — the zoning. This is the judgement, and it cannot be automated. + +**The Driver's rule, in their words:** geography FIRST so the poster maps make sense, balanced with +LANGUAGE — because languages usually cluster geographically, the zones are already a mixture of the +two, grouped manually. + +That balance is a human call. What the pattern can do is give you the vocabulary to express the call +you made, so the next reader can see it. UPG-100's 29 entries use **seven selector types**, and the +type you pick IS the statement of how you zoned: + +| selector | zones | what it says | +|---|---|---| +| `wagf_block` | 3 | a named regional bloc (west/east/central Africa) — pure geography | +| `wagf_region` | 1 | a whole region (europe) — geography, coarsest | +| `country_code` | 3 | one country is the territory (SDN, BRA, LAO) | +| `country_codes` | 6 | several countries make one territory (middle-east-core, maghreb-egypt) | +| `union` | 2 | a composite that no single rule expresses (oceania-indonesia) | +| `pg_slugs` | 12 | an explicit roster — every Indian state and Chinese province | +| `deaf_predicate` | 2 | not geographic at all (see the exception below) | + +★ **THE `pg_slugs` BLOCK IS THE HONEST PART, NOT THE LAZY PART.** All 12 of India's states and +China's provinces are enumerated group-by-group rather than derived from a rule. That is not +technical debt: below the country level there is no ADM1 field on the records to select by, so a +rule would be a *fiction* that looks principled. An explicit roster is checkable; an invented +predicate is not. Expect a sub-national list to need the same, and do not let a reviewer talk you +into a clever selector that cannot be verified. + +### 29 entries, 28 zones — the collapse is real, count the LEAVES + +``` +buildLegendNodes(clusters) → leafZones(...) ⇒ 28 +``` +The two `deaf_lists` entries (`deaf-old-world`, `deaf-new-world-asia-pacific`) collapse into ONE +`deaf` leaf. So `clusters.json` has 29 list entries and the product has 28 posters/packets/zones. + +⚠ **Never count zones from `clusters.json`'s buckets.** I did, called it 27 (forgetting `deaf` +entirely), and later deleted 30 files believing there were 28. `leafZones(buildLegendNodes(...))` +is the one answer that poster, packet and legend all agree on — ask it, don't count by hand. + +### The deaf poster is the known exception. Leave it unsolved. + +The deaf zone is global and non-geographic by construction (affinity bloc A017, 123 groups spanning +every region), so it breaks the geography-first rule the rest of the pattern rests on. **The Driver +has not solved this and said so.** It is recorded here as an open exception, not papered over with +an invented zoning. If your list has a similar affinity-not-territory group, do the same: give it +its own bucket, let it collapse to one leaf, and say plainly that it is not geographic. + +--- + +## What you get for free once your `clusters.json` exists + +`documentation/STANDARD-GROUP-SET.md` is the fork contract for the offline generator: point it at +one folder and it produces a packet per zone, localized, with no code change. Its falsifier +(`internal/check-group-set-standard.mjs`) runs a synthetic 3-list toy set end-to-end on every gate +pass, so the contract cannot rot silently. Read it next; it covers data shapes, `PACKET_LANG`, +`ONLY=`, and which pages omit cleanly when optional repo data is absent. + +## Not solved by this pattern + +- **Posters for arbitrary sets.** The poster generator needs geo/bbox inputs this pattern does not + define. Packets are generative today; posters are not. +- **Zone TITLES in other languages.** Tracked separately — only 10 of 28 titles are composable from + already-translated parts, and `Intl.DisplayNames({type:'language'})` resolves just 9.5% of ISO + 639-3 codes and ECHOES the code on a miss, so it would print raw codes on a poster silently. +- **Multi-territory maps** and any authoring UI.