Write down how a project list becomes ~100 zones, and gate the numbers
Project 7000, 4700, 2000 and 700 each need their own clusters of about 100 groups, and the next one should be built by following a pattern rather than by copying this plugin. The document separates two halves that turn out to be independent: which GROUPS are in the list (library/lists/, shared, already built and gated by check-list-prop-unit) and which ZONES they are cut into (this plugin's data/clusters.json). A new list needs the second half only if it wants its own territories. Measured rather than described: 28 leaf zones from 29 clusters.json entries -- the two deaf buckets collapse into one leaf. Never count zones from the buckets; ask leafZones(buildLegendNodes(...)), which is the one answer poster, packet and legend all agree on. Counting by hand produced 27 once (forgetting deaf) and a delete of 30 files another time. 5 registry lists, all present, with the membership counts they actually have (unreached 2082, unengaged 1929, frontier 1745, frontier-500k 71). doxa has no membership file, correctly: it is the unfiltered default. 7 selector types, and which type a zone uses IS the statement of how it was zoned -- from wagf_block (pure geography) through to an explicit pg_slugs roster. The 12 pg_slugs zones are the honest case, not the lazy one: below country level there is no ADM1 field to select by, so a rule would be a fiction that looks principled. The deaf poster stays recorded as the known exception. It is global and non-geographic by construction, the Driver has not solved it and said so, and the pattern says that rather than inventing a zoning. check-list-pattern-doc.mjs re-derives every number from clusters.json, lists.json and the membership files and compares them to the document's own text. Neither side is hardcoded twice, so it fails in both directions: a doc that drifts, and data that moves while the doc sits still. Shown capable of failing, five ways: leaf count 28->27, a membership count, a selector count, and a real new zone (which trips three claims at once). The fifth is the useful one -- malformed data now exits 75, NOT MEASURED, because an uncaught throw exits 1 and would read exactly like the document being wrong. Found by mutation-testing; the first draft crashed with a stack trace I would have mistaken for a pass at catching drift. Also fixed in the harness: the registry-id parser matched any backticked lowercase table cell and read the selector table's `union` as an unknown list -- a false failure against a correct doc. Each table is now parsed from its own section.
This commit is contained in:
Doxa Maps
parent
32d2e4dc59
commit
a60eb3b5ac
2 changed files with 285 additions and 0 deletions
163
internal/check-list-pattern-doc.mjs
Normal file
163
internal/check-list-pattern-doc.mjs
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* check-list-pattern-doc — the numbers in PATTERN-A-LIST-OF-ZONES.md must still be true.
|
||||
*
|
||||
* That document tells the next agent how to turn a project list (7000/4700/2000/700) into ~100
|
||||
* zones, and it does it by QUOTING MEASUREMENTS: 28 leaf zones, 5 registry lists, 7 selector types,
|
||||
* four membership files with specific slug counts. Prose like that is the first thing to rot — the
|
||||
* data moves, the sentence does not, and a confident wrong number in a how-to is worse than no
|
||||
* how-to, because the reader has no reason to doubt it.
|
||||
*
|
||||
* ★ IT RE-DERIVES EVERY NUMBER FROM THE SOURCE AND COMPARES TO THE DOC'S OWN TEXT. It does not
|
||||
* assert the numbers are 28/5/7 — hardcoding the expected value on both sides is the tautology this
|
||||
* whole class of gate falls into, where the check restates the doc instead of testing it. One side
|
||||
* is parsed out of the markdown, the other is computed from clusters.json / lists.json / the
|
||||
* membership files. They must agree.
|
||||
*
|
||||
* So this gate fails in BOTH useful directions:
|
||||
* · someone adds a zone or a list and does not update the doc → mismatch
|
||||
* · someone edits the doc's numbers without the data changing → mismatch
|
||||
*
|
||||
* ⚠ The deaf collapse (29 cluster entries → 28 leaves) is asserted explicitly, because it is the
|
||||
* exact discrepancy that has already caused two real errors: a zone count of 27 that forgot `deaf`,
|
||||
* and a delete of 30 files in the belief there were 28.
|
||||
*
|
||||
* Exit 0 pass · 1 fail · 75 NOT MEASURED.
|
||||
*/
|
||||
import { readFileSync, existsSync, readdirSync } from 'node:fs'
|
||||
import { resolve, dirname, join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url))
|
||||
const REPO = resolve(HERE, '..')
|
||||
const PLUGIN = join(REPO, 'plugin-designs/my-upg-100-list')
|
||||
const DOC = join(PLUGIN, 'documentation/PATTERN-A-LIST-OF-ZONES.md')
|
||||
|
||||
const fails = []
|
||||
const notes = []
|
||||
const ok = (label, got) => notes.push(` ok ${label.padEnd(58)} ${got}`)
|
||||
const bad = (label, expected, got) => fails.push(`${label}\n doc says: ${expected}\n source is: ${got}`)
|
||||
|
||||
if (!existsSync(DOC)) {
|
||||
console.error('LIST-PATTERN: NOT MEASURED (exit 75) — the document is absent; nothing to check.')
|
||||
process.exit(75)
|
||||
}
|
||||
const doc = readFileSync(DOC, 'utf8')
|
||||
|
||||
// ── 1. leaf zones. The doc's headline count, and the one with a history of being wrong. ──────────
|
||||
const clusters = JSON.parse(readFileSync(join(PLUGIN, 'data/clusters.json'), 'utf8'))
|
||||
const tree = await import(join(PLUGIN, 'data/legend-tree.js'))
|
||||
|
||||
// ⚠ buildLegendNodes THROWS on a malformed zone (assertUniformLeafShape guards the consumer shape),
|
||||
// and an uncaught throw here would exit 1 — the same code as a real FAIL, so "the doc is wrong" and
|
||||
// "the data will not even load" would look identical to a reader and to CI. They need different
|
||||
// answers: one is a doc to fix, the other is a broken clusters.json that this gate cannot speak to
|
||||
// at all. Found by mutation-testing: adding a zone without its `count` field crashed the run with a
|
||||
// stack trace and exit 1, which I would otherwise have read as the gate correctly failing.
|
||||
let leaves
|
||||
try {
|
||||
leaves = tree.leafZones(tree.buildLegendNodes(clusters))
|
||||
} catch (e) {
|
||||
console.error('LIST-PATTERN: NOT MEASURED (exit 75) — clusters.json does not build a legend tree,')
|
||||
console.error(' so nothing can be said about whether the document matches it. Fix the data first;')
|
||||
console.error(' the zone-shape gate (assertUniformLeafShape) is the one describing the problem:')
|
||||
console.error(` ${e.message.split('\n').slice(0, 3).join('\n ')}`)
|
||||
process.exit(75)
|
||||
}
|
||||
|
||||
const docLeaves = Number((doc.match(/⇒\s*(\d+)/) || [])[1])
|
||||
if (docLeaves !== leaves.length) bad('leaf-zone count', docLeaves, leaves.length)
|
||||
else ok('leaf zones, re-derived through leafZones()', leaves.length)
|
||||
|
||||
// The entry count, and that the difference is exactly the deaf collapse.
|
||||
const entries = Object.values(clusters).filter(Array.isArray).flat().length
|
||||
const docEntries = Number((doc.match(/(\d+)\s+entries,\s+\d+\s+zones/) || [])[1])
|
||||
if (docEntries !== entries) bad('clusters.json entry count', docEntries, entries)
|
||||
else ok('clusters.json list entries', entries)
|
||||
|
||||
const deafLeaves = leaves.filter((z) => /deaf/i.test(z.id || z.key || ''))
|
||||
const deafEntries = (clusters.deaf_lists || []).length
|
||||
if (deafLeaves.length !== 1) bad('the deaf collapse yields ONE leaf', 1, deafLeaves.length)
|
||||
else if (entries - leaves.length !== deafEntries - 1) {
|
||||
bad('entries minus leaves is explained by the deaf collapse alone', `${deafEntries - 1}`, `${entries - leaves.length}`)
|
||||
} else ok('the 29→28 collapse is exactly the deaf buckets merging', `${deafEntries} entries → 1 leaf`)
|
||||
|
||||
// ── 2. the registry. Every list the doc tables must exist, with those aliases. ────────────────────
|
||||
const registry = JSON.parse(readFileSync(join(REPO, 'library/lists/lists.json'), 'utf8'))
|
||||
|
||||
/**
|
||||
* Slice the markdown to ONE table, by the heading it sits under.
|
||||
* ⚠ NOT OPTIONAL. The first version of this gate matched `^| \`id\` |` across the WHOLE document
|
||||
* and swept up the SELECTOR table too, reporting `union` as an unknown registry list — a false
|
||||
* failure against a correct doc. Two tables in one file share a row shape; the only thing that
|
||||
* distinguishes them is where they are.
|
||||
*/
|
||||
function section(fromHeading, toHeading) {
|
||||
const a = doc.indexOf(fromHeading)
|
||||
if (a < 0) return ''
|
||||
const b = toHeading ? doc.indexOf(toHeading, a + 1) : -1
|
||||
return doc.slice(a, b < 0 ? undefined : b)
|
||||
}
|
||||
const listSection = section('## Half 1', '## Half 2')
|
||||
const zoneSection = section('## Half 2', '## What you get')
|
||||
|
||||
const docIds = [...listSection.matchAll(/^\|\s*`([a-z0-9-]+)`\s*\|/gm)].map((m) => m[1])
|
||||
const realIds = registry.lists.map((l) => l.id)
|
||||
const missing = realIds.filter((id) => !docIds.includes(id))
|
||||
const extra = docIds.filter((id) => !realIds.includes(id))
|
||||
if (missing.length) bad('every registry list appears in the doc table', `missing ${missing.join(', ')}`, realIds.join(', '))
|
||||
else if (extra.length) bad('the doc table lists only real registry ids', `unknown ${extra.join(', ')}`, realIds.join(', '))
|
||||
else ok('registry lists, all present in the doc table', realIds.length)
|
||||
|
||||
if (registry.default !== 'doxa') bad('the default list', 'doxa', registry.default)
|
||||
else ok('the default list is the unfiltered one', registry.default)
|
||||
|
||||
// ── 3. selector types. The doc's table is the vocabulary the pattern hands the next author. ───────
|
||||
const selCounts = {}
|
||||
for (const v of Object.values(clusters)) {
|
||||
if (!Array.isArray(v)) continue
|
||||
for (const l of v) {
|
||||
const t = l?.selector?.type
|
||||
if (t) selCounts[t] = (selCounts[t] || 0) + 1
|
||||
}
|
||||
}
|
||||
for (const [type, n] of Object.entries(selCounts)) {
|
||||
const row = zoneSection.match(new RegExp(`^\\|\\s*\`${type}\`\\s*\\|\\s*(\\d+)\\s*\\|`, 'm'))
|
||||
if (!row) bad(`selector type "${type}" has a row in the doc`, 'a row', `${n} zones use it, undocumented`)
|
||||
else if (Number(row[1]) !== n) bad(`selector "${type}" zone count`, row[1], n)
|
||||
}
|
||||
const docSelTypes = [...zoneSection.matchAll(/^\|\s*`(\w+)`\s*\|\s*\d+\s*\|/gm)].map((m) => m[1])
|
||||
.filter((t) => t in selCounts || /^(wagf_|country_|union|pg_slugs|deaf_)/.test(t))
|
||||
for (const t of docSelTypes) {
|
||||
if (!(t in selCounts)) bad(`doc documents selector "${t}"`, 'in use', 'no zone uses it')
|
||||
}
|
||||
if (!fails.length) ok('selector types and their zone counts', Object.keys(selCounts).length + ' types')
|
||||
|
||||
// ── 4. membership files. The doc quotes four slug counts; doxa must have none. ────────────────────
|
||||
const MEM = join(REPO, 'public/dx-maps/assets/people-groups/lists')
|
||||
if (!existsSync(MEM)) {
|
||||
console.error('LIST-PATTERN: NOT MEASURED (exit 75) — membership directory absent, so the counts')
|
||||
console.error(` in the doc cannot be checked either way: ${MEM}`)
|
||||
process.exit(75)
|
||||
}
|
||||
for (const f of readdirSync(MEM).filter((f) => f.endsWith('.json'))) {
|
||||
const id = f.replace(/\.json$/, '')
|
||||
const n = (JSON.parse(readFileSync(join(MEM, f), 'utf8')).slugs || []).length
|
||||
const row = listSection.match(new RegExp(`^${id}\\s+(\\d+)`, 'm'))
|
||||
if (!row) bad(`membership "${id}" is quoted in the doc`, 'a row', `${n} slugs, undocumented`)
|
||||
else if (Number(row[1]) !== n) bad(`membership "${id}" slug count`, row[1], n)
|
||||
}
|
||||
if (existsSync(join(MEM, 'doxa.json'))) {
|
||||
bad('doxa has NO membership file (it is the unfiltered default)', 'absent', 'present')
|
||||
} else ok('doxa has no membership file, as the default requires', 'absent')
|
||||
|
||||
// ── report ───────────────────────────────────────────────────────────────────────────────────────
|
||||
for (const n of notes) console.log(n)
|
||||
if (fails.length) {
|
||||
console.error('LIST-PATTERN: FAIL — PATTERN-A-LIST-OF-ZONES.md disagrees with the data it describes.')
|
||||
for (const f of fails) console.error(` ✗ ${f}`)
|
||||
console.error('\n Fix whichever is wrong. If the DATA changed, update the doc; if the DOC drifted,')
|
||||
console.error(' correct the number. Do not delete the claim to make this pass.')
|
||||
process.exit(1)
|
||||
}
|
||||
console.log(`LIST-PATTERN: PASS — ${notes.length} claims in PATTERN-A-LIST-OF-ZONES.md re-derived from clusters.json, lists.json and the membership files.`)
|
||||
|
|
@ -0,0 +1,122 @@
|
|||
# PATTERN — a project list becomes ~100 zones (what UPG-100 actually does)
|
||||
|
||||
Project 7000, 4700, 2000 and 700 each need their own clusters of roughly 100 groups. This document
|
||||
is so the next one is built by FOLLOWING a pattern rather than by copying this plugin.
|
||||
|
||||
Two halves, and they are independent — that separation is the pattern's main structural claim:
|
||||
|
||||
| half | what it decides | where it lives | gated by |
|
||||
|---|---|---|---|
|
||||
| **which groups** | which people groups are in the list at all | `library/lists/` (shared) | `internal/check-list-prop-unit.mjs` |
|
||||
| **which zones** | how those groups are cut into ~100-group territories | your plugin's `data/clusters.json` | `internal/check-group-set-standard.mjs` |
|
||||
|
||||
A new list needs the SECOND half only if it wants its own territories. The first half already
|
||||
carries all five lists.
|
||||
|
||||
---
|
||||
|
||||
## Half 1 — the list prop. Already built; do not rebuild it.
|
||||
|
||||
`library/lists/peopleGroupList.js` is shared machinery, not UPG-100's. The registry
|
||||
(`library/lists/lists.json`) already defines every list the Driver named:
|
||||
|
||||
| id | aliases | label |
|
||||
|---|---|---|
|
||||
| `doxa` | `2000`, `project-2000` | Project 2000: DOXA's people groups (the default) |
|
||||
| `unreached` | `7000`, `project-7000` | Project 7000: unreached people groups |
|
||||
| `frontier` | `4700`, `project-4700` | Project 4700: frontier people groups |
|
||||
| `frontier-500k` | `700`, `project-700` | Project 700: frontier people groups of 500,000 or more |
|
||||
| `unengaged` | — | Unengaged unreached people groups |
|
||||
|
||||
`resolvePeopleGroupList(profileConfig)` picks one, **first answer wins**: profile-config `list` →
|
||||
`?list=` → `window.MAP_LIST` → `doxa`. An unknown value falls back to the default and warns once,
|
||||
naming every known id.
|
||||
|
||||
MEASURED, so you can trust the counts rather than re-derive them — membership files under
|
||||
`public/dx-maps/assets/people-groups/lists/`:
|
||||
|
||||
```
|
||||
unreached 2082 slugs
|
||||
unengaged 1929
|
||||
frontier 1745
|
||||
frontier-500k 71
|
||||
doxa (no file, and correctly so — the default is the unfiltered API result)
|
||||
```
|
||||
|
||||
★ **The filter FAILS CLOSED.** If a non-default list's membership cannot be loaded, the map shows
|
||||
NO rows and the console names the list and the URL. A map that silently showed all 2,107 while the
|
||||
embed asked for Project 700 would be showing the wrong list with no sign of it — the failure mode
|
||||
this design exists to prevent. Do not "improve" this into a fall-back-to-everything.
|
||||
|
||||
**To add a list:** a registry entry + a membership file from
|
||||
`library/scripts/people-groups/build-lists.mjs` + label/definition keys in every
|
||||
`library/i18n/locales/<lc>/lists.json`. The gate checks all three (24 cases) and will tell you which
|
||||
locale you missed.
|
||||
|
||||
---
|
||||
|
||||
## Half 2 — the zoning. This is the judgement, and it cannot be automated.
|
||||
|
||||
**The Driver's rule, in their words:** geography FIRST so the poster maps make sense, balanced with
|
||||
LANGUAGE — because languages usually cluster geographically, the zones are already a mixture of the
|
||||
two, grouped manually.
|
||||
|
||||
That balance is a human call. What the pattern can do is give you the vocabulary to express the call
|
||||
you made, so the next reader can see it. UPG-100's 29 entries use **seven selector types**, and the
|
||||
type you pick IS the statement of how you zoned:
|
||||
|
||||
| selector | zones | what it says |
|
||||
|---|---|---|
|
||||
| `wagf_block` | 3 | a named regional bloc (west/east/central Africa) — pure geography |
|
||||
| `wagf_region` | 1 | a whole region (europe) — geography, coarsest |
|
||||
| `country_code` | 3 | one country is the territory (SDN, BRA, LAO) |
|
||||
| `country_codes` | 6 | several countries make one territory (middle-east-core, maghreb-egypt) |
|
||||
| `union` | 2 | a composite that no single rule expresses (oceania-indonesia) |
|
||||
| `pg_slugs` | 12 | an explicit roster — every Indian state and Chinese province |
|
||||
| `deaf_predicate` | 2 | not geographic at all (see the exception below) |
|
||||
|
||||
★ **THE `pg_slugs` BLOCK IS THE HONEST PART, NOT THE LAZY PART.** All 12 of India's states and
|
||||
China's provinces are enumerated group-by-group rather than derived from a rule. That is not
|
||||
technical debt: below the country level there is no ADM1 field on the records to select by, so a
|
||||
rule would be a *fiction* that looks principled. An explicit roster is checkable; an invented
|
||||
predicate is not. Expect a sub-national list to need the same, and do not let a reviewer talk you
|
||||
into a clever selector that cannot be verified.
|
||||
|
||||
### 29 entries, 28 zones — the collapse is real, count the LEAVES
|
||||
|
||||
```
|
||||
buildLegendNodes(clusters) → leafZones(...) ⇒ 28
|
||||
```
|
||||
The two `deaf_lists` entries (`deaf-old-world`, `deaf-new-world-asia-pacific`) collapse into ONE
|
||||
`deaf` leaf. So `clusters.json` has 29 list entries and the product has 28 posters/packets/zones.
|
||||
|
||||
⚠ **Never count zones from `clusters.json`'s buckets.** I did, called it 27 (forgetting `deaf`
|
||||
entirely), and later deleted 30 files believing there were 28. `leafZones(buildLegendNodes(...))`
|
||||
is the one answer that poster, packet and legend all agree on — ask it, don't count by hand.
|
||||
|
||||
### The deaf poster is the known exception. Leave it unsolved.
|
||||
|
||||
The deaf zone is global and non-geographic by construction (affinity bloc A017, 123 groups spanning
|
||||
every region), so it breaks the geography-first rule the rest of the pattern rests on. **The Driver
|
||||
has not solved this and said so.** It is recorded here as an open exception, not papered over with
|
||||
an invented zoning. If your list has a similar affinity-not-territory group, do the same: give it
|
||||
its own bucket, let it collapse to one leaf, and say plainly that it is not geographic.
|
||||
|
||||
---
|
||||
|
||||
## What you get for free once your `clusters.json` exists
|
||||
|
||||
`documentation/STANDARD-GROUP-SET.md` is the fork contract for the offline generator: point it at
|
||||
one folder and it produces a packet per zone, localized, with no code change. Its falsifier
|
||||
(`internal/check-group-set-standard.mjs`) runs a synthetic 3-list toy set end-to-end on every gate
|
||||
pass, so the contract cannot rot silently. Read it next; it covers data shapes, `PACKET_LANG`,
|
||||
`ONLY=`, and which pages omit cleanly when optional repo data is absent.
|
||||
|
||||
## Not solved by this pattern
|
||||
|
||||
- **Posters for arbitrary sets.** The poster generator needs geo/bbox inputs this pattern does not
|
||||
define. Packets are generative today; posters are not.
|
||||
- **Zone TITLES in other languages.** Tracked separately — only 10 of 28 titles are composable from
|
||||
already-translated parts, and `Intl.DisplayNames({type:'language'})` resolves just 9.5% of ISO
|
||||
639-3 codes and ECHOES the code on a miss, so it would print raw codes on a poster silently.
|
||||
- **Multi-territory maps** and any authoring UI.
|
||||
Loading…
Reference in a new issue