gate wave: a SKIP is not a pass, and this printed them identically
Some checks are pending
build-gate / build (push) Waiting to run

run-parallel grades on the exit code, and a gate that declines to evaluate
exits 0. So "35/35 passed" read the same whether 35 gates graded something or
31 did, and the summary line — the one people quote — was the most misleading
part of the output.

This is the generalisation of the packet-sweep fix two commits ago. That closed
ONE always-skipping gate; the reason nobody could see it for months was that the
runner reported it green. 18 of the 35 steps in gates:post CAN skip, every one
on a precondition a fresh checkout does not have, so the gap is structural
rather than specific to packets.

Measured on a genuinely fresh checkout — `git archive HEAD | tar -x`, which is
what CI sees. Before: "31/35 passed". After: "28/35 passed, 4 SKIPPED", naming
check:packets-all, check:zone-census, check:done-evidence, check:backlog-status.

All four are legitimate, and I checked rather than assumed. packets-all skips
until packet:ci runs, which CI does first. zone-census needs generated posters.
The two backlog gates read _qa-cycle, which lives outside any repo by the
deliberate decision that PM markdown never sits inside a code repo — and both
already say so in their own output. So the finding is the REPORTING, not a
broken gate.

Exit semantics are deliberately unchanged, and verified: a wave whose only step
skips still exits 0. Failing on a SKIP would fail every CI build on the backlog
gates and teach people to disable the reporting.

The detector is mutation-tested against seven real output shapes. It matches
"ZONE-CENSUS: SKIP — …" and the indented variant, and correctly does not match
prose mentioning skip, nor "SHARE-POLICY: PASS — 12 buttons, 0 skipped".

A correction to my own method along the way: I first censused skips in
doxa-maps-plugin-studio as a proxy for a fresh checkout. It is not one — it sits
many commits behind main with a 21-step gates:post that lacks 14 of the steps
main has, so the two censuses were of different waves. The fresh-archive test
replaced it.

Also documents check:zone-census's precondition, the one conditional gate in the
wave that had none — including why it cannot be fixtured the way packets-all
was: it asserts coverage of real generated output, so a fixture would only
assert that the fixture is complete.
This commit is contained in:
doxa-maps-88655c70 2026-09-24 14:50:00 +00:00
parent ca8a501942
commit a361da0a2e
2 changed files with 23 additions and 3 deletions

View file

@ -133,10 +133,20 @@ async function main() {
if (!step) return
const r = await runStep(step)
results.push(r)
// ★ A SKIP IS NOT A PASS, AND THIS PRINTED THEM IDENTICALLY. The grade is the exit
// code, and a gate that declines to evaluate exits 0 — so "35/35 passed" read the
// same whether 35 gates graded something or 17 did. MEASURED 2026-09-24: 18 of the
// 35 steps in gates:post CAN skip, every one of them on a precondition (a served
// tree, generated posters, a built manifest) that a fresh checkout does not have.
// check:packets-all SKIPped on every CI build for exactly this reason and nobody
// could see it, because the wave reported it green. An always-taken SKIP is
// indistinguishable from an unwired gate unless the runner says so.
const skipped = /(^|\n)\s*[A-Z][A-Z0-9-]*:\s*SKIP\b/.test(r.out)
console.log(
` ${r.code === 0 ? '✓' : '✗'} ${r.name.padEnd(26)} ${String(r.ms).padStart(7)}ms` +
` ${r.code !== 0 ? '✗' : skipped ? '⊘' : '✓'} ${r.name.padEnd(26)} ${String(r.ms).padStart(7)}ms` +
(r.code === 0 ? '' : ` exit ${r.code}`)
)
r.skipped = skipped
}
}
await Promise.all(Array.from({ length: Math.min(CONCURRENCY, steps.length) }, worker))
@ -150,10 +160,20 @@ async function main() {
// a truncated explanation is how a real defect gets re-diagnosed as a flake.
else for (const r of failed) console.log(`\n───── FAILED: ${r.name} (exit ${r.code}) ─────\n${r.out}`)
// COUNT THE SKIPS IN THE SUMMARY, because the summary is the line a reader quotes and
// the one that lands in a report. "35/35 passed" was true of a wave where a third of the
// gates declined to look at anything.
const skips = results.filter((r) => r.skipped)
console.log(
`\n▶ ${LABEL}: ${results.length - failed.length}/${results.length} passed in ${(wall / 1000).toFixed(1)}s wall ` +
`\n▶ ${LABEL}: ${results.length - failed.length - skips.length}/${results.length} passed` +
(skips.length ? `, ${skips.length} SKIPPED` : '') +
` in ${(wall / 1000).toFixed(1)}s wall ` +
`(${(serial / 1000).toFixed(1)}s of work — ${(serial / Math.max(wall, 1)).toFixed(1)}x)`
)
if (skips.length) {
console.log(` ⊘ evaluated nothing: ${skips.map((s) => s.name).join(', ')}`)
console.log(' Each declined on a precondition it names in its own output (run it alone to see).')
}
if (failed.length) {
console.error(`✗ ${LABEL}: ${failed.length} step(s) FAILED — ${failed.map((f) => f.name).join(', ')}`)
process.exit(1)

View file

@ -68,7 +68,7 @@
"check:provider-switch": "node internal/check-provider-switch.mjs",
"check:mfe-two-tab": "node internal/check-mfe-two-tab.mjs",
"check:share-policy": "node internal/check-share-policy.mjs",
"check:zone-census": "node internal/check-zone-census.mjs",
"check:zone-census": "node internal/check-zone-census.mjs # every zone a USER CAN SELECT must have a poster (asks the LEGEND, not clusters.json — two collapsed deaf lists are unreachable and not a gap). CONDITIONAL: SKIPs without out/posters/, which is gitignored — run `bun run posters` first. It cannot be fixtured like check:packets-all, because it asserts COVERAGE of real generated output and a fixture would only assert the fixture is complete.",
"check:print-geometry": "node internal/check-print-geometry.mjs",
"check:packet-browser": "node internal/check-packet-browser.mjs",
"check:packet-render": "node internal/check-packet-render.mjs",