gate wave: a SKIP is not a pass, and this printed them identically
Some checks are pending
build-gate / build (push) Waiting to run
Some checks are pending
build-gate / build (push) Waiting to run
run-parallel grades on the exit code, and a gate that declines to evaluate exits 0. So "35/35 passed" read the same whether 35 gates graded something or 31 did, and the summary line — the one people quote — was the most misleading part of the output. This is the generalisation of the packet-sweep fix two commits ago. That closed ONE always-skipping gate; the reason nobody could see it for months was that the runner reported it green. 18 of the 35 steps in gates:post CAN skip, every one on a precondition a fresh checkout does not have, so the gap is structural rather than specific to packets. Measured on a genuinely fresh checkout — `git archive HEAD | tar -x`, which is what CI sees. Before: "31/35 passed". After: "28/35 passed, 4 SKIPPED", naming check:packets-all, check:zone-census, check:done-evidence, check:backlog-status. All four are legitimate, and I checked rather than assumed. packets-all skips until packet:ci runs, which CI does first. zone-census needs generated posters. The two backlog gates read _qa-cycle, which lives outside any repo by the deliberate decision that PM markdown never sits inside a code repo — and both already say so in their own output. So the finding is the REPORTING, not a broken gate. Exit semantics are deliberately unchanged, and verified: a wave whose only step skips still exits 0. Failing on a SKIP would fail every CI build on the backlog gates and teach people to disable the reporting. The detector is mutation-tested against seven real output shapes. It matches "ZONE-CENSUS: SKIP — …" and the indented variant, and correctly does not match prose mentioning skip, nor "SHARE-POLICY: PASS — 12 buttons, 0 skipped". A correction to my own method along the way: I first censused skips in doxa-maps-plugin-studio as a proxy for a fresh checkout. It is not one — it sits many commits behind main with a 21-step gates:post that lacks 14 of the steps main has, so the two censuses were of different waves. The fresh-archive test replaced it. Also documents check:zone-census's precondition, the one conditional gate in the wave that had none — including why it cannot be fixtured the way packets-all was: it asserts coverage of real generated output, so a fixture would only assert that the fixture is complete.
This commit is contained in:
parent
ca8a501942
commit
a361da0a2e
2 changed files with 23 additions and 3 deletions
|
|
@ -133,10 +133,20 @@ async function main() {
|
|||
if (!step) return
|
||||
const r = await runStep(step)
|
||||
results.push(r)
|
||||
// ★ A SKIP IS NOT A PASS, AND THIS PRINTED THEM IDENTICALLY. The grade is the exit
|
||||
// code, and a gate that declines to evaluate exits 0 — so "35/35 passed" read the
|
||||
// same whether 35 gates graded something or 17 did. MEASURED 2026-09-24: 18 of the
|
||||
// 35 steps in gates:post CAN skip, every one of them on a precondition (a served
|
||||
// tree, generated posters, a built manifest) that a fresh checkout does not have.
|
||||
// check:packets-all SKIPped on every CI build for exactly this reason and nobody
|
||||
// could see it, because the wave reported it green. An always-taken SKIP is
|
||||
// indistinguishable from an unwired gate unless the runner says so.
|
||||
const skipped = /(^|\n)\s*[A-Z][A-Z0-9-]*:\s*SKIP\b/.test(r.out)
|
||||
console.log(
|
||||
` ${r.code === 0 ? '✓' : '✗'} ${r.name.padEnd(26)} ${String(r.ms).padStart(7)}ms` +
|
||||
` ${r.code !== 0 ? '✗' : skipped ? '⊘' : '✓'} ${r.name.padEnd(26)} ${String(r.ms).padStart(7)}ms` +
|
||||
(r.code === 0 ? '' : ` exit ${r.code}`)
|
||||
)
|
||||
r.skipped = skipped
|
||||
}
|
||||
}
|
||||
await Promise.all(Array.from({ length: Math.min(CONCURRENCY, steps.length) }, worker))
|
||||
|
|
@ -150,10 +160,20 @@ async function main() {
|
|||
// a truncated explanation is how a real defect gets re-diagnosed as a flake.
|
||||
else for (const r of failed) console.log(`\n───── FAILED: ${r.name} (exit ${r.code}) ─────\n${r.out}`)
|
||||
|
||||
// COUNT THE SKIPS IN THE SUMMARY, because the summary is the line a reader quotes and
|
||||
// the one that lands in a report. "35/35 passed" was true of a wave where a third of the
|
||||
// gates declined to look at anything.
|
||||
const skips = results.filter((r) => r.skipped)
|
||||
console.log(
|
||||
`\n▶ ${LABEL}: ${results.length - failed.length}/${results.length} passed in ${(wall / 1000).toFixed(1)}s wall ` +
|
||||
`\n▶ ${LABEL}: ${results.length - failed.length - skips.length}/${results.length} passed` +
|
||||
(skips.length ? `, ${skips.length} SKIPPED` : '') +
|
||||
` in ${(wall / 1000).toFixed(1)}s wall ` +
|
||||
`(${(serial / 1000).toFixed(1)}s of work — ${(serial / Math.max(wall, 1)).toFixed(1)}x)`
|
||||
)
|
||||
if (skips.length) {
|
||||
console.log(` ⊘ evaluated nothing: ${skips.map((s) => s.name).join(', ')}`)
|
||||
console.log(' Each declined on a precondition it names in its own output (run it alone to see).')
|
||||
}
|
||||
if (failed.length) {
|
||||
console.error(`✗ ${LABEL}: ${failed.length} step(s) FAILED — ${failed.map((f) => f.name).join(', ')}`)
|
||||
process.exit(1)
|
||||
|
|
|
|||
|
|
@ -68,7 +68,7 @@
|
|||
"check:provider-switch": "node internal/check-provider-switch.mjs",
|
||||
"check:mfe-two-tab": "node internal/check-mfe-two-tab.mjs",
|
||||
"check:share-policy": "node internal/check-share-policy.mjs",
|
||||
"check:zone-census": "node internal/check-zone-census.mjs",
|
||||
"check:zone-census": "node internal/check-zone-census.mjs # every zone a USER CAN SELECT must have a poster (asks the LEGEND, not clusters.json — two collapsed deaf lists are unreachable and not a gap). CONDITIONAL: SKIPs without out/posters/, which is gitignored — run `bun run posters` first. It cannot be fixtured like check:packets-all, because it asserts COVERAGE of real generated output and a fixture would only assert the fixture is complete.",
|
||||
"check:print-geometry": "node internal/check-print-geometry.mjs",
|
||||
"check:packet-browser": "node internal/check-packet-browser.mjs",
|
||||
"check:packet-render": "node internal/check-packet-render.mjs",
|
||||
|
|
|
|||
Loading…
Reference in a new issue