a02458a171
238 checks; 148 run anywhere, 87 need a real browser, 3 shell out. A newcomer
cannot tell from the name -- check:packet-render sounds like a file check and
drives a browser; check:composites-mount sounds like it needs one and does not.
The test is one grep for playwright/chromium, and the doc says so.
The part worth writing down is exit 75. run-parallel.mjs:262 defines three
outcomes, not two: 75 (EX_TEMPFAIL) is NOT MEASURED -- the gate could not look --
and a gate that exits 0 while announcing SKIP is also NOT MEASURED. Only another
non-zero is a FAIL. Two traps follow: exit 0 alone does not mean a gate looked at
anything, and your own `timeout` produces 124, which says you stopped it, not
that the code failed.
Also states the rule the split exists for -- one machine, one browser seat, so
one person runs them -- and why it is not just logistics: a source read and a
render are different measurements. Four catalogue gates can pass while a page
still paints English; and of 16 cards parked as "needs a browser", 7 were about
markup, a path, a manifest field or a gate, and settled headlessly.
Every number measured on main d8baaefd2 rather than estimated, including
correcting gates:pre from 13 to 15 after re-running it.
5.2 KiB
Which gates need a browser
There are 238 checks in this repo. 148 run anywhere. 87 need a real browser. If you are not the person who owns the browser seat, that second group is not yours to run — and knowing which is which saves you from a whole class of wrong answer.
The short version. A gate that reads files is headless. A gate that asks what the page actually painted needs a browser. Run the first kind freely; hand the second kind to whoever owns the browser.
Telling them apart
Open the gate's file and look for one import:
grep -l 'playwright\|chromium' internal/check-*.mjs
If it imports playwright or launches chromium, it needs a browser. That is the whole test — the
name does not tell you. check:packet-render sounds like a file check and drives a browser;
check:composites-mount sounds like it needs one and does not (it mounts every composite through
Vite's SSR loader, no browser at all).
Measured on main (2026-09-28)
| Count | |
|---|---|
| headless — run these yourself | 148 |
| browser — not yours unless you own the seat | 87 |
neither (shells out to bun/php) |
3 — check:mobile, check:wp-embed, smoke:playground |
The browser group is, broadly: everything that renders a map (check:map-paints,
check:map-data-painted, check:auto-engine, check:maplibre-drops), everything about layout
(check:fits-screen, check:shell-scroll, check:filter-width, check:scrim-viewport),
everything that prints (check:poster-prints, check:print-geometry, check:packet-render),
everything about right-to-left shaping (check:rtl-shaping, check:embedded-rtl), the English-leak
family (check:english-leaks, check:leaf-english-leaks, check:host-locale-contract), and every
smoke:*.
Why the split exists, and why it is not negotiable
One machine, one browser seat. Two agents running Playwright at once fight over it, and the loser does not fail — it reports that it could not look, which is a third outcome most people's mental model does not have (see below). So the rule is: one person runs the browser gates. In this repo that is the QA agent; ask them rather than starting a second browser.
This is not a limitation to work around. A source read and a render are different measurements, and confusing them is the most common way to report something false:
check:leaf-english-leakscan only find a leak by comparing what the catalogue says with what the page shows. Four catalogue gates can all pass — keys exist, differ per locale, are reachable, are in scope — while a page still paints English. Catalogue-green is not render-green.- Equally, a headless check often settles a question people assume needs pixels. Of 16 cards parked as "needs a browser", 7 turned out to be about markup, a path, a manifest field or a gate — all readable without one.
Ask what property the claim is about, then pick the tool. Not the other way round.
★ Exit 75 means NOT MEASURED. It is not a pass and it is not a failure.
This is the one that bites hardest, because it looks like success.
internal/run-parallel.mjs:262 states the contract:
THREE OUTCOMES, never two. Exit 75 (
EX_TEMPFAIL) from ANY step is NOT MEASURED: the gate could not look (no browser seat, no slot, no built tree), which is neither a pass nor a failure. A gate that exits 0 while announcing SKIP is also NOT MEASURED. Only any other non-zero exit is a FAIL.
Two traps follow from that, and both have caught people here:
- Exit 0 is not enough. A gate can exit 0 and print
SOMETHING: SKIP— it looked at nothing. The runner marks that NOT MEASURED; a human reading only$?records a pass. - Your own
timeoutproduces exit 124, which is also not a failure. Killing a gate at 180 s because you were impatient tells you nothing about the code. If you must bound a run, bound it generously and read exit 124 as "I stopped it", never as "it failed".
The runner prints all three: ✓ pass, ✗ fail, ⊘ not measured. Read the symbol, not the tail of
the log — a truncated log with a green last line looks exactly like a pass.
What to do, depending on who you are
Landing a change: run bun run gates:pre (all headless, 15 checks, ~30 s on a fresh clone) plus
whichever headless gates cover what you touched. Do not run the browser group.
You think a browser gate would catch your bug: say so, name the gate, and say what result would change your mind. A prediction someone else can falsify is worth more than a run you did yourself.
You own the browser seat: you are the only one. When a gate comes back NOT MEASURED, say "not measured" — resist reporting it as either colour.
A worked example of getting this wrong
check:packets-all reads generated packets from a gitignored scratch directory. It went red on 43
files left over from a run three days earlier, reporting "0 prayer tick boxes" about packets
generated before tick boxes existed. Nothing was broken. The gate was grading leftovers.
The fix was not to loosen it: it now skips any packet older than the renderer that produces it, and says so. A stale input is NOT MEASURED, not FAIL — the same three-outcome rule, one level down.