diff --git a/documentation/HOWTO-which-gates-need-a-browser.md b/documentation/HOWTO-which-gates-need-a-browser.md new file mode 100644 index 000000000..45a8bb0c4 --- /dev/null +++ b/documentation/HOWTO-which-gates-need-a-browser.md @@ -0,0 +1,105 @@ +# Which gates need a browser + +There are **238** checks in this repo. **148 run anywhere. 87 need a real browser.** If you are not +the person who owns the browser seat, that second group is not yours to run — and knowing which is +which saves you from a whole class of wrong answer. + +> **The short version.** A gate that reads *files* is headless. A gate that asks *what the page +> actually painted* needs a browser. Run the first kind freely; hand the second kind to whoever owns +> the browser. + +--- + +## Telling them apart + +Open the gate's file and look for one import: + +```bash +grep -l 'playwright\|chromium' internal/check-*.mjs +``` + +If it imports `playwright` or launches `chromium`, it needs a browser. That is the whole test — the +name does not tell you. `check:packet-render` sounds like a file check and drives a browser; +`check:composites-mount` sounds like it needs one and does not (it mounts every composite through +Vite's SSR loader, no browser at all). + +### Measured on `main` (2026-09-28) + +| | Count | +|---|---:| +| headless — run these yourself | **148** | +| **browser — not yours unless you own the seat** | **87** | +| neither (shells out to `bun`/`php`) | 3 — `check:mobile`, `check:wp-embed`, `smoke:playground` | + +The browser group is, broadly: everything that renders a map (`check:map-paints`, +`check:map-data-painted`, `check:auto-engine`, `check:maplibre-drops`), everything about layout +(`check:fits-screen`, `check:shell-scroll`, `check:filter-width`, `check:scrim-viewport`), +everything that prints (`check:poster-prints`, `check:print-geometry`, `check:packet-render`), +everything about right-to-left shaping (`check:rtl-shaping`, `check:embedded-rtl`), the English-leak +family (`check:english-leaks`, `check:leaf-english-leaks`, `check:host-locale-contract`), and every +`smoke:*`. + +## Why the split exists, and why it is not negotiable + +One machine, one browser seat. Two agents running Playwright at once fight over it, and the loser +does not fail — **it reports that it could not look**, which is a third outcome most people's mental +model does not have (see below). So the rule is: **one person runs the browser gates.** In this +repo that is the QA agent; ask them rather than starting a second browser. + +This is not a limitation to work around. A source read and a render are *different measurements*, +and confusing them is the most common way to report something false: + +- `check:leaf-english-leaks` can only find a leak by comparing what the catalogue says with **what + the page shows**. Four catalogue gates can all pass — keys exist, differ per locale, are reachable, + are in scope — while a page still paints English. Catalogue-green is not render-green. +- Equally, a headless check often settles a question people assume needs pixels. Of 16 cards parked + as "needs a browser", **7 turned out to be about markup, a path, a manifest field or a gate** — + all readable without one. + +**Ask what property the claim is about**, then pick the tool. Not the other way round. + +--- + +## ★ Exit 75 means NOT MEASURED. It is not a pass and it is not a failure. + +This is the one that bites hardest, because it looks like success. + +`internal/run-parallel.mjs:262` states the contract: + +> THREE OUTCOMES, never two. Exit **75** (`EX_TEMPFAIL`) from ANY step is **NOT MEASURED**: the gate +> could not look (no browser seat, no slot, no built tree), which is neither a pass nor a failure. A +> gate that exits **0 while announcing SKIP** is also NOT MEASURED. Only any other non-zero exit is a +> FAIL. + +Two traps follow from that, and both have caught people here: + +1. **Exit 0 is not enough.** A gate can exit 0 and print `SOMETHING: SKIP` — it looked at nothing. + The runner marks that NOT MEASURED; a human reading only `$?` records a pass. +2. **Your own `timeout` produces exit 124, which is also not a failure.** Killing a gate at 180 s + because you were impatient tells you nothing about the code. If you must bound a run, bound it + generously and read exit 124 as "I stopped it", never as "it failed". + +The runner prints all three: `✓` pass, `✗` fail, `⊘` not measured. **Read the symbol, not the tail of +the log** — a truncated log with a green last line looks exactly like a pass. + +--- + +## What to do, depending on who you are + +**Landing a change:** run `bun run gates:pre` (all headless, 15 checks, ~30 s on a fresh clone) plus +whichever headless gates cover what you touched. Do not run the browser group. + +**You think a browser gate would catch your bug:** say so, name the gate, and say what result would +change your mind. A prediction someone else can falsify is worth more than a run you did yourself. + +**You own the browser seat:** you are the only one. When a gate comes back NOT MEASURED, say +"not measured" — resist reporting it as either colour. + +## A worked example of getting this wrong + +`check:packets-all` reads generated packets from a gitignored scratch directory. It went red on 43 +files left over from a run three days earlier, reporting *"0 prayer tick boxes"* about packets +generated before tick boxes existed. Nothing was broken. The gate was grading leftovers. + +The fix was not to loosen it: it now skips any packet older than the renderer that produces it, and +says so. **A stale input is NOT MEASURED, not FAIL** — the same three-outcome rule, one level down.