A five-second studio check, so a dev-surface regression cannot ride along

One commit shipped the requested change and a regression together: a base prefix applied to an already-rooted URL 404'd all 29 studio tabs, and finding it took 32 minutes. The root-cause analysis attributes the largest single share of that to there being no gate that exercises the dev server at all — of 46 gate-ish scripts and 13 browser gates, none did. The built manifest hands relative refs and the dev manifest hands rooted ones, so a base-join bug is invisible to every build-time gate and fatal to every developer. The check finishes in about four seconds against an already-running dev server, which is the point: the full sweep costs about seventy-five, so it only ever gets run at the end of a change, which is exactly how a regression rides along unnoticed. It activates one tab rather than twenty-nine because a single mount already exercises the whole resolver path — manifest fetch, script resolve, module load, define, mount. If the resolver is wrong, the first tab fails like all of them. It asserts the five things that have actually broken: that the root serves the studio and not a staging page, that the element both mounts and registers, that the tab list is not empty, that no request failed, and that the activated pane holds real content — because an element can register and attach while its content never arrives, which is what a missing API base looks like. An unreachable dev server exits 2 rather than 1. Conflating "no server" with "broken product" is how a gate earns a reputation for flakiness and gets switched off. Negative-tested: removing the tree base makes it fail on the empty tab list, and an unreachable URL exits 2; restoring returns to green.
2026-09-22 14:48:33 +00:00 · 2026-09-22 14:48:33 +00:00 · f3cdc1ad8d
commit f3cdc1ad8d
parent 371119a3f8
2 changed files with 182 additions and 2 deletions
--- a/internal/smoke-studio.mjs
+++ b/internal/smoke-studio.mjs
@ -0,0 +1,179 @@
+#!/usr/bin/env node
+/**
+ * smoke-studio.mjs — the FIVE-SECOND studio check, runnable on every edit.
+ *
+ * WHY THIS EXISTS (measured, 2026-09-22). A single commit shipped the requested change AND a
+ * regression: a base prefix applied to an already-rooted URL produced
+ *     404 /__plugin-gallery//plugin-designs/<bundle>/index.js
+ * and 404'd ALL 29 studio tabs. Finding it took 32 minutes. The root-cause analysis
+ * (_qa-cycle/01-ideation-research/RESEARCH-why-plugin-studio-changes-are-slow.md) attributes the
+ * single largest share of that to there being NO DEV-SURFACE GATE. The census it measured:
+ *     package.json scripts ........................ 76
+ *     gate-ish (check:/verify:/smoke:/probe:) ..... 46
+ *     gates that launch a real browser ............ 13
+ *     gates in verify:browser-all .................  7
+ *     gates that exercise the DEV SERVER ..........  0   <- the hole this closes
+ *
+ * ★ WHY THE BUILD COULD NEVER HAVE CAUGHT IT. The BUILT manifest hands RELATIVE refs
+ * ("./big-picture/big-picture.js"); the DEV manifest hands ROOTED ones
+ * ("/plugin-designs/big-picture/index.js"). A `base + ref` join is therefore correct forever in the
+ * built tree and broken instantly in dev — every build-time gate stays green while every developer's
+ * studio is broken. A product that must work on ten laptops cannot be verified by the build alone.
+ * See REFERENCE-the-four-surface-contract.md.
+ *
+ * ★ WHY IT ACTIVATES ONE TAB, NOT 29. The full sweep costs ~75s (29 tabs x ~2.5s), so it only ever
+ * gets run at the END of a change — which is exactly how a regression rides along undetected. One
+ * mount exercises the whole resolver path: manifest fetch -> scriptUrl resolve -> module load ->
+ * customElements.define -> mount. If the resolver is wrong, tab 1 fails like all 29. Cheap enough to
+ * run on every edit is worth more than thorough enough to run once.
+ *
+ * WHAT IT ASSERTS — each one maps to a defect that actually happened:
+ *   1. `/` serves the STUDIO, not a staging page   (three staging surfaces have masqueraded as the product)
+ *   2. the element MOUNTS and its tag is DEFINED   (a classic <script> on a dev ES module fires
+ *                                                   onload, THEN throws — the tag is never defined
+ *                                                   while the loader reports success)
+ *   3. the tab list is NON-EMPTY                   (a missing tree-base mounts an EMPTY shell; zero
+ *                                                   tabs is a PASS for any check that only looks for errors)
+ *   4. ZERO 4xx/5xx during load + one activation   (this alone catches the double-slash 404)
+ *   5. the activated pane holds REAL CONTENT       ("mounts" and "works" are different assertions —
+ *                                                   a dashboard with no MAP_APP_API_URL mounts fine
+ *                                                   and renders nothing)
+ *
+ * USAGE:   bun run smoke:studio                 (expects `bun run dev` already running on :5173)
+ *          bun run smoke:studio --url http://localhost:5199
+ * EXIT:    0 pass · 1 a real failure · 2 the dev server is not reachable (NOT a product failure)
+ *
+ * NEGATIVE-TESTED: re-introducing the base-concatenation bug makes this FAIL naming the 404 URL;
+ * removing tree-base makes it FAIL on the empty tab list. A gate that cannot fail is worse than none.
+ */
+import { chromium } from 'playwright'
+
+const argOf = (flag, dflt) => {
+  const i = process.argv.indexOf(flag)
+  return i > -1 && process.argv[i + 1] ? process.argv[i + 1] : dflt
+}
+const BASE = String(argOf('--url', 'http://localhost:5173')).replace(/\/+$/, '')
+const BUDGET_MS = Number(process.env.SMOKE_BUDGET_MS || 20000)
+
+// Deep walker — SHADOW DOM DOES NOT PIERCE. A flat querySelectorAll lies about this component, and a
+// probe that cannot see inside a shadow root reports "empty" for a perfectly healthy studio.
+const DEEP = `window.__q=(s)=>{const o=[],z=new Set();const w=r=>{if(!r||z.has(r))return;z.add(r);
+ for(const e of r.querySelectorAll('*')){if(e.matches&&e.matches(s))o.push(e);if(e.shadowRoot)w(e.shadowRoot);}};w(document);return o;}`
+
+const t0 = Date.now()
+const fail = (msg, detail) => {
+  console.error(`SMOKE-STUDIO: FAIL — ${msg}`)
+  if (detail) console.error(detail)
+  process.exit(1)
+}
+
+// Reachability is NOT a product failure. Exit 2 so a caller can tell "dev server is down" from
+// "the studio is broken" — conflating them is how a gate gets muted for being flaky.
+let probe
+try { probe = await fetch(BASE + '/', { redirect: 'follow' }) } catch { probe = null }
+if (!probe || !probe.ok) {
+  console.error(`SMOKE-STUDIO: dev server not reachable at ${BASE} — is \`bun run dev\` running?`)
+  process.exit(2)
+}
+const html = await probe.text()
+
+// ── assertion 1: / serves the STUDIO ─────────────────────────────────────────
+if (!/<plugin-studio\b/.test(html)) {
+  fail(`${BASE}/ does not serve <plugin-studio>.`,
+    '  The dev root must mount the capability-5 studio element. Three different staging pages have\n' +
+    '  occupied this route historically; each returned HTTP 200 while mounting no custom element.')
+}
+if (/One shared core|paste-anywhere maps|scroll the strip/i.test(html)) {
+  fail(`${BASE}/ is serving a STAGING GALLERY, not the studio.`,
+    '  That page is a chooser: it renders real-looking content and mounts nothing. See\n' +
+    '  _archive/staging-surface-2026-09-22/WHY-THIS-WAS-ARCHIVED.md.')
+}
+
+const browser = await chromium.launch()
+const ctx = await browser.newContext({ viewport: { width: 1280, height: 900 } })
+const page = await ctx.newPage()
+const bad = []
+const errs = []
+page.on('response', r => { if (r.status() >= 400) bad.push(`${r.status()} ${r.url().replace(BASE, '')}`) })
+page.on('pageerror', e => errs.push(String(e).slice(0, 160)))
+
+await page.goto(BASE + '/', { waitUntil: 'domcontentloaded', timeout: BUDGET_MS })
+// Wait for the tag to define rather than sleeping a fixed amount: the studio is ready when its
+// element is registered, and polling that is both faster and stricter than a timer.
+let defined = false
+try {
+  await page.waitForFunction(() => !!customElements.get('plugin-studio'), null, { timeout: BUDGET_MS })
+  defined = true
+} catch { /* handled below with the full diagnostic */ }
+
+await page.evaluate(DEEP)
+const state = await page.evaluate(() => ({
+  present: !!document.querySelector('plugin-studio'),
+  tabs: window.__q('.vtl-tab').length,
+}))
+
+// ── assertion 2: the element mounts AND its tag is defined ───────────────────
+if (!defined || !state.present) {
+  await browser.close()
+  fail(`<plugin-studio> ${!state.present ? 'is absent from the DOM' : 'never registered'}.`,
+    '  A classic <script> pointed at a dev ES module fires onload SUCCESSFULLY and only then throws\n' +
+    '  "Cannot use import statement outside a module" — the tag is never defined while the loader\n' +
+    '  reports success. Gate mounting on customElements.whenDefined, never on script onload.\n' +
+    (errs.length ? '  page errors:\n    ' + [...new Set(errs)].slice(0, 3).join('\n    ') : ''))
+}
+
+// ── assertion 3: the tab list is non-empty ───────────────────────────────────
+if (state.tabs === 0) {
+  await browser.close()
+  fail('the studio mounted but has ZERO tabs.',
+    '  It builds its tab list by fetching <tree-base>manifest.json. With no tree-base it falls back to\n' +
+    '  a path relative to its own script URL, which does not exist under the dev server — so the shell\n' +
+    '  renders EMPTY. Zero tabs is a PASS for any check that only looks for errors, which is why this\n' +
+    '  assertion is explicit.\n' +
+    (bad.length ? '  4xx/5xx:\n    ' + [...new Set(bad)].slice(0, 5).join('\n    ') : ''))
+}
+
+// ── assertion 5 (setup): activate ONE tab ────────────────────────────────────
+await page.evaluate(DEEP)
+await page.evaluate(() => { const t = window.__q('.vtl-tab')[0]; if (t) t.click() })
+await page.waitForTimeout(2500)
+await page.evaluate(DEEP)
+const pane = await page.evaluate(() => {
+  const p = window.__q('.ps-pane').find(x => !x.hasAttribute('hidden'))
+  if (!p) return { none: true }
+  const hint = p.querySelector('.ps-hint')
+  const slot = p.querySelector('.ps-slot')
+  return {
+    hint: hint ? hint.textContent.replace(/\s+/g, ' ').trim().slice(0, 80) : '',
+    kids: slot ? slot.children.length : 0,
+  }
+})
+
+await browser.close()
+
+// ── assertion 4: zero 4xx/5xx ────────────────────────────────────────────────
+// Deliberately placed AFTER activation so it covers the mount path too — the double-slash 404 fired
+// when a pane loaded its bundle, not on the initial document.
+const uniq = [...new Set(bad)]
+if (uniq.length) {
+  fail(`${uniq.length} failed request(s) while loading the studio.`,
+    '    ' + uniq.slice(0, 6).join('\n    ') +
+    '\n\n  A doubled slash means a base was prefixed onto an already-rooted URL — see\n' +
+    '  check:url-resolver and AP-a-base-prefix-must-never-be-applied-to-an-already-rooted-url.\n' +
+    '  A relative /api/... means a global is missing (MAP_APP_API_URL); Vite answers it with\n' +
+    '  SPA-fallback HTML at 200, so the component mounts and silently renders nothing.')
+}
+
+// ── assertion 5: the pane holds real content ─────────────────────────────────
+if (pane.none) fail('no pane became active after clicking the first tab.')
+if (/could not load|failed/i.test(pane.hint || '')) fail(`the first tab reports an error: "${pane.hint}"`)
+if (!pane.kids) {
+  fail('the first tab activated but mounted NOTHING (empty slot).',
+    '  "Mounts" and "works" are different assertions. An element can register and attach while its\n' +
+    '  content never arrives — that is what a missing API base or a failed child import looks like.')
+}
+
+const secs = ((Date.now() - t0) / 1000).toFixed(1)
+console.log(`SMOKE-STUDIO: PASS in ${secs}s — ${BASE}/ mounts <plugin-studio>, ${state.tabs} tab(s), first tab holds ${pane.kids} node(s), 0 failed requests.`)
+console.log('  This is the DEV surface the build-time gates structurally cannot see: the built manifest')
+console.log('  hands relative refs, so a base-join bug is invisible there and fatal here.')
--- a/package.json
+++ b/package.json
@ -13,7 +13,7 @@
    "clean": "node internal/clean.mjs",
    "build:bundle": "bun x vite build",
    "tree": "node internal/generate-tree.mjs",
-    "gallery": "node internal/generate-gallery.mjs   # writes public/index.html, the built plugin gallery (the one root page), from public/doxa-maps/manifest.json",
+    "gallery": "node internal/generate-gallery.mjs   # writes public/__gallery/index.html — the ARCHIVED bundle-index gallery, from public/doxa-maps/manifest.json. It is NO LONGER the root page: as of 2026-09-22 `/` mounts <plugin-studio> in BOTH trees (the built root page is written by `bun run tree`), and the gallery is the opt-in index at /__gallery. Archived, not deleted — the dev server imports renderGallery()/readBuildConfig() from this module.",
    "check:drop": "node internal/drop-portability-check.mjs",
    "check:parse": "node internal/inline-parse-check.mjs",
    "verify": "node library/scripts/research/verify-all.mjs",
@ -81,7 +81,8 @@
    "check:shot-fonts": "node internal/check-shot-fonts.mjs   # can THIS machine render the emoji the UI uses? Run BEFORE `bun run shots` — no font = tofu boxes in every committed PNG, which every other shot gate passes. Deliberately NOT in build:unlocked: it reports an ENVIRONMENT condition (this box has no emoji font), and a gate that always fails in the build makes the build useless.",
    "smoke:share-all": "sh -c 'B=${BASE_URL:-http://127.0.0.1:4173}; exec node internal/share-roundtrip-smoke.mjs --url \"$B/doxa-maps/cities-internet/connectivity-map/index.html?tk=$MAPBOX_TOKEN\" --url \"$B/doxa-maps/country-posters/countries-map/index.html?tk=$MAPBOX_TOKEN\" --url \"$B/doxa-maps/demo-map/demo-map/index.html?tk=$MAPBOX_TOKEN\" --url \"$B/doxa-maps/doxa-research-map/research-map/index.html?tk=$MAPBOX_TOKEN\" --url \"$B/doxa-maps/doxa-simple-map/doxa-simple-map/index.html?tk=$MAPBOX_TOKEN\" --url \"$B/doxa-maps/gospel-resources/resources-map/index.html?tk=$MAPBOX_TOKEN\" --url \"$B/doxa-maps/my-upg-100-list/upg-100-list/index.html?tk=$MAPBOX_TOKEN\" --url \"$B/doxa-maps/my-upg-100-list/upg-100-mfe/index.html?tk=$MAPBOX_TOKEN\" --browser chromium'",
    "check:sfc-parse": "node internal/check-sfc-script-parses.mjs   # every .vue script block must parse as the BUILD parses it — catches the LIVE-CSS backtick that check:template-literal cannot see (it reads CSS comments only). 0 false positives on 83 files, where 3 lookahead heuristics scored 98 and 136.",
-    "check:url-resolver": "node internal/check-url-resolver.mjs   # cap-5 resolve() must not prefix a rooted URL (the dev-surface 404 class)"
+    "check:url-resolver": "node internal/check-url-resolver.mjs   # cap-5 resolve() must not prefix a rooted URL (the dev-surface 404 class)",
+    "smoke:studio": "node internal/smoke-studio.mjs   # 5s DEV-surface check: / mounts the studio, tabs>0, 0 4xx (exit 2 = dev server down)"
  },
  "devDependencies": {
    "@capacitor/android": "^7.6.9",