be3620ef1c
Two defects, both reproduced before the fix and re-measured after. 1. It assigned BASE for its lock path. A variable the caller exported stays exported in bash, so every gate run through the slot saw BASE=/tmp/doxa-maps-browser-gate.lock. Seven gates read process.env.BASE; check:upg-map-data-painted printed "SKIP — nothing served at /tmp/doxa-maps-browser-gate.lock/…" and exited 0. Renamed to SLOT_BASE. Proof: with BASE=/x set, a child printed the lock path before and /x after. 2. With every slot busy, waiters queued on slot 1 alone, so a slot that freed was never taken and two slots behaved as one queue. It now polls every slot with flock -n every 2 s until the deadline and still exits 75 on timeout. Proof, in a private slot dir with slot 1 held 12 s and slot 2 freed at ~3 s: a caller waited 11.5 s before and 4.3 s after. A free slot is still taken at once (66 ms). check-upg-map-data-painted: its unreachable-server SKIP exited 0 and its wrong-tree refusal exited 2. Both now exit 75, the repo's "not measured" code. Its existing tree-identity check is what makes the :4173 default safe, since another agent's tree there is refused.
82 lines
3.9 KiB
Bash
Executable file
82 lines
3.9 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# browser-slot.sh — run ONE browser command inside a memory-floored, N-slot rendezvous.
|
|
#
|
|
# WHY A SEMAPHORE AND NOT A LOCK. A single lock made every agent's browser proof wait behind one
|
|
# chrome: measured 2026-09-25, the queue sat 10-19 deep for hours. The real constraint is MEMORY,
|
|
# not exclusivity — a headless chrome costs ~300-500MB — so the right shape is "as many browsers as
|
|
# the box can hold", which is a counting semaphore with a memory floor, not a mutex.
|
|
#
|
|
# HOW: N slot files. Try each with flock -n (no wait). If all are busy, WAIT on slot 1. Before
|
|
# taking any slot, require MemAvailable above the floor — a slot that is free while the box is out
|
|
# of memory is an invitation to be OOM-killed mid-measurement, which reads as a flaky gate.
|
|
#
|
|
# ★ EXIT 75 MEANS "NOT MEASURED", NEVER "FAILED". flock returns 1 on timeout, the same code a
|
|
# failing gate uses, and that once turned a busy queue into five red product gates. 75 is
|
|
# EX_TEMPFAIL: distinct from 0 and from any gate's own verdict, so a caller can report it as the
|
|
# precondition it is.
|
|
#
|
|
# Usage: internal/browser-slot.sh <command> [args...]
|
|
# Env: BROWSER_SLOTS (default 2), BROWSER_SLOT_DIR (default /tmp),
|
|
# BROWSER_MEM_FLOOR_MB (default 2500), BROWSER_LOCK_WAIT (default 900),
|
|
# BROWSER_MEM_WAIT (default 300)
|
|
set -u
|
|
|
|
SLOTS="${BROWSER_SLOTS:-2}"
|
|
DIR="${BROWSER_SLOT_DIR:-/tmp}"
|
|
# SLOT_BASE, never BASE: a variable the CALLER exported stays exported in bash, so assigning BASE
|
|
# here rewrote the caller's BASE for every gate this runs. Seven gates read process.env.BASE, and
|
|
# check:upg-map-data-painted measured /tmp/doxa-maps-browser-gate.lock/... and SKIPPED with exit 0.
|
|
SLOT_BASE="$DIR/doxa-maps-browser-gate.lock"
|
|
FLOOR_MB="${BROWSER_MEM_FLOOR_MB:-2500}"
|
|
WAIT_S="${BROWSER_LOCK_WAIT:-900}"
|
|
MEM_WAIT_S="${BROWSER_MEM_WAIT:-300}"
|
|
BUSY=75
|
|
|
|
slot_path() { [ "$1" = "1" ] && echo "$SLOT_BASE" || echo "$SLOT_BASE$1"; }
|
|
|
|
mem_available_mb() {
|
|
awk '/^MemAvailable:/ { printf "%d", $2 / 1024 }' /proc/meminfo 2>/dev/null || echo 999999
|
|
}
|
|
|
|
# Hold off while the box is below the floor. Proceeding anyway is not an option here the way it is
|
|
# for a load average: load makes a measurement slow, memory makes it get killed halfway through.
|
|
waited=0
|
|
while [ "$(mem_available_mb)" -lt "$FLOOR_MB" ]; do
|
|
if [ "$waited" -ge "$MEM_WAIT_S" ]; then
|
|
echo "browser-slot: MemAvailable $(mem_available_mb)MB below floor ${FLOOR_MB}MB after ${waited}s — not measuring" >&2
|
|
exit "$BUSY"
|
|
fi
|
|
sleep 10
|
|
waited=$((waited + 10))
|
|
done
|
|
|
|
# Fast path: any free slot, in a FIXED order so two processes can never take them in opposite
|
|
# orders. -n means "do not wait": if this slot is busy we move on rather than queue behind it.
|
|
i=1
|
|
while [ "$i" -le "$SLOTS" ]; do
|
|
exec 9>"$(slot_path "$i")" 2>/dev/null || true
|
|
if flock -n 9; then
|
|
exec "$@"
|
|
fi
|
|
i=$((i + 1))
|
|
done
|
|
|
|
# Everything busy: keep POLLING EVERY SLOT until the deadline, and report a timeout as UNMEASURED
|
|
# rather than as a verdict. This used to queue on slot 1 alone (flock -w on slot 1), so a slot that
|
|
# freed up was never taken: measured 2026-09-25 (dx-shepherd), ten waiters were piled on slot 1 while
|
|
# slot 2 sat free, and with slot 1 held 12 s and slot 2 freed at ~3 s a caller waited 11.5 s — the
|
|
# two-slot semaphore had collapsed into ONE queue. Polling takes the first slot that frees (~4 s in
|
|
# the same test). Trade-off, accepted: polling is not first-come-first-served, so under heavy
|
|
# contention a waiter can lose a few rounds; an idle slot beside a queue is strictly worse.
|
|
deadline=$(( $(date +%s) + WAIT_S ))
|
|
while [ "$(date +%s)" -lt "$deadline" ]; do
|
|
i=1
|
|
while [ "$i" -le "$SLOTS" ]; do
|
|
exec 9>"$(slot_path "$i")" 2>/dev/null || true
|
|
if flock -n 9; then exec "$@"; fi
|
|
i=$((i + 1))
|
|
done
|
|
sleep 2
|
|
done
|
|
echo "browser-slot: no slot within ${WAIT_S}s (${SLOTS} slots, floor ${FLOOR_MB}MB) — not measuring" >&2
|
|
exit "$BUSY"
|