Rudder is built for a thin client: by default the whole budget goes to speech-to-text and nothing of the
robot voice runs or even loads.
- Settings → Robot voice is marked "experimental". Ticking it first shows a warning (in the page, not a
browser dialog the desktop webview may block): it runs an extra voice engine, uses more CPU and ~150 MB
while it speaks, and can slow voice-to-text on a smaller laptop or shared dev server. "Turn on anyway"
switches to High performance mode; the settings show the current mode.
- Engine: POST /api/rudder/mode {robot} switches thin-client ↔ high performance, and refuses the robot when
the machine has under ~500 MB available (/proc/meminfo) — a voice tool must never crash someone's laptop.
/api/rudder/tts refuses in thin-client mode, so an old saved setting can't start the robot either. In
thin-client mode Piper would yield the CPU; in high performance mode it runs at normal priority.
- The voice list loads only when the robot voice is turned on; a robot voice left on re-enters high
performance mode on the next visit (the engine re-checks memory).
- Verified on the engine: starts thin-client; speaking refused (409); on → high-performance; speaks; off →
thin-client; speaking refused again.
Optimising the default for an average laptop, not more CPU (the speech model is hard-wired to 4 threads in
NeMo-Speech.cpp, session.cpp:911 — already a thin-client budget).
- RECONNECT: a live recording ended with "speech server connection lost: … broken pipe" when the stream to
the speech server broke. Now the engine opens a new stream and keeps going; the words still in flight on
the old one are finalised so the box keeps them; up to 5 attempts, then a plain message. Verified against
a fake speech server that drops the first stream after 3 s: 6 words finalised, reconnected in 2 ms, 14 more
words streamed, clean Stop, 0 data races (go -race).
- FALLING BEHIND: when the mic clearly carries speech but no words have come back for 10 s (the server is
starved of CPU — the stream used to freeze with no message), say so once; it clears when words resume.
Verified with a fake server that returns no words: warned after 10 s.
- The robot voice (Piper) runs at nice 10 on Linux, so its ~1 s burst per sentence never makes the words
you are speaking fall behind.
- Engine: start the speech server with --asr.endpointing.enable=true. End-of-sentence detection is OFF in
NeMo-Speech.cpp by default, and session.update's endpointing_ms only sets the pause length, so every
recording came back as ONE final when Stop was pressed (measured: a minute of speech → one 437-character
final; the robot voice then spoke a minute late). Now each sentence is final ~1 s after it ends: in a
Firefox run the robot spoke the first sentence 13 s before Stop.
- Settings menu in the full layout too (⚙, closed by default, works at phone width): Language, Microphone
and Robot voice live in it; Record and the level meter stay visible.
- Robot voice: exactly ten presets, five male then five female, chosen by measured pitch (men ~92–108 Hz,
women ~206–237 Hz); Male voice 1 is the default. The browser's own voices appear only as a fallback when
the offline voices are missing, at most ten. A blocked or failed playback is shown instead of silence.
Settings → Robot voice: a checkbox and a voice dropdown. When on, every finished sentence is spoken back
(sentences, not words: TTS needs context for its intonation). speak(text) is the single entry point, so a
future conversational mode can reuse the same voice.
- Providers, both on this computer only: Piper (offline neural TTS run by the engine on the CPU; six
LibriTTS-R speakers offered as "Offline voice A–F") and the browser's own voices, restricted to
localService === true (some browsers synthesise "online" voices on a server — never offered).
- backend/rudder-engine/tts.go: POST /api/rudder/tts (text → audio/wav, 1–2000 characters, one synthesis
at a time), GET /api/rudder/tts/voices, POST /api/rudder/duck. The spoken text is never logged, only
its length. Piper 2023.11.14-2 (MIT) bundles ONNX Runtime 1.14.1: no telemetry client; a synthesis
traced with strace made 0 network calls. ~150 MB RAM; a 10.7 s sentence rendered in 2.2 s on CPU.
- While the robot speaks through loudspeakers, the engine feeds the transcriber silence, so it never
transcribes itself (and that silence is not mistaken for a dead mic). With a Bluetooth headset or
headphones it keeps listening — pausing would only lose the person's words.
- setup-runtime.sh: Piper and the en_US-libritts_r-medium voice (CC BY 4.0), pinned by SHA-256.
- Verified in Firefox against a dev server: dropdown lists A–F, each finished sentence is synthesised
(200 audio/wav) between duck on/off, zero requests to any other host.
The engine did not compile outside Linux: Pdeathsig (kill children with the
engine) is Linux-only. It now sits behind a build tag, with a no-op elsewhere
where the desktop shell owns the engine's lifetime. The default speech-server path
names this platform's NeMo-Speech.cpp release folder instead of the Linux one.
The /stream WebSocket accepts the Tauri origins (tauri://localhost,
http://tauri.localhost) and RUDDER_ALLOW_ORIGIN as well as the page's own host,
and still refuses any other site. Cross-compiled for darwin/arm64, darwin/amd64
and windows/amd64.
The streaming model holds back the last fraction of a second until more audio
or a pause arrives, and the commit alone did not flush it, so a Stop
mid-sentence lost its final words. The engine now sends 1.2 s of silence before
the commit, and keeps reading after the commit until the final text or 6 s of
quiet. Measured: a stream cut at 6 s of the test clip now ends "...ask not what
your" instead of "...ask not".
On a loaded server the last words can land seconds after Stop, and the widget
read "Stop" until they did. Stop now releases the mic and returns to idle at
once while the session's socket stays open for the tail. Each engine session
sends its events to its own reader, so a session still draining never writes
into the next one, and Record again simply lets go of the old tail.
With RUDDER_AUDIO=browser the engine records no local microphone. The page
captures the reader's mic, converts it to 16 kHz mono PCM in an AudioWorklet,
and streams it over a WebSocket at /api/rudder/stream into the same speech
session a local recorder feeds. The words come back on that socket to that
reader only, never on the shared event stream. One session at a time; a second
reader is told the engine is busy. The engine honours RUDDER_HOME, accepts the
public host named in RUDDER_ALLOW_HOST, and refuses a socket from another
origin. The dev proxy now carries WebSockets to backends.
deployment/cloud: PocketBase (pinned, sha256-checked) as dev (/pb/, signed in; its /_/ dashboard for
OIDC_ADMIN_GROUPS only) and prod (/api-prod/api/ public, dashboard never exposed), with the schema
as migrations in backend/pocketbase/pb_migrations and data on the /data volume; superuser from env.
admin.mjs routes /cdn/, /api-prod/, /pb/ and /admin/ behind oauth2-proxy (public: /cdn/ and
/api-prod/api/ only). build-cdn.sh: flock (409 on a second build), a snapshot of the pulled commit,
the public exclusions applied BEFORE the build, a heap-capped build, a check of the output that
REFUSES the swap if anything excluded remains (proved with planted fixtures: a field-access file,
'Visa Requirements' content, and key-cities one level down each fail; clean passes), then an atomic
swap; runtime-config.js carries the public API URL. public-exclusions.txt is the ONE list the
giveaway export and the CDN build share. Any git host via REPO_URL/BRANCH/GIT_TOKEN.
Rudder is the first plugin that isn't a map: it shows that one plugin-designs/ folder, developed live
in the Plugin Studio, can ship as a desktop app with its own local backend.
- plugin-designs/rudder-voice: <rudder-voice> element. Words stream into the end of a text area while
you type/paste anywhere (typing holds the stream; nothing duplicated or lost). Mic picker with
hot-plug, 3-second mic test, plain-language errors, info panel, privacy-safe debug report.
meta.json declares "targets": ["desktop"].
- backend/rudder-engine (Go): supervises NVIDIA NeMo-Speech.cpp (ggml, CPU) with the Nemotron 3.5 ASR
Streaming GGUF over a loopback WebSocket; records with parec/pw-record only while recording; recovers
dead Bluetooth mic links (reopen -> CVSD voice profile -> optional RUDDER_BT_RESET) and starts
headsets on CVSD. No telemetry: traced with strace, zero sockets/DNS while transcribing.
- Privacy: log lines redact MAC addresses and the home folder, transcripts are never logged, Host check
against DNS rebinding, CORS only for the desktop webview, random API key for the speech server.
- internal/backend-dev-plugin.mjs + vite.config.js: `bun run dev` starts built plugin backends and
proxies /api/<name> same-origin, so a plugin calls the same API in the studio and on desktop.
- backend/rudder-engine/setup-runtime.sh: fetches the runtime and model pinned by SHA-256.
Not yet: the Tauri desktop build (sidecar) and the PocketBase word-correction table.
