[Feature] Speech support: voice input to prompt and read-aloud of agent responses

[n0tlu5]

Motivation

Codeg has no voice path today. Two directions would help, especially when you're away from the keyboard or reviewing long agent output:

1. Speech → prompt: dictate into the chat composer instead of typing.
2. Agent response → speech: have an agent's reply read aloud.

I'd like to contribute both, and I'm opening this issue first to agree on scope with you before writing code.

Relation to #732

Open PR #732 adds a voice-input button to the composer. It only covers speech → prompt. In its current state it:

• hardcodes engine: "gemini", which points at a local endpoint (127.0.0.1:8318) with a credential embedded in the source, so it won't work for other users;
• has no settings UI for choosing an engine or entering a Whisper endpoint or key;
• only ships en / zh-CN strings, with one status string hardcoded in Chinese;
• includes unrelated changes: createUpdaterArtifacts: false in tauri.conf.json and a new build-windows.yml workflow.

Its UX ideas are good: a pulsing indicator while recording, a live preview of the in-progress transcript, and Esc to cancel. I'd reuse them. I'd rather not duplicate work, though, so tell me whether you want a fresh implementation that covers both directions or would rather see #732 reworked first.

Proposed behavior

Speech → prompt

• A mic button in the composer. Click to start and click to stop, with a keyboard shortcut in Settings → Shortcuts.
• A live transcript preview while speaking. The final text goes in at the caret and is never sent automatically. Esc cancels.
• The recognition language defaults to the UI locale and can be overridden.

Agent response → speech

• A "Read aloud" action on each finished agent message, with play/stop.
• An optional "auto-read agent replies when a turn completes" toggle, off by default.
• Speech covers the prose only. Code blocks, tool calls, diffs and raw markdown syntax are skipped or summarized ("code block omitted").
• Playback stops when the user starts typing, sends a new prompt or switches sessions.
• The audio output is released when idle, to avoid the same problem as [Bug] 提示音效的 AudioContext 从不释放：coreaudiod 常驻 ~6% CPU 并全程持有 PreventUserIdleSystemSleep #837.

Settings (a new "Speech" section, or under General)

• Engine selection for each direction. Everything is off by default, so nothing changes for existing users.
• A configurable OpenAI-compatible endpoint, key and model for cloud engines (e.g. Whisper /v1/audio/transcriptions, /v1/audio/speech), stored the way codeg already stores provider credentials.
• Voice, rate and language options.
• Strings in all 10 supported locales.

Both runtimes

• Desktop (Tauri) and server/web mode behave the same wherever the platform allows. Where an engine is unavailable, the UI says so and doesn't fail silently.

Open questions for you

1. feat(chat): add voice input to message composer with dual-mode recognition #732: fresh implementation, or build on feat(chat): add voice input to message composer with dual-mode recognition #732?
2. Engines: browser speech APIs (SpeechRecognition / speechSynthesis) are free and need no setup, but availability varies across Tauri's webviews. Speech recognition in particular isn't reliably available on WebKitGTK (Linux) or WebView2. Are you OK with:
   • (a) a browser engine where available, plus an OpenAI-compatible cloud engine configured by the user, or
   • (b) also adding an offline/native engine on the Rust side (e.g. local whisper), which increases binary size and build complexity?
3. Placement: a dedicated Settings → Speech page, or a section inside General?
4. PR shape: one PR, or two (speech → prompt first, then read-aloud)?

I'll start implementation once you've confirmed the direction.

[n0tlu5]

Implementation plan (1/2)

I've drafted a full plan for this feature and am posting it here for feedback before any code is written. The scope has grown beyond the original description. The plan now has four stacked PRs:

1. Dictation: speech to prompt in the chat composer, with a browser engine plus an optional OpenAI-compatible cloud engine proxied through the backend. The key is stored in the OS keyring.
2. Read-aloud: agent replies spoken on demand or automatically.
3. Workspace assistant: one persistent, backend-owned assistant session that can see every running session. With opt-in switches it can message, stop or start sessions and answer permission prompts, but every action needs a codeg-built confirmation, and permissions are allow once / deny once only.
4. Voice mode: a hands-free, Jarvis-style conversation with that assistant. It covers streaming speech, talking over it to interrupt, fixed voice commands, spoken confirmations, workspace-wide status announcements and an orb UI.

No new npm or cargo dependencies. The PRs are split so each one is reviewable on its own; PR1 and PR2 don't depend on the later ones. Feedback on scope, especially whether the assistant/voice-mode parts (PR3/PR4) are welcome upstream, would be much appreciated before I start.

Full plan, part 1 (overview, scope, verification, todos 1-10)

speech-support - Work Plan

TL;DR (For humans)

Who this is for and what changes for them: Codeg users on desktop (macOS, Windows, Linux) and in the browser/server version can talk to the chat box instead of typing, and can have an agent's reply read out loud. Anyone who doesn't turn it on sees no change.

What you'll get: A microphone button (plus a keyboard shortcut) in the chat box that turns speech into text you can edit before sending; a "read aloud" button on every finished agent reply; an optional "read replies automatically" switch; a new Speech page in Settings to choose the engine, language, voice and an optional cloud speech service; and, in later steps, a Jarvis-style workspace assistant: one persistent assistant conversation (running the agent you choose) that sits above all your tabs and folders. In voice mode you just talk to it: it sends when you stop, speaks its answer while it is still being written, then listens again, and switching tabs never interrupts it. It knows what every session is doing ("the Codex session in api is waiting for approval"), can bring a session to the front, and, if you switch it on, can send a message to another session, stop one, start a new one, or approve/deny a permission prompt. Every such action first shows a confirmation card that codeg builds from real state, which you can answer by saying "confirm" or "cancel". It also announces when sessions anywhere finish, fail or need you, you can talk over it to cut it off, it understands a few fixed commands ("stop", "cancel", "repeat that", "exit voice mode"), and a glowing orb shows whether it is listening, thinking or speaking.

Why this approach: The browser's built-in speech recognition does not work inside Codeg's desktop app on Windows and Linux, so a cloud speech service (any OpenAI-compatible one, configured by the user) is the reliable path there, while the free built-in engines are used wherever they work. The cloud key is kept in the operating system's secure key store and all cloud calls go through Codeg's backend, so the key never sits in the web page and network proxies are respected.

What it will NOT do: No offline/built-in speech model, no "Hey Codeg" wake word, no talking to Codeg from other apps (global hotkey), no free-form voice commands interpreted by an AI (only the assistant agent interprets requests, and its actions on other sessions always need your confirmation), no "always allow" anywhere, no acting on other sessions unless you switch it on, the plain dictation button never sends on its own, and it never reads code blocks or tool output aloud.

Effort: XL
Risk: Medium-High - desktop microphone permissions differ per OS, macOS/Windows cannot be exercised on the Linux build host, talking over the assistant relies on the browser's echo cancellation (weaker for speakers than headphones; it can be switched off), and the assistant adds a new agent-facing tool surface, which is why every action on another session goes through a codeg-built confirmation and both control switches start off.
Decisions to sanity-check: fresh implementation rather than reworking PR #732; cloud engine proxied through the backend; dedicated Settings page; four pull requests (dictation, read-aloud, workspace assistant, voice mode); the assistant is one persistent chat conversation spawned by the backend with the agent you pick (codex, claude, opencode, ...; Pi is excluded because it cannot receive codeg's tools); session control and permission answers off by default, "allow once"/"deny once" only; shortcuts Mod+Shift+M (dictate) and Mod+Shift+J (voice mode); wake word and global hotkey left out; voice mode no longer dictates into the active tab (dictation button does that).

---

TL;DR (machine): XL, Medium-High risk; 20 todos in 4 PRs: (PR1) speech prefs+capabilities, backend OpenAI-compatible transcribe with keyring key, desktop mic enablement, composer dictation, Settings -> Speech; (PR2) synthesize, speakable-text, player, read-aloud button, auto-read; (PR3) workspace assistant: backend-owned persistent assistant chat session (owner label assistant, agent from settings), codeg-mcp assistant tool group (list/focus/send/cancel/answer_permission/start) validated at the listener, mutating tools behind a codeg-built confirmation card, allow_once/reject_once only; (PR4) voice mode bound to the assistant session: single-AudioContext VAD, streaming sentence speech, barge-in, fixed commands + spoken confirmations, workspace announcements from pet://sessions, orb + top-bar button + Mod+Shift+J + assistant settings. Upstream: #844.

Scope

Affected user and ideal state

Affected user: (a) desktop users on macOS / Windows / Linux who want to dictate prompts or listen to replies; (b) server/web-mode users including phone browsers; (c) existing users who never enable speech; (d) the upstream maintainer (xintaofei) reviewing the PRs against issue #844. Today none of them has any speech path: rg -i "SpeechRecognition|speechSynthesis|getUserMedia" src returns nothing on main (2774a7e).

Row	Statement	Reason
IS-1	Dictation into the chat composer works on every desktop OS and in web mode; final text lands at the caret as literal text and is never auto-sent	the core ask; auto-send on a misheard phrase is destructive
GAP-1	No dictation exists; the browser engine alone cannot work in WebView2 / WebKitGTK	needs a cloud path plus desktop mic enablement
IS-2	When an engine or the mic is unavailable, the UI says why and what to do (configure cloud, allow mic, use https) instead of failing silently	PR #732 fails silently off its author's machine
GAP-2	No capability detection or messaging exists	users would see a dead button
IS-3	Any finished agent reply can be played / stopped, speaking prose only (no code, tool output, raw markdown syntax)	code read aloud is noise
GAP-3	No read-aloud action or speakable-text extraction exists	core ask
IS-4	Optional auto-read of the ACTIVE tab's reply when its turn ends cleanly; off by default	hands-free use without background tabs talking over each other
GAP-4	No turn-end speech hook exists	core ask
IS-5	Mic and playback never linger: playback stops on send, on typing, on tab switch, on a new playback; mic tracks stopped and object URLs revoked on every exit path; dictation and read-aloud create no AudioContext, and voice mode's single AudioContext is closed on exit	issue #837 shows the cost of leaked audio devices
GAP-5	n/a today; must hold for every new path	regression guard
IS-6	The cloud API key lives only in the OS keyring (desktop) or the 0600 token store (server); never in localStorage, SQLite, JS, or logs	repo convention: src-tauri/src/commands/config_sync/credentials.rs
GAP-6	No speech credential storage exists	security
IS-7	Users who do not opt in see no new composer or message chrome and no behavior change	notification-sound-prefs.ts precedent: master switch starts OFF
GAP-7	Every new surface must be gated	regression guard
IS-8	Every string exists in all 10 locales; buttons are keyboard reachable with aria-labels and tooltips	src/i18n/messages.test.ts enforces key parity
GAP-8	New keys needed	upstream acceptance
IS-9	Upstream-acceptable diff: desktop + server + codeg-mcp builds compile, clippy clean, lint/tests/build green, no unrelated file changes	maintainer ack on #844
GAP-9	n/a; enforced by final wave	upstream acceptance
IS-10	Voice mode is a hands-free loop with ONE workspace assistant, independent of the open tab or folder: speak -> sent to the assistant when you stop (held and sent next if it is still answering) -> reply spoken sentence by sentence while it streams -> mic reopens; switching tabs/folders never interrupts it; exit by button, shortcut, Esc or "exit voice mode", and after exit no microphone track stays live	"ironman's jarvis"; "binding the conversation into a session feels half hearted"
GAP-10	No end-of-speech detection, no streaming speech, no loop	new PR4
IS-11	Talking over the assistant (or Esc / "stop") cuts playback without cancelling its turn; "cancel" cancels the assistant's turn	interruption is what makes it feel conversational
GAP-11	No barge-in path	new PR4
IS-12	While voice mode is on, the user hears short announcements for any session in any folder (finished, needs approval, failed), never over a reply	ambient awareness across agents
GAP-12	Only OS notifications exist (notifyDesktop); pet://sessions is emitted on desktop only	new PR4
IS-13	The assistant acts on another session (send a message, stop a turn, start a session, approve/deny a permission) only after a confirmation card that codeg builds from real state (agent, title, exact action), answerable by click or by saying "confirm"/"cancel"; permission answers pick only allow_once/reject_once; session control and permission answers are separate switches, both off by default	safety: the LLM never acts on other sessions unconfirmed
GAP-13	Other sessions' permissions are answerable only by click in their tab or the pet panel	new PR3/PR4
IS-14	A visible orb shows starting / listening / thinking / speaking / confirming and reacts to the voice level; voice mode is reachable by a top-bar button and Mod+Shift+J and configurable in Settings -> Speech (assistant agent, control switches)	presence and discoverability
GAP-14	No voice-mode UI	new PR4
IS-15	The assistant is one persistent conversation per install, running the agent the user picks, spawned and owned by the backend (never by a tab), resumed across restarts, never reaped while voice mode is on; its transcript is an ordinary conversation the user can open; "start fresh" replaces it	"feels half hearted" when bound to a tab
GAP-15	No workspace-level assistant session exists; every ACP session belongs to a tab, the task engine or a delegation	new PR3
IS-16	The assistant knows the workspace: it lists every running session (title, agent, folder, status, what it waits on), reads a session's recent turns (existing get_session_info), and brings a session to the front	cross-session awareness
GAP-16	codeg-mcp tools only see the calling session (get_session_info needs an id; no listing, no control)	new PR3
IS-17	Assistant tools exist only on the assistant's own connection: the companion flag is injected only for owner label assistant, and the listener re-checks that the token belongs to the assistant connection	least privilege
GAP-17	New tool surface must not leak to ordinary sessions	new PR3

Must have

• Engines: browser (Web Speech SpeechRecognition / speechSynthesis) and cloud (OpenAI-compatible /audio/transcriptions and /audio/speech), plus auto which picks browser when usable, otherwise cloud when configured, otherwise reports unavailable.
• Voice mode (PR4) reuses the PR1/PR2 engines, player and prefs and talks only to the backend-owned assistant session (PR3); the assistant reaches other sessions only through codeg-mcp assistant tools that call existing ConnectionManager APIs (send_prompt_linked_with_message_id, cancel, respond_permission, spawn_agent) and the existing question-card machinery for confirmations.
• Cloud calls executed by the Rust backend (_core fns shared by Tauri commands and Axum handlers), reqwest with system proxy env honored (reqwest default).
• Per-device preferences in localStorage (engine, language, browser voice, rate, enable switches, auto-read); cloud endpoint/models/voice in app_metadata; cloud key in keyring named secret.
• Desktop mic enablement: macOS Info.plist usage strings + audio-input entitlement; Linux WebKitGTK media stream enabled with audio-only permission grant on app windows.
• All new strings in all 10 locale files in the same commit that introduces them.

Must NOT have (guardrails, anti-slop, scope boundaries)

• NO hardcoded endpoints, API keys, models or locale fallbacks other than the documented defaults (https://api.openai.com/v1, whisper-1, tts-1, alloy, UI locale).
• NO AudioContext / Web Audio outside src/lib/voice-mode/audio-frontend.ts; that file owns exactly one AudioContext, created on voice-mode entry and close()d on exit (the [Bug] 提示音效的 AudioContext 从不释放：coreaudiod 常驻 ~6% CPU 并全程持有 PreventUserIdleSystemSleep #837 failure mode). Dictation and read-aloud use CSS only.
• NO auto-send from the dictation button; NO voice-mode binding to the active tab or MessageInput (dictation is the per-composer path; voice mode speaks only with the assistant).
• NO changes to src/lib/notification-sound.ts ([Bug] 提示音效的 AudioContext 从不释放：coreaudiod 常驻 ~6% CPU 并全程持有 PreventUserIdleSystemSleep #837 is a separate fix), to src-tauri/src/browser/** (built-in browser keeps its own permission policy), to tauri.conf.json fields other than bundle.macOS.entitlements, or to .github/workflows/**.
• NO dictation in the to-do task composer (src/components/tasks/task-message-composer.tsx) or welcome hero; dictation lives in the chat composer (MessageInput) only. Voice mode is workspace-level and has no composer button.
• NO reading of tool calls, code blocks, images, tables, HTML, or reasoning blocks.
• NO native/offline STT or TTS crates; NO new npm or cargo dependencies (everything needed is already present: unified, remark-parse, remark-gfm, reqwest with multipart, base64, webkit2gtk, keyring, axum). This rules out tauri-plugin-global-shortcut and any wake-word model.
• NO speaking while a turn streams outside voice mode (read-aloud and auto-read speak finished replies only).
• NO wake word / always-on listening; the mic is open only while voice mode is on or dictation is running.
• NO LLM-interpreted voice commands; commands are a fixed, localized phrase list matched against the whole utterance.
• NO allow_always / reject_always selected by any assistant or voice path; NO assistant action on another session without the codeg-built confirmation; NO session control or permission answering unless the user enabled the matching switch (both default off).
• NO assistant companion tools on any connection not spawned by the backend with owner label assistant; NO frontend-initiated assistant spawn; NO second live assistant connection.
• NO answering the assistant's own ask_user_question or any non-confirmation card by voice; NO polling for workspace status (use pet://sessions).

Verification strategy

Zero human intervention - all verification is agent-executed.

• Test decision: tests-after, colocated with each implementation todo. Frontend: vitest + @testing-library/react (pnpm test). Rust: #[cfg(test)] modules with #[tokio::test], a mock OpenAI-compatible server bound on 127.0.0.1:0 using axum (already a dependency), and a process-local secret store swapped in under cfg(test) exactly like config_sync/credentials.rs mod store.
• Scratch QA mock (NOT committed): /tmp/speech-mock/server.ts run with bun, listening on 127.0.0.1:18731, serving POST /v1/audio/transcriptions -> {"text":"hello from mock"} (401 when the bearer is not test-key) and POST /v1/audio/speech -> a fixed 1-second 8 kHz silent WAV with content-type: audio/wav.
• Web-mode QA: pnpm build then cargo run --no-default-features --bin codeg-server with CODEG_STATIC_DIR=../out CODEG_PORT=3480 CODEG_TOKEN=qa, driven by the browser skill in Chromium launched with --use-fake-ui-for-media-stream --use-fake-device-for-media-stream --use-file-for-fake-audio-capture=/tmp/speech-mock/hello.wav.
• Linux desktop QA: pnpm tauri dev on this host (Linux x64), screenshot evidence via the visual-qa skill.
• macOS and Windows desktop cannot run on this host: the PR descriptions must say so explicitly and ask the maintainer for a smoke test; this is the only unverified surface.
• Evidence: .omo/evidence/speech-support/task-<N>-speech-support.<ext>.

Execution strategy

Parallel execution waves

PR1 "speech to prompt" (branch feat/speech-input, todos 1-6):

• Wave A: 1, 2, 3
• Wave B: 4, 6
• Wave C: 5

PR2 "read aloud" (branch feat/speech-read-aloud, based on PR1's branch until PR1 merges; todos 7-11):

• Wave D: 7, 8
• Wave E: 9
• Wave F: 10, 11

PR3 "workspace assistant" (branch feat/workspace-assistant, stacked on PR2 until it merges; todos 14-16):

• Wave G: 14 (may run in parallel with Wave J's 12, 13; they touch disjoint files)
• Wave H: 15
• Wave I: 16

PR4 "voice mode" (branch feat/voice-mode, stacked on PR3; todos 12, 13, 17-20):

• Wave J: 12, 13
• Wave K: 17
• Wave L: 18, 19, 20

Dependency matrix

Todo	Depends on	Blocks	Can parallelize with
1	-	4, 5, 6, 9	2, 3
2	-	4, 6, 7	1, 3
3	-	F3	1, 2
4	1, 2	5, 12	6
5	4	F-wave	-
6	1, 2	10	4
7	2	9	8
8	-	9	7
9	1, 7, 8	10, 11	-
10	6, 9	20	11
11	9	17	10
12	1, 3, 4	17	13, 14, 15
13	8, 9	17	12, 14, 15
14	-	15, 17, 20	12, 13
15	14	16	12, 13
16	15	18	17
17	11, 12, 13, 14	18, 19, 20	16
18	16, 17	F-wave	19, 20
19	17	F-wave	18, 20
20	10, 14, 17	F-wave	18, 19

Todos

Implementation + Test = ONE todo. Never separate.

• 1. Speech preferences store and capability detection
  Recommended task executor category: deep-low
  What to do:

  • Create src/lib/speech-prefs.ts cloned in shape from src/lib/notification-sound-prefs.ts (lines 1-232): key settings:speech:v1, event codeg:speech-prefs-changed, memoized snapshot, storage listener, useSyncExternalStore hook useSpeechPrefs(), getSpeechPrefs(), saveSpeechPrefs(), parseSpeechPrefs(raw) validating every field independently, resetSpeechPrefsCacheForTests().
  • Shape (PR1 ships only input; todo 9 adds output with the same parse discipline):
    SpeechPrefs = { input: { enabled: boolean /*false*/, engine: "auto"|"browser"|"cloud" /*"auto"*/, language: string /*"" = follow UI locale*/ } }.
  • Create src/lib/speech-capabilities.ts exporting pure functions taking an injectable env object (so tests need no globals):
    • detectSpeechCapabilities(env) -> { browserStt: boolean, mediaCapture: boolean, secureContext: boolean }. browserStt = constructor SpeechRecognition ?? webkitSpeechRecognition exists AND NOT (desktop runtime AND os is windows or linux) - WebView2 and WebKitGTK expose no working recognizer. mediaCapture = navigator.mediaDevices?.getUserMedia exists AND window.isSecureContext.
    • resolveInputEngine(pref, caps, cloudConfigured) -> { engine: "browser"|"cloud" } | { engine: null, reason: "no-mic"|"insecure-context"|"no-engine"|"cloud-not-configured" } following: explicit browser -> browser if browserStt else reason no-engine; explicit cloud -> cloud if mediaCapture && cloudConfigured, else insecure-context / no-mic / cloud-not-configured; auto -> browser if usable, else cloud if usable, else the most actionable reason (insecure-context before cloud-not-configured before no-engine).
    • resolveSpeechLanguage(pref, uiLocale) -> BCP-47 tag: pref when non-empty else map the next-intl locale (en->en-US, zh-CN, zh-TW, ja->ja-JP, ko->ko-KR, es->es-ES, de->de-DE, fr->fr-FR, pt->pt-BR, ar->ar-SA).
  • Desktop/OS detection: reuse isDesktop() (defined src/lib/transport/index.ts:64, re-exported by src/lib/platform.ts:13) and detectPlatform() from src/hooks/use-platform.ts:7 (returns "macos"|"windows"|"linux"|"unknown"); pass both into env, do not add a second OS sniffer.
  • Tests: src/lib/speech-prefs.test.ts (defaults, per-field fallback, unknown engine -> default, cross-window invalidation via storage event), src/lib/speech-capabilities.test.ts (full decision table of resolveInputEngine, WebView2/WebKitGTK exclusion, insecure context, locale mapping).
    Must NOT do: store anything secret; import React in speech-capabilities.ts.
    Closes: GAP-2, GAP-7
    Parallelization: Wave A | Blocked by: - | Blocks: 4, 5, 6, 9
    References: src/lib/notification-sound-prefs.ts:1-232; src/lib/transport/index.ts:64 (isDesktop); src/lib/platform.ts:13,22; src/hooks/use-platform.ts:5-32; src/i18n/messages/ locale list.
    Acceptance criteria: pnpm vitest run src/lib/speech-prefs.test.ts src/lib/speech-capabilities.test.ts passes; pnpm lint src/lib/speech-prefs.ts src/lib/speech-capabilities.ts clean.
    QA scenarios: happy - vitest run prints all cases passing; failure - temporarily flip the WebView2 exclusion in a scratch copy and confirm the matching test fails (mutation check, revert). Evidence .omo/evidence/speech-support/task-1-speech-support.txt.
    Commit: Y | feat(speech): add speech preferences and capability detection

• 2. Backend cloud speech settings, key storage and transcription
  Recommended task executor category: deep-low
  What to do:

  • New module src-tauri/src/commands/speech.rs (register pub mod speech; in src-tauri/src/commands/mod.rs).
  • Settings struct SpeechCloudSettings { base_url: String /*"https://api.openai.com/v1"*/, stt_model: String /*"whisper-1"*/, tts_model: String /*"tts-1"*/, tts_voice: String /*"alloy"*/ } (serde camelCase to match the frontend convention in src/lib/types.ts), persisted as JSON in app_metadata under key speech_cloud_settings via crate::db::service::app_metadata_service::{get_value, upsert_value} (pattern: config_sync/webdav_sync.rs:275-434). Missing/invalid JSON -> defaults.
  • View SpeechCloudSettingsView { settings, api_key_set: bool }. Update params { settings: SpeechCloudSettings, api_key: Option<String> }: None = leave key unchanged, Some("") = delete, Some(v) = store. Secret name constant SPEECH_CLOUD_API_KEY = "speech-cloud-api-key"; storage through a private mod store that calls crate::keyring_store::{get_secret,set_secret,delete_secret} and is swapped for a process-local map under #[cfg(test)], copying src-tauri/src/commands/config_sync/credentials.rs:57-100 verbatim in structure. A read error must NOT be treated as "absent" on save (same rule as credentials::read).
  • Validate base_url: trimmed, http/https scheme only, trailing / stripped -> AppCommandError::invalid_input otherwise.
  • speech_transcribe_core(conn, audio_base64: String, mime_type: String, language: Option<String>) -> Result<String, AppCommandError>: decode base64 (base64 crate); reject empty or > 25 MiB with invalid_input; key missing -> configuration_missing; POST multipart to {base_url}/audio/transcriptions with parts file (filename speech.<ext> where ext maps audio/webm->webm, audio/ogg->ogg, audio/mp4->m4a, audio/wav->wav, else webm), model=stt_model, response_format=json, and language = primary subtag of the BCP-47 tag when provided; bearer auth; 60 s timeout. 401/403 -> authentication_failed; other non-2xx -> network with status and at most 500 chars of body as detail; transport error -> network. Return trimmed text field. Never log the key or audio.
  • HTTP client: a module-local OnceLock<reqwest::Client> built with a codeg/<version> user agent (reqwest reads HTTP(S)_PROXY env that network/proxy.rs maintains).
  • Tauri commands (#[cfg(feature = "tauri-runtime")] + #[cfg_attr(feature = "tauri-runtime", tauri::command)], taking State<'_, AppDatabase>): speech_get_settings, speech_update_settings, speech_transcribe. Register in the generate_handler! list in src-tauri/src/lib.rs (next to config_sync::config_sync_get_settings, ~line 1977).
  • Axum: new src-tauri/src/web/handlers/speech.rs (register in web/handlers/mod.rs), routes /speech_get_settings, /speech_update_settings, /speech_transcribe in src-tauri/src/web/router.rs beside the config_sync_* routes (~line 607); /speech_transcribe gets .layer(DefaultBodyLimit::max(36 * 1024 * 1024)) (25 MiB base64 + envelope), following the background_set precedent at router.rs ~1672-1682.
  • Frontend: types SpeechCloudSettings, SpeechCloudSettingsView in src/lib/types.ts; speechGetSettings, speechUpdateSettings, speechTranscribe in src/lib/api.ts using getTransport().call(...) like getSystemRenderingSettings (api.ts:1830).
  • Tests in speech.rs #[cfg(test)]: settings round-trip + defaults on bad JSON (use existing db test helpers, db::test_helpers); key tri-state (None/empty/value) against the test store; unreadable store on save leaves key untouched; transcription against an axum mock on 127.0.0.1:0 asserting multipart fields, bearer header, language subtag, 401 -> authentication_failed, 500 -> network, empty audio -> invalid_input, invalid base_url -> invalid_input.
    Must NOT do: expose the key in any response (only api_key_set); add dependencies; touch keyring_store.rs.
    Closes: GAP-1 (cloud path), GAP-6
    Parallelization: Wave A | Blocked by: - | Blocks: 4, 6, 7
    References: src-tauri/src/commands/config_sync/credentials.rs:1-100; src-tauri/src/keyring_store.rs:254-314; src-tauri/src/commands/config_sync/webdav_sync.rs:43,275-434; src-tauri/src/db/service/app_metadata_service.rs:9-41; src-tauri/src/app_error.rs:186-240; src-tauri/src/web/handlers/config_sync.rs:38-70; src-tauri/src/web/router.rs:530-570,607-644,1672-1682; src-tauri/src/lib.rs:1476,1977; src/lib/api.ts:1830-1837.
    Acceptance criteria (run in src-tauri/): cargo test --features test-utils speech:: passes; cargo test --no-default-features --bin codeg-server --lib speech passes; cargo clippy --all-targets --features test-utils -- -D warnings and cargo clippy --no-default-features --bin codeg-server --lib -- -D warnings clean; cargo check --no-default-features --bin codeg-mcp clean.
    QA scenarios: happy - start the scratch mock, curl -s -X POST localhost:3480/api/speech_update_settings (auth CODEG_TOKEN) with base_url http://127.0.0.1:18731/v1 and key test-key, then speech_transcribe with a base64 WAV returns "hello from mock"; speech_get_settings shows apiKeySet:true and no key; failure - set key wrong -> error code for authentication failure; rg -n "test-key" ~/.local/share/codeg*/ -g '*.db' finds nothing (key not in SQLite). Evidence .omo/evidence/speech-support/task-2-speech-support.txt.
    Commit: Y | feat(speech): add OpenAI-compatible cloud transcription backend

• 3. Desktop microphone enablement for macOS and Linux
  Recommended task executor category: deep-low
  What to do:

  • macOS: create src-tauri/Info.plist (Tauri 2 merges it into the bundle automatically) containing only NSMicrophoneUsageDescription = "Codeg uses the microphone to turn your speech into chat prompts." and NSSpeechRecognitionUsageDescription = "Codeg uses speech recognition to turn your speech into chat prompts.". Create src-tauri/Entitlements.plist with com.apple.security.device.audio-input = true, and add "macOS": { "entitlements": "./Entitlements.plist" } under bundle in src-tauri/tauri.conf.json (release builds are signed in .github/workflows/release.yml:334-424, i.e. hardened runtime, where audio input requires this entitlement). wry 0.55.1 already grants WKWebView media-capture requests (wkwebview/class/wry_web_view_ui_delegate.rs requestMediaCapturePermissionForOrigin -> Grant), so no Rust change on macOS.
  • Linux: in src-tauri/src/commands/windows.rs post_window_setup (line 284) add a #[cfg(all(feature = "tauri-runtime", target_os = "linux"))] call to a new fn enable_linux_audio_capture(window) that uses window.with_webview(|wv| ...) to get the webkit2gtk::WebView, then: settings.set_enable_media_stream(true) and set_enable_mediasource(true) via webkit2gtk::SettingsExt, and connect_permission_request that, for a webkit2gtk::UserMediaPermissionRequest where is_for_audio_device() is true and is_for_video_device() is false, calls allow() and returns true; every other request returns false (default handling). wry 0.55.1 enables neither (webkitgtk/mod.rs:414-450 only sets webgl/webaudio/clipboard/page-cache), which is why getUserMedia fails today. Confirm the method names against the pinned webkit2gtk =2.0.2 (feature v2_40) docs before writing; src-tauri/src/browser/shim/linux.rs:61 shows the import style.
  • Both the main window (src-tauri/src/lib.rs:1265) and remote-workspace windows (src-tauri/src/commands/remote_workspace.rs ~line 245) already call post_window_setup, so the hook there covers both; the settings and other auxiliary windows built in commands/windows.rs also pass through it, which is harmless (audio-only grant). Do NOT apply to src-tauri/src/browser/** webviews.
  • Windows: no change (WebView2 shows its own mic prompt; wry webview2/mod.rs:500-507 only auto-allows clipboard). Document this in the PR description.
    Must NOT do: grant video/camera; change security.csp; change any other tauri.conf.json field.
    Closes: GAP-1 (desktop mic)
    Parallelization: Wave A | Blocked by: - | Blocks: F3
    References: src-tauri/src/commands/windows.rs:284-286; src-tauri/src/lib.rs:1245-1266; src-tauri/Cargo.toml:26-40,247 (webkit2gtk gated by tauri-runtime); src-tauri/src/browser/shim/linux.rs:61; src-tauri/tauri.conf.json:19-45.
    Acceptance criteria (in src-tauri/): cargo check and cargo clippy --all-targets --features test-utils -- -D warnings clean on Linux; cargo check --no-default-features --bin codeg-server clean; plutil -lint equivalent via python3 -c "import plistlib,sys;[plistlib.load(open(p,'rb')) for p in sys.argv[1:]]" src-tauri/Info.plist src-tauri/Entitlements.plist exits 0.
    QA scenarios: happy - pnpm tauri dev, open devtools console of the main window (devtools feature is on), run await navigator.mediaDevices.getUserMedia({audio:true}).then(s=>{s.getTracks().forEach(t=>t.stop());return "ok"}) -> "ok" (requires a PulseAudio/PipeWire source; create one with pactl load-module module-null-sink sink_name=qa if none); failure - getUserMedia({video:true}) still rejects. Evidence .omo/evidence/speech-support/task-3-speech-support.png + console transcript.
    Commit: Y | feat(desktop): allow microphone capture in app windows

• 4. Speech input hook with browser and cloud engines
  Recommended task executor category: deep-low
  What to do:

  • Create src/components/chat/composer/use-speech-input.ts exporting useSpeechInput({ onFinalText, onError }) returning { status: "idle"|"listening"|"transcribing"|"unavailable", interimText: string, unavailableReason, start(), stop(), cancel(), toggle() }.
  • Engine resolution each start(): resolveInputEngine(prefs.input.engine, detectSpeechCapabilities(...), view.apiKeySet) from todo 1, with speechGetSettings() fetched once on mount and refetched on start() when last result said not configured; language from resolveSpeechLanguage(prefs.input.language, useLocale()).
  • Browser engine: continuous = true, interimResults = true, lang; each final result -> onFinalText(transcript); interim -> interimText; onerror not-allowed/service-not-allowed -> onError("mic-denied"), network/language-not-supported -> onError("engine-failed"); onend while still listening (browser auto-stop) -> status idle.
  • Cloud engine: getUserMedia({ audio: true }) -> MediaRecorder with the first supported of audio/webm;codecs=opus, audio/ogg;codecs=opus, audio/mp4; collect chunks; auto-stop at 120 s; on stop() -> status transcribing, blob -> base64 (FileReader data URL, strip prefix) -> speechTranscribe(b64, mimeType, language) -> onFinalText(text) when non-empty; on error map AppCommandError codes (authentication_failed -> "cloud-auth", configuration_missing -> "cloud-not-configured", else "engine-failed"). cancel() discards chunks and skips the request.
  • Release on every exit path (stop, cancel, error, unmount, 120 s cap): track.stop() on all tracks, recognition.abort(), drop recorder references. No AudioContext.
  • Tests use-speech-input.test.ts with fake SpeechRecognition / MediaRecorder / getUserMedia classes injected on globalThis and vi.mock("@/lib/api"): browser final + interim flow; mic denied; cloud stop -> transcribe called with correct mime/language -> onFinalText; cancel sends nothing and stops tracks; unmount mid-recording stops tracks; auth error mapping. Drive the fakes by calling their handlers directly (no timers); for the 120 s cap use vi.useFakeTimers() and advanceTimersByTime (time is the behavior under test).
    Must NOT do: insert text into the editor itself (that is todo 5); use AudioContext; add a Whisper-in-browser fallback.
    Closes: GAP-1, GAP-2, GAP-5
    Parallelization: Wave B | Blocked by: 1, 2 | Blocks: 5
    References: todo 1 and 2 outputs; src/components/chat/composer/use-composer-shortcuts.ts (hook style); src/lib/app-error.ts (toErrorMessage, error code access).
    Acceptance criteria: pnpm vitest run src/components/chat/composer/use-speech-input.test.ts passes; pnpm lint src/components/chat/composer clean.
    QA scenarios: happy/failure are the vitest cases above; evidence is the vitest output. Evidence .omo/evidence/speech-support/task-4-speech-support.txt.
    Commit: Y | feat(chat): add speech input hook with browser and cloud engines

• 5. Composer microphone button, text insertion and shortcut
  Recommended task executor category: visual-engineering
  What to do:

  • Create src/components/chat/composer/composer-speech-button.tsx: ghost icon button h-8 w-8 (same size as send), lucide Mic idle, Square while listening with a CSS animate-pulse ring, Loader2 animate-spin while transcribing, MicOff + disabled when unavailable; tooltip/aria-label from i18n (idle, listening "stop", transcribing, unavailable reason); a small floating bubble above the button showing interimText while listening (max 2 lines, truncated), with "Esc to cancel" hint.
  • Wire into src/components/chat/message-input.tsx: render only when useSpeechPrefs().input.enabled; place it immediately before {actionButtons} inside the existing <div className="shrink-0"> container at ~line 2040 by rendering <div className="flex items-center gap-1">{speechButton}{actionButtons}</div> INSIDE that shrink-0 div (keep the existing wrapper and its classes untouched; commit 09eeac5 asserts composer wrapper classes in message-input.test.tsx).
  • onFinalText: editor.chain().focus().insertContent(textToInlineContent(prefix + text)).run() where prefix is " " when the character before the caret is non-whitespace and the doc is non-empty; then setComposerEmpty(isComposerEmpty(editor)). Use textToInlineContent from src/components/chat/composer/plain-text-content.ts (literal text; see the comment at message-input.tsx:1415-1419). Never call handleSend.
  • Esc while listening/transcribing: cancel() and swallow the key only in that state (add to the composer keydown path without affecting IME: check isImeCompositionKey first, as the file already does).
  • Shortcut: add "toggle_voice_input" to ShortcutActionId, SHORTCUT_DEFINITIONS, DEFAULT_SHORTCUTS ("mod+shift+m", verified unused on 2774a7e) in src/lib/keyboard-shortcuts.ts. Handle it in message-input.tsx with a window keydown listener registered only while isActive && prefs.input.enabled, reading the binding from useShortcutSettings() (src/hooks/use-shortcut-settings.ts) and matching with matchShortcutEvent, the same way src/components/layout/workspace-chrome-controller.tsx:63-86 does (note: use-composer-shortcuts.ts is the skill-insertion hook, not a key handler - do not put it there). Add title/description i18n under the same parent object as toggle_search (en.json ~line 408) in all 10 locales.
  • i18n: new keys under Folder.chat.messageInput: speechStart, speechStop, speechTranscribing, speechCancelHint, speechMicDenied, speechUnavailableNoEngine, speechUnavailableInsecure, speechUnavailableNoMic, speechUnavailableCloud, speechCloudAuthFailed, speechFailed - all 10 locales. Error toasts via sonner toast.error like the attachment toasts.
  • Tests: composer-speech-button.test.tsx (states, aria-labels, disabled when unavailable) and additions to message-input.test.tsx: hidden when prefs disabled; visible when enabled; a mocked useSpeechInput delivering final text inserts literal <b>x</b> as text (not HTML) and does not call onSend; existing wrapper-class tests still pass.
    Must NOT do: change actionButtons markup; add speech to task-message-composer.tsx; alter send/queue logic.
    Closes: GAP-1, GAP-2, GAP-7, GAP-8
    Parallelization: Wave C | Blocked by: 4 | Blocks: F-wave
    References: src/components/chat/message-input.tsx:124,214,364-380,1415-1470,1533-1583,1935-2040,2218; src/components/chat/composer/plain-text-content.ts:25; src/components/layout/workspace-chrome-controller.tsx:63-86; src/hooks/use-shortcut-settings.ts:21; src/lib/keyboard-shortcuts.ts:1-170; src/lib/ime-composition.ts; src/i18n/messages/*.json.
    Acceptance criteria: pnpm test fully green (includes src/i18n/messages.test.ts parity); pnpm lint . clean; pnpm build exits 0.
    QA scenarios: happy - web-mode QA setup (see Verification strategy), enable Speech input with engine cloud pointing at the mock, click mic, wait 2 s, click again -> composer contains "hello from mock", message NOT sent; screenshot at 1280x800 and 390x844. Failure - set wrong key -> toast shows the cloud-auth message and composer unchanged; open over plain http://<LAN-IP> -> button disabled with the insecure-context tooltip. Evidence .omo/evidence/speech-support/task-5-speech-support-{desktop,mobile,auth,insecure}.png.
    Commit: Y | feat(chat): add voice dictation button to the composer

• 6. Settings Speech page with voice input and cloud sections
  Recommended task executor category: visual-engineering
  What to do:

  • Nav: add "speech" to SettingsNavItem.labelKey union and an entry { href: "/settings/speech", labelKey: "speech", icon: AudioLines } after the general item in src/components/settings/settings-shell.tsx; add SettingsShell.nav.speech in all 10 locales.
  • Page: src/app/settings/speech/page.tsx rendering <SpeechSettings /> (mirror src/app/settings/general/page.tsx).
  • Component src/components/settings/speech-settings.tsx using SettingsSection, SettingCard, SettingRow, Switch, Select, Input exactly as general-settings.tsx does. Sections for PR1:
    • "Voice input": enable switch; engine select (Auto / Browser / Cloud) with a live status line from resolveInputEngine (e.g. "Using: browser" or the unavailable reason); language select ("Follow app language" + the 10 mapped tags + free-text custom field).
    • "Cloud speech service": base URL input, API key password input showing "Saved" placeholder when apiKeySet, a "Remove key" button (sends apiKey: ""), STT model input; save button calling speechUpdateSettings with apiKey: undefined when the key field is untouched; success/failed toasts. Explanatory copy: key stored in the system keychain; audio is sent to this endpoint only when the Cloud engine is used.
  • i18n under a new top-level SpeechSettings namespace, all 10 locales.
  • Tests speech-settings.test.tsx: toggling saves prefs; untouched key field sends apiKey undefined; "Remove key" sends ""; status line shows the unavailable reason with mocked capabilities.
    Must NOT do: put speech controls into general-settings.tsx; show or log the key value.
    Closes: GAP-2, GAP-6, GAP-8
    Parallelization: Wave B | Blocked by: 1, 2 | Blocks: 10
    References: src/components/settings/settings-shell.tsx:40-140; src/app/settings/general/page.tsx; src/components/settings/general-settings.tsx:1-60,360-522; src/components/settings/notification-sound-settings.tsx (prefs-backed section pattern); src/components/settings/config-sync-settings.tsx (secret field pattern).
    Acceptance criteria: pnpm vitest run src/components/settings/speech-settings.test.tsx src/i18n/messages.test.ts passes; pnpm lint . clean; pnpm build exits 0 and out/settings/speech/index.html exists.
    QA scenarios: happy - web-mode, open /settings/speech, save base URL + key, reload -> key shows "Saved", engine status reads "Using: cloud"; failure - base URL ftp://x -> error toast, nothing saved. Screenshots desktop + mobile. Evidence .omo/evidence/speech-support/task-6-speech-support-*.png.
    Commit: Y | feat(settings): add Speech settings page

• 7. Backend speech synthesis command
  Recommended task executor category: deep-low
  What to do:

  • In src-tauri/src/commands/speech.rs add speech_synthesize_core(conn, text: String, speed: f32) -> Result<SpeechAudio, AppCommandError> where SpeechAudio { audio_base64: String, mime_type: String }: reject empty or > 4096 chars (invalid_input); POST JSON { model: tts_model, voice: tts_voice, input, response_format: "mp3", speed: clamp(speed, 0.25, 4.0) } to {base_url}/audio/speech; same auth/error mapping/timeout as transcription; mime_type from the response content-type (fallback audio/mpeg).
  • Tauri command speech_synthesize, Axum handler + route /speech_synthesize, speechSynthesize in src/lib/api.ts, SpeechAudio type in src/lib/types.ts.
  • Settings UI gets TTS model and voice inputs in todo 10 (fields already exist on the struct from todo 2).
  • Tests: mock server asserts JSON body (model, voice, speed clamped), returns WAV bytes -> base64 round-trips; 401 -> authentication_failed; 4097 chars -> invalid_input.
    Must NOT do: stream audio; cache audio on disk.
    Closes: GAP-3 (cloud path)
    Parallelization: Wave D | Blocked by: 2 | Blocks: 9
    References: todo 2 code; src-tauri/src/web/router.rs speech routes.
    Acceptance criteria: same cargo test/clippy/check commands as todo 2 all clean.
    QA scenarios: happy - curl /api/speech_synthesize against the scratch mock returns mimeType: "audio/wav" and non-empty base64; failure - wrong key -> auth error. Evidence .omo/evidence/speech-support/task-7-speech-support.txt.
    Commit: Y | feat(speech): add cloud speech synthesis backend

• 8. Speakable text extraction from agent markdown
  Recommended task executor category: deep-low
  What to do:

  • Create src/lib/speakable-text.ts exporting toSpeakableText(markdown: string, labels: { codeOmitted: string; tableOmitted: string }): string and chunkSpeakableText(text: string, maxLen: number): string[].
  • Parse with unified().use(remarkParse).use(remarkGfm) (both already dependencies). Walk the mdast: code -> labels.codeOmitted sentence; table -> labels.tableOmitted; inlineCode -> its value; link -> its text children only; image, html, footnoteDefinition, thematicBreak -> dropped; heading/paragraph/listItem/blockquote -> their text followed by a sentence break; delete/emphasis/strong -> text only. Collapse whitespace; drop bare URLs (https?://\S+) from text nodes.
  • chunkSpeakableText: split on sentence boundaries (. ! ? 。 ！ ？ and newlines), pack sentences greedily up to maxLen, hard-split any single sentence longer than maxLen at the last space before it (or at maxLen for CJK).
  • Tests speakable-text.test.ts: fenced code replaced; inline code kept; link text kept, URL dropped; table replaced; headings/lists read as sentences; CJK chunking; a 10 000-char input yields chunks all <= maxLen whose join equals the normalized text.
    Must NOT do: read reasoning/tool blocks (callers pass only assistant text parts).
    Closes: GAP-3
    Parallelization: Wave D | Blocked by: - | Blocks: 9
    References: package.json (unified, remark-parse, remark-gfm); src/components/ai-elements/remark-*.ts (existing remark usage style).
    Acceptance criteria: pnpm vitest run src/lib/speakable-text.test.ts passes; lint clean.
    QA scenarios: vitest output (happy + edge cases above). Evidence .omo/evidence/speech-support/task-8-speech-support.txt.
    Commit: Y | feat(speech): extract speakable text from agent replies

• 9. Read-aloud preferences and speech player
  Recommended task executor category: deep-low
  What to do:

  • Extend src/lib/speech-prefs.ts with output: { enabled: boolean /*false*/, engine: "auto"|"browser"|"cloud" /*"auto"*/, browserVoiceUri: string /*"" = default voice for language*/, rate: number /*1, clamp 0.5..2*/, autoRead: boolean /*false*/ }, parsed field by field; extend tests.
  • Extend src/lib/speech-capabilities.ts: browserTts = speechSynthesis exists AND at least one voice (voices may arrive via voiceschanged; expose waitForVoices(synth, timeoutMs=1500) that resolves on the event or the timeout); resolveOutputEngine(pref, caps, cloudConfigured) mirroring resolveInputEngine.
  • Create src/lib/speech-player.ts, a module singleton: speak(id: string, markdown: string, labels), stopSpeech(), getSpeechPlayerState() -> { playingId: string | null, status: "idle"|"loading"|"playing" }, subscribeSpeechPlayer(fn), useSpeechPlayer() (useSyncExternalStore).
    • speak always calls stopSpeech() first. Text = chunkSpeakableText(toSpeakableText(markdown, labels), engine === "cloud" ? 4000 : 220).
    • Browser: queue one SpeechSynthesisUtterance per chunk with lang = resolved language, rate, chosen voice; state playing until the last onend; onerror (other than interrupted/canceled) -> stop + error callback.
    • Cloud: for each chunk sequentially speechSynthesize(chunk, rate) -> Blob -> URL.createObjectURL -> one reusable HTMLAudioElement; ended -> revoke URL, next chunk; prefetch the next chunk while the current plays.
    • stopSpeech(): speechSynthesis.cancel(), pause audio, removeAttribute("src") + load(), revoke every outstanding object URL, abort pending fetches via a generation counter, state idle. No AudioContext.
  • Tests speech-player.test.ts with fake speechSynthesis and fake Audio/URL.createObjectURL: browser chunk queueing + state; cloud sequential play + prefetch + URL revocation; speak while playing stops the previous; stopSpeech mid-fetch discards the late response (generation check).
    Must NOT do: autoplay anything on import; hold a MediaStream or AudioContext.
    Closes: GAP-3, GAP-5, GAP-7
    Parallelization: Wave E | Blocked by: 1, 7, 8 | Blocks: 10, 11
    References: todo 1, 7, 8 outputs; src/lib/notification-sound-prefs.ts snapshot pattern.
    Acceptance criteria: pnpm vitest run src/lib/speech-prefs.test.ts src/lib/speech-capabilities.test.ts src/lib/speech-player.test.ts passes; lint clean.
    QA scenarios: vitest output. Evidence .omo/evidence/speech-support/task-9-speech-support.txt.
    Commit: Y | feat(speech): add read-aloud player

• 10. Read-aloud button on agent replies and settings section
  Recommended task executor category: visual-engineering
  What to do:

  • In src/components/message/turn-stats.tsx, when useSpeechPrefs().output.enabled && hasCopy, render a tooltip icon button right after the Copy button (same iconButtonClass, h-3.5 w-3.5 icon): Volume2 idle, Loader2 animate-spin loading, Square while this turn is the playingId; click toggles speak(turnKey, copyText, labels) / stopSpeech(). turnKey = a stable id already available to TurnStats (use its turn id prop; if none exists, add an optional speechId prop and pass forkPoint?.id ?? group key from HistoricalMessageGroup in message-list-view.tsx ~line 969-990).
  • Only render for completed responses (isResponseComplete), never on the live streaming turn.
  • i18n keys under Folder.chat.messageList: readAloud, stopReading, readAloudLoading, readAloudFailed, speechCodeOmitted, speechTableOmitted - all 10 locales.
  • Settings: add a "Read aloud" section to speech-settings.tsx: enable switch, engine select with status line from resolveOutputEngine, browser voice select (populated after waitForVoices, filtered to the resolved language first), rate slider 0.5-2, auto-read switch (disabled unless enabled); add TTS model + voice inputs to the Cloud section. i18n in all 10 locales.
  • Tests: turn-stats test for hidden-when-disabled, shown-when-enabled, click calls speak with copyText, second click calls stopSpeech; settings test for the new controls.
    Must NOT do: add the button to user messages or system messages; change existing TurnStats buttons.
    Closes: GAP-3, GAP-7, GAP-8
    Parallelization: Wave F | Blocked by: 6, 9 | Blocks: 20
    References: src/components/message/turn-stats.tsx:24-215; src/components/message/message-list-view.tsx:896-990 (copyText={extractTextFromParts(group.parts)} at ~983); todo 6 component.
    Acceptance criteria: pnpm test fully green; pnpm lint . clean; pnpm build exits 0.
    QA scenarios: happy - web-mode with Chromium, enable Read aloud with engine cloud -> mock, click the button on a reply -> document.querySelector("audio") shows !paused via page eval and the button shows Stop; click Stop -> paused and src removed; engine browser in Chromium -> speechSynthesis.speaking === true via page eval. Failure - wrong key -> readAloudFailed toast, button returns to idle. Screenshots desktop + mobile. Evidence .omo/evidence/speech-support/task-10-speech-support-*.{png,txt}.
    Commit: Y | feat(chat): add read-aloud action to agent replies

[n0tlu5]

Implementation plan (2/2)

Full plan, part 2 (todos 11-20, final verification, commit strategy, success criteria)

• 11. Auto-read at turn end and playback stop triggers
  Recommended task executor category: deep-low
  What to do:

  • In src/contexts/acp-connections-context.tsx case "turn_complete" (~line 5171): capture the assistant text BEFORE any later state clears it, from turnConn.liveMessage.content blocks with type === "text" and no parentToolUseId, joined with "\n" (the same turnConn read at ~5199). Then, inside the existing if (nc) success branch where notifyDesktop("turn_complete", ...) is called (~5260, i.e. not quiet, stop_reason === "end_turn", no terminal failure), call maybeAutoRead(contextKey, text): speaks only when getSpeechPrefs().output.enabled && .autoRead, contextKey === useTabStore.getState().activeTabId (the ACP contextKey is the tab id: conversation-detail-panel.tsx:624), document.visibilityState === "visible", and text is non-empty. Put maybeAutoRead in src/lib/speech-player.ts taking the active id and visibility as arguments so it is unit-testable.
  • Stop triggers: (a) handleSend in message-input.tsx calls stopSpeech() at its top; (b) handleComposerChange (message-input.tsx ~2218) calls stopSpeech() when the doc changes and the player is not idle; (c) in AcpConnectionsProvider, a useEffect subscribing to useTabStore activeTabId changes calls stopSpeech(); (d) starting dictation (useSpeechInput.start) calls stopSpeech() so the mic never records the speaker.
  • Tests: speech-player.test.ts cases for maybeAutoRead (disabled, background tab, hidden document, empty text, happy); message-input.test.tsx send and typing call stopSpeech; acp-connections-context.test.tsx a clean turn_complete on the active tab with auto-read enabled calls speak once, a cancelled stop_reason and a non-active tab do not.
    Must NOT do: speak during streaming; speak background tabs; alter notification or sound behavior on that path.
    Closes: GAP-4, GAP-5
    Parallelization: Wave F | Blocked by: 9 | Blocks: 17
    References: src/contexts/acp-connections-context.tsx:219-267 (LiveContentBlock), 5171-5267; src/stores/tab-store.ts:1200 (useTabStore); src/components/conversations/conversation-detail-panel.tsx:624; src/components/chat/message-input.tsx:1533,2218; src/contexts/acp-connections-context.test.tsx (mock harness).
    Acceptance criteria: pnpm test fully green; pnpm lint . clean.
    QA scenarios: happy - web-mode with a real connected agent (any installed ACP agent on this host) or the existing test harness, auto-read on, send "reply with one short sentence" -> audio starts at turn end (page eval shows playing), typing in the composer stops it; failure - switch to another tab before the turn ends -> nothing plays. Evidence .omo/evidence/speech-support/task-11-speech-support.txt.
    Commit: Y | feat(chat): auto-read agent replies at turn end

• 12. Voice-mode audio front end with end-of-speech detection
  Recommended task executor category: deep-low
  What to do:

  • Extend src/lib/speech-prefs.ts with voiceMode: { enabled: boolean /*false*/, endSilenceMs: number /*900, clamp 500..2500*/, bargeIn: boolean /*true*/, announce: "off"|"background"|"all" /*"all"*/, voiceApprovals: boolean /*false*/ }, parsed field by field; extend speech-prefs.test.ts.
  • Create src/lib/voice-mode/vad.ts: a pure, clock-injected state machine createVad({ endSilenceMs, now }) with push(rmsDb: number): VadEvent[] where VadEvent = "speech-candidate" | "candidate-dropped" | "speech-start" | "speech-end". Adaptive noise floor (slow EMA of frames below threshold, initial -60 dBFS); speech threshold = floor + 12 dB; speech-candidate on the first frame above threshold (the host begins recording here so the first syllable is not clipped), candidate-dropped if it falls below before the start window elapses, speech-start after 150 ms continuously above; speech-end after endSilenceMs continuously below following a start; setPlaybackActive(bool) raises the threshold by a further 10 dB and the start window to 300 ms while the assistant is speaking (barge-in resistance to its own echo).
  • Create src/lib/voice-mode/audio-frontend.ts, the ONLY file allowed to touch Web Audio: startAudioFrontend({ onLevel, onVadEvent, vad }) -> { stream, stop(): Promise<void> }. Steps: getUserMedia({ audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true } }); one AudioContext; an AudioWorkletNode whose processor source is an inline string loaded via a Blob URL (revoked right after addModule) that posts the RMS of every 20 ms window; main thread converts to dBFS, feeds vad.push, reports onLevel(0..1). stop() disconnects nodes, stops all tracks, await ctx.close(). AudioWorklet (not rAF polling) so detection keeps working while the window is hidden or throttled.
  • Create src/lib/voice-mode/utterance-recorder.ts: given the front-end stream and a resolved input engine, beginUtterance() / endUtterance(): Promise<string> / abort(). Cloud: a MediaRecorder on the shared stream (same mime selection as todo 4) started at beginUtterance, stopped at endUtterance, sent through speechTranscribe; hard cap 60 s. Browser: a SpeechRecognition (continuous, interimResults) running for the utterance; endUtterance resolves with the accumulated finals plus the last interim, then abort()s it. Reuse todo 4's engine helpers by extracting them from use-speech-input.ts into src/lib/speech-engines.ts (pure move, todo 4 tests keep passing) instead of duplicating.
  • Tests: vad.test.ts drives synthetic dB sequences with an injected clock (start/end timing, floor adaptation, playback threshold); audio-frontend.test.ts with fake AudioContext/AudioWorkletNode/getUserMedia asserting stop() stops every track, closes the context exactly once and revokes the Blob URL; utterance-recorder.test.ts for both engines incl. abort() sending nothing.
    Must NOT do: create a second AudioContext; keep the mic open outside voice mode; add a VAD library.
    Closes: GAP-10, GAP-11, GAP-5
    Parallelization: Wave J | Blocked by: 1, 3, 4 | Blocks: 17
    References: todo 1 and 4 outputs; src/lib/notification-sound.ts (the leak pattern to avoid, read only; see issue [Bug] 提示音效的 AudioContext 从不释放：coreaudiod 常驻 ~6% CPU 并全程持有 PreventUserIdleSystemSleep #837).
    Acceptance criteria: pnpm vitest run src/lib/voice-mode src/lib/speech-prefs.test.ts src/components/chat/composer/use-speech-input.test.ts passes; lint clean.
    QA scenarios: vitest output; failure case - a test where stop() is called twice asserts one close() and no throw. Evidence .omo/evidence/speech-support/task-12-speech-support.txt.
    Commit: Y | feat(voice): add voice-mode audio front end and end-of-speech detection

• 13. Streaming sentence speech for live replies
  Recommended task executor category: deep-low
  What to do:

  • Create src/lib/voice-mode/sentence-stream.ts: createSentenceStream(labels) with push(delta: string): string[] and flush(): string[]. Buffers streamed markdown; emits a segment when a sentence boundary (. ! ? 。 ！ ？ followed by whitespace/end, or a blank line) is reached OUTSIDE a fenced block; tracks triple-backtick / ~~~ fences so a whole fence produces exactly one labels.codeOmitted segment when it closes; a buffer over 400 chars without a boundary is emitted at the last space. Each emitted segment passes through toSpeakableText (todo 8) and empty results are dropped.
  • Extend src/lib/speech-player.ts with a streaming queue: beginSpeechStream(id), enqueueSpeech(id, text) (appends; ignored when id is not the current stream), endSpeechStream(id), and onSpeechDrained(cb) fired when a stream has ended and its queue is empty. speak() / stopSpeech() end any stream. Cloud mode keeps its one-ahead prefetch; browser mode queues one utterance per segment.
  • Tests: sentence-stream.test.ts (deltas split mid-word and mid-fence, CJK punctuation, one code-omitted per fence, flush of the tail, links/inline code via toSpeakableText); speech-player.test.ts additions (enqueue order, stale-id enqueue ignored, drained fires once after end, stopSpeech mid-stream clears the queue and no drained callback fires).
    Must NOT do: speak thinking or subagent (parent_tool_use_id) text; change the existing speak() contract.
    Closes: GAP-10
    Parallelization: Wave J | Blocked by: 8, 9 | Blocks: 17
    References: todo 8, 9 outputs; src/lib/types.ts:2448 (content_delta shape).
    Acceptance criteria: pnpm vitest run src/lib/voice-mode/sentence-stream.test.ts src/lib/speech-player.test.ts passes; lint clean.
    QA scenarios: vitest output. Evidence .omo/evidence/speech-support/task-13-speech-support.txt.
    Commit: Y | feat(voice): speak replies sentence by sentence while they stream

• 14. Backend-owned workspace assistant session
  Recommended task executor category: deep-low
  What to do:

  • Create src-tauri/src/commands/assistant.rs (register in src-tauri/src/commands/mod.rs). Constants: ASSISTANT_OWNER_LABEL = "assistant" (pub, re-used by todos 15-16), metadata keys assistant.agent_type, assistant.conversation_id, assistant.allow_session_control ("1"/"0", default "0"), assistant.allow_permission_answers ("1"/"0", default "0"). Read/write them with app_metadata_service::get_value / upsert_value.
  • assistant_get_settings_core(db) -> AssistantSettings { agent_type: Option<AgentType>, allow_session_control: bool, allow_permission_answers: bool } and assistant_set_settings_core(db, AssistantSettings). Changing agent_type clears assistant.conversation_id, because a conversation is bound to one agent.
  • assistant_ensure_core(db, manager, emitter, data_dir) -> AssistantSession { connection_id, conversation_id, folder_id, agent_type }:
    1. No agent_type set -> AppCommandError "assistant agent not configured".
    2. Refuse agents that cannot receive the codeg-mcp companion: make agent_delivers_wire_mcp (acp/connection.rs:4863) pub(crate) and reject agents where it is false (Pi) or where the agent has no MCP support.
    3. If assistant.conversation_id is set and the conversation still exists (not deleted), use it and its folder. Otherwise call create_chat_conversation_core(conn, data_dir, agent_type, Some(title), None) (commands/conversations.rs:2464), where title = "Codeg Assistant". Store the new id.
    4. Reuse a live connection when find_connection_by_conversation_id (manager.rs:3718) returns one whose owner_window_label == ASSISTANT_OWNER_LABEL. Otherwise spawn exactly like work_task/engine.rs:1227-1262:
       • build_session_runtime_env(db, agent_type, resume_id, data_dir), where resume_id = the conversation's external_id;
       • verify_agent_installed;
       • manager.spawn_agent(agent_type, Some(folder_path), resume_id, env, ASSISTANT_OWNER_LABEL.to_string(), emitter.clone(), None, None).
         On a resume error, fall back to a fresh session in the same conversation, as the engine does.
    5. Serialize concurrent calls with a tokio::sync::Mutex<()> held in a OnceLock, so two windows cannot spawn two assistants.
  • assistant_reset_core: disconnect the live assistant connection (if any) and clear assistant.conversation_id. The old conversation stays in history.
  • First-turn system context: when ensure creates a NEW conversation, prepend one text block to the first prompt the frontend sends (not a separate turn). Store it as a const ASSISTANT_PRIMER in this file, English, with this content: you are Codeg's workspace assistant; you are spoken to by voice; answer in 1-3 short spoken sentences without markdown or code; use the codeg-mcp tools list_sessions, get_session_info, focus_session, send_to_session, cancel_session, answer_permission and start_session to act on the user's other sessions; never claim an action happened unless the tool result says so. Expose it via AssistantSession.primer: Option<String>, which is Some only for a freshly created conversation.
  • Injection: in acp/connection.rs next to tasks_enabled (~6135), add assistant_enabled = owner_window_label == crate::commands::assistant::ASSISTANT_OWNER_LABEL. Add assistant: bool to CompanionFeatureFlags (5119) and emit the "assistant" token in companion_features_arg (5147). Like tasks, it injects the companion even when every settings-gated group is off, and it also forces sessions on, so get_session_info is present.
  • Surfaces:
    • #[cfg_attr(feature = "tauri-runtime", tauri::command)] wrappers, registered in lib.rs generate_handler! (1476);
    • web handlers in a new src-tauri/src/web/handlers/assistant.rs (register in handlers/mod.rs);
    • routes /assistant_ensure, /assistant_reset, /assistant_get_settings, /assistant_set_settings in web/router.rs;
    • frontend wrappers assistantEnsure, assistantReset, assistantGetSettings, assistantSetSettings in src/lib/api.ts, plus AssistantSession / AssistantSettings in src/lib/types.ts.
  • Tests (#[cfg(test)] in assistant.rs, in-memory DB as other _core tests do):
    • settings round-trip and defaults;
    • agent change clears the conversation id;
    • ensure without an agent errors;
    • ensure with a Pi agent errors;
    • companion_features_arg with assistant yields a string containing assistant and sessions (extend the existing companion_features_arg_inject_skip_decision test at connection.rs:26635);
    • CompanionFeatures::parse(Some("assistant")) sets the flag. This todo adds the assistant field and parse arm to CompanionFeatures (acp/delegation/companion.rs:193-256); todo 15 adds the tool gating.
      Must NOT do: spawn from the frontend (the owner label is what gates the tools); create more than one live assistant connection; store the primer in the DB; add dependencies.
      Closes: GAP-15, GAP-17
      Parallelization: Wave G | Blocked by: - | Blocks: 15, 17, 20
      References: src-tauri/src/work_task/engine.rs:1215-1290; src-tauri/src/commands/conversations.rs:2464; src-tauri/src/db/service/app_metadata_service.rs:9,35; src-tauri/src/acp/connection.rs:4863,5119,5147,6131-6182; src-tauri/src/acp/delegation/companion.rs:193-256; src-tauri/src/acp/manager.rs:680,3718.
      Acceptance criteria: cargo test --features test-utils assistant passes; cargo clippy --all-targets --features test-utils -- -D warnings clean; cargo check --no-default-features --bin codeg-server and --bin codeg-mcp pass.
      QA scenarios: happy - web mode, curl -H 'Authorization: Bearer qa' -X POST :3480/api/assistant_set_settings with agent codex, then /api/assistant_ensure twice. Both return the same connection_id, and ps -o args for the spawned codeg-mcp shows --features containing assistant. Failure - assistant_ensure with no agent set returns the configured error; with a Pi agent it returns the unsupported-agent error. Evidence .omo/evidence/speech-support/task-14-speech-support.txt.
      Commit: Y | feat(assistant): add backend-owned workspace assistant session

• 15. Assistant tool group: wire and read-only tools
  Recommended task executor category: deep-low
  What to do:

  • acp/delegation/companion.rs:
    • in allows_tool: "list_sessions" | "focus_session" | "send_to_session" | "cancel_session" | "answer_permission" | "start_session" => self.assistant;
    • dispatch arms beside the browser_* arms (~800-830), each building a BrokerRequest.
  • acp/delegation/tool_schema.json: add the six tools in the existing entry format.
    • list_sessions {};
    • focus_session {session_id: integer};
    • send_to_session {session_id: integer, text: string (1..=8000 chars)};
    • cancel_session {session_id: integer};
    • answer_permission {session_id: integer, decision: enum["approve","deny"]};
    • start_session {folder_id: integer, agent_type: string, task: string}.
    • session_id is codeg's conversation PK, the same meaning as in get_session_info (transport.rs:199).
    • Descriptions state that mutating tools show the user a confirmation and return declined if the user says no.
  • acp/delegation/transport.rs BrokerMessage: add AssistantListSessions, AssistantFocusSession, AssistantSendToSession, AssistantCancelSession, AssistantAnswerPermission and AssistantStartSession, each carrying token + args. Response types: AssistantSessionList, and AssistantActionResult { outcome: "ok"|"declined"|"busy"|"not_running"|"not_found"|"disabled"|"no_pending_permission"|"unsupported", message: String }.
  • New src-tauri/src/acp/assistant_tools.rs (register in acp/mod.rs), modelled on acp/session_info.rs:133:
    • #[async_trait] pub trait AssistantToolAccess: Send + Sync with is_assistant_connection(conn_id) -> bool, list_sessions(exclude_conn_id) -> AssistantSessionList, focus_session(session_id) -> AssistantActionResult, and the four mutating methods implemented in todo 16;
    • AssistantSessionList { sessions: Vec<AssistantSessionEntry{session_id, title, agent_type, folder_id, folder_name, status: "idle"|"working"|"needs_approval"|"needs_answer"|"error", pending_action: Option<String>}>, folders: Vec<{folder_id, name, path}> }.
  • src-tauri/src/commands/assistant_tools.rs: DbAssistantToolAccess { manager, db, emitter }.
    • list_sessions: add ConnectionManager::list_linked_sessions() beside list_active_sessions (manager.rs:2868). It returns every live connection with both conversation_id and folder_id plus status, pending permission summary and pending_question.is_some(). It excludes the assistant itself and connections owned by work_task. Titles and folder names are filled from the DB as pet_list_active_sessions_core does (commands/pet.rs:216). folders comes from folder_service::list_folders. pending_action = the permission tool call's command ?? title, truncated to 160 chars.
    • focus_session: resolve folder + agent from the conversation row and emit_event(emitter, "workspace://focus-conversation", {folderId, conversationId, agent}). That is the payload PetFocusBridge (src/components/workspace/deep-link-bootstrap.tsx:222-236) already consumes through the transport, so it works in web mode too. On desktop, also bring the main window forward the way commands/windows.rs:1803's caller does, under #[cfg(feature = "tauri-runtime")].
  • acp/delegation/listener.rs:
    • add assistant: Arc<dyn AssistantToolAccess> to DelegationListener::new (168) and wire every constructor call: lib.rs:1024 (desktop, DbAssistantToolAccess), bin/codeg_server.rs:366 (server, same impl), and the test constructions at acp/delegation/service.rs:490 (its shared stub at :264 implements the new trait too) and listener.rs:1748-1867, each test stub returning empty/not_found;
    • every Assistant* arm: tokens.lookup(&req.token) -> None gives not_found; !assistant.is_assistant_connection(&entry.parent_connection_id) gives unsupported. This is defence in depth: a hand-edited --features assistant on a normal session still cannot act.
  • Tests:
    • allows_tool for all six names;
    • tool_schema.json parses and lists each once (extend the existing schema test);
    • listener: bad token gives not_found; non-assistant token gives unsupported; assistant token lists sessions via the stub;
    • list_linked_sessions excludes the assistant and work_task connections (manager test style at manager.rs:6970).
      Must NOT do: expose these tools on any non-assistant spawn; include transcript text in list_sessions (that is get_session_info's job); add dependencies.
      Closes: GAP-16, GAP-17
      Parallelization: Wave H | Blocked by: 14 | Blocks: 16
      References: src-tauri/src/acp/delegation/{companion.rs:193-256,transport.rs:197,339,listener.rs:96,168,423-614,831,tool_schema.json}; src-tauri/src/acp/session_info.rs:133; src-tauri/src/commands/session_info.rs:92; src-tauri/src/commands/pet.rs:216; src-tauri/src/acp/manager.rs:2849,2868.
      Acceptance criteria: cargo test --features test-utils passes; both clippy commands in AGENTS.md clean; cargo clippy --no-default-features --bin codeg-mcp -- -D warnings clean.
      QA scenarios: happy - web mode with the assistant from todo 14 on codex plus one normal codex tab. Ask the assistant by text "what sessions are running?" and it answers naming the tab. Then ask "open it" and the tab is focused (screenshot). Failure - pipe a raw tools/call list_sessions frame to a codeg-mcp started with --features assistant but a normal session's token (printf into the binary's stdin); the response contains unsupported. Evidence .omo/evidence/speech-support/task-15-speech-support.{txt,png}.
      Commit: Y | feat(assistant): add assistant tool group with list and focus

• 16. Assistant mutating tools behind a codeg confirmation
  Recommended task executor category: deep-high
  What to do:

  • Confirmation gate (commands/assistant_tools.rs, used by all four mutating methods):
    • Build ONE QuestionSpec in the backend from REAL state, never from LLM text alone:
      • header "Confirm";
      • question = "{agent} · {title}: {action}", where action is, per tool:
        • send: the first 300 chars of text;
        • cancel: "stop the current turn";
        • answer_permission: "approve|deny: " + the pending permission's command ?? title;
        • start: agent + folder name + first 300 chars of task;
      • two options, labels from a Rust confirm_labels(AppLocale) match in the style of commands/windows.rs:494 (e.g. en "Confirm"/"Cancel"), locale from load_system_language_settings(conn) (windows.rs:601).
    • Register it on the ASSISTANT's own connection via the existing SessionQuestionAccess::register_question (acp/question.rs:162). Await the answer with the same peer-close racing as the Ask arm (listener.rs:485-540), so a cancelled tool call tears the card down.
    • Only the confirm label proceeds. Cancel, declined, or cancelled -> declined.
    • register_question returning None (a question is already pending) -> busy.
  • send_to_session:
    • allow_session_control off -> disabled with no card;
    • target has no live connection -> not_running;
    • target turn in flight -> busy with no card;
    • else confirm, then send_prompt_linked_with_message_id(db, conn, [Text{text}], folder_id, conversation_id, None, None) (manager.rs:1116);
    • an in-flight rejection at send time -> busy.
  • cancel_session: allow_session_control gate, confirm, then manager.cancel(db, conn_id) (manager.rs:1814).
  • answer_permission:
    • allow_permission_answers off -> disabled;
    • no pending permission -> no_pending_permission;
    • map approve -> the option with kind == AllowOnce, deny -> RejectOnce; the kind is absent -> unsupported (the card stays for a click);
    • confirm, re-check that the SAME request_id is still pending (else no_pending_permission), then respond_permission (manager.rs:1872).
  • start_session:
    • allow_session_control gate;
    • validate the folder exists (folder_service::get_folder_by_id) and the agent is installed;
    • confirm;
    • conversation_service::create(conn, folder_id, agent_type, Some(title_from_task), None), then spawn with owner label "main" via spawn_agent + build_session_runtime_env, send task as the first prompt linked to the new conversation, and emit the focus event so the tab opens and attaches to the live connection through the normal find-connection-for-conversation path.
  • Auto-allow: extend codeg_ask_auto_allow_option (acp/connection.rs:~7026) with a sibling is_codeg_assistant_tool_name in acp/question.rs (same normalization as is_codeg_ask_tool_name, :699, names from todo 15), applied ONLY when the requesting connection's owner label is ASSISTANT_OWNER_LABEL. It selects allow_once only. Rationale: the tool's own codeg confirmation is the consent, so the agent's generic permission card would ask twice.
  • Tests (in commands/assistant_tools.rs, with a fake SessionQuestionAccess that answers confirm/cancel/None):
    • each gate (disabled, not_running, busy, no_pending_permission, unsupported);
    • confirm -> action taken;
    • cancel -> no action;
    • answer_permission never selects AllowAlways/RejectAlways even when offered first;
    • stale request_id after confirmation -> no respond;
    • is_codeg_assistant_tool_name positive/negative cases;
    • auto-allow not applied for a non-assistant owner.
      Must NOT do: act on any tool without the confirmation card; select allow_always/reject_always; queue prompts server-side; let the question text come solely from LLM arguments (agent, title and permission action are read from state).
      Closes: GAP-13, GAP-16
      Parallelization: Wave I | Blocked by: 15 | Blocks: 18
      References: src-tauri/src/acp/question.rs:44-58,151-176,308,699; src-tauri/src/acp/delegation/listener.rs:485-540; src-tauri/src/acp/connection.rs:7010-7060; src-tauri/src/acp/manager.rs:1116,1192-1206,1814,1872,3376; src-tauri/src/db/service/conversation_service.rs:14; src-tauri/src/commands/windows.rs:494,601.
      Acceptance criteria: cargo test --features test-utils passes; all AGENTS.md clippy commands clean.
      QA scenarios: happy - web mode, assistant on codex, a second codex tab idle. Type to the assistant "tell the other session to run ls". The confirmation card appears in the assistant conversation with the real tab title. Click Confirm and the other tab receives the prompt (screenshot of both). Failure - repeat and click Cancel: the other tab receives nothing and the assistant reports it was declined. With allow_session_control off the tool returns disabled and no card shows. A pending permission offering only allow_always returns unsupported and the card stays. Evidence .omo/evidence/speech-support/task-16-speech-support.{txt,png}.
      Commit: Y | feat(assistant): add confirmed session control tools

• 17. Voice-mode host bound to the assistant session
  Recommended task executor category: deep-high
  What to do:

  • src/lib/voice-mode/voice-mode-store.ts (module store + useSyncExternalStore, same pattern as the player): { phase: "off"|"starting"|"listening"|"capturing"|"transcribing"|"waiting"|"speaking"|"confirming", level: number, assistant: AssistantSession | null, lastSpoken: string | null, queued: string | null, error: string | null }, plus requestVoiceMode(on: boolean).
  • src/lib/voice-mode/assistant-session.ts: the ONLY voice-mode file allowed to call sendPrompt.
    • useAssistantSession() exposes ensure(): assistantEnsure() (todo 14), then attachDelegationChild({ connectionId, parentConnectionId: connectionId, parentToolUseId: "voice-assistant", agentType, hydrate: true }), exactly as src/components/tasks/task-transcript-dialog.tsx:250-272 attaches a non-tab connection (contextKey = connectionId), then registerLiveSurfaceKeys("voice-assistant", new Set([connectionId])) so the idle sweep (acp/idle_sweep.rs, 180 s default) never reaps it while voice mode is on.
    • release(): detach and register an empty set.
    • send(text): sendPrompt(connectionId, blocks, { folderId, conversationId }), where blocks = [primer?, text] (primer only on the first send after an ensure that returned one).
  • src/components/chat/voice-mode/voice-mode-host.tsx, mounted once in src/app/workspace/layout.tsx after <ConversationStatusEventBridge /> (1294), so it lives across tabs and folders. State machine:
    1. Entry: resolve input and output engines (todos 1, 9); if either is unavailable, or assistantGetSettings().agent_type is unset, toast the reason (with a link to Settings -> Speech) and stay off. stopSpeech(), phase = starting, ensure(); failure -> toast + off. Start the audio front end (todo 12), phase = listening.
    2. speech-candidate -> beginUtterance; candidate-dropped -> abort(); speech-start -> capturing; speech-end -> transcribing, endUtterance(). Empty text -> listening. Otherwise text -> the command matcher (todo 18) first, else submit(text).
    3. submit(text): if the assistant connection's status is prompting, keep it in queued (replacing any earlier queued text, and toast "queued") and return to listening. Else send(text), phase = waiting, beginSpeechStream(turnId). On turn_complete, if queued is set, send it next.
    4. useAcpEvent: for envelopes with connection_id === assistant.connectionId only (events reach subscribers because the connection is attached):
       • content_delta without parent_tool_use_id -> sentenceStream.push -> enqueueSpeech; phase = speaking on the first segment;
       • turn_complete -> flush + endSpeechStream; if nothing was spoken, speak voiceTurnEndedEmpty for non-end_turn stop reasons;
       • onSpeechDrained -> listening;
       • keep the joined spoken text as lastSpoken.
    5. Barge-in: while waiting/speaking the VAD stays live with setPlaybackActive(true). When prefs.voiceMode.bargeIn is on and speech-start fires, stopSpeech() (the assistant turn is NOT cancelled; later deltas of that turn are dropped by a turn-id check) and go to capturing.
    6. Exit (button, shortcut, Esc outside inputs, "exit" command, pagehide): stop the VAD front end (closes its AudioContext and tracks), stopSpeech(), release(), phase = off.
  • Debug: window.__codegVoiceDebug = { state, inject(text) }, only when localStorage["codeg:voice-debug"] === "1"; inject runs step 2 with a transcript, for QA without a microphone.
  • Tests (voice-mode-host.test.tsx, mocked ACP context + fake front end):
    • entry without an agent stays off and toasts;
    • an utterance calls sendPrompt with the assistant contextKey and primer only once;
    • a delta stream is spoken sentence by sentence;
    • an utterance while prompting is queued and sent after turn_complete;
    • barge-in stops speech without cancel;
    • events for other connection ids are ignored;
    • exit releases the surface key and stops the front end.
      Must NOT do: bind to the active tab or MessageInput; call sendPrompt for any connection other than the assistant; create an AudioContext here; cancel the assistant turn on barge-in.
      Closes: GAP-10, GAP-11, GAP-15
      Parallelization: Wave K | Blocked by: 11, 12, 13, 14 | Blocks: 18, 19, 20
      References: src/contexts/acp-connections-context.tsx:2973-3048,3205,5745-5800,7116-7150; src/components/tasks/task-transcript-dialog.tsx:250-272; src/app/workspace/layout.tsx:1294; src-tauri/src/acp/idle_sweep.rs:19; todo 11-14 outputs.
      Acceptance criteria: pnpm vitest run src/lib/voice-mode src/components/chat/voice-mode passes; pnpm lint . clean.
      QA scenarios: happy - web mode with the fake-audio Chromium from the verification strategy (the clip says "hello from mock"), assistant on codex. Voice mode on -> the assistant conversation receives the prompt, the reply is spoken (player log in __codegVoiceDebug.state), and the phase returns to listening. Switching tabs and folders mid-reply does not interrupt it. Exit -> navigator.mediaDevices tracks all ended. Failure - no assistant agent configured -> toast, no mic prompt, phase stays off. Evidence .omo/evidence/speech-support/task-17-speech-support.{txt,png}.
      Commit: Y | feat(voice): add workspace voice mode bound to the assistant session

• 18. Fixed voice commands and spoken confirmations
  Recommended task executor category: deep-low
  What to do:

  • src/lib/voice-mode/voice-commands.ts:
    • normalizeUtterance(s): NFKC, lowercase, strip punctuation incl. CJK, collapse spaces, trim;
    • matchVoiceCommand(utterance, phrases): VoiceCommand | null, with VoiceCommand = "stop" | "cancel" | "repeat" | "exit" | "confirm" | "reject";
    • only a WHOLE-utterance match counts;
    • phrases = the current locale's VoiceMode.commands.<id> split on |, plus the English list (always accepted);
    • confirm/reject are considered only while phase === "confirming".
  • Host wiring before submit:
    • stop -> stopSpeech();
    • cancel -> cancel(assistant.connectionId) + stopSpeech();
    • repeat -> speak lastSpoken;
    • exit -> exit.
  • Spoken confirmations: when the assistant connection's pendingQuestion becomes non-null AND it is a codeg confirmation (single question, header equal to the todo 16 header for the UI locale, two options):
    • stopSpeech(), phase = confirming;
    • speak VoiceMode.confirmPrompt with {action} = the question text truncated to 200 chars;
    • confirm -> answerQuestion(connectionId, question_id, { answers: [{ question_id: q.id, labels: [q.options[0].label] }], declined: false });
    • reject -> the same with options[1];
    • anything else -> speak confirmRepeat once, then on the second non-match speak confirmUseScreen and leave the card on screen.
  • Any other pending question on the assistant (the agent's own ask_user_question) is spoken as "{question}" plus "answer on screen", never answered by voice.
  • Spoken confirmations exist regardless of settings, because the tools themselves are gated by todo 14's settings.
  • i18n: VoiceMode.commands.{stop,cancel,repeat,exit,confirm,reject} (pipe-separated) and VoiceMode.{confirmPrompt,confirmRepeat,confirmUseScreen,voiceTurnEndedEmpty,questionOnScreen} in all 10 locales. Phrase lists must not contain { }; ICU args must pass src/i18n/messages.test.ts.
  • Tests:
    • normalization; "stop the server" is NOT a command;
    • confirm picks options[0], reject options[1];
    • two non-matches -> no answerQuestion;
    • the card answered by click while confirming returns to listening;
    • confirming utterances never reach sendPrompt;
    • non-confirmation questions are never answered.
      Must NOT do: fuzzy/LLM matching; answer permission cards directly (only the assistant's confirmation card, whose content codeg built); answer the agent's own ask_user_question by voice.
      Closes: GAP-11, GAP-13
      Parallelization: Wave L | Blocked by: 16, 17 | Blocks: F-wave
      References: todo 16 (confirmation shape), todo 17 host; src/components/chat/ask-question-card.tsx:28-93; src/lib/types.ts:1373-1386; src/contexts/acp-connections-context.tsx:3023.
      Acceptance criteria: pnpm vitest run src/lib/voice-mode src/components/chat/voice-mode src/i18n passes; lint clean.
      QA scenarios: happy - todo 16's send scenario driven by __codegVoiceDebug.inject("tell the other session to run ls"), then inject("confirm"). The other tab receives the prompt. Failure - inject("maybe") twice -> confirmUseScreen spoken, card still pending, nothing sent. inject("stop the server") is sent to the assistant as a prompt, not treated as a command. Evidence .omo/evidence/speech-support/task-18-speech-support.txt.
      Commit: Y | feat(voice): add fixed voice commands and spoken confirmations

• 19. Workspace-wide spoken announcements
  Recommended task executor category: deep-low
  What to do:

  • Server mode: spawn codeg_lib::pet_sessions::pet_sessions_subscriber_task(bus, broadcaster, emitter, manager, db_conn) in src-tauri/src/bin/codeg_server.rs beside pet_state_subscriber_task (482). The fn is not Tauri-gated; desktop already spawns it at lib.rs:937. If a comment near that spawn calls it desktop-only, update it.
  • src/lib/voice-mode/session-watch.ts: subscribe to "pet://sessions" via getTransport().subscribe, and seed with pet_list_active_sessions as src/app/pet/_hooks/usePetSessions.ts does (reuse its fetch/subscribe helpers if exported; otherwise mirror them).
    • Diff consecutive PetSessionsPayloads by connection_id into events:
      • needs_approval (pending newly non-null);
      • failed (status newly error);
      • finished (an entry that was prompting without pending leaves the list).
    • Ignore the assistant's own connection and entries with a parent (delegation children).
  • src/components/chat/voice-mode/voice-announcer.tsx (rendered by the host):
    • while voice mode is on and prefs.voiceMode.announcements !== "off", queue short phrases VoiceMode.announce.{finished,needsApproval,failed} with {agent} (getAgentLabel) and {title} truncated to 60 chars;
    • speak them only when phase === "listening" (never over a reply);
    • collapse repeats per session within 10 s; drop the queue on exit.
  • Tests: the diff function (all three events, delegation children ignored, assistant ignored); announcer speaks only in listening; off setting is silent.
    Must NOT do: poll; use per-tab useAcpEvent mapping for other sessions; speak tool output.
    Closes: GAP-12
    Parallelization: Wave L | Blocked by: 17 | Blocks: F-wave
    References: src-tauri/src/pet_sessions.rs:86; src-tauri/src/lib.rs:918-945; src-tauri/src/bin/codeg_server.rs:470-487; src/app/pet/_hooks/usePetSessions.ts; src/lib/pet/types.ts:148-175; src-tauri/src/models/pet.rs:306.
    Acceptance criteria: pnpm vitest run src/lib/voice-mode passes; cargo check --no-default-features --bin codeg-server and its clippy clean.
    QA scenarios: happy - web mode, voice mode on, a codex tab in another folder runs echo hi via a typed prompt. When it finishes, __codegVoiceDebug.state.spoken contains the finished announcement with that tab's title. Failure - announcements off -> nothing spoken; an announcement arriving during speaking waits until listening. Evidence .omo/evidence/speech-support/task-19-speech-support.txt.
    Commit: Y | feat(voice): announce session status across the workspace

• 20. Voice orb, entry points and assistant settings
  Recommended task executor category: visual-engineering
  What to do:

  • src/components/chat/voice-mode/voice-orb.tsx, rendered by the host while phase !== "off":
    • a fixed, pointer-events-limited orb bottom-center (respect env(safe-area-inset-bottom)), ~56 px, layered radial gradients using theme tokens;
    • per-phase styling: starting/waiting = slow shimmer; listening = calm breathing; capturing scales with level via a CSS custom property; speaking = pulse; confirming = amber ring;
    • prefers-reduced-motion disables animation;
    • a visually-hidden aria-live="polite" phase label;
    • a close button, and an "open transcript" button that opens the assistant conversation as a tab with useTabActions().openTab, the same call PetFocusBridge (src/components/workspace/deep-link-bootstrap.tsx:135-260) makes for a focus request.
  • Entry points:
    • a workspace-chrome button (AudioLines icon) in the top bar, shown only when prefs.voiceMode.enabled;
    • shortcut toggle_voice_mode = "mod+shift+j" (verified unused on 2774a7e) added to ShortcutActionId / SHORTCUT_DEFINITIONS / DEFAULT_SHORTCUTS, handled by the host with a window keydown listener using useShortcutSettings() + matchShortcutEvent, as src/components/layout/workspace-chrome-controller.tsx:63-86 does.
  • Settings, a "Voice assistant" section in speech-settings.tsx:
    • enable switch with a status line (needs input AND output engine AND an assistant agent; say which is missing);
    • an assistant agent select listing installed agents from src/hooks/use-acp-agents.ts, excluding agents without MCP delivery, saved via assistantSetSettings;
    • "Let the assistant send to and stop other sessions" switch (allow_session_control, default off);
    • "Let the assistant answer permission prompts (allow once / deny once only, always confirmed)" switch (allow_permission_answers, default off);
    • end-of-speech pause slider 0.5-2.5 s;
    • barge-in switch with the hint "works best with headphones";
    • announcements select (off / on);
    • a "Start a fresh assistant conversation" button -> assistantReset().
    • All strings in 10 locales.
  • Tests: voice-orb.test.tsx (phase labels, close exits, reduced-motion class, aria-live); settings tests (agent select persists, switches call assistantSetSettings, reset calls assistantReset); the top-bar button is hidden when disabled.
    Must NOT do: add a canvas/WebGL visualizer or a second analyser (level comes from the store); render in the pet window (follow-up); put a voice-mode button in MessageInput (voice mode is workspace-level; dictation stays there).
    Closes: GAP-14, GAP-8
    Parallelization: Wave L | Blocked by: 10, 14, 17 | Blocks: F-wave
    References: todo 6, 10, 14, 17 outputs; src/components/layout/workspace-chrome-controller.tsx:63-86; src/hooks/use-acp-agents.ts; src/lib/pet/api.ts:182; src/app/globals.css.
    Acceptance criteria: pnpm test fully green; pnpm lint . clean; pnpm build exits 0.
    QA scenarios: happy - web mode, screenshots of each phase at 1280x800 and 390x844, dark and light (phases reached with __codegVoiceDebug.inject and todo 16's confirmation scenario). Mod+Shift+J toggles; "open transcript" opens the assistant conversation. Failure - agent unset -> the status line names the missing assistant agent and the top-bar button toast links to settings, with no orb. Evidence .omo/evidence/speech-support/task-20-speech-support-*.png.
    Commit: Y | feat(voice): add voice orb, entry points and assistant settings

Final verification wave

Runs in parallel after ALL todos. ALL must APPROVE. Surface results and wait for the user's explicit okay before declaring complete.

• F1. Plan compliance audit
  Check every todo's Must / Must NOT against the diff (git diff main...feat/voice-mode --stat and per-file); confirm no changes under src-tauri/src/browser/**, .github/**, src/lib/notification-sound.ts, and that tauri.conf.json differs only by bundle.macOS.entitlements; rg -l "AudioContext" src --glob '!*.test.*' lists only src/lib/notification-sound.ts (pre-existing) and src/lib/voice-mode/audio-frontend.ts; rg -n "allow_always|reject_always" src/lib/voice-mode src/components/chat/voice-mode empty; rg -l "sendPrompt|acpSendPrompt" src/lib/voice-mode src/components/chat/voice-mode --glob '!*.test.*' lists only src/lib/voice-mode/assistant-session.ts; rg -n "AllowAlways|RejectAlways" src-tauri/src/commands/assistant_tools.rs src-tauri/src/acp/assistant_tools.rs hits only #[cfg(test)] code and !=/negative matches; rg -n '"assistant"' src-tauri/src/acp/connection.rs shows the flag set only from ASSISTANT_OWNER_LABEL; every BrokerMessage::Assistant* arm in listener.rs calls is_assistant_connection; rg -n "api\\.openai\\.com" src empty (default lives in Rust only); git diff main...feat/voice-mode -- package.json pnpm-lock.yaml src-tauri/Cargo.toml shows no dependency changes.
• F2. Code quality review
  Run the full AGENTS.md matrix: pnpm lint ., pnpm test, pnpm build; in src-tauri/: cargo check, cargo test --features test-utils, cargo clippy --all-targets --features test-utils -- -D warnings, cargo check --no-default-features --bin codeg-server, cargo test --no-default-features --bin codeg-server --lib, cargo clippy --no-default-features --bin codeg-server --lib -- -D warnings, cargo check --no-default-features --bin codeg-mcp, cargo clippy --no-default-features --bin codeg-mcp -- -D warnings. All exit 0.
• F3. Real manual QA
  Web mode (Chromium with fake media flags + scratch mock): dictation cloud + browser engines, read-aloud cloud + browser, auto-read, stop triggers, insecure-context message, assistant text control (list, focus, confirmed send/cancel/start/permission, declined and disabled paths, a normal session's token refused), voice-mode loop across a tab and folder switch, barge-in, commands, spoken confirmation, announcement from another folder, orb phases, desktop and mobile widths. Linux desktop (pnpm tauri dev): dictation and one voice-mode round trip via cloud engine with a virtual PipeWire/Pulse source, read-aloud via cloud; after exiting voice mode pactl list source-outputs shows no codeg capture stream. Record macOS/Windows as unverified in the PR descriptions.
• F4. Ideal-state fidelity
  Walk IS-1..IS-17 against evidence 1:1; any shortfall becomes a new numbered todo.

Commit strategy

• One conventional commit per todo, messages as listed. Execution and commit order: 1-6 (PR1), 7-11 (PR2), 14-16 (PR3), then 12, 13, 17-20 (PR4). No commit touches files outside its todo.
• PR1 feat/speech-input = todos 1-6, title "feat(chat): voice dictation for the composer ([Feature] Speech support: voice input to prompt and read-aloud of agent responses #844)", body links [Feature] Speech support: voice input to prompt and read-aloud of agent responses #844, lists engines, the desktop permission changes, what was verified on Linux/web, and asks for a macOS + Windows smoke test.
• PR2 feat/speech-read-aloud = todos 7-11, stacked on PR1, title "feat(chat): read agent replies aloud ([Feature] Speech support: voice input to prompt and read-aloud of agent responses #844)".
• PR3 feat/workspace-assistant = todos 14-16, stacked on PR2, title "feat(assistant): workspace assistant session with confirmed session control ([Feature] Speech support: voice input to prompt and read-aloud of agent responses #844)". Its body explains the owner-label gate, the listener re-check, the confirmation card, the default-off switches, and that the settings UI lands in PR4 (PR3 is exercisable through the API and by opening the assistant conversation as a tab).
• PR4 feat/voice-mode = todos 12, 13, 17-20, stacked on PR3, title "feat(voice): hands-free voice mode with the workspace assistant ([Feature] Speech support: voice input to prompt and read-aloud of agent responses #844)". Its body explains the single-AudioContext lifecycle (and why it does not reintroduce [Bug] 提示音效的 AudioContext 从不释放：coreaudiod 常驻 ~6% CPU 并全程持有 PreventUserIdleSystemSleep #837) and the headphone caveat for barge-in.
• [Feature] Speech support: voice input to prompt and read-aloud of agent responses #844 currently scopes dictation + read-aloud only. Before PR3 is opened, a comment describing the workspace assistant and voice mode must be posted on [Feature] Speech support: voice input to prompt and read-aloud of agent responses #844 - drafted by the executor, posted only after the user okays the wording.
• Do not push or open PRs until the user approves the F-wave results.

Success criteria

One row per IS row. The plan is complete only when every IS row has a delivering todo and a proving QA scenario; F4 checks the delivered behavior against these rows 1:1, and a shortfall becomes new - [ ] N. rows, never a note.
| IS | Delivering todo(s) | Proving QA scenario | Evidence |
| --- | --- | --- | --- |
| IS-1 | 2, 3, 4, 5 | task 5 happy (text inserted, not sent); task 3 Linux getUserMedia ok; F3 Linux desktop dictation | task-5-speech-support-desktop.png, task-3-speech-support.png |
| IS-2 | 1, 5, 6 | task 5 insecure + auth failure; task 6 status line | task-5-speech-support-insecure.png, task-6-speech-support-.png |
| IS-3 | 7, 8, 9, 10 | task 10 happy (cloud + browser); task 8 code omitted | task-10-speech-support-.txt, task-8-speech-support.txt |
| IS-4 | 11 | task 11 happy + background-tab failure | task-11-speech-support.txt |
| IS-5 | 4, 9, 11 | task 4 unmount/cancel stops tracks; task 9 stop revokes URLs; task 11 typing stops playback; F1 no AudioContext | task-4/9/11 evidence |
| IS-6 | 2, 6 | task 2 key absent from SQLite and responses | task-2-speech-support.txt |
| IS-7 | 1, 5, 9, 10 | task 5 hidden when disabled; task 10 hidden when disabled | vitest output |
| IS-8 | 5, 6, 10 | src/i18n/messages.test.ts parity; aria-label tests | F2 output |
| IS-9 | all | F1 + F2 | F1/F2 output |
| IS-10 | 12, 13, 17 | task 17 happy loop incl. tab + folder switch mid-reply; exit leaves no live track; F3 Linux pactl check | task-17-speech-support.{txt,png} |
| IS-11 | 12, 17, 18 | task 17 barge-in test (speech stopped, cancel not called); task 18 "cancel" command | vitest output, task-18-speech-support.txt |
| IS-12 | 19 | task 19 other-folder finish announced; announcements off silent | task-19-speech-support.txt |
| IS-13 | 16, 18, 20 | task 16 confirm/cancel/disabled/unsupported; task 18 spoken confirm + two non-matches; F1 AllowAlways grep | task-16-speech-support.{txt,png}, task-18-speech-support.txt, F1 output |
| IS-14 | 20 | task 20 phase screenshots desktop/mobile + Mod+Shift+J + agent-unset failure | task-20-speech-support-*.png |
| IS-15 | 14, 17, 20 | task 14 ensure twice returns one connection; task 17 no reap while on; task 20 "open transcript" + reset | task-14-speech-support.txt, task-20 evidence |
| IS-16 | 15 | task 15 list + focus happy | task-15-speech-support.{txt,png} |
| IS-17 | 14, 15 | task 14 --features contains assistant only for the assistant; task 15 normal-token unsupported | task-14/15 evidence, F1 output |

[n0tlu5]

Update: the whole speech stack is now up for review, one PR per step, each stacked on the previous one:

1. feat(speech): speech to prompt (dictation) #853 speech to prompt (dictation), targeting main
2. feat(speech): read agent replies aloud #854 read agent replies aloud
3. feat(assistant): workspace assistant session #855 workspace assistant session: a backend-owned assistant with a confirmed assistant tool group, available only to that session
4. feat(voice): hands-free voice mode for the workspace assistant #856 hands-free voice mode for the assistant

GitHub can't base a PR on a fork branch, so #854 to #856 target main, and each diff includes the PRs below it. Each description lists only its own commits to review, and I'll rebase them as the earlier ones merge. Reviewing #853 and #854 first makes sense: they stand on their own.

Since the plan above, #853 has one fix: on plain-http web deployments, browser dictation is now reported as unavailable, because Chromium exposes SpeechRecognition there but every session fails with not-allowed. Before, the mic button failed as soon as it was clicked. The cloud engine still works over http.

If you'd rather keep #844 to dictation and read-aloud and move the assistant and voice mode (#855, #856) to a separate issue, I'm happy to do that. No new dependencies anywhere in the stack.