From 927509d99ddab38382b8cf17214337fc5da1ac78 Mon Sep 17 00:00:00 2001 From: dozycat Date: Tue, 25 Aug 2026 09:23:22 +0800 Subject: [PATCH 1/5] =?UTF-8?q?docs:=20POCKET=5FLIVE.md=20=E2=80=94=20desi?= =?UTF-8?q?gn=20rationale=20and=20alternatives=20for=20the=20native=20char?= =?UTF-8?q?acter=20widget?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Companion to WIDGET.md: what pocket-character/Pocket Live chose at every layer (shell, VRM crates, QuickJS guest, data-driven plugins, MediaPipe sidecar, CMIO camera extension), the frequency-partitioning principle the choices follow from, the roads not taken, and an honest process count. Co-Authored-By: Claude Fable 5 --- docs/POCKET_LIVE.md | 351 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 351 insertions(+) create mode 100644 docs/POCKET_LIVE.md diff --git a/docs/POCKET_LIVE.md b/docs/POCKET_LIVE.md new file mode 100644 index 00000000..4d592081 --- /dev/null +++ b/docs/POCKET_LIVE.md @@ -0,0 +1,351 @@ +# Pocket Live — the Electron digital human, rebuilt native + +*How the Electron "digital human" was rebuilt on the Pocket runtime family — +every technology choice, the principle they all follow from, and for each +layer, the roads not taken and why. The widget itself measures 1 process / +118 MB / 3.9 % CPU against airi's 8 / 2184 MB / 44.4 %; tracking and the +virtual camera add processes — deliberately, at isolation boundaries — and +§10 accounts for them honestly.* + +This document is the design-rationale companion to +[pocket-character](https://github.com/pocket-stack/pocket-character) (the +repo Pocket Live grew out of) and to [WIDGET.md](WIDGET.md), which +generalizes what it proved. WIDGET.md names the *capability*; this page +explains the *choices* — with alternatives, so the next runtime doesn't have +to re-derive them. + +## 1. The category, and the question + +There is a category of app that is delightful in concept and horrifying in +Activity Monitor: the desktop companion character. A little anime figure +floats in a transparent, always-on-top window — idling, blinking every few +seconds, eyes darting in small saccades, hair swaying with physics. You can +drag her around your screen. Behind her sit two bigger pieces: driving her +expressions from the real webcam (face tracking), and presenting her +composed program as a camera to Zoom (the virtual camera). + +The reference implementation is [airi](https://github.com/moeru-ai/airi), +an open-source Electron app that does all of the above. Pocket Live began +as an experiment with a sharp question behind it: *same character, same +model file, same blink timing, same window geometry — what does it cost on +the Pocket architecture instead?* + +The measured answer (same machine, same ≥60 s steady-state methodology, +full process tree; **parity scope** — the idle widget alone, no face +tracking, no virtual camera, on both sides): + +| | airi (Electron, VRM stage) | Pocket Live widget | +| --- | --- | --- | +| Processes | 8 | 1 | +| RSS | 2184 MB | 118 MB | +| CPU (of one core) | 44.4 % | 3.9 % | + +Better than an order of magnitude, and no single heroic optimization +explains it. It falls out of one architectural principle applied +consistently. + +## 2. Why Electron costs what it costs + +First, a defense of airi: it isn't slow because it's badly written. It's +slow because of what it stands on. Electron means every app ships a +complete Chrome browser; even to draw one little character, you pay for the +whole vehicle: + +- **A process tree.** Main process, GPU process, network service, audio + service, a renderer per window — that's the 8 processes, each with its + own memory baseline. Gigabytes follow. +- **A web rendering path.** three.js maintains the entire 3D scene graph in + JavaScript. Every frame — 60 times a second, rAF-driven — JS computes + spring-bone physics, expression lerps, and skeletal matrices, then hands + the result to WebGL. Per-frame JS allocates; the garbage collector chases + it forever. That's where the CPU goes. +- **Pure waste at the edges.** airi's main process polls the global cursor + at 60 Hz even when nothing consumes it. + +The expensive part was never "draw a character." The character's math is +tiny. The expensive part is the vehicle. + +## 3. The one idea everything else follows from + +Every technology choice in the repo is a corollary of a single rule: + +> **Things that happen 60 times per second run in Rust. Things that happen +> occasionally run in JavaScript. Things that don't change at all are +> data.** + +What the character actually does each frame is a fixed pipeline with zero +decisions in it: + +``` +sample clip → apply blink/expression → solve springs → skin → draw +``` + +Pure math, no branching on intent — so the whole pipeline lives native +(`crates/pocket-character-core` plus the engine crates), compiled, +allocation-free, GC-free. *Decisions* — which motion plays on click, when +to swap expressions, whether to track the mouse — happen a few times per +second at most. Those live in a QuickJS guest: per tick the core pushes a +handful of read-only facts (`blink`, `hovered`, `fps`) through +`character.__dispatch(state, events)`, and the guest queues commands +(`SetTracking`, `PlayClip`, `SetExpression`, `Quit`) that the core applies +at the end of the frame. This is the RUNTIMES.md ⟨Cores, Surfaces, Guest⟩ +shape with exactly one core and one surface. + +The analogy that sticks: **Rust is the cerebellum, JavaScript is the +cortex.** Walking, blinking, heartbeat — continuous control that never +touches conscious thought — versus "that's a friend, wave." Electron's +architecture makes the cortex operate every heartbeat by hand. + +A side effect that became the product: a character's *personality* is just +a guest bundle. Swap the bundle, get a different character; the host +doesn't recompile. The airi-parity personality is deliberately near-empty — +32 lines — because airi's default behavior barely requires decisions. + +## 4. The shell: native Rust, not Electron, not Tauri + +The product is a single binary: a winit window configured transparent, +undecorated, always-on-top, 450×600 (airi's exact stage geometry), rendered +with wgpu, frame-paced at 60 fps — *paced* meaning the loop sleeps until +the next deadline rather than spinning. The window plumbing (`AppConfig` +widget mode, transparent clear, `max_fps` pacing) went upstream in +[#125](https://github.com/pocket-stack/pocketjs/pull/125) and is being +generalized as [WIDGET.md](WIDGET.md)'s `shell`. + +The alternatives, and why they lose *here*: + +| Shell | Principle | Why not | +| --- | --- | --- | +| Electron | bundle Chromium + Node; full web ecosystem | the measured baseline being compared against | +| Tauri | reuse the system WebView, Rust backend | fixes the memory story, not the rendering one — per-frame 3D is still rAF-driven JS inside a WebView, its worst workload | +| Unity / Godot | mature VRM ecosystems (UniVRM, godot-vrm) | a game engine's resting heart rate is antithetical to "idle costs almost nothing"; frameless transparent overlays are second-class there | +| Swift/AppKit + Metal | leanest possible build | platform lock-in, and loses the reusable core/surface/guest pattern the runtime family shares | + +## 5. VRM rendering: engine crates, not three-vrm, not bevy_vrm + +A primer if VRM is new: it's an open humanoid-avatar standard (a glTF +extension) specifying skeleton naming, blend-shape expressions, and — +crucially — the spring-bone physics that make hair and clothes sway. +Everything is documented; you can implement it from the spec. + +That matters because airi's *actual* out-of-the-box character is +**Live2D**, a 2.5D mesh-deformation format ubiquitous in VTubing — and +rendering its `.moc3` files requires linking Live2D's proprietary, +closed-source Cubism Core, which cannot be vendored into an open runtime. +So the parity target is airi's VRM stage, with the identical model +(VRoid's AvatarSample_A), identical idle animation, identical source URLs. +Apples to apples. + +VRM support was built as generic crates and PR'd upstream (#125): +`pocket-vrm` (VRM 0.x parsing, VRMA retargeting, spring-bone verlet solver, +eye look-at) and morph-target/pose machinery in `pocket3d`. Two details +carry most of the performance story: + +- **Blinking is free while it isn't happening.** Facial expressions use + morph targets: the artist pre-sculpts a "closed eyes" mesh and the + runtime interpolates vertex positions by a weight. The naive approach + interpolates and re-uploads every frame; here morph deltas are computed + and uploaded **only when a weight changes**. A blink is a 0.2 s sine + pulse every 1–6 s — over 95 % of frames pay literally zero for the face. +- **Hair physics is a few vector ops.** Spring bones are chains of point + masses solved with verlet integration — velocity represented implicitly + as "current minus last position," a couple of adds per particle per + frame. The math was never expensive; airi pays for running it in + allocating, GC'd JavaScript. Here it's preallocated Rust arrays stepping + allocation-free. + +The alternatives: + +- [@pixiv/three-vrm](https://github.com/pixiv/three-vrm) — the most mature + VRM runtime anywhere, and it's three.js: airi's path. +- [bevy_vrm](https://github.com/unavi-xyz/bevy_vrm) — essentially the only + off-the-shelf Rust option (VRM 0.0/1.0), but it brings all of Bevy's ECS + and render graph with it. The game-engine tax again. +- UniVRM — official, excellent, requires Unity. +- Writing the solver from the public spec was a bounded amount of work in + exchange for a crate any Pocket app can now use. + +## 6. The scripting layer: QuickJS, not V8 + +JS-engine selection has a simple intuition: V8 is fast because of its JIT, +and the JIT costs tens of megabytes of baseline memory plus a heavyweight +runtime. But this guest reads a few properties per tick — **throughput is +irrelevant; footprint is everything**. QuickJS is ~1 MB, JIT-free, and +starts in microseconds: the exact sweet spot for "policy only." + +Why JS rather than Lua or Rhai (both even cheaper to embed)? Because plugin +authors write **TypeScript** — the typed SDK (`plugin-sdk/character.ts`) is +the plugin contract, and Bun bundles it. The type system is part of the +product surface. WASM (wasmtime, Extism) deserves a mention: strongest +sandbox, language-agnostic, and wrong for this shape of problem — every +host↔guest exchange crosses a serialization boundary, exactly the wrong +trade when the pattern is "hand the guest a state object every tick." +QuickJS mounts plain JS objects directly. + +## 7. Content: plugins are data, not code + +The content architecture has three layers: + +``` +runtime host + ├─ character plugin (VRM + VRMA + policy bundle + render framing) + ├─ background plugin (one WGSL pixel function + compositor defaults) + └─ vibe preset (character id + background id + tracking/output) +``` + +A character plugin is a `plugin.json` manifest pointing at a model, +animations, a policy bundle, and framing parameters (FOV, anchor height, +camera distance). A background plugin is a manifest plus one WGSL shader — +literally a pure function from pixel coordinates to color. A vibe is a JSON +file containing only IDs: "this character + this stage + these settings." + +The load-bearing rule: **the host recognizes no specific character.** It +consumes manifest paths. A character whose model has licensing restrictions +lives outside the repo via `.gitignore`, but architecturally it is not a +special case — just another manifest the host has never heard of. + +Both places where plugins get to *execute* anything — the QuickJS bundle +and the WGSL function — are sandboxes; neither can touch the host process. +Compare the classic alternative, dynamic-library plugins (`dlopen`): +unlimited power, and a plugin crash is a host crash with no security +boundary anywhere. The WASM component model sandboxes as well but brings a +heavy toolchain. For "swap the skin, swap the personality," data-driven is +exactly the right amount of power. + +## 8. Face tracking: a MediaPipe sidecar over the narrowest pipe + +To drive the character from a real face, a Python sidecar runs Google's +MediaPipe and emits **52 blendshape coefficients** — semantic values like +"left eye closed: 0.8," "mouth smile: 0.3." The 52 names were defined by +Apple's ARKit and became the industry's de facto face-tracking vocabulary, +which is convenient: they map almost directly onto VRM expressions. + +Three deliberate moves: + +1. **Process isolation behind a narrow protocol.** The sidecar emits + newline-delimited JSON and nothing else; MediaPipe's internal result + objects are explicitly not allowed to leak into the host. Swap the + tracker tomorrow — the host doesn't change. No networking; frames are + never persisted. +2. **Shared-memory frames.** The camera is opened once, by the host; + frames reach Python through an mmap ring. No second camera claim, no + copies. +3. **A hybrid escape hatch.** A `--face-only` flag keeps body and hands on + Apple's native Vision path. Why hybrid: Vision gives facial geometry but + *not* blendshape coefficients — expressions would have to be + reverse-engineered from landmarks. MediaPipe gives the coefficients + directly. Each does what it's best at. + +The alternatives map the landscape: ARKit's TrueDepth tracking is the +quality ceiling, but Macs have no TrueDepth camera — VTubers bridge an +iPhone as a peripheral, a fine hobbyist workflow and a terrible onboarding +story. OpenSeeFace is the beloved community veteran that newer models have +outrun. NVIDIA Maxine needs an NVIDIA GPU; on a Mac, that's a no. +(Kalidokit-style libraries solve the *next* link — landmark → rig solving — +which here is done on the Rust side.) + +## 9. The virtual camera: convincing Zoom you're a webcam + +The most platform-flavored piece (design in the pocket-character repo's +`docs/virtual-camera.md`; Swift host + extension skeleton in +`native/PocketLiveCamera/`). The goal: a "Pocket Live Camera" entry in +Zoom's device list showing the final composed program — character, virtual +background, effects — never the raw webcam, never the debug HUD. + +Zoom only trusts devices in the OS camera list, so you must register a fake +camera with the system. macOS has a before-and-after story here. The old +mechanism, DAL plugins, worked by **injecting your code into Zoom's own +process** — unsandboxed, routinely rejected by hardening policies, +deprecated by Apple in macOS 12.3. The modern mechanism is the +**CoreMediaIO Camera Extension** (macOS 12.3+): the fake camera is a +separate, sandboxed system-extension process. OBS migrated to it in v28 for +the same reasons. Depending on OBS's virtual camera instead would make +"user has OBS installed" a product prerequisite and route the program +through OBS; Syphon/NDI share textures between production tools but never +appear as a camera to conferencing apps. + +The design works like a post office with the OS as the mail carrier. The +extension publishes one device with two same-format streams: + +``` +Pocket Live host ──frames──▶ sink stream ("Pocket Live In") + │ OS handles IPC, validation, fan-out + ▼ + extension keeps only the newest frame + │ re-emitted on a steady 30 fps clock + ▼ + source stream ("Pocket Live Camera") ──▶ Zoom / Teams / FaceTime +``` + +Host and Zoom never touch. Permission prompts, buffer validation, multiple +simultaneous readers — all the OS's problem. + +Three decisions worth stealing: + +- **The render thread never waits for the camera.** A GPU frame reaches the + system via GPU→CPU readback, which is slow; the design rotates three + staging buffers with async map callbacks, and if all three are busy the + frame is dropped and a counter incremented. For live video, dropping + beats queueing — latency is the product. +- **Prove the platform before optimizing.** v1 accepts a full readback plus + one row-copy per frame (720p30 ≈ 111 MB/s — comfortably affordable) and + explicitly refuses wgpu's unstable HAL for zero-copy IOSurface interop. + Step one of the plan: get a template extension signed, notarized, + installed, and *enumerated by Zoom* — because signing and system-extension + approval are where the project can actually die, not throughput. +- **Privacy as a state machine, not a policy.** The extension never opens + the real camera (it doesn't even request the entitlement). If the host + stops publishing for 500 ms, viewers see a "paused" placeholder generated + inside the extension. No failure mode — none — falls back to the real + face. + +## 10. The whole thing on one napkin + +``` +┌─ desktop widget (the 1-process core) ────────────────────┐ +│ │ +│ Rust @ 60 Hz QuickJS sandbox (low-freq) │ +│ anim → blink → springs ◀── commands ── policy bundle │ +│ → wgpu draw ── facts ──▶ (new character = │ +│ │ new bundle) │ +│ │ +│ content is data: plugin.json + VRM + WGSL + vibe.json │ +└──────┬──────────────────────────────▲────────────────────┘ + │ 720p BGRA frames │ blendshapes (NDJSON) + ▼ │ + CMIO system extension MediaPipe sidecar + (the fake camera) ▲ shared-memory frames + │ │ + ▼ │ + Zoom real camera (tracking only) +``` + +An honest process count first: with everything on, this is **three +processes of our code** (host, MediaPipe sidecar, camera extension), not +one. The 1-process figure is the parity-scope widget, and stays true +whenever tracking and the camera are off. The distinction worth defending +is *why* each extra process exists. Electron's 8 arrive before you draw +anything — they are the vehicle. Pocket Live's extras each mark a boundary +that genuinely wants to be a process: the sidecar quarantines a Python/ML +runtime behind a JSON pipe (crash it, the character keeps idling), and the +camera extension is OS-mandated — sandboxed, running even when the host is +dead, which is exactly what lets it show a placeholder instead of your real +face when Pocket Live crashes. Processes bought at isolation boundaries, +not paid as vehicle tax — and both terminate with their feature; idle cost +returns to the one-process baseline. + +Every seam in this diagram is a **narrow, boring protocol**: plain JS +objects (facts/commands), JSON lines, BGRA bytes behind a C struct. Any box +can be replaced without the others noticing. + +That is also the honest way to summarize the alternatives question. +Electron, three-vrm, bevy_vrm, V8, OBS's virtual camera, ARKit-over-iPhone +— none of them are wrong, and most are the *pragmatic* choice for someone +shipping fast. They just each violate one of this project's two invariants +somewhere: **idle costs almost nothing**, and **the host knows nothing +about specific content**. Hold those two lines, and the rest of the +architecture more or less designs itself. + +Measurement methodology, from-scratch build steps, and the full airi +comparison — including how to script airi into VRM mode for a fair fight, +and the CPU-percentage footgun in Activity Monitor — live in the +pocket-character repo's README and REPORT. From e18624a3cce49122a66ba1d0bac4ca41524c8c22 Mon Sep 17 00:00:00 2001 From: dozycat Date: Tue, 25 Aug 2026 09:48:05 +0800 Subject: [PATCH 2/5] docs(pocket-live): recenter on the live chain, not the Electron comparison Tracking, stage compositor modes (virtual/camera/matte/clean/split), program-texture output, and virtual-camera delivery are now the spine; the pocket-character parity result is one paragraph pointing at WIDGET.md. Adds the honest multi-process measurement story (measure-live.ts roles). Co-Authored-By: Claude Fable 5 --- docs/POCKET_LIVE.md | 537 +++++++++++++++++++------------------------- 1 file changed, 235 insertions(+), 302 deletions(-) diff --git a/docs/POCKET_LIVE.md b/docs/POCKET_LIVE.md index 4d592081..5edfc2d9 100644 --- a/docs/POCKET_LIVE.md +++ b/docs/POCKET_LIVE.md @@ -1,270 +1,189 @@ -# Pocket Live — the Electron digital human, rebuilt native +# Pocket Live — a native live-performance runtime -*How the Electron "digital human" was rebuilt on the Pocket runtime family — -every technology choice, the principle they all follow from, and for each -layer, the roads not taken and why. The widget itself measures 1 process / -118 MB / 3.9 % CPU against airi's 8 / 2184 MB / 44.4 %; tracking and the -virtual camera add processes — deliberately, at isolation boundaries — and -§10 accounts for them honestly.* +*A VRM character, driven by your real face, performing on a composed stage, +delivered to Zoom as a camera — the full livestreaming chain as one small +native runtime. This page walks the chain stage by stage: what each piece +chose, the alternatives, and the principles that decided between them.* This document is the design-rationale companion to [pocket-character](https://github.com/pocket-stack/pocket-character) (the repo Pocket Live grew out of) and to [WIDGET.md](WIDGET.md), which -generalizes what it proved. WIDGET.md names the *capability*; this page -explains the *choices* — with alternatives, so the next runtime doesn't have -to re-derive them. - -## 1. The category, and the question - -There is a category of app that is delightful in concept and horrifying in -Activity Monitor: the desktop companion character. A little anime figure -floats in a transparent, always-on-top window — idling, blinking every few -seconds, eyes darting in small saccades, hair swaying with physics. You can -drag her around your screen. Behind her sit two bigger pieces: driving her -expressions from the real webcam (face tracking), and presenting her -composed program as a camera to Zoom (the virtual camera). - -The reference implementation is [airi](https://github.com/moeru-ai/airi), -an open-source Electron app that does all of the above. Pocket Live began -as an experiment with a sharp question behind it: *same character, same -model file, same blink timing, same window geometry — what does it cost on -the Pocket architecture instead?* - -The measured answer (same machine, same ≥60 s steady-state methodology, -full process tree; **parity scope** — the idle widget alone, no face -tracking, no virtual camera, on both sides): - -| | airi (Electron, VRM stage) | Pocket Live widget | -| --- | --- | --- | -| Processes | 8 | 1 | -| RSS | 2184 MB | 118 MB | -| CPU (of one core) | 44.4 % | 3.9 % | - -Better than an order of magnitude, and no single heroic optimization -explains it. It falls out of one architectural principle applied -consistently. +generalizes its widget shell. The widget was the seed; the product is the +**live chain** built on top of it. -## 2. Why Electron costs what it costs +## 1. The product is a chain -First, a defense of airi: it isn't slow because it's badly written. It's -slow because of what it stands on. Electron means every app ships a -complete Chrome browser; even to draw one little character, you pay for the -whole vehicle: +A user picks a *vibe* — "牛来 · 百花奖舞台" is one JSON file: a character +id, a stage id, `tracking: camera`, `output_size: 1920x1080` — and goes +live. From that point five stages run continuously: -- **A process tree.** Main process, GPU process, network service, audio - service, a renderer per window — that's the 8 processes, each with its - own memory baseline. Gigabytes follow. -- **A web rendering path.** three.js maintains the entire 3D scene graph in - JavaScript. Every frame — 60 times a second, rAF-driven — JS computes - spring-bone physics, expression lerps, and skeletal matrices, then hands - the result to WebGL. Per-frame JS allocates; the garbage collector chases - it forever. That's where the CPU goes. -- **Pure waste at the edges.** airi's main process polls the global cursor - at 60 Hz even when nothing consumes it. +``` +real camera ──▶ tracking (blendshapes + skeleton, NDJSON) + │ + ▼ + character sim (VRM: expressions, motion, springs) + │ + ▼ + stage compositor (background mode: virtual · camera · + │ matte · clean · split · transparent) + ▼ + program output (fixed-size texture, e.g. 1920×1080) + │ + ├──▶ on-screen preview / fullscreen per monitor + └──▶ virtual camera ──▶ Zoom / Teams / FaceTime / OBS +``` -The expensive part was never "draw a character." The character's math is -tiny. The expensive part is the vehicle. +Every arrow is a narrow protocol, every stage is replaceable, and the +sections below take them in order. The design constraint that shapes all of +them: this chain runs for **hours**, next to the user's real meeting or +stream, on their own machine — so idle cost, latency, and privacy are +product features, not engineering hygiene. -## 3. The one idea everything else follows from +## 2. The foundation, briefly -Every technology choice in the repo is a corollary of a single rule: +The rendering core descends from the pocket-character parity experiment — +the same character widget that costs 8 processes / 2184 MB / 44 % CPU on +Electron measured 1 process / 118 MB / 3.9 % on the Pocket stack. That +result, and the shell/VRM/guest split behind it, are covered in +[WIDGET.md](WIDGET.md) and the pocket-character repo; here it earns one +paragraph because everything below inherits its one rule: > **Things that happen 60 times per second run in Rust. Things that happen > occasionally run in JavaScript. Things that don't change at all are > data.** -What the character actually does each frame is a fixed pipeline with zero -decisions in it: +Concretely: winit + wgpu host, engine crates for VRM (`pocket-vrm`: +parsing, VRMA retargeting, verlet spring bones — upstreamed in +[#125](https://github.com/pocket-stack/pocketjs/pull/125)), and a QuickJS +guest that receives per-tick facts and queues intent commands. A +character's personality is a guest bundle, not a build. For the live +chain this foundation matters for one reason: **the character itself is +nearly free**, so the budget can be spent where live actually needs it — +tracking inference and video delivery. + +## 3. Tracking: the performer's face and body + +The chain's input stage turns the real camera into semantic control +signals. The launcher exposes it as `--tracking off | mock | camera`: +`off` is the idle widget, `mock` replays synthetic signals (develop and +demo the whole chain with no camera at all), `camera` runs the real +pipeline. + +The real pipeline is two cooperating processes beside the host: + +- **`mediapipe_face_bridge.py`** runs Google's MediaPipe Face/Pose/Hand + Landmarker and emits **52 blendshape coefficients** — semantic values + like "left eye closed 0.8," "mouth smile 0.3" — plus a small set of + skeletal points. The 52 names were defined by Apple's ARKit and became + the industry's face-tracking vocabulary; they map almost directly onto + VRM expressions, which is the whole reason the character can mirror the + performer without a hand-authored mapping layer. +- **`pocket-vision-bridge`** is the native Apple Vision path; a + `--face-only` flag on the MediaPipe side keeps body and hands here. The + hybrid exists because Vision gives facial *geometry* but not blendshape + coefficients (expressions would have to be reverse-engineered from + landmarks), while MediaPipe gives coefficients directly. Each does what + it's best at. + +Three rules keep the stage honest: + +1. **Narrow protocol.** The sidecar emits newline-delimited JSON and + nothing else; MediaPipe's internal result objects may not leak into the + host. Swap the tracker tomorrow — the host doesn't change. +2. **One camera claim.** The host opens the camera once; frames reach the + sidecar through a shared-memory ring (mmap, no copies, no second + permission prompt, no device contention with the meeting app). +3. **No persistence, no network.** Frames are never written and never + leave the machine. Tracking output is numbers, not pixels. + +Alternatives, and why they lost: + +| Tracker | Principle | Why not | +| --- | --- | --- | +| ARKit (`ARFaceAnchor`) | TrueDepth depth camera; the quality ceiling | Macs have no TrueDepth — VTubers bridge an iPhone as a peripheral; fine for hobbyists, terrible onboarding | +| Apple Vision only | native, zero extra runtime | no blendshape output; expressions from raw landmarks is a research project | +| OpenSeeFace | the beloved community veteran, pure RGB/CPU | newer models have outrun it | +| NVIDIA Maxine | best-in-class GPU inference | needs an NVIDIA GPU; not on a Mac | + +## 4. The stage: characters, backgrounds, and the compositor + +What the audience sees behind the character is the **background mode**, a +first-class launcher/vibe setting: ``` -sample clip → apply blink/expression → solve springs → skin → draw +--background-mode transparent | virtual | camera | matte | clean | split ``` -Pure math, no branching on intent — so the whole pipeline lives native -(`crates/pocket-character-core` plus the engine crates), compiled, -allocation-free, GC-free. *Decisions* — which motion plays on click, when -to swap expressions, whether to track the mouse — happen a few times per -second at most. Those live in a QuickJS guest: per tick the core pushes a -handful of read-only facts (`blink`, `hovered`, `fps`) through -`character.__dispatch(state, events)`, and the guest queues commands -(`SetTracking`, `PlayClip`, `SetExpression`, `Quit`) that the core applies -at the end of the frame. This is the RUNTIMES.md ⟨Cores, Surfaces, Guest⟩ -shape with exactly one core and one surface. - -The analogy that sticks: **Rust is the cerebellum, JavaScript is the -cortex.** Walking, blinking, heartbeat — continuous control that never -touches conscious thought — versus "that's a friend, wave." Electron's -architecture makes the cortex operate every heartbeat by hand. - -A side effect that became the product: a character's *personality* is just -a guest bundle. Swap the bundle, get a different character; the host -doesn't recompile. The airi-parity personality is deliberately near-empty — -32 lines — because airi's default behavior barely requires decisions. - -## 4. The shell: native Rust, not Electron, not Tauri - -The product is a single binary: a winit window configured transparent, -undecorated, always-on-top, 450×600 (airi's exact stage geometry), rendered -with wgpu, frame-paced at 60 fps — *paced* meaning the loop sleeps until -the next deadline rather than spinning. The window plumbing (`AppConfig` -widget mode, transparent clear, `max_fps` pacing) went upstream in -[#125](https://github.com/pocket-stack/pocketjs/pull/125) and is being -generalized as [WIDGET.md](WIDGET.md)'s `shell`. - -The alternatives, and why they lose *here*: - -| Shell | Principle | Why not | -| --- | --- | --- | -| Electron | bundle Chromium + Node; full web ecosystem | the measured baseline being compared against | -| Tauri | reuse the system WebView, Rust backend | fixes the memory story, not the rendering one — per-frame 3D is still rAF-driven JS inside a WebView, its worst workload | -| Unity / Godot | mature VRM ecosystems (UniVRM, godot-vrm) | a game engine's resting heart rate is antithetical to "idle costs almost nothing"; frameless transparent overlays are second-class there | -| Swift/AppKit + Metal | leanest possible build | platform lock-in, and loses the reusable core/surface/guest pattern the runtime family shares | - -## 5. VRM rendering: engine crates, not three-vrm, not bevy_vrm - -A primer if VRM is new: it's an open humanoid-avatar standard (a glTF -extension) specifying skeleton naming, blend-shape expressions, and — -crucially — the spring-bone physics that make hair and clothes sway. -Everything is documented; you can implement it from the spec. - -That matters because airi's *actual* out-of-the-box character is -**Live2D**, a 2.5D mesh-deformation format ubiquitous in VTubing — and -rendering its `.moc3` files requires linking Live2D's proprietary, -closed-source Cubism Core, which cannot be vendored into an open runtime. -So the parity target is airi's VRM stage, with the identical model -(VRoid's AvatarSample_A), identical idle animation, identical source URLs. -Apples to apples. - -VRM support was built as generic crates and PR'd upstream (#125): -`pocket-vrm` (VRM 0.x parsing, VRMA retargeting, spring-bone verlet solver, -eye look-at) and morph-target/pose machinery in `pocket3d`. Two details -carry most of the performance story: - -- **Blinking is free while it isn't happening.** Facial expressions use - morph targets: the artist pre-sculpts a "closed eyes" mesh and the - runtime interpolates vertex positions by a weight. The naive approach - interpolates and re-uploads every frame; here morph deltas are computed - and uploaded **only when a weight changes**. A blink is a 0.2 s sine - pulse every 1–6 s — over 95 % of frames pay literally zero for the face. -- **Hair physics is a few vector ops.** Spring bones are chains of point - masses solved with verlet integration — velocity represented implicitly - as "current minus last position," a couple of adds per particle per - frame. The math was never expensive; airi pays for running it in - allocating, GC'd JavaScript. Here it's preallocated Rust arrays stepping - allocation-free. - -The alternatives: - -- [@pixiv/three-vrm](https://github.com/pixiv/three-vrm) — the most mature - VRM runtime anywhere, and it's three.js: airi's path. -- [bevy_vrm](https://github.com/unavi-xyz/bevy_vrm) — essentially the only - off-the-shelf Rust option (VRM 0.0/1.0), but it brings all of Bevy's ECS - and render graph with it. The game-engine tax again. -- UniVRM — official, excellent, requires Unity. -- Writing the solver from the public spec was a bounded amount of work in - exchange for a crate any Pocket app can now use. - -## 6. The scripting layer: QuickJS, not V8 - -JS-engine selection has a simple intuition: V8 is fast because of its JIT, -and the JIT costs tens of megabytes of baseline memory plus a heavyweight -runtime. But this guest reads a few properties per tick — **throughput is -irrelevant; footprint is everything**. QuickJS is ~1 MB, JIT-free, and -starts in microseconds: the exact sweet spot for "policy only." - -Why JS rather than Lua or Rhai (both even cheaper to embed)? Because plugin -authors write **TypeScript** — the typed SDK (`plugin-sdk/character.ts`) is -the plugin contract, and Bun bundles it. The type system is part of the -product surface. WASM (wasmtime, Extism) deserves a mention: strongest -sandbox, language-agnostic, and wrong for this shape of problem — every -host↔guest exchange crosses a serialization boundary, exactly the wrong -trade when the pattern is "hand the guest a state object every tick." -QuickJS mounts plain JS objects directly. - -## 7. Content: plugins are data, not code - -The content architecture has three layers: +- `transparent` — the desktop-widget mode: no stage, character over your + desktop. +- `virtual` — a procedural stage: one WGSL pixel function per background + plugin (the 百花奖 stage is ~a page of shader: film strips, gold + particles, spotlight wash). No video decode, no image assets, resolution + independent, costs microseconds per frame. +- `camera` — the real camera as the backdrop (character over your room). +- `matte` / `clean` — background replacement without a green screen. The + clean-plate approach is the notable one: the background manifest carries + `clean_plate_delay_seconds` — step out of frame, the compositor captures + the empty room, and from then on "you" can be subtracted from the feed + by difference against the plate. The alternative is ML person + segmentation (MediaPipe selfie-segmentation and friends), which needs no + choreography but costs continuous inference and produces the familiar + hair-eating halo; a captured plate is free per-frame and pixel-exact for + a static camera, which a desk setup is. +- `split` — side-by-side (real feed + character), the format for "show + the performer and the puppet" comparisons; character plugins carry a + separate `split_camera_distance` framing parameter for it. + +Content above the compositor is **data, not code** — the three-layer +plugin architecture: ``` -runtime host - ├─ character plugin (VRM + VRMA + policy bundle + render framing) - ├─ background plugin (one WGSL pixel function + compositor defaults) - └─ vibe preset (character id + background id + tracking/output) +character plugin plugin.json → VRM + VRMA + policy bundle + framing +background plugin plugin.json → one WGSL function + compositor defaults +vibe preset ids only: character + background + tracking + output ``` -A character plugin is a `plugin.json` manifest pointing at a model, -animations, a policy bundle, and framing parameters (FOV, anchor height, -camera distance). A background plugin is a manifest plus one WGSL shader — -literally a pure function from pixel coordinates to color. A vibe is a JSON -file containing only IDs: "this character + this stage + these settings." - -The load-bearing rule: **the host recognizes no specific character.** It -consumes manifest paths. A character whose model has licensing restrictions -lives outside the repo via `.gitignore`, but architecturally it is not a -special case — just another manifest the host has never heard of. - -Both places where plugins get to *execute* anything — the QuickJS bundle -and the WGSL function — are sandboxes; neither can touch the host process. -Compare the classic alternative, dynamic-library plugins (`dlopen`): -unlimited power, and a plugin crash is a host crash with no security -boundary anywhere. The WASM component model sandboxes as well but brings a -heavy toolchain. For "swap the skin, swap the personality," data-driven is -exactly the right amount of power. - -## 8. Face tracking: a MediaPipe sidecar over the narrowest pipe - -To drive the character from a real face, a Python sidecar runs Google's -MediaPipe and emits **52 blendshape coefficients** — semantic values like -"left eye closed: 0.8," "mouth smile: 0.3." The 52 names were defined by -Apple's ARKit and became the industry's de facto face-tracking vocabulary, -which is convenient: they map almost directly onto VRM expressions. - -Three deliberate moves: - -1. **Process isolation behind a narrow protocol.** The sidecar emits - newline-delimited JSON and nothing else; MediaPipe's internal result - objects are explicitly not allowed to leak into the host. Swap the - tracker tomorrow — the host doesn't change. No networking; frames are - never persisted. -2. **Shared-memory frames.** The camera is opened once, by the host; - frames reach Python through an mmap ring. No second camera claim, no - copies. -3. **A hybrid escape hatch.** A `--face-only` flag keeps body and hands on - Apple's native Vision path. Why hybrid: Vision gives facial geometry but - *not* blendshape coefficients — expressions would have to be - reverse-engineered from landmarks. MediaPipe gives the coefficients - directly. Each does what it's best at. - -The alternatives map the landscape: ARKit's TrueDepth tracking is the -quality ceiling, but Macs have no TrueDepth camera — VTubers bridge an -iPhone as a peripheral, a fine hobbyist workflow and a terrible onboarding -story. OpenSeeFace is the beloved community veteran that newer models have -outrun. NVIDIA Maxine needs an NVIDIA GPU; on a Mac, that's a no. -(Kalidokit-style libraries solve the *next* link — landmark → rig solving — -which here is done on the Rust side.) - -## 9. The virtual camera: convincing Zoom you're a webcam - -The most platform-flavored piece (design in the pocket-character repo's -`docs/virtual-camera.md`; Swift host + extension skeleton in -`native/PocketLiveCamera/`). The goal: a "Pocket Live Camera" entry in -Zoom's device list showing the final composed program — character, virtual -background, effects — never the raw webcam, never the debug HUD. - -Zoom only trusts devices in the OS camera list, so you must register a fake -camera with the system. macOS has a before-and-after story here. The old -mechanism, DAL plugins, worked by **injecting your code into Zoom's own -process** — unsandboxed, routinely rejected by hardening policies, -deprecated by Apple in macOS 12.3. The modern mechanism is the -**CoreMediaIO Camera Extension** (macOS 12.3+): the fake camera is a -separate, sandboxed system-extension process. OBS migrated to it in v28 for -the same reasons. Depending on OBS's virtual camera instead would make -"user has OBS installed" a product prerequisite and route the program -through OBS; Syphon/NDI share textures between production tools but never -appear as a camera to conferencing apps. - -The design works like a post office with the OS as the mail carrier. The -extension publishes one device with two same-format streams: +The host recognizes no specific character; it consumes manifest paths. A +model with licensing restrictions stays out of the repo via `.gitignore` +yet is architecturally unremarkable — just another manifest. The two +places plugins *execute* anything — the QuickJS policy bundle and the WGSL +function — are both sandboxes. The dlopen-style alternative (native plugin +libraries) offers unlimited power and no boundary: a plugin crash is a +show crash, mid-stream. For "swap the performer, swap the stage," +data-driven is exactly the right amount of power — and it makes a *show* +reproducible: a vibe file is a complete, versionable description of a +performance setup. + +## 5. Program output: the show is a texture, not a window + +A window is a terrible video contract: it can be transparent, resized, +Retina-scaled, occluded, minimized, portrait-shaped. So the live chain +promotes the **program** to a first-class render target — a fixed-size +texture (vibes carry `output_size`, e.g. 1920×1080) that the scene, +stage, and effects render into. The on-screen window becomes a *preview* +blitted from that texture (`--fullscreen --monitor "DELL U2723QE"` for a +dedicated display), and every consumer downstream sees the same pixels +regardless of what the user does to the window. + +The program contract is also a privacy line: final composition happens +*before* the texture, alpha forced to 1, debug skeletons / tracking +points / FPS overlays / window chrome excluded by construction, and the +raw camera feed can never become an implicit background — it participates +in tracking and in the modes that explicitly request it, nothing else. + +## 6. Delivery: convincing Zoom you're a webcam + +The most platform-flavored stage. Zoom only trusts devices in the OS +camera list, so the program must be registered as a camera. macOS has a +before-and-after story: the old mechanism, **DAL plugins**, injected your +code into Zoom's own process — unsandboxed, routinely rejected by +hardening policies, deprecated in macOS 12.3. The modern mechanism is the +**CoreMediaIO Camera Extension** (12.3+): the fake camera is a separate, +sandboxed system-extension process. OBS migrated to it in v28 for the +same reasons. + +The design works like a post office with the OS as the carrier. One +device, two same-format streams: ``` Pocket Live host ──frames──▶ sink stream ("Pocket Live In") @@ -276,76 +195,90 @@ Pocket Live host ──frames──▶ sink stream ("Pocket Live In") source stream ("Pocket Live Camera") ──▶ Zoom / Teams / FaceTime ``` -Host and Zoom never touch. Permission prompts, buffer validation, multiple -simultaneous readers — all the OS's problem. - -Three decisions worth stealing: - -- **The render thread never waits for the camera.** A GPU frame reaches the - system via GPU→CPU readback, which is slow; the design rotates three - staging buffers with async map callbacks, and if all three are busy the - frame is dropped and a counter incremented. For live video, dropping - beats queueing — latency is the product. -- **Prove the platform before optimizing.** v1 accepts a full readback plus - one row-copy per frame (720p30 ≈ 111 MB/s — comfortably affordable) and - explicitly refuses wgpu's unstable HAL for zero-copy IOSurface interop. - Step one of the plan: get a template extension signed, notarized, - installed, and *enumerated by Zoom* — because signing and system-extension - approval are where the project can actually die, not throughput. -- **Privacy as a state machine, not a policy.** The extension never opens - the real camera (it doesn't even request the entitlement). If the host - stops publishing for 500 ms, viewers see a "paused" placeholder generated - inside the extension. No failure mode — none — falls back to the real - face. - -## 10. The whole thing on one napkin +Host and Zoom never touch. Permission prompts, buffer validation, +multiple simultaneous readers — the OS's problem. Decisions worth +stealing: + +- **The render thread never waits for the camera.** GPU→CPU readback runs + through three rotating staging buffers with async map callbacks; if all + three are busy the frame is dropped and counted. For live video, + dropping beats queueing — latency is the product. +- **720p30 for v1, on purpose.** One readback plus one row-copy is + ~111 MB/s — comfortably affordable — and most conferencing links + re-scale and re-encode anyway. Zero-copy IOSurface interop via wgpu's + unstable HAL is explicitly deferred until measurement demands it. +- **Prove the platform first.** Step one of the plan is a template + extension, signed and notarized, *enumerated by Zoom* — because signing + and system-extension approval are where this can actually die, not + throughput. +- **Privacy as a state machine.** The extension never opens the real + camera (it doesn't even hold the entitlement). Host silent for 500 ms → + a locally-generated "paused" placeholder. No failure mode falls back to + the performer's real face — the extension outliving a host crash is + precisely what makes that guarantee enforceable. + +The alternatives: depending on **OBS's virtual camera** makes "user has +OBS installed" a product prerequisite and routes the show through OBS; +**Syphon/NDI** share textures between production tools but never appear +as a camera to conferencing apps; **screen-sharing the window** surrenders +resolution, framing, and the privacy line all at once. + +## 7. Measuring a live chain honestly + +The parity experiment's methodology carries over, extended for the +multi-process reality: `measure-live.ts` walks the launcher's full process +tree, tags each process by role — `renderer`, `vision`, `mediapipe`, +`launcher` — and reports median/p10/p90 per role over a settled sampling +window. There is no pretending the chain is one process: **with +everything on, it is three processes of our code** (host, MediaPipe +sidecar, camera extension) plus the Vision bridge when active. + +The distinction worth defending is *why* each process exists. Electron's +eight arrive before you draw anything — they are the vehicle. The live +chain's extras each mark a boundary that genuinely wants to be a process: +the sidecar quarantines a Python/ML runtime behind a JSON pipe (crash it, +the show keeps rendering and the character keeps idling), and the camera +extension is OS-mandated, sandboxed, and alive even when the host is dead +— which is exactly what §6's placeholder guarantee is made of. Processes +bought at isolation boundaries, not paid as vehicle tax; every one of +them terminates with its feature, and idle cost returns to the +one-process baseline. + +## 8. The whole thing on one napkin ``` -┌─ desktop widget (the 1-process core) ────────────────────┐ +┌─ host (the 1-process core) ──────────────────────────────┐ │ │ │ Rust @ 60 Hz QuickJS sandbox (low-freq) │ │ anim → blink → springs ◀── commands ── policy bundle │ -│ → wgpu draw ── facts ──▶ (new character = │ -│ │ new bundle) │ -│ │ +│ → stage compositor ── facts ──▶ (new character = │ +│ → program texture new bundle) │ +│ │ │ │ content is data: plugin.json + VRM + WGSL + vibe.json │ └──────┬──────────────────────────────▲────────────────────┘ - │ 720p BGRA frames │ blendshapes (NDJSON) + │ BGRA frames │ blendshapes (NDJSON) ▼ │ - CMIO system extension MediaPipe sidecar + CMIO system extension MediaPipe / Vision sidecars (the fake camera) ▲ shared-memory frames │ │ ▼ │ Zoom real camera (tracking only) ``` -An honest process count first: with everything on, this is **three -processes of our code** (host, MediaPipe sidecar, camera extension), not -one. The 1-process figure is the parity-scope widget, and stays true -whenever tracking and the camera are off. The distinction worth defending -is *why* each extra process exists. Electron's 8 arrive before you draw -anything — they are the vehicle. Pocket Live's extras each mark a boundary -that genuinely wants to be a process: the sidecar quarantines a Python/ML -runtime behind a JSON pipe (crash it, the character keeps idling), and the -camera extension is OS-mandated — sandboxed, running even when the host is -dead, which is exactly what lets it show a placeholder instead of your real -face when Pocket Live crashes. Processes bought at isolation boundaries, -not paid as vehicle tax — and both terminate with their feature; idle cost -returns to the one-process baseline. - -Every seam in this diagram is a **narrow, boring protocol**: plain JS -objects (facts/commands), JSON lines, BGRA bytes behind a C struct. Any box -can be replaced without the others noticing. - -That is also the honest way to summarize the alternatives question. -Electron, three-vrm, bevy_vrm, V8, OBS's virtual camera, ARKit-over-iPhone -— none of them are wrong, and most are the *pragmatic* choice for someone -shipping fast. They just each violate one of this project's two invariants -somewhere: **idle costs almost nothing**, and **the host knows nothing -about specific content**. Hold those two lines, and the rest of the -architecture more or less designs itself. - -Measurement methodology, from-scratch build steps, and the full airi -comparison — including how to script airi into VRM mode for a fair fight, -and the CPU-percentage footgun in Activity Monitor — live in the -pocket-character repo's README and REPORT. +Every seam is a narrow, boring protocol: plain JS objects +(facts/commands), JSON lines, BGRA bytes behind a C struct. Any box can +be replaced without the others noticing — and each section above listed +what it would be replaced *with*. + +Two invariants decided every contested choice: **idle costs almost +nothing** (the show runs for hours beside real work), and **the host +knows nothing about specific content** (a performance is data — a vibe +file — not a build). The alternatives that lost — three-vrm, ML matting, +OBS's virtual camera, ARKit-over-iPhone, dlopen plugins — are mostly the +pragmatic choices for someone shipping fast; they just each break one of +those two lines somewhere. + +Measurement methodology and build steps live in the pocket-character +repo's README and REPORT; the virtual-camera design in its +`docs/virtual-camera.md`; the plugin contract in its +`docs/PLUGIN_ARCHITECTURE.md`. From 8c495b2d0ea9b365c9d9ede93bb83c5e086de1b2 Mon Sep 17 00:00:00 2001 From: dozycat Date: Tue, 25 Aug 2026 09:50:55 +0800 Subject: [PATCH 3/5] docs(pocket-live): drop Chinese text and the unimplemented character example Co-Authored-By: Claude Fable 5 --- docs/POCKET_LIVE.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/POCKET_LIVE.md b/docs/POCKET_LIVE.md index 5edfc2d9..67256cde 100644 --- a/docs/POCKET_LIVE.md +++ b/docs/POCKET_LIVE.md @@ -13,9 +13,9 @@ generalizes its widget shell. The widget was the seed; the product is the ## 1. The product is a chain -A user picks a *vibe* — "牛来 · 百花奖舞台" is one JSON file: a character -id, a stage id, `tracking: camera`, `output_size: 1920x1080` — and goes -live. From that point five stages run continuously: +A user picks a *vibe* — one JSON file naming a character id, a stage id, +`tracking: camera`, `output_size: 1920x1080` — and goes live. From that +point five stages run continuously: ``` real camera ──▶ tracking (blendshapes + skeleton, NDJSON) @@ -117,9 +117,9 @@ first-class launcher/vibe setting: - `transparent` — the desktop-widget mode: no stage, character over your desktop. - `virtual` — a procedural stage: one WGSL pixel function per background - plugin (the 百花奖 stage is ~a page of shader: film strips, gold - particles, spotlight wash). No video decode, no image assets, resolution - independent, costs microseconds per frame. + plugin (the checked-in award-stage shader is about a page: film strips, + drifting gold particles, a spotlight wash). No video decode, no image + assets, resolution independent, costs microseconds per frame. - `camera` — the real camera as the backdrop (character over your room). - `matte` / `clean` — background replacement without a green screen. The clean-plate approach is the notable one: the background manifest carries From b0242c71b7b2baf23aac069da7dcd175f4cd5b8c Mon Sep 17 00:00:00 2001 From: dozycat Date: Tue, 25 Aug 2026 09:56:59 +0800 Subject: [PATCH 4/5] =?UTF-8?q?blog:=20Pocket=20Live=20=E2=80=94=20the=20l?= =?UTF-8?q?ivestreaming=20chain=20as=20a=20native=20runtime?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adapts docs/POCKET_LIVE.md into the site's blog voice: vibe/plugin content system, MediaPipe sidecar tracking, compositor background modes and clean-plate matting, program-texture output, the CMIO virtual-camera design (marked as design, not shipped), and role-tagged process measurement. Registers the post in BLOG_POSTS; links the Electron story to the existing pocket-character post instead of retelling it. Co-Authored-By: Claude Fable 5 --- site/content/blog/pocket-live.md | 85 ++++++++++++++++++++++++++++++++ site/nav.ts | 8 +++ 2 files changed, 93 insertions(+) create mode 100644 site/content/blog/pocket-live.md diff --git a/site/content/blog/pocket-live.md b/site/content/blog/pocket-live.md new file mode 100644 index 00000000..29aa6fc7 --- /dev/null +++ b/site/content/blog/pocket-live.md @@ -0,0 +1,85 @@ +Pick a character. Pick a stage. Point your webcam at your face, and join the meeting as neither — a VRM performer on a composed set, mirroring your expressions in real time, delivered to Zoom as an ordinary camera device. That's Pocket Live: the livestreaming chain — tracking, character, stage, program, delivery — built as a small native runtime instead of a streaming app's worth of processes. + +[Pocket Character](/blog/pocket-character/) was the seed: airi's idle VRM widget rebuilt as one native process, 118 MB and 4 % of a core where the Electron tree spends 2.2 GB and 44 %. That post ends where the character starts breathing. This one is about everything a *live* character needs that an idle one doesn't — a face to mirror, a set to stand on, a video contract to honor, and an audience on the other side of a conferencing app that only trusts things which look like webcams. + +## A performance is a file + +The unit of use is a *vibe* — one JSON file naming a character, a stage, and the run parameters: + +```json +{ + "kind": "vibe", + "character": "pocket-live.default-character", + "background": "pocket-live.hundred-flowers-stage", + "tracking": "camera", + "output_size": "1920x1080" +} +``` + +Underneath it, the content system is three kinds of plugin, all data: + +- A **character plugin** is a manifest pointing at a VRM model, its animations, a QuickJS policy bundle (the personality — hot-swappable, [as before](/blog/pocket-character/)), and framing parameters: camera FOV, anchor height, and a separate framing distance for split-screen mode. +- A **background plugin** is a manifest plus one WGSL pixel function. The checked-in award-stage set is about a page of shader — film strips, drifting gold particles, a spotlight wash — no video decode, no image assets, resolution-independent, microseconds per frame. +- A **vibe** composes the two by id and adds tracking and output settings. + +The host recognizes none of them specifically; it consumes manifest paths. A launcher resolves ids and cross-combines freely: + +```bash +bun run vibe:list # everything discovered +bun run vibe -- --character golden-horn --background japanese-station +bun run vibe -- --vibe default --background hundred-flowers-stage # override one axis +``` + +The payoff is that a show is *reproducible*: a vibe file is a complete, versionable description of a performance setup. And the two places plugin content gets to execute anything — the policy bundle and the pixel function — are both sandboxes. The alternative everyone reaches for, native plugin libraries, offers unlimited power and no boundary: a plugin crash is a show crash, mid-stream. + +## Tracking without giving up your camera + +The chain's input stage turns the real camera into semantic control signals, and it's the one place we deliberately spend a second process. `--tracking off | mock | camera`: `off` is the idle widget, `mock` replays synthetic signals so the whole chain can be developed and demoed with no camera attached, and `camera` runs the real pipeline — a MediaPipe sidecar beside a native Apple Vision bridge. + +The sidecar runs Google's Face/Pose/Hand Landmarker models and emits **52 blendshape coefficients** — semantic values like "left eye closed 0.8," "mouth smile 0.3" — plus a small set of skeletal points. Those 52 names were coined by Apple's ARKit and became the industry's face-tracking vocabulary, and they map almost directly onto VRM expressions; that shared vocabulary is the entire reason the character can mirror a performer without a hand-authored mapping layer. The Vision bridge exists because the hybrid is genuinely better than either half: Apple's framework gives facial *geometry* but no blendshape coefficients (recovering expressions from raw landmarks is a research project), while MediaPipe gives the coefficients directly — so a `--face-only` flag keeps the face on MediaPipe and lets body and hands ride the native path. + +Three rules keep this stage honest. The sidecar speaks newline-delimited JSON and nothing else — MediaPipe's result objects may not leak into the host, so the tracker is swappable without touching it. The camera is claimed **once**, by the host, and frames reach the sidecar through a shared-memory ring — no copies, no second permission prompt, no fighting the meeting app for the device. And nothing persists: frames are never written, never networked; what leaves the tracking stage is numbers, not pixels. + +The alternatives map the landscape. ARKit's TrueDepth tracking is the quality ceiling, but Macs have no TrueDepth camera — VTubers bridge an iPhone as a peripheral, which is a fine hobbyist workflow and a terrible onboarding story. OpenSeeFace is the beloved community veteran that newer models have outrun. NVIDIA Maxine wants a GPU this machine doesn't have. + +## The show is a texture, not a window + +A window is a terrible video contract: transparent, resizable, Retina-scaled, occludable, sometimes portrait. So the live chain promotes the **program** — the pixels the audience sees — to a first-class render target: a fixed-size texture (the vibe's `output_size`) that scene, stage, and effects render into. The desktop window is demoted to a *preview* blitted from that texture (there's a `--fullscreen --monitor` path for a dedicated display), and every downstream consumer sees the same pixels no matter what the user does to the window. + +What stands behind the character is the compositor's **background mode**, a first-class setting rather than a stack of ad-hoc toggles: + +``` +--background-mode transparent | virtual | camera | matte | clean | split +``` + +`transparent` is the desktop-widget mode. `virtual` is a procedural WGSL stage. `camera` puts the real feed behind the character. `split` renders performer and puppet side by side — the honest format for "show me the human and the character," with its own framing parameter in the character manifest. The interesting pair is `matte` and `clean`: background replacement without a green screen. The clean-plate approach asks the performer to step out of frame once; the compositor captures the empty room (the background manifest carries the delay), and from then on the person can be separated by difference against the plate — free per-frame, pixel-exact for a static desk camera. The road not taken is ML person-segmentation, which needs no choreography but costs continuous inference and produces the familiar hair-eating halo. For a camera that doesn't move, the plate wins on both axes. + +The program contract doubles as a privacy line, enforced by construction rather than policy: final composition happens *before* the texture with alpha forced opaque; debug skeletons, tracking points, FPS counters and window chrome can't reach it; and the raw camera feed participates only in tracking and in the modes that explicitly name it — it can never become an implicit background because someone toggled the wrong thing mid-meeting. + +## Becoming a webcam + +Delivery is the most platform-shaped stage, and the design is worth describing even though it's the part still being built (the Xcode skeleton is in-tree; the design doc is in the repo). Zoom trusts only devices in the OS camera list, so the program must *be* a camera. macOS has a before-and-after story here: the old mechanism, DAL plugins, injected your code into Zoom's own process — unsandboxed, routinely rejected by hardening policies, deprecated since macOS 12.3. The modern mechanism is the **CoreMediaIO Camera Extension**: the fake camera is a separate, sandboxed system-extension process, the same path OBS moved to in v28. + +It works like a post office with the OS as the carrier. The extension publishes one device with two same-format streams — a *sink* only Pocket Live writes, and a *source* that Zoom, Teams and FaceTime read. The host pushes frames into the sink; the extension keeps only the newest and re-emits it on a steady 30 fps clock; the OS handles the IPC, buffer validation, permission prompts, and fan-out to multiple simultaneous readers. Host and meeting app never touch. + +The design decisions worth stealing, in order of how much pain they prevent: + +- **The render thread never waits for the camera.** GPU→CPU readback goes through three rotating staging buffers with async callbacks; if all three are busy, the frame is dropped and counted. For live video, dropping beats queueing — latency is the product. +- **720p30 on purpose.** One readback plus one row copy is ~111 MB/s, comfortably affordable, and most conferencing links re-scale and re-encode anyway. Zero-copy IOSurface interop through wgpu's unstable HAL is explicitly deferred until measurement demands it — and step one of the plan is a signed, notarized template extension *enumerated by Zoom*, because code signing and system-extension approval are where projects like this actually die, not throughput. +- **Privacy as a state machine.** The extension never opens the real camera — it doesn't even hold the entitlement. If the host goes silent for 500 ms, viewers get a locally-generated "paused" placeholder. No failure mode falls back to the performer's real face; the extension outliving a host crash is precisely what makes that guarantee enforceable. + +The alternatives: depending on OBS's virtual camera makes "user has OBS installed" a product prerequisite and routes the show through OBS; Syphon and NDI share textures between production tools but never appear as cameras to conferencing apps; screen-sharing the preview window surrenders resolution, framing, and the privacy line in one move. + +## Measuring a multi-process show honestly + +The [parity experiment's](/blog/pocket-character/) methodology carries over, extended for the multi-process reality: the measurement script walks the launcher's full process tree, tags each process by role — renderer, vision, mediapipe, launcher — and reports median and p10/p90 per role over a settled sampling window. There is no pretending the chain is one process; with everything on, it's the host plus the tracking sidecars, plus the camera extension when it ships. + +The distinction worth defending is *why* each process exists. An Electron tree's processes arrive before you draw anything — they're the vehicle. The live chain's extras each mark a boundary that genuinely wants to be a process: the sidecar quarantines a Python/ML runtime behind a JSON pipe, so crashing the tracker leaves the show rendering and the character idling; the camera extension is OS-mandated, sandboxed, and alive even when the host is dead — which is exactly what the paused-placeholder guarantee is made of. Processes bought at isolation boundaries, not paid as vehicle tax. Every one of them terminates with its feature, and idle cost returns to the one-process baseline the widget started from. + +## What this doesn't claim + +Status, honestly: the character runtime, plugin/vibe system, tracking pipeline and background modes run today; the virtual camera is a completed design with its extension skeleton in-tree, staged behind the prove-the-platform-first plan above — this post describes that design, not a shipped device. No audio: the microphone stays whatever the meeting app already uses. And none of this makes Pocket Live a streaming *platform* — there's no scene switching, no overlays-as-apps, no RTMP. It's the narrower bet that the live chain itself — track, perform, compose, deliver — fits in a runtime small enough to sit beside your actual work, on the machine you're already using, for hours. + +--- + +*Pocket Live builds on [pocket-character](https://github.com/pocket-stack/pocket-character) and the [PocketJS engine family](https://github.com/pocket-stack/pocketjs) — the VRM crate, morph targets, and widget windowing shipped upstream. Follow [@pocket_js](https://x.com/pocket_js) for the parts still landing.* diff --git a/site/nav.ts b/site/nav.ts index e4d6a8be..ed36b6b3 100644 --- a/site/nav.ts +++ b/site/nav.ts @@ -56,6 +56,14 @@ export interface BlogPost { } export const BLOG_POSTS: BlogPost[] = [ + { + slug: "pocket-live", + title: "Pocket Live: The Livestreaming Chain as a Native Runtime", + date: "2026-08-25", + description: + "Pocket Character learned to go live: webcam-driven blendshape tracking through a quarantined MediaPipe sidecar, stages and performers as data-only plugins composed by one vibe file, six compositor background modes including green-screen-free clean-plate matting, the show promoted from a window to a fixed-size program texture, and a CoreMediaIO camera-extension design that delivers it to Zoom without ever being able to leak the performer's real face. Plus role-tagged process-tree measurement — because a live chain that honestly needs three processes should say so.", + author: { name: "dozycat", url: "https://github.com/dozycat" }, + }, { slug: "blackberry-classic", title: "One Square Screen, Two Native Stacks: PocketJS on the BlackBerry Classic", From 35f9a8bba2c9d3e2ab2267a6122b0e19b176c5c8 Mon Sep 17 00:00:00 2001 From: dozycat Date: Tue, 25 Aug 2026 12:26:25 +0800 Subject: [PATCH 5/5] blog(pocket-live): fold the docs page into the post, drop docs/POCKET_LIVE.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One rendition instead of two: the chain diagram, the frequency rule, the sink/source relay diagram and the two-invariants close move into the blog post in its own voice. The docs page was rationale narrative, not a normative contract — WIDGET.md and the pocket-character repo's own docs already carry the contracts. Co-Authored-By: Claude Fable 5 --- docs/POCKET_LIVE.md | 284 ------------------------------- site/content/blog/pocket-live.md | 36 +++- 2 files changed, 34 insertions(+), 286 deletions(-) delete mode 100644 docs/POCKET_LIVE.md diff --git a/docs/POCKET_LIVE.md b/docs/POCKET_LIVE.md deleted file mode 100644 index 67256cde..00000000 --- a/docs/POCKET_LIVE.md +++ /dev/null @@ -1,284 +0,0 @@ -# Pocket Live — a native live-performance runtime - -*A VRM character, driven by your real face, performing on a composed stage, -delivered to Zoom as a camera — the full livestreaming chain as one small -native runtime. This page walks the chain stage by stage: what each piece -chose, the alternatives, and the principles that decided between them.* - -This document is the design-rationale companion to -[pocket-character](https://github.com/pocket-stack/pocket-character) (the -repo Pocket Live grew out of) and to [WIDGET.md](WIDGET.md), which -generalizes its widget shell. The widget was the seed; the product is the -**live chain** built on top of it. - -## 1. The product is a chain - -A user picks a *vibe* — one JSON file naming a character id, a stage id, -`tracking: camera`, `output_size: 1920x1080` — and goes live. From that -point five stages run continuously: - -``` -real camera ──▶ tracking (blendshapes + skeleton, NDJSON) - │ - ▼ - character sim (VRM: expressions, motion, springs) - │ - ▼ - stage compositor (background mode: virtual · camera · - │ matte · clean · split · transparent) - ▼ - program output (fixed-size texture, e.g. 1920×1080) - │ - ├──▶ on-screen preview / fullscreen per monitor - └──▶ virtual camera ──▶ Zoom / Teams / FaceTime / OBS -``` - -Every arrow is a narrow protocol, every stage is replaceable, and the -sections below take them in order. The design constraint that shapes all of -them: this chain runs for **hours**, next to the user's real meeting or -stream, on their own machine — so idle cost, latency, and privacy are -product features, not engineering hygiene. - -## 2. The foundation, briefly - -The rendering core descends from the pocket-character parity experiment — -the same character widget that costs 8 processes / 2184 MB / 44 % CPU on -Electron measured 1 process / 118 MB / 3.9 % on the Pocket stack. That -result, and the shell/VRM/guest split behind it, are covered in -[WIDGET.md](WIDGET.md) and the pocket-character repo; here it earns one -paragraph because everything below inherits its one rule: - -> **Things that happen 60 times per second run in Rust. Things that happen -> occasionally run in JavaScript. Things that don't change at all are -> data.** - -Concretely: winit + wgpu host, engine crates for VRM (`pocket-vrm`: -parsing, VRMA retargeting, verlet spring bones — upstreamed in -[#125](https://github.com/pocket-stack/pocketjs/pull/125)), and a QuickJS -guest that receives per-tick facts and queues intent commands. A -character's personality is a guest bundle, not a build. For the live -chain this foundation matters for one reason: **the character itself is -nearly free**, so the budget can be spent where live actually needs it — -tracking inference and video delivery. - -## 3. Tracking: the performer's face and body - -The chain's input stage turns the real camera into semantic control -signals. The launcher exposes it as `--tracking off | mock | camera`: -`off` is the idle widget, `mock` replays synthetic signals (develop and -demo the whole chain with no camera at all), `camera` runs the real -pipeline. - -The real pipeline is two cooperating processes beside the host: - -- **`mediapipe_face_bridge.py`** runs Google's MediaPipe Face/Pose/Hand - Landmarker and emits **52 blendshape coefficients** — semantic values - like "left eye closed 0.8," "mouth smile 0.3" — plus a small set of - skeletal points. The 52 names were defined by Apple's ARKit and became - the industry's face-tracking vocabulary; they map almost directly onto - VRM expressions, which is the whole reason the character can mirror the - performer without a hand-authored mapping layer. -- **`pocket-vision-bridge`** is the native Apple Vision path; a - `--face-only` flag on the MediaPipe side keeps body and hands here. The - hybrid exists because Vision gives facial *geometry* but not blendshape - coefficients (expressions would have to be reverse-engineered from - landmarks), while MediaPipe gives coefficients directly. Each does what - it's best at. - -Three rules keep the stage honest: - -1. **Narrow protocol.** The sidecar emits newline-delimited JSON and - nothing else; MediaPipe's internal result objects may not leak into the - host. Swap the tracker tomorrow — the host doesn't change. -2. **One camera claim.** The host opens the camera once; frames reach the - sidecar through a shared-memory ring (mmap, no copies, no second - permission prompt, no device contention with the meeting app). -3. **No persistence, no network.** Frames are never written and never - leave the machine. Tracking output is numbers, not pixels. - -Alternatives, and why they lost: - -| Tracker | Principle | Why not | -| --- | --- | --- | -| ARKit (`ARFaceAnchor`) | TrueDepth depth camera; the quality ceiling | Macs have no TrueDepth — VTubers bridge an iPhone as a peripheral; fine for hobbyists, terrible onboarding | -| Apple Vision only | native, zero extra runtime | no blendshape output; expressions from raw landmarks is a research project | -| OpenSeeFace | the beloved community veteran, pure RGB/CPU | newer models have outrun it | -| NVIDIA Maxine | best-in-class GPU inference | needs an NVIDIA GPU; not on a Mac | - -## 4. The stage: characters, backgrounds, and the compositor - -What the audience sees behind the character is the **background mode**, a -first-class launcher/vibe setting: - -``` ---background-mode transparent | virtual | camera | matte | clean | split -``` - -- `transparent` — the desktop-widget mode: no stage, character over your - desktop. -- `virtual` — a procedural stage: one WGSL pixel function per background - plugin (the checked-in award-stage shader is about a page: film strips, - drifting gold particles, a spotlight wash). No video decode, no image - assets, resolution independent, costs microseconds per frame. -- `camera` — the real camera as the backdrop (character over your room). -- `matte` / `clean` — background replacement without a green screen. The - clean-plate approach is the notable one: the background manifest carries - `clean_plate_delay_seconds` — step out of frame, the compositor captures - the empty room, and from then on "you" can be subtracted from the feed - by difference against the plate. The alternative is ML person - segmentation (MediaPipe selfie-segmentation and friends), which needs no - choreography but costs continuous inference and produces the familiar - hair-eating halo; a captured plate is free per-frame and pixel-exact for - a static camera, which a desk setup is. -- `split` — side-by-side (real feed + character), the format for "show - the performer and the puppet" comparisons; character plugins carry a - separate `split_camera_distance` framing parameter for it. - -Content above the compositor is **data, not code** — the three-layer -plugin architecture: - -``` -character plugin plugin.json → VRM + VRMA + policy bundle + framing -background plugin plugin.json → one WGSL function + compositor defaults -vibe preset ids only: character + background + tracking + output -``` - -The host recognizes no specific character; it consumes manifest paths. A -model with licensing restrictions stays out of the repo via `.gitignore` -yet is architecturally unremarkable — just another manifest. The two -places plugins *execute* anything — the QuickJS policy bundle and the WGSL -function — are both sandboxes. The dlopen-style alternative (native plugin -libraries) offers unlimited power and no boundary: a plugin crash is a -show crash, mid-stream. For "swap the performer, swap the stage," -data-driven is exactly the right amount of power — and it makes a *show* -reproducible: a vibe file is a complete, versionable description of a -performance setup. - -## 5. Program output: the show is a texture, not a window - -A window is a terrible video contract: it can be transparent, resized, -Retina-scaled, occluded, minimized, portrait-shaped. So the live chain -promotes the **program** to a first-class render target — a fixed-size -texture (vibes carry `output_size`, e.g. 1920×1080) that the scene, -stage, and effects render into. The on-screen window becomes a *preview* -blitted from that texture (`--fullscreen --monitor "DELL U2723QE"` for a -dedicated display), and every consumer downstream sees the same pixels -regardless of what the user does to the window. - -The program contract is also a privacy line: final composition happens -*before* the texture, alpha forced to 1, debug skeletons / tracking -points / FPS overlays / window chrome excluded by construction, and the -raw camera feed can never become an implicit background — it participates -in tracking and in the modes that explicitly request it, nothing else. - -## 6. Delivery: convincing Zoom you're a webcam - -The most platform-flavored stage. Zoom only trusts devices in the OS -camera list, so the program must be registered as a camera. macOS has a -before-and-after story: the old mechanism, **DAL plugins**, injected your -code into Zoom's own process — unsandboxed, routinely rejected by -hardening policies, deprecated in macOS 12.3. The modern mechanism is the -**CoreMediaIO Camera Extension** (12.3+): the fake camera is a separate, -sandboxed system-extension process. OBS migrated to it in v28 for the -same reasons. - -The design works like a post office with the OS as the carrier. One -device, two same-format streams: - -``` -Pocket Live host ──frames──▶ sink stream ("Pocket Live In") - │ OS handles IPC, validation, fan-out - ▼ - extension keeps only the newest frame - │ re-emitted on a steady 30 fps clock - ▼ - source stream ("Pocket Live Camera") ──▶ Zoom / Teams / FaceTime -``` - -Host and Zoom never touch. Permission prompts, buffer validation, -multiple simultaneous readers — the OS's problem. Decisions worth -stealing: - -- **The render thread never waits for the camera.** GPU→CPU readback runs - through three rotating staging buffers with async map callbacks; if all - three are busy the frame is dropped and counted. For live video, - dropping beats queueing — latency is the product. -- **720p30 for v1, on purpose.** One readback plus one row-copy is - ~111 MB/s — comfortably affordable — and most conferencing links - re-scale and re-encode anyway. Zero-copy IOSurface interop via wgpu's - unstable HAL is explicitly deferred until measurement demands it. -- **Prove the platform first.** Step one of the plan is a template - extension, signed and notarized, *enumerated by Zoom* — because signing - and system-extension approval are where this can actually die, not - throughput. -- **Privacy as a state machine.** The extension never opens the real - camera (it doesn't even hold the entitlement). Host silent for 500 ms → - a locally-generated "paused" placeholder. No failure mode falls back to - the performer's real face — the extension outliving a host crash is - precisely what makes that guarantee enforceable. - -The alternatives: depending on **OBS's virtual camera** makes "user has -OBS installed" a product prerequisite and routes the show through OBS; -**Syphon/NDI** share textures between production tools but never appear -as a camera to conferencing apps; **screen-sharing the window** surrenders -resolution, framing, and the privacy line all at once. - -## 7. Measuring a live chain honestly - -The parity experiment's methodology carries over, extended for the -multi-process reality: `measure-live.ts` walks the launcher's full process -tree, tags each process by role — `renderer`, `vision`, `mediapipe`, -`launcher` — and reports median/p10/p90 per role over a settled sampling -window. There is no pretending the chain is one process: **with -everything on, it is three processes of our code** (host, MediaPipe -sidecar, camera extension) plus the Vision bridge when active. - -The distinction worth defending is *why* each process exists. Electron's -eight arrive before you draw anything — they are the vehicle. The live -chain's extras each mark a boundary that genuinely wants to be a process: -the sidecar quarantines a Python/ML runtime behind a JSON pipe (crash it, -the show keeps rendering and the character keeps idling), and the camera -extension is OS-mandated, sandboxed, and alive even when the host is dead -— which is exactly what §6's placeholder guarantee is made of. Processes -bought at isolation boundaries, not paid as vehicle tax; every one of -them terminates with its feature, and idle cost returns to the -one-process baseline. - -## 8. The whole thing on one napkin - -``` -┌─ host (the 1-process core) ──────────────────────────────┐ -│ │ -│ Rust @ 60 Hz QuickJS sandbox (low-freq) │ -│ anim → blink → springs ◀── commands ── policy bundle │ -│ → stage compositor ── facts ──▶ (new character = │ -│ → program texture new bundle) │ -│ │ │ -│ content is data: plugin.json + VRM + WGSL + vibe.json │ -└──────┬──────────────────────────────▲────────────────────┘ - │ BGRA frames │ blendshapes (NDJSON) - ▼ │ - CMIO system extension MediaPipe / Vision sidecars - (the fake camera) ▲ shared-memory frames - │ │ - ▼ │ - Zoom real camera (tracking only) -``` - -Every seam is a narrow, boring protocol: plain JS objects -(facts/commands), JSON lines, BGRA bytes behind a C struct. Any box can -be replaced without the others noticing — and each section above listed -what it would be replaced *with*. - -Two invariants decided every contested choice: **idle costs almost -nothing** (the show runs for hours beside real work), and **the host -knows nothing about specific content** (a performance is data — a vibe -file — not a build). The alternatives that lost — three-vrm, ML matting, -OBS's virtual camera, ARKit-over-iPhone, dlopen plugins — are mostly the -pragmatic choices for someone shipping fast; they just each break one of -those two lines somewhere. - -Measurement methodology and build steps live in the pocket-character -repo's README and REPORT; the virtual-camera design in its -`docs/virtual-camera.md`; the plugin contract in its -`docs/PLUGIN_ARCHITECTURE.md`. diff --git a/site/content/blog/pocket-live.md b/site/content/blog/pocket-live.md index 29aa6fc7..bafa3a17 100644 --- a/site/content/blog/pocket-live.md +++ b/site/content/blog/pocket-live.md @@ -1,6 +1,24 @@ Pick a character. Pick a stage. Point your webcam at your face, and join the meeting as neither — a VRM performer on a composed set, mirroring your expressions in real time, delivered to Zoom as an ordinary camera device. That's Pocket Live: the livestreaming chain — tracking, character, stage, program, delivery — built as a small native runtime instead of a streaming app's worth of processes. -[Pocket Character](/blog/pocket-character/) was the seed: airi's idle VRM widget rebuilt as one native process, 118 MB and 4 % of a core where the Electron tree spends 2.2 GB and 44 %. That post ends where the character starts breathing. This one is about everything a *live* character needs that an idle one doesn't — a face to mirror, a set to stand on, a video contract to honor, and an audience on the other side of a conferencing app that only trusts things which look like webcams. +[Pocket Character](/blog/pocket-character/) was the seed: airi's idle VRM widget rebuilt as one native process, 118 MB and 4 % of a core where the Electron tree spends 2.2 GB and 44 %. That post ends where the character starts breathing. This one is about everything a *live* character needs that an idle one doesn't — a face to mirror, a set to stand on, a video contract to honor, and an audience on the other side of a conferencing app that only trusts things which look like webcams. End to end: + +``` +real camera ──▶ tracking (blendshapes + skeleton, NDJSON) + │ + ▼ + character sim (VRM: expressions, motion, springs) + │ + ▼ + stage compositor (background mode: virtual · camera · + │ matte · clean · split · transparent) + ▼ + program output (fixed-size texture, e.g. 1920×1080) + │ + ├──▶ on-screen preview / fullscreen per monitor + └──▶ virtual camera ──▶ Zoom / Teams / FaceTime / OBS +``` + +The widget's one rule carries over unchanged — things that happen sixty times a second run in Rust, things that happen occasionally run in JavaScript, things that don't change at all are data — and it matters here for a new reason: **the character itself is nearly free, so the budget goes where live actually needs it**, tracking inference and video delivery. Every arrow in that diagram is a narrow, boring protocol (JSON lines, plain objects, BGRA bytes behind a C struct), every box is replaceable, and the chain is built to run for *hours* next to your real meeting on your own machine — which makes idle cost, latency, and privacy product features rather than engineering hygiene. ## A performance is a file @@ -60,7 +78,19 @@ The program contract doubles as a privacy line, enforced by construction rather Delivery is the most platform-shaped stage, and the design is worth describing even though it's the part still being built (the Xcode skeleton is in-tree; the design doc is in the repo). Zoom trusts only devices in the OS camera list, so the program must *be* a camera. macOS has a before-and-after story here: the old mechanism, DAL plugins, injected your code into Zoom's own process — unsandboxed, routinely rejected by hardening policies, deprecated since macOS 12.3. The modern mechanism is the **CoreMediaIO Camera Extension**: the fake camera is a separate, sandboxed system-extension process, the same path OBS moved to in v28. -It works like a post office with the OS as the carrier. The extension publishes one device with two same-format streams — a *sink* only Pocket Live writes, and a *source* that Zoom, Teams and FaceTime read. The host pushes frames into the sink; the extension keeps only the newest and re-emits it on a steady 30 fps clock; the OS handles the IPC, buffer validation, permission prompts, and fan-out to multiple simultaneous readers. Host and meeting app never touch. +It works like a post office with the OS as the carrier. The extension publishes one device with two same-format streams — a *sink* only Pocket Live writes, and a *source* that Zoom, Teams and FaceTime read: + +``` +Pocket Live host ──frames──▶ sink stream ("Pocket Live In") + │ OS handles IPC, validation, fan-out + ▼ + extension keeps only the newest frame + │ re-emitted on a steady 30 fps clock + ▼ + source stream ("Pocket Live Camera") ──▶ Zoom / Teams / FaceTime +``` + +The host pushes frames into the sink; the extension keeps only the newest and re-emits it on a steady clock; the OS handles the IPC, buffer validation, permission prompts, and fan-out to multiple simultaneous readers. Host and meeting app never touch. The design decisions worth stealing, in order of how much pain they prevent: @@ -80,6 +110,8 @@ The distinction worth defending is *why* each process exists. An Electron tree's Status, honestly: the character runtime, plugin/vibe system, tracking pipeline and background modes run today; the virtual camera is a completed design with its extension skeleton in-tree, staged behind the prove-the-platform-first plan above — this post describes that design, not a shipped device. No audio: the microphone stays whatever the meeting app already uses. And none of this makes Pocket Live a streaming *platform* — there's no scene switching, no overlays-as-apps, no RTMP. It's the narrower bet that the live chain itself — track, perform, compose, deliver — fits in a runtime small enough to sit beside your actual work, on the machine you're already using, for hours. +If you're keeping score on the choices, two invariants decided every contested one: **idle costs almost nothing** (the show runs for hours beside real work), and **the host knows nothing about specific content** (a performance is a vibe file, not a build). The roads not taken — three-vrm, ML matting, OBS's virtual camera, ARKit-over-iPhone, dlopen plugins — are mostly the pragmatic picks for someone shipping fast; each just breaks one of those two lines somewhere. + --- *Pocket Live builds on [pocket-character](https://github.com/pocket-stack/pocket-character) and the [PocketJS engine family](https://github.com/pocket-stack/pocketjs) — the VRM crate, morph targets, and widget windowing shipped upstream. Follow [@pocket_js](https://x.com/pocket_js) for the parts still landing.*