Research: keep Polyester CLJS animation mixer-first, no runtime update loop #16

Context

Follow-up to the Loom3 runtime/update discussion, focused on Polyester because this is where the CLJS version will be changed.

The core question: should Polyester have a runtime shaped like l.update() / runtime.update(delta), or should the CLJS agencies schedule animation work into the Three/Loom3 mixer layer and stay out of per-frame evaluation?

Short answer from the current architecture: do not add a Polyester per-frame runtime loop. Polyester should remain the planner/orchestrator that emits scheduled snippets and control effects. Three.js AnimationMixer still needs to be advanced every frame, but that clock belongs to Loom3 / the renderer host, not to the CLJS agencies.

Findings

Polyester CLJS is already mostly event/schedule-oriented

The CLJS runtime applies agency outputs by calling host methods such as scheduleSnippet, schedule, updateSnippet, removeSnippet, seekSnippet, pauseSnippet, resumeSnippet, play, pause, stop, and parameter setters. It does not currently own a render loop or curve-sampling loop.

Relevant files:

• src-cljs/latticework/runtime.cljs:31 - apply-animation-effect!
• src-cljs/latticework/runtime.cljs:35 - scheduleSnippet / schedule host dispatch
• src-cljs/latticework/runtime.cljs:41 - update/remove/seek/pause/resume/play/pause/stop/set params host dispatch
• src-cljs/latticework/runtime.cljs:95 - apply-output!
• src-cljs/latticework/runtime.cljs:101 - handles scheduleSnippet via host scheduling and cleanup registration
• src-cljs/latticework/runtime.cljs:118 - handles animation/prosodic/vocal/lipsync effects and cleanup plans
• src-cljs/latticework/runtime.cljs:335 - in-process animation agency wraps animation/load!, schedule!, remove!, play!, pause!, etc.; no tick/update command
• src-cljs/latticework/runtime.cljs:626 - worker-client animation pass-through; no tick/update command
• src-cljs/latticework/worker.cljs:54 - worker dispatches commands to agencies; no frame loop

The only worker-side timing found is blink auto-scheduling (src-cljs/latticework/worker.cljs:41), which is a wall-clock event timer, not a render-frame animation runtime.

The CLJS animation agency is state plus effects, not a runtime evaluator

src-cljs/latticework/animation.cljs stores snippet state, normalizes snippet metadata, and emits schedule/control effects. It does not step the snippets frame-by-frame.

Relevant files:

• src-cljs/latticework/animation.cljs:6 - default state holds snippets/order/schedule/playback metadata
• src-cljs/latticework/animation.cljs:71 - normalize-snippet preserves mixer metadata such as loop mode, reverse playback, blend mode, weight, and category
• src-cljs/latticework/animation.cljs:173 - schedule! emits protocol/emit-schedule-snippet
• src-cljs/latticework/animation.cljs:391 - command handler supports load/schedule/update/remove/play/pause/etc.; no STEP, tick, or update command

This is the right shape for the CLJS side. The agency should describe intent and schedule data-rich snippets, not become an evaluator competing with Three/Loom3.

Existing CLJS agencies already build snippets for mixer scheduling

Several agencies already produce curve snippets that can be compiled into mixer clips by the host.

Gaze:

• src-cljs/latticework/gaze.cljs:112 - builds axis curves
• src-cljs/latticework/gaze.cljs:125 - builds snippets with category eyeHeadTracking, clamp metadata, and priority
• src-cljs/latticework/gaze.cljs:190 - target scheduling removes old snippets and schedules new snippets

Blink:

• src-cljs/latticework/blink.cljs:89 - builds blink curves
• src-cljs/latticework/blink.cljs:103 - builds blink snippet
• src-cljs/latticework/blink.cljs:126 - emits scheduleSnippet

Prosodic:

• src-cljs/latticework/prosodic.cljs:136 - builds additive animation snippets
• src-cljs/latticework/prosodic.cljs:177 - start-talking schedules brow/head snippets
• src-cljs/latticework/prosodic.cljs:197 - pulse removes/reschedules snippets
• src-cljs/latticework/prosodic.cljs:226 - stop-talking emits fade/remove plan

Lipsync:

• src-cljs/latticework/lipsync.cljs:143 - builds viseme curves
• src-cljs/latticework/lipsync.cljs:173 - builds viseme snippets with category visemeSnippet
• src-cljs/latticework/lipsync.cljs:200 - schedules snippets plus cleanup plans
• src-cljs/latticework/lipsync.cljs:217 - schedules per-word snippets
• src-cljs/latticework/lipsync.cljs:294 - schedules one Azure viseme timeline snippet

Vocal:

• src-cljs/latticework/vocal.cljs:509 - samples curves while building expressive curves, not during render playback
• src-cljs/latticework/vocal.cljs:667 - builds jaw curve
• src-cljs/latticework/vocal.cljs:708 - builds a combined vocal snippet with articulated curves and jaw curve
• src-cljs/latticework/vocal.cljs:814 - schedules the vocal timeline snippet plus cleanup plan
• src-cljs/latticework/vocal.cljs:878 - word boundary handling mainly seeks the scheduled snippet on drift

The TypeScript root animation service already documents the desired boundary

The TS-side animation service is currently mixer-first and says Loom3 owns runtime playback, mixer timing, looping, scrubbing, keyframe detection, and completion.

Relevant files:

• src/animation/README.md:3 - animation agency is the UI/orchestration adapter for Loom3 reusable clips
• src/animation/README.md:5 - Loom3 owns runtime playback and mixer timing
• src/animation/README.md:8 - no local animation runtime, scheduler polling loop, or requestAnimationFrame tick in the adapter
• src/animation/README.md:85 - delegates snippet playback to loom3.buildClip
• src/animation/README.md:133 - UI time comes from Loom3 stream events, not polling
• src/animation/animationService.ts:380 - service comment says Loom3 clip handles are the runtime source of truth
• src/animation/animationService.ts:541 - clip playback runner
• src/animation/animationService.ts:548 - fails if host.buildClip is unavailable
• src/animation/animationService.ts:566 - calls host.buildClip(...)
• src/animation/animationService.ts:592 - fails fast when the returned clip handle lacks subscribe
• src/animation/animationService.ts:605 - starts playback through the clip handle
• src/animation/animationService.ts:617 - awaits handle completion / stop promise and emits completion
• src/animation/animationService.ts:898 - schedule starts playback runner when auto-play is enabled
• src/animation/animationService.ts:990 - schedule snapshots read time from the clip handle rather than local polling

There is also a test that enforces the no-polling contract:

• src/animation/__tests__/animationService.test.ts:793 - should fail fast instead of polling when Loom3 clip streams are missing
• src/animation/__tests__/animationService.test.ts:827 - confirms a clip was built
• src/animation/__tests__/animationService.test.ts:828 - confirms playback does not start without stream support
• src/animation/__tests__/animationService.test.ts:830 - confirms cleanup is called
• src/animation/__tests__/animationService.test.ts:832 - asserts the required Loom3 clip stream error

Documentation conflict to clean up

src/animation/MIXER_MIGRATION_README.md appears partially stale or conflicting with the current service README.

• src/animation/MIXER_MIGRATION_README.md:1 describes the mixer migration goal
• src/animation/MIXER_MIGRATION_README.md:6 says the latest revert/current state is legacy runtime baseline, which conflicts with the current TS service behavior and README
• src/animation/MIXER_MIGRATION_README.md:22 describes a desired lifecycle where STEP only advances the mixer
• src/animation/MIXER_MIGRATION_README.md:43 says the only per-frame work in mixer mode should be mixer.update

We should reconcile or retire this doc so the CLJS and TS docs agree.

What update still means

There is still one unavoidable frame update somewhere: Three.js AnimationMixer.update(delta) must be called from the render loop. The important boundary is that this update should live in Loom3 / the renderer host.

Polyester CLJS should not introduce a second runtime loop. Its job should be:

1. Receive chat / behavior / multimodal events.
2. Convert them into agency commands.
3. Emit scheduled snippets and control effects.
4. Let the host compile those snippets into Loom3 clip handles.
5. Observe Loom3 stream/completion events where needed.

Risks / wrong turns to avoid

1. Adding STEP, tick, or update(delta) to CLJS agencies

That would recreate the old dual-runtime problem. The agency layer would start sampling curves or applying morphs while Loom3/Three is also running actions. This increases drift, makes blending harder to reason about, and makes chat/nonverbal timing less reliable.

2. Letting procedural transitionAU / transitionViseme paths become the normal host implementation

Those are useful legacy/dev fallbacks, but they should not be the production path for CLJS scheduled playback. The normal path should compile snippets into Loom3 clips through buildClip and drive playback through ClipHandle streams.

3. Treating command timers as a render runtime

There are cleanup/fade timers in the runtime/client path, especially around prosodic fade plans. These are wall-clock orchestration timers, not a frame update loop. Still, where possible, fade-outs should move into mixer action weights or explicit clips/handles so the host owns the timing consistently.

Relevant references:

• src-cljs/latticework/runtime.cljs:129 - fallback timer for prosodic fade plan when the host does not handle it
• src-cljs/latticework/prosodic.cljs:226 - stop-talking fade/remove plan

4. Per-word lipsync clip churn

lipsync.cljs can schedule per-word snippets. That is acceptable for compatibility, but the mixer-first path should prefer the combined vocal.cljs timeline or the Azure full-viseme timeline path so the host creates fewer short-lived clips and cleanup timers.

5. High-frequency gaze updates creating many replacement clips

Gaze currently schedules remove + schedule on target changes. That is fine for discrete target updates, but mouse/webcam/high-frequency gaze sources could generate too many clips. We should use coalescing, replacement-by-name, or a host-supported retarget/update capability for high-frequency targets while still avoiding a CLJS frame loop.

Proposed architecture boundary

Polyester CLJS owns

• Agency state and behavior decisions
• Appraisal / intent / stimulus integration
• Snippet construction
• Scheduling and control commands
• Cleanup plans and coarse orchestration
• Optional metadata for blending, priority, category, and expressive intent

Loom3 / Three host owns

• AnimationMixer.update(delta) in the render loop
• Clip/action creation from snippet curves
• Mixer timing, looping, weights, fades, and completion
• Runtime stream events (subscribe)
• Action cleanup and disposal
• Scrubbing / seeking / pause / resume

Required host capability contract

Normal CLJS playback should require a host that supports:

• scheduleSnippet / schedule
• buildClip or equivalent Loom3 clip construction
• ClipHandle.play()
• ClipHandle.pause()
• ClipHandle.resume()
• ClipHandle.stop()
• ClipHandle.setTime()
• ClipHandle.getTime()
• ClipHandle.getDuration()
• ClipHandle.setWeight()
• ClipHandle.setPlaybackRate()
• ClipHandle.setLoop()
• ClipHandle.finished
• ClipHandle.subscribe(...)

If a host cannot supply stream events, fail fast instead of silently falling back to polling.

Implementation plan

1. Audit the CLJS host adapter used by LoomLarge/Polyester and confirm scheduleSnippet routes to Loom3 clip construction, not procedural morph transitions.
2. Add or tighten tests that verify CLJS animation outputs do not create a STEP, tick, requestAnimationFrame, interval-based evaluator, or per-frame curve sampler.
3. Add worker/client boundary tests proving scheduleSnippet, updateSnippet, removeSnippet, seekSnippet, pauseSnippet, and resumeSnippet remain control effects only.
4. Prefer vocal.cljs combined timeline scheduling over lipsync.cljs per-word scheduling for production chat playback.
5. For prosodic fade plans, either move fades into mixer weights/handles or document the existing timer fallback as temporary host-compatibility code.
6. Add gaze coalescing or replacement semantics for high-frequency gaze inputs so target changes do not flood the mixer with short-lived clips.
7. Reconcile src/animation/README.md and src/animation/MIXER_MIGRATION_README.md so the documented architecture is unambiguous.

Acceptance criteria

• No Polyester CLJS agency exposes or depends on a frame-level update(delta), tick, STEP, requestAnimationFrame, or polling loop.
• CLJS animation/prosodic/gaze/blink/lipsync/vocal agencies emit scheduled snippets and control effects only.
• The host adapter for normal playback compiles snippets into Loom3/Three mixer clips.
• Missing Loom3 clip stream support fails fast, consistent with animationService.test.ts.
• transitionAU / transitionViseme style fallbacks are not used for normal production CLJS playback.
• Prosodic fades are either mixer-owned or explicitly documented as temporary orchestration timers.
• High-frequency gaze sources are coalesced, replaced by stable snippet names, or retargeted through a host capability without introducing a CLJS frame loop.
• Docs clearly state: Polyester schedules and orchestrates; Loom3/Three owns mixer advancement and per-frame playback.

Open questions

• Which LoomLarge host adapter currently receives scheduleSnippet from the Polyester CLJS worker in production chat?
• Do we want the host contract to hard-require ClipHandle.subscribe, or should there be a separately named degraded mode for tests/dev only?
• Should gaze retargeting be represented as updateSnippet, a new retargetSnippet, or replacement-by-name with coalescing?
• Should prosodic fade plans become explicit generated snippets, clip action fades, or both depending on the expression category?

Bottom line

Polyester is closer to the desired architecture than Loom3 was in the original concern. The change here should be to preserve and tighten the scheduler/orchestrator boundary, not to add a CLJS runtime. The only frame update should remain the renderer/Loom3 mixer update.