feat(producer): calibration-aware heartbeat + worker-death terminal-error contract

[vanceingalls]

Description

Two additive parallel-capture observability improvements that make healthy runs distinguishable from broken ones in the render trace:

Fix 1 — Calibration-aware heartbeat label. observeRenderStage now accepts an optional heartbeatMessage override, and the browser_probe + capture_calibration call sites pass "browser calibrating (frames not started)" so their heartbeats no longer share the "stage still running" phrasing with post-capture stages. A new stagePhase: "calibrating" | "capturing" field flows through the observation data (added to ALLOWED_STRING_DATA_KEYS, defaulted inside captureStageObservationData, overridden at the calibration call site) so structured consumers can classify the heartbeat without string-matching the message.

Fix 2 — Worker-death terminal-error contract. formatWorkerFailure now synthesizes a terminal error string when result.error is falsy (undefined or empty), naming the framesCaptured shortfall, the expected count, the range, the field-signal reference, and the --workers=1 isolation hint. executeParallelCapture now also detects under-captured but no-error results — a worker returning success with framesCaptured < expected gets an error string written to result.error in place, so the downstream results.filter((r) => r.error) doesn't silently accept it. Two exports added to support both branches: expectedFramesForTask(task) (stride-aware) and synthesizeSilentWorkerExitError(result, expected).

Both changes are additive — the existing "stage still running" message is preserved everywhere else, and workers that already surface a bare error still get their explicit message untouched (the synthesized string only fills in when result.error is falsy).

Context

Two field signals define the observability failure surface. Both concern healthy or broken runs that the current instrumentation cannot tell apart:

• ts=1784019503 (#hyperframes-cli-feedback): parallel-capture heartbeat/observability during startup underinstruments healthy runs. On a ~64s browser calibration (SwiftShader warm-up class), the heartbeat emitted at 30s and 60s carried framesCompleted: 0 — read as a broken zero-frame stall — while calibration probe frames were legitimately being written to disk. The stage phase is capture_calibration (which uses captureStageObservationData — that helper's framesCompleted getter returns job.framesRendered ?? 0, and calibration doesn't touch job.framesRendered because it writes probe frames only). Directly covered here — the calibration call sites now override the heartbeat message and pass stagePhase: "calibrating" in the data, so consumers can distinguish healthy pre-capture waits from mid-capture zero-frame stalls.

• ts=1784042064 (#hyperframes-cli-feedback): a 1292s Windows render hard-exited during video frame extraction after heartbeatIndex: 3 (~180s) with no final error string. The parallel-worker path in parallelCoordinator.ts treats a WorkerResult with error === undefined as success and never surfaces the failure — the parent's executeParallelCapture filters results.filter((r) => r.error) so a silent no-error result gets dropped from the aggregated failure message entirely. Directly covered here — under-captured no-error results now get a synthesized error string in place, and formatWorkerFailure also fills in a synthesized message when it's asked to format a result whose error field is missing or empty. Both paths cite the field signal + --workers=1 isolation hint so operators grepping logs can classify the failure.

The heartbeat message + stagePhase field are backward-compatible additions — no consumer that reads message === "stage still running" regresses (the default is unchanged), and RenderObservationData remains a Record<string, RenderObservationValue>. The worker-death contract keeps the current happy-path behavior for workers that emit real error strings; the synthesized message only fills gaps where the current code emits "unknown error" (in formatWorkerFailure) or silently passes the result through (in executeParallelCapture).

Testing

• packages/producer/src/services/render/observability.test.ts — 11 pass, 0 fail. Two new cases:
  • uses a custom heartbeat message during calibration stages — verifies the custom message replaces "stage still running" (no both-emission), that stagePhase: "calibrating" and framesCompleted: 0 flow through the heartbeat data, and that heartbeatIndex: 1 still increments as expected.
  • keeps the default heartbeat message when no override is passed — verifies "stage still running" is unchanged when the fifth arg is omitted, and that stagePhase: "capturing" flows through when the caller sets it.
• packages/engine/src/services/parallelCoordinator.test.ts — 34 pass, 0 fail. Six new cases across three describes:
  • synthesizes a terminal error when a worker exits with no error string — verifies the synthesized message names the worker, the shortfall (12 captured / 60 expected), the range [0, 60), the field signal ts=1784042064, and the --workers=1 hint.
  • prefers an explicit error string over the synthesized silent-exit message — verifies real errors (e.g. Target closed) are unchanged and the field-signal reference does NOT appear.
  • treats an empty error string as a silent exit for the message contract — closes the "empty-string is falsy but not undefined" gap.
  • expectedFramesForTask — contiguous range, stride-3 interleaved (30 → 10 frames), remainder round-up (31 → 11), empty range → 0, reversed range → 0.
  • synthesizeSilentWorkerExitError — direct-call test verifying every field lands in the message.
• bun run typecheck — clean on both @hyperframes/producer and @hyperframes/engine.
• oxfmt --write — clean on all 5 touched files.
• Pre-existing ffprobe.test.ts failures (3 cases about HDR PNG cICP metadata) are unrelated to this PR — they touch packages/engine/src/utils/ffprobe.ts and predate via/overlay-count-lint.

Enterprise release / feature flag holdout

• No enterprise release / feature flag holdout required — observability-only additions (log message text + one new optional string data field + a synthesized error string). No user-visible surface, no runtime behavior change on healthy paths.
• Enterprise release / feature flag holdout required — coordinated with enterprise team.

UX/Screenshot recording

• No UI impact — heartbeat log text + structured trace data + operator-facing error message only.

Stack: PR #6 of 9. Base: via/overlay-count-lint (#2508).

— Via

Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

[miga-heygen]

Review — #2510 feat(producer): calibration-aware heartbeat + worker-death terminal-error contract

Verdict: LGTM — two clean fixes for two distinct field signals, well-tested.

SSOT check

1. Silent worker exit

Item	Owner
Expected-frame count	expectedFramesForTask — single function, honors stride
Synthesized error message	synthesizeSilentWorkerExitError — single function
Error synthesis call site	executeParallelCapture — single in-place mutation before the error filter

formatWorkerFailure now falls through to synthesizeSilentWorkerExitError when result.error is falsy or empty. The in-place mutation in executeParallelCapture ensures the downstream results.filter(r => r.error) catches the synthesized failure. Clean.

expectedFramesForTask correctly handles stride: ceil((endFrame - startFrame) / stride). Contiguous (stride=1), interleaved (stride>1), empty ranges (start >= end → 0) all tested.

2. Calibration heartbeat

Item	Owner
Heartbeat message override	ObserveRenderStageOptions.heartbeatMessage
Stage phase discrimination	stagePhase: "calibrating" | "capturing" in observation data

The heartbeatMessage option fully replaces the default "stage still running" — not appended. Test verifies no "stage still running" heartbeat emits during a calibration stage. The stagePhase field is spread with the default first (stagePhase: "capturing" in captureStageObservationData) so the calibration call site can override via extra.

Tests

• Silent exit: formatWorkerFailure with no error (synthesizes message), with explicit error (prefers it), with empty string error (treated as silent). expectedFramesForTask: contiguous, interleaved, stride remainder (ceil), empty ranges. synthesizeSilentWorkerExitError: field signal reference, frame shortfall details, reruns hint.
• Heartbeat: Custom message during calibration (no "stage still running" leak), default message without override.

No blockers. Ship it.