Skip to content

Show process telemetry on the build progress tab - #198

Merged
PavelMakarchuk merged 7 commits into
mainfrom
build-progress-process-telemetry
Oct 1, 2026
Merged

PavelMakarchuk merged 7 commits into
mainfrom
build-progress-process-telemetry

Conversation

@PavelMakarchuk

Copy link
Copy Markdown
Contributor

Stacked on #197. GitHub retargets it to main once #197 merges.

Summary

The Build progress tab now reads the process telemetry that PolicyEngine/microcosm#1082's follow-up adds to US release builds, branch staging-telemetry-p0. Runs that don't report it look exactly as before.

  • Cores and memory per stage. Cores are CPU seconds divided by wall seconds between the stage's start and end resources. Memory is the larger resident set at either end. Both appear on timeline rows, and as medians and maxima in "Where build time goes". The overview adds a line for the running stage: "1.0 of 18 cores busy in Target compilation · memory 62 GB of 128 GB · peak 81 GB · commit …".
  • Heartbeat. A run that sends heartbeats is marked stalled after five minutes without one (HEARTBEAT_STALE_MS), not six hours. That makes an out-of-memory or SIGKILL death visible quickly.
  • Measured batch progress. work in the progress document gives batches done, the total and the time spent so far.
    • The current stage's remaining time follows that measured rate once 3 units are done.
    • History still forecasts what comes after the stage.
    • Calibration's epoch rate stays a floor only, because size search repeats passes.
  • Failure classes. A failed run shows its class (gate refused, terminated, interrupted, out of memory, refused, error). A new "Why runs stop" table counts runs by class, together with the median compute already spent. Blocked runs count as gate refusals and silent runs as "stopped without a final event".
  • Identity. The run manifest's commit, runtime versions, CPU count and memory are shown.

Testing

  • bun test: 529 pass, including new tests for cores from resources, measured batch rate with and without history, heartbeat staleness, failure-class counts and identity parsing. tsc --noEmit passes.
  • End to end: I drove microcosm's real StagingTelemetry from the staging-telemetry-p0 branch through a simulated US release, local only with no uploads, and watched it on the local tab.
    • Batch progress appeared, and the remaining time followed the measured rate.
    • Cores and memory appeared per stage, along with the heartbeat age, the host and the commit.
    • The run ended in a classed gate refusal with its failure lines.

🤖 Generated with Claude Code

Reads what instrumented runs report: cores and memory per stage (and their
medians across runs), a heartbeat that marks a silent run stalled after five
minutes, batches done of a stage's total (the current stage's remaining time
then follows the measured rate), failure classes with the compute each threw
away, and the run's commit, runtime and machine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
calibration-diagnostics Ready Ready Preview Oct 1, 2026 8:09pm UTC

Request Review

A US run whose write check failed records delivery_check; a UK run's
delivery block counts uploads. Either way, a run that stayed on its build
machine is flagged, with the re-stage command for US runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs from before failure classes get one inferred from their error type,
message and failing stage (marked inferred). A run that went silent within two
minutes of starting is told apart from a kill as a test or aborted launch.
Each kind lists the stages runs stopped in and their most common failure
messages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PavelMakarchuk and others added 2 commits October 1, 2026 15:57
The epoch, solver pass and seconds per epoch move into the current-stage
summary, which is what a running build needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A run silent since July drew a timeline months long: its elapsed time ran to
now and the axis drew a tick per day. Elapsed time now stops at the last
telemetry, the axis keeps to about eight ticks at any length, and a run that
went silent within two minutes reads as a test or aborted launch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A running build first, otherwise the newest run that got past its first two
minutes, instead of whichever run is newest (often a test or an abort).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Short stages were folded into a count; each now gets its own row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PavelMakarchuk
PavelMakarchuk changed the base branch from build-monitor to main October 1, 2026 21:20
@PavelMakarchuk
PavelMakarchuk marked this pull request as ready for review October 1, 2026 21:20
@PavelMakarchuk
PavelMakarchuk merged commit f38305a into main Oct 1, 2026
5 checks passed

This branch was successfully deployed

1 active deployment
Preview — 90c6620a Deployed Oct 1, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant