Skip to content

Telemetry off: no repository size reads around retention runs, no pool acquire events - #901

Merged
schmitch merged 2 commits into
mainfrom
claude/549-telemetry-off-measurements
Nov 17, 2026
Merged

schmitch merged 2 commits into
mainfrom
claude/549-telemetry-off-measurements

Conversation

@schmitch

Copy link
Copy Markdown
Member

Fixes #549

Telemetry is off by default, but two of its measurements cost work regardless. Both are now gated on whether the metrics record (forge_telemetry::enabled()).

What changed

Retention size reads (crates/core/src/retention/worker.rs)

  • The worker read repository_size() before every claimed run and again after a run that deleted something, only to feed forge.retention.freed. It now reads the size only with the metrics on.
  • With the metrics on, a dry run (dry_run_only) and a disabled policy (enabled: false) no longer read it either: they free nothing, and the issue names both. They still count in forge.retention.runs, with 0 bytes freed, as before.

Pool acquire events (crates/db/src/pool.rs, crates/server/src/main.rs, crates/server/src/telemetry.rs)

  • forge_db::connect_with asks sqlx for the per-acquire TRACE event only with the metrics on; without them it keeps sqlx's default (no event). A slow acquire (2 s) still logs its WARN either way.
  • main installs the AcquireTimes layer only with the metrics on, through the new telemetry::acquire_times(). Without it the subscriber's level is no longer raised to TRACE for that target.
  • Both read forge_telemetry::enabled(), so they depend on main's existing order: Telemetry::install() before the subscriber and before the database connection. The doc comments say so.

Nothing in the exporter setup (Telemetry::new) is touched, to stay clear of #558 (#900). No admin API changes, so no web UI screen.

Tests

Metrics are a process-wide switch, so the "off" cases are test binaries of their own:

Metrics off Metrics on
Retention crates/core/tests/retention_metrics_off.rs: a run that deletes a version sends no size query crates/core/tests/retention_metrics.rs: the same run sends exactly two size queries and forge.retention.freed counts the deleted bytes; a dry run and a disabled policy send none
Pool crates/server/tests/telemetry_off.rs: acquire_times() is None, and the server's pool (connect_with) emits no acquire event to a subscriber that asks for them at TRACE; a pool told to emit them does reach that subscriber, so the check can fail crates/server/tests/telemetry.rs: five acquires on the server's pool add five to db.client.connection.wait_time

The retention tests count size queries from sqlx's own statement log (sqlx::query events), recognising a statement that starts with SELECT COALESCE(SUM(, which all three repository_size queries do.

Verification

Run locally, all passing:

  • cargo xtask check (fmt and clippy with -D warnings)
  • cargo xtask unused, cargo xtask doc-check, cargo xtask changelog --check
  • .github/scripts/test-parallel.sh --workspace: 137 test binaries, 2066 tests passed, 0 failed, 2 ignored (Docker with testcontainers, images through mirror.gcr.io)

Not run: cargo xtask e2e and the client e2e scripts (oci-e2e, raw-e2e, apt-e2e, npm-e2e, image smoke), and nothing measured the saved work on a large repository or a busy pool.

The tests were built with CARGO_PROFILE_DEV_DEBUG=0 to fit the session's disk; that changes debug info only.

Changelog and milestone

docs/changes/2.2.0/549.fixed.md. The issue was in Backlog; since the fix ships in the next release, I moved it to the 2.2.0 milestone.


Generated by Claude Code

…l acquire events

Two measurements cost work with the metrics off (#549):

- The retention worker read the repository's size before every claimed
  run and again after one that deleted something, only to feed
  `forge.retention.freed`. It now reads the size only when the metrics
  record (`forge_telemetry::enabled()`), and then not for a dry run or a
  disabled policy, which free nothing.
- `forge_db::connect_with` had sqlx emit a `TRACE` event per acquire, and
  `main` always installed the `AcquireTimes` layer for it. The pool now
  emits the event, and `main` installs the layer
  (`telemetry::acquire_times()`), only with the metrics on. A slow acquire
  still logs its `WARN`.

Tests: `retention_metrics_off.rs` and `telemetry_off.rs` are binaries
without the metrics, where a run sends no size query and the server's pool
emits no acquire event; `retention_metrics.rs` and `telemetry.rs` show the
two size reads with the freed bytes, and the acquire times in
`db.client.connection.wait_time`, with the metrics on.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJYg356rtc9YTDE6RrF
The statement recorder is a bare `tracing::Subscriber` instead of a
`tracing-subscriber` layer, so forge-core gains no dev-dependency and the
lockfile stays as on main.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJYg356rtc9YTDE6RrF
@schmitch
schmitch added this pull request to the merge queue Nov 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 17, 2026
@schmitch
schmitch added this pull request to the merge queue Nov 17, 2026

Copy link
Copy Markdown
Member Author

The merge queue removed this pull request after its test job failed (run 44060834451). One test failed, 52 of 53 in crates/server/tests/registry.rs passed, and every other job that had finished passed:

a_stalled_pull_of_a_downloading_layer_releases_its_staging_space_once_stored
panicked at crates/server/tests/registry.rs:2903: arrived before the upstream sent the rest: Elapsed(())

That is read_some's 10 s bound on the first MiB of a proxied 32 MiB layer. The binary took 67 s in that run, against 23 to 32 s here, so the bound ran out under the load of the test binaries running side by side.

I do not think this change causes it:

  • The registry tests build their pool with PgPoolOptions directly, not forge_db::connect_with, so the pool change does not reach them.
  • The only other code this diff changes is the retention worker, where it removes two queries when the metrics are off. The proxy's layer streaming is untouched.
  • On the merge of this branch with main (099c841), the registry binary passed three times in a row locally, 53 of 53 each time.
  • The same commit's pull-request run passed test.

I have not changed the test here: it belongs to #763, and a fix for it (more slack in read_some, as DEVELOPMENT.md advises for timing bounds under test-parallel.sh) is a pull request of its own. This one needs re-queueing, which I cannot do.


Generated by Claude Code

Merged via the queue into main with commit 40c1530 Nov 17, 2026
20 checks passed
@schmitch
schmitch deleted the claude/549-telemetry-off-measurements branch November 17, 2026 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telemetry off: retention still reads each repository's size around every run, and the pool emits an acquire event per connection

2 participants