[MaxGhenis]

Problem

Every build that supplies the same queue path shares one telemetry spool, and every service's delivery worker walked every pending run in it (CollectorDelivery.flush_once over spool.pending_runs()). For each run it read its own process's Hugging Face credential. With none, it called _make_local_only(registration, missing_huggingface_credential), which permanently sets upload_state = local_only on that run, another build's run included. The same applied to a token exchange the collector refused (401/403), a registration refused with 403 or 409, and an events batch refused with 403.

So one build started without a credential (a CI shell, a terminal with no HF_TOKEN, another login) permanently disabled hosted telemetry for every other build's queued events. The new two-process test reproduces it on this repo's main (6e3cfd1): with the service subprocesses running main's code, build B (no credential) leaves build A's run local_only for missing_huggingface_credential.

This is a port of PolicyEngine/microcosm#1177 (head 615d84f89), which fixes the same bug in Microcosm's copy of this service. The lease files are identical, so services from both packages that share one spool recognise each other's leases.

How the collector authorizes (read for this change)

From PolicyEngine/calibration-diagnostics telemetry-service/telemetry_collector/ at main (280459c8). app.py is identical at the 80a2a4ac pinned by this repo's collector job; storage.py there differs only in how it materializes blocked runs.

• POST /v1/auth/huggingface/exchange: an invalid token gets 401, a non-member of policyengine gets 403, and an unavailable Hugging Face gets 503. Success returns a JWT whose sub is the HF user id.
• POST /v1/runs: the first registration of a run id records the caller as owner_hf_id (INSERT … ON CONFLICT DO NOTHING, then a locked read). A registration by another HF user gets 403 ("Run belongs to another Hugging Face user"). The owner with different metadata gets 409.
• POST /v1/runs/{run_id}/events: an unknown run gets 404, checked before the owner. An owner mismatch or an unregistered producer gets 403.

So a run belongs to whoever registers it first. Delivering another producer's run with your own credential has three outcomes:

• It works, if you are the same HF user.
• It gets 403, if another user registered the run.
• If the run is not registered yet, it claims the run for your user. The real producer's own registration then gets 403.

Decision (same as microcosm#1177)

1. A live producer's run belongs to its own service. No other service sends any request for it.
2. Another producer's run is adopted only once its service has exited. Any service may adopt it, with its own credential, one adopter at a time. Runs orphaned by dead services keep being delivered.
3. An answer about the caller's credential decides only the caller's own run. Such answers are: no credential, exchange 401/403, registration 403, events 403. An adopted run goes local-only only for an answer about the run itself. In this service that is a registration 409: the collector gives it only to the run's owner, so no owner's registration can be accepted, and any other caller gets 403 and passes the run over. Otherwise the run stays pending for a service whose credential can deliver it. This service does not offer it again under the same credential, and offers it again once its login changes.

Liveness: per-producer advisory leases, no spool migration

• Each service takes an exclusive flock on <spool>.leases/<sha256(run_id NUL producer_id)>.lock when TelemetryModule.initialize builds its delivery. That happens before the run is registered, before its started event, and before the host binds the socket, so the run never has an event without a live lease. The service holds the lease for its whole life; TelemetryModule.close releases it after the drain, and the kernel drops it on any exit, SIGKILL included. A failed initialization releases it too.
• Another producer's pending run is adoptable only if its lease can be taken without waiting. The adopter holds the lease only while delivering that run.
• After taking a lock, the code checks that the open file is still the path's inode. If a sweeper unlinked the file in between, it retries.
• No lease file means an older service, or a free lease that was swept. Such a run is adopted, under a lease the adopter creates, only after ORPHAN_IDLE_SECONDS (15 min) without an update. Live services append a heartbeat every 60 s by default.
• A periodic sweep (every 10 min) removes free lease files of runs with nothing pending.
• Same files as Microcosm. Directory suffix, digest, file suffix, modes (0700 directory, 0600 file) and lock type (flock, LOCK_EX | LOCK_NB) match microcosm#1177. A test pins the path for a known id pair. A local differential check against microcosm#1177's leases.py confirmed identical paths and mutual exclusion across real processes (below).
• Why not a schema change: every checkout and build on a host shares this spool, and a new Alembic revision makes older ones refuse it. Leases live outside the database. upload_state values and reason strings are unchanged.

Change

• leases.py (new): ProducerLeases (hold, try_acquire, sweep), ProducerLease and OwnLeaseUnavailableError; microcosm#1177's code apart from imports.
  • hold() repeats an attempt that errs until its deadline. A failed attempt can leave a lease file nobody holds, which other services read as an exited producer's.
  • _lock closes its descriptor if the inode check itself raises.
• collector.py:
  • CollectorDelivery(spool, registration, …) now knows its own run and holds its lease; close() releases it. Where advisory locks exist, a delivery that cannot take its lease raises OwnLeaseUnavailableError rather than serve without it.
  • flush_once claims each run first (_claim): the own run always; another run only under its lease, or under the idle fallback.
  • Then, while holding the run, it checks again that the run is still pending (run_has_deliverable) before sending anything. A pass reads its list of pending runs once, then can wait seconds on the collector; meanwhile a run's own service may have made it local-only and exited, freeing its lease.
  • _make_local_only(…, caller_dependent=…) is the one gate. Registration 403 is now split from 409.
  • A refused credential is remembered as an in-memory SHA-256 fingerprint and not exchanged again.
  • A pass that has passed-over runs first notes the credential its requests would use (_note_current_credential), so a run passed over under one login is offered again once the login changes. A session the collector issued stays in use until it expires, as for the own run, so the new login counts from then.
  • The no-credential notice and the rejected-credential warning, both about "this run", print only for the build's own run. An adopted run's reason is kept in the spool.
  • The per-run memory (_registered, _passed_over, _warned_denied) is keyed by the (run_id, producer_id) pair. It was keyed by "run_id:producer_id", and ids may contain colons.
• module.py:
  • Passes the registration to the delivery.
  • The shutdown drain waits only for the build's own run (EventSpool.run_has_deliverable). Before, every shutdown could wait out its full drain while another build had queued events.
  • The lease is released in close and when initialize fails.
  • When the lease cannot be taken, initialize prints why (OWN_LEASE_UNAVAILABLE_MESSAGE, Microcosm's wording) and re-raises. The host then exits before readiness, and the build adapter falls back to running without telemetry, as for any failed start.
• spool.py: run_has_deliverable and last_updated, both SQLAlchemy ORM queries.
• hypothesis joins the dev dependency group.

Differences from microcosm#1177

• Settled 4xx answers. This service predates microcosm#1147's rule that makes a settled collector rejection local-only (Microcosm commit 0e80b0361). Here an events answer other than 202/401/403, and a registration answer other than 201/401/403/409, still defer and retry. So the only caller-independent local-only in this service is a registration 409, and an adopted run whose batch the collector refuses with 422 stays queued for retry. That behavior is unchanged by this PR; Make a settled collector rejection local-only instead of retrying it forever #6, stacked on this branch, ports #1147's rule.
• Wiring. The delivery is built in TelemetryModule.initialize rather than a main(), and the drain is TelemetryModule.close. A service that cannot take its lease fails initialize (the host exits before readiness) where Microcosm's main() returns 1.
• Tests.
  • They read and write the spool through the ORM rather than raw SQL (AGENTS.md).
  • The property test models registration conflicts (409) as well as rejected batches (422). It also advances a fake monotonic clock past any retry delay before each flush, because here a 422 sets a backoff. So it checks what passes eventually do, not real retry timing.
  • Its two last services start under generated logins and make a pass before they become Alice and Bob, so recovery by a service that was refused first is checked.
  • It runs with database=None, so no .hypothesis/ directory lands in checkouts.
  • Additional tests: the lease layout pinned to Microcosm's; the module holding its lease from initialize to close; a failed initialize releasing it; an adopted run's events 403 passed over; a passed-over run offered again after a login change, at once with no session and after expiry with one; a lease whose inode check fails not left locked; runs whose ids join alike kept apart (refusal and registration); a lock refused once at startup not exposing the run; hold() waiting out refused locks; a module, and a real service, that cannot take the lease not starting.

Invariants

Each is checked by tests/telemetry/test_foreign_runs.py, within the fake collector's model of the hosted one. The first five are also checked after every step of a Hypothesis property test (150 examples in CI) over 2–4 running builds plus up to 30 random steps:

• the steps: start, emit, flush, kill, change login, and run one service while another's collector request is in flight (that other service may exit then);
• logins: none, outsider, expired, alice, bob;
• shared run ids;
• run ids whose registration the collector answers with 409, and runs whose batches it answers with 422.

1. No foreign local-only from a caller's credential. A service never makes another producer's run local-only because of its own credential (missing, refused, or not the run's owner). Every foreign local-only follows a registration 409 for that run, received by that service.
2. No requests about a live producer's run. While a producer's service is alive and holds its lease, no other service sends any collector request about that producer's run. Where advisory locks exist, a service that cannot take its lease does not start, so its run gets no event. A run from an older version that has no lease file is protected only by the 15-minute idle rule, which is a heuristic.
3. Every local-only follows a justifying answer. When a service makes a run local-only, its latest collector answer is the refusal that justifies it: a registration 409 for another producer's run; a registration 403/409 or an events 403 for its own. Two cases have no fresh answer: a missing credential, and a credential the collector refused to that service earlier, which is not exchanged again.
4. A refused credential is exchanged once per service.
5. Local-only is final. No request about a run reaches the collector after the run goes local-only, even from a service that listed the run before.
6. Orphans are delivered. Once every service has gone, every run left pending is delivered in full by a later service whose login owns it. That includes a service that was refused for the run under an earlier login, once any session from that login has expired. Two exceptions: a run id the collector answers with 409 goes local-only with run_registration_rejected; a run whose batches get 422 stays pending and queued.
7. The own run behaves as before. It goes local-only for the same reasons, with the same reason strings and warnings. The existing test_delivery.py cases pass with only the new constructor argument. The test_module.py fixture now closes the real delivery before replacing it with a mock, so its lease is not leaked.
8. The shutdown drain waits only for the build's own run. It loops only while that run has deliverable events. A pass it makes may still deliver an adopted run, and a pass in flight is not cut off at the deadline.

Verification

All local runs below are on 093efb1, macOS, Python 3.14.7.

• Touched test files. tests/telemetry/test_foreign_runs.py, test_delivery.py and test_module.py: 95 passed. ruff check and ruff format --check are clean.
• The bugs, on main's code. Two tests ran with their service subprocesses on main (6e3cfd1); both pass on this branch.
  • The two-process test fails with {'build-a': 'missing_huggingface_credential'} != {'build-a': 'pending'}.
  • The held-lease test fails because the service started (assert not True), since main has no leases.
• Lease interop with Microcosm. A local check loaded microcosm#1177's leases.py beside this one:
  • Both give the same lease path for 5,005 id pairs, including empty, unicode and NUL-containing ids.
  • A lease held by either package in another process cannot be taken by the other, and the other's sweep keeps it.
  • SIGKILL frees it.
• Mutation check. 23 runtime mutants were injected through a pytest plugin that patches module globals, so no tracked file was edited. Each ran against the three touched test files, and the tests that target it killed it. The 12 from microcosm#1177 are joined by 5 for this port's wiring and layout, 3 for review round 1, and 3 for the fixes ported from #1177's review. Only no_own_lease had errors: with no lease ever granted, every service now refuses to start, so the 46 test_module.py fixtures error. It is also killed by assertion failures.

Mutant	What it removes	Killing tests
original_bug	both the claim and the gate	21, including the e7 regression and the stale-list test
no_gate	the _make_local_only gate	11: orphan tests, the 409 orphan test, the login-change tests, property
no_claim	lease and idle checks (deliver every pending run)	10: live run (alice, bob), other owner (register 403, events 403), shared run id, lease-less idle, similar ids, session wait-out, property
gate_all_foreign	over-correction: never make a foreign run local-only	2: the 409 orphan test, property
conflict_caller_dependent	409 treated as an answer about the caller	2: the 409 orphan test, property
no_recheck	the state check under the claim	2: the stale-list test, property
no_passed_over	memory of a 403 pass-over	4: other owner at registration ([403, 403, 403] != [403]) and at events, similar ids, session wait-out
no_credential_memo	refused-credential memo	5: orphan-wait and login-change exchange counts (outsider, expired), property
no_credential_refresh	noting the current login before pass-over filtering	4: login change (outsider, expired), session wait-out, property
refresh_ignores_session	over-correction: a new login counts while a session is live	session wait-out ([403, 403] != [403])
hold_gives_up_on_error	retrying a lock error until the deadline	2: lock refused once at startup, own lease waits out refused locks
serve_without_own_lease	refusing to serve without the own lease	2: the module that cannot take its lease (locks refused, lease held): DID NOT RAISE
joined_run_keys	the id pair as the per-run key (the old "run_id:producer_id")	2: the two similar-ids tests
no_own_lease	any own lease at all	36 failures (most raise OwnLeaseUnavailableError at start; lease exclusivity and sweep assert) and 46 fixture errors
always_idle	the idle wait for lease-less runs	lease-less idle test
drain_everything	main's drain over every run	drain test (60 flushes for the other build's run)
close_keeps_lease	releasing the lease in close	15, including the module lease test
failed_start_keeps_lease	main's initialize failure path	failed-start test
sweep_held	sweep safety	13
trust_replaced_file	the inode check	replaced-file test
unchecked_lease_leak	closing the descriptor when the inode check raises	unchecked-lease test
lease_name_without_nul	the lease name Microcosm uses	4: the layout pin, and three tests whose child process or held lease uses the real layout
warn_for_foreign	"this run" warning scoped to the own run	the 409 orphan test

• Deep property run. 2,000 examples (CI runs 150) passed, with 0 failing and 42 invalid. Share of examples reaching each path:

Path	Share
Adopted orphan, register 201	77%
Adopted orphan, events 202	73%
Adopted orphan, register 409	17%
Adopted run local-only for that 409	17%
Adopted orphan, register 403 (another user's run)	13%
Adopted orphan, events 422	6%
Adopted orphan, events 403	2%
Own run local-only: credential rejected	29%
Own run local-only: no credential	18%
Own run local-only: registration rejected	7%

• CI. All jobs passed on 07b2c36 and on each earlier head except the first: quality, tests on Linux and macOS with Python 3.13 and 3.14, the package build, and the real collector against PostgreSQL. 231a26a failed only in the collector job, because tests/test_collector_contract.py built CollectorDelivery without the new registration argument; 0e6328d fixed that test.

Evidence (scripts, logs, mutant plugin and results, review prompts and reports) is in a local folder, ~/reviews/microcosm-emitter-foreign-local-only-2026-10-09/.

Review

Round 1 (GPT-6.1 Sol, at 94ca5c1) returned REQUEST_CHANGES with one should-fix and two nits. The reviewer could not execute (free disk was under its floor), so each was checked here first. 07b2c36 addresses all three.

1. Should-fix, reproduced. A service passed over another build's run under a login the collector refused, and its own run went local-only. After its login changed to the run's owner, it still never offered the run: the pass-over check compared against a credential fingerprint that only a delivery refreshes, and nothing was being delivered. Three new tests fail on 94ca5c1 and pass now. The property test's last two services now start under generated logins before becoming Alice and Bob, so it covers this path; with the fix removed it fails.
   • The reviewer also suggested dropping a cached session when the login changes. This PR does not: a session the collector issued stays in use until it expires, which is what main does for the own run, and a test pins it.
2. Nit. test_delivery.py left its deliveries' leases held; a fixture now closes them.
3. Nit. The "justifying answer" assertions accepted any earlier answer. They now require the deciding service's latest answer to be the refusal, and the wording here matches.

Also from its notes:

• leases._lock now closes its descriptor if the inode check itself raises.
• The two-process test no longer claims build B made two passes: A's exchange is held until B's first pass has made B's own run local-only, and for 1.5 s more.
• Invariants 2, 3, 6 and 8 and the 409 sentence are scoped to what is checked.

Round 2 (Sol, at 07b2c36) approved with no findings. It executed the recovery path across refused logins, a None login, session expiry and 401s, plus all 25 rescue-login pairs and a differential of the own run's requests against main.

After that, the review of microcosm#1177 found two more defects in the code this PR copies. 093efb1 ports both (see Change), each with a test that fails on 07b2c36:

• Serving without the own lease. A service whose first lock attempt erred ran without its lease. The attempt had left a lease file nobody held, and other services adopted the live run.
• Colliding run keys. Run keys joined the ids with a colon, and ids may contain colons.

Round 3 is a delta review of 07b2c36..093efb1.

Limits and follow-ups

• Older services. A service from a version without this change still runs the old logic on shared rows. Until those update, it can still mark other builds' runs local-only, or deliver a live build's run under its own login. New services treat an older service's run (no lease file) as live until it has been idle for 15 minutes.
• Platforms without fcntl. There are no leases there: a service adopts only idle runs and has no exclusion. The host already needs Unix sockets, so no supported platform lacks fcntl.
• microcosm#1177 is still open. This port matches its head 615d84f89 apart from the #1147 lines and the wiring above. If its review changes the design again, this port should follow.
• Add durable Orrery publication and graph readers to the local provider #4 (draft, conflicting) edits the collector, module and spool under an older packages/provider-telemetry layout; whichever lands second rebases.

axiom: n/a: telemetry infrastructure

🤖 Generated with Claude Code

[MaxGhenis]

Two more findings from microcosm#1177's review round 1 that apply to this port. The code here at 07b2c36 is the same as Microcosm's was at 662fd4acb, and both were executed there by the reviewer (GPT-6.1 Sol). Microcosm fixes them in 1c9bd671b; each has a test that fails before the fix.

1. A failed own-lease lock leaves a live run adoptable. ProducerLeases.hold returns None on the first OSError, and the delivery then runs without a lease. But the failed attempt has already created the lease file (os.open(O_CREAT) comes before flock), and a lease file nobody holds is how other services recognise an exited producer. With an injected ENOLCK at A's start, B took the free file, registered A's live run under B's login and uploaded it; A's own registration then got 403 and A's run went local-only. The 15-minute idle rule never applies, because the file exists. The same holds when another process still has the lease at the deadline.
   • Microcosm's fix: hold repeats an attempt that raises OSError until its deadline, and a service that still has no lease does not start (OwnLeaseUnavailableError from the delivery's constructor; main() prints one line and exits 1). Only a platform without fcntl runs lease-less. Here, TelemetryModule.initialize already releases everything when the delivery raises, so the same raise would leave the build without telemetry for that run.
2. Run keys collide. _registration_key joins run_id:producer_id, and the collector's id pattern (^[A-Za-z0-9][A-Za-z0-9._:-]{0,191}$) allows colons in both. So ("a:b", "c") and ("a", "b:c") share a key:
   • in _passed_over, Bob's 403 for Alice's a:b/c passed over Bob's own deliverable a/b:c (reproduced);
   • in _registered, registering one counts for the other, so the second run's registration is skipped, its events get 404 and the run goes local-only (reproduced in Microcosm's tests).
   • Microcosm's fix keys all three per-run memories (_passed_over, _registered, _warned_denied) by the (run_id, producer_id) tuple. The lease path is unaffected: it hashes run_id NUL producer_id, and ids cannot contain NUL.

The lease files are unchanged, so services from both packages still recognise each other's leases.

🤖 Generated with Claude Code