[anth-volk]

Fixes #35

Stacked on #34. The base is claude/modal-telemetry-token; GitHub retargets this PR to main once #34 merges. This branch contains #34 up to a6cf212, through the merge commits 85ee23d, 98dcbab and c99c53e (see "Merges of #34"). This repository's pull request workflow runs for every pull request, so CI runs here too: all 10 checks pass on 132be00.

Summary

A build on Modal leaves undelivered events in its spool database when its given collector token expires, is refused, or the collector is unavailable (#34 keeps those runs pending). The Modal runner will copy that spool to the runs volume. This PR adds the two commands that report on and deliver such a copy on the operator's workstation.

1. microcosm-emitter status --spool <path> prints {"runs": [{"run_id", "producer_id", "upload_state", "reason", "undelivered_events"}]}, sorted by run and producer. reason is the run's local_only_reason column: why a local_only run is excluded, or why a pending one waits (collector_token_expired, collector_token_rejected). It makes no network request and does not change the file (see "Pruning").
2. microcosm-emitter deliver --spool <path> [--time-limit-seconds N] delivers the copy's pending runs with this machine's own Hugging Face login, then prints the same JSON and exits.
   • File first, then login. It first checks the file as status does: this user's permissions, then, in SQLite's read-only mode, that it is a spool database of a revision this emitter knows (exit 9 otherwise, file unchanged). It then exchanges the login, and never uses a collector token given to builds (CollectorSession(origin, use_given_token=False)). Without a login, with a saved login it cannot read, with a refused or unsendable login or with an unreachable collector, it exits before opening the file for writing. Otherwise the existing delivery code would make every run in the copy permanently local_only with missing_huggingface_credential or huggingface_credential_rejected.
   • A login that fails partway. After a 401, the session exchanges the login again. If the operator has since logged out or lost access, or the saved login can no longer be read, deliver sends nothing more, leaves every run not yet delivered pending, and exits 3 or 4. _CopiedSpoolDelivery answers the session itself for this, instead of inheriting the build's handling, which made those runs local_only and exited 0.
   • A copy left with an unfinished write (see the section of that name): deliver checks a recovered copy first, then rolls the write back and delivers; status exits 9 and says why.
   • A copy from another machine. The spool records no paths for telemetry, so a copy delivers from any directory. The run was registered at the collector by the container, with a token obtained from the same operator's login. The collector accepts a second registration from the same Hugging Face user with the same metadata, and counts events it already holds as duplicates, so deliver can also be repeated. The test drives this end to end: a token from exchange_huggingface_credential, a container spool that registers the run and delivers one event before the collector refuses the token, then a copy in another directory delivered by deliver.
   • A run that belongs to another Hugging Face user. The collector gives a run to the Hugging Face user who registers it first, and answers anyone else with 403. When an operator other than the one who launched the build runs deliver, that run stays pending in the copy, unchanged, so the launching operator can still deliver it. deliver names the run on standard error, says that the operator who launched the build should deliver it, and exits 8. The same test runs deliver with another operator's login first, then with the launching operator's, which delivers every event.
   • Other refusals (409, other permanent 4xx) still make a run local_only, as in a build. Open Keep one build's telemetry login from deciding other builds' runs #5 to Say what the collector refused, fit requests to its limit, register a lost run again #7 change the build's handling of collector refusals. Runs already local_only in the copy are not sent.
   • What it changes beside events: no permissions. It opens the copy with EventSpool(path, as_copy=True), which neither creates the file nor sets the 0600 and 0700 modes that a build's own spool gets. Opening it runs the spool migrations, which bring a copy an older emitter wrote up to date and leave a lock file beside it (events.migration.lock).
3. The message a build prints when its given token stops working now names deliver --spool.

Exit codes

Every command's exit codes are now one table, ExitCode in cli.py, marked @unique: importing the module fails if two members share a value, so a new code cannot take a meaning another command already gives it. The table across this PR and #34:

Exit code	Commands	Meaning
0	all	telemetry-token: the token is printed. status: printed. deliver: nothing it may deliver is pending
1	host	The host could not start. telemetry-token, status and deliver exit 7, not Python's 1, for an unexpected error
2	all	Usage error, including a --spool that is not an existing file
3	telemetry-token, deliver	No Hugging Face login, or a saved login this user cannot read; nothing was sent. deliver: the copy is unchanged, or, when the login is gone at an exchange partway through delivery, the runs not yet delivered stay pending
4	telemetry-token, deliver	The collector refused the login, or the login contains characters that cannot be sent as a bearer token and was not sent. deliver: as for 3
5	telemetry-token, deliver	The collector was unreachable, its answer was not valid HTTP, or it gave no usable token (copy unchanged); for deliver, also: the time limit, which bounds the whole delivery, passed with events still pending
6	telemetry-token	The collector does not accept lifetime_seconds (#34)
7	telemetry-token, status, deliver	An unexpected error; only the error's type is printed
8	deliver	The collector refused this login (403) for a run another Hugging Face user registered; that run stays pending in the copy. Every other run was delivered, or excluded from upload for a 409 or another permanent 4xx
9	status, deliver	Changed from 7. This user lacks permission for the file (status needs to read it; deliver also to write it and to create files in its directory), or it is not a spool database, or one with a revision this emitter does not know (written by a newer release). status also: the journal beside the copy holds a write that did not finish. The message says which

When the time limit passes and a run belongs to another user, deliver exits 5 and prints both messages. A login failure partway takes precedence over both: deliver prints the login's message and that the runs stay pending, and exits 3 or 4. deliver_spool reports a login failure as LoginFailure (missing, unsendable, rejected or unavailable), and maps any other session answer, which its session cannot give, to unavailable; CREDENTIAL_FAILURES has a code and a message for each, and a test checks that. A saved login that huggingface_hub cannot read (OSError or UnicodeDecodeError, with neither HF_TOKEN nor HUGGINGFACE_TOKEN set) raises UnreadableLoginError, at the start or partway; deliver prints the message telemetry-token prints for it, with only the error's type, and exits 3.

Merges of #34

85ee23d merges #34's branch at 25d40c4 with a merge commit. Conflicts:

• README.md: Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34's rewritten bullets on given tokens, followed by this PR's section on a copied spool.
• cli.py: both sides' imports, codes and messages. telemetry-token keeps Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34's exchange_huggingface_credential call and its exit 7 for unexpected failures, with this PR's --development-collector-url argument and its CREDENTIAL_FAILURES table.

The commits after it:

• 738dd51: Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34 removed CollectorSession's environment parameter, through which deliver ignored a given token. After the merge, deliver sent a build's token left in the operator's shell instead of the login. CollectorSession now takes use_given_token (true by default); deliver passes false.
• e12a6a5, 3a99f2f: both PRs used exit 7. Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34 merges first, so 7 keeps its meaning, and the spool file errors exit 9, the next code no command uses. 8 was free and stays. All codes are now the ExitCode table above.
• 30afcac: SpoolDelivery.login took Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34's SessionResult. Round 2 replaced that with login_failure, typed by the four failures deliver reports (536a49b, b32cbf3). 85ee23d, not 30afcac, adapted the tests to Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34: they obtain the build's token with exchange_huggingface_credential and set the given-token variables with Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34's given_token_variables.

98dcbab merges #34's branch at ca37144 (#34's round 2 fixes) with a merge commit. Conflicts:

• cli.py: this PR's ExitCode table and messages, with Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34's unsendable-credential entry (exit 4), unreadable-login message (exit 3) and ExchangeFailure check. CREDENTIAL_FAILURES is typed by the exchange's failure reasons and the session's answers, which telemetry-token and deliver look up.
• README.md: Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34's corrected paragraph on a token that stops working, followed by this PR's section.

After it, b32cbf3 makes the session answer unsendable rather than rejected for a credential that cannot be sent as a bearer token, so that deliver prints #34's message for it rather than saying the collector refused it.

c99c53e merges #34's branch at a6cf212 (#34's round 3 fixes) with a merge commit. #34 now makes the session answer unsendable itself, so that the telemetry module can print its own warning for it. Conflicts:

• core/auth.py: both sides added unsendable to the session's answers; Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34's comment is kept, as it says how each caller handles it.
• cli.py: this PR's ExitCode table and CREDENTIAL_FAILURES, with Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34's corrected wording for the unsendable message and the comment above it.

Also resolved in the merge: telemetry/service/collector.py merged without a conflict, but with this PR's result in ("rejected", "unsendable") ahead of #34's new unsendable branch, which would have hidden #34's warning; the rejected branch matches only rejected again. test_copied_spool checks the unsendable message's new wording.

Orrery's receipts. The Orrery module records the session's answer as credential_<answer>, so a graph job whose Hugging Face credential cannot be sent records credential_unsendable rather than credential_rejected, and is retried with backoff, as before. The telemetry module still makes the run local_only with huggingface_credential_rejected, now with its own warning, and collector submissions still end failed with CREDENTIAL_REJECTED. The session's comment, the README and the changelog (from #34), and a comment where the Orrery queue records the code (132be00), say so.

Relation to #5

Open #5 states the same rule for builds that share one spool: an answer about the caller's credential (no credential, a refused exchange, registration 403, events 403) must not make another producer's run local_only. It does this by changing CollectorDelivery's constructor to take the build's own registration and hold a lease for it, and by adding a caller_dependent argument to _make_local_only.

This PR does not change the constructor. CollectorDelivery now sends both 403 answers through one method, _forbidden, which a build still answers by making its run local_only, with the same reason and warning as before. deliver_spool uses a subclass whose _forbidden records the run instead, writes nothing, and sends nothing more for it, and whose _collector_token stops delivery, writing nothing, when this login is missing or refused. When #5 is brought up to date with main, its caller_dependent=True calls for the two 403 answers go inside _forbidden, and deliver_spool must decide which registration it passes as its own, since every run in a copy is another process's.

Pruning: what EventSpool does to a copy, and what I decided

EventSpool.__init__ runs the spool migrations and then prune(). That deletes events older than 7 days (by the time they were queued in the container), then the oldest events beyond 100 MiB, then runs older than 7 days that have no events left. Opening a copy that way 8 days after the build would delete exactly the events it was copied to keep, before anything is sent.

• status never opens the copy with EventSpool. It opens it in SQLite's read-only mode (core.database.create_read_only_spool_engine), checks that its revision is one this emitter knows, and reads the runs through the ORM. Nothing is migrated, pruned, chmod-ed or created beside it. The test checks the file's SHA-256 and the directory listing are unchanged, with events aged past 7 days, while every socket connection fails the test and a login is set.
• deliver opens the copy with EventSpool(path, as_copy=True), and never calls prune(). An event leaves the copy only when the collector acknowledges it, however old it is. It still migrates an older revision forward, as any emitter would. The test ages the copy's events to 8 days and checks all of them reach the collector, and that the copy and its directory keep modes 0644 and 0755.
• The emitter host's own spool keeps pruning on open, and keeps its private modes, as before.

A copy left with an unfinished write

An emitter stopped during a commit leaves a journal beside the spool database (events.sqlite3-journal) holding the write that did not finish. SQLite rolls it back the first time a connection that may write reads the file; a read-only connection refuses the file instead (SQLITE_READONLY_ROLLBACK). Round 1's read-only check therefore made deliver refuse such a copy with exit 9, where it had recovered it before.

• deliver: when the read-only check fails for that reason, it copies the file and its journal into a temporary directory that only this user may open, lets SQLite roll the write back there, and checks the revision of the result. Only when that passes, and the login works, does it open the copy itself, which rolls back the same write, and deliver it. The copy is never written before then, and the temporary files are deleted. I chose this over opening the copy read-write at once (SQLite's mode=rw), which would roll the write back before anything had shown the file to be a spool database. It costs one extra copy of the file, and only for such a copy.
• status still never writes, and does not recover the write elsewhere either. It exits 9 and says that the journal holds a write that did not finish, that status changes nothing, and that deliver recovers it; before, it said the file was not a spool database.
• The error is recognised by the driver exception's sqlite_errorname; the emitter's source imports no database driver (tests/test_boundaries.py).

emitter#27's submission rows on Modal

The tracker asked whether emitter#27's collector submission rows and their status checks would be lost when a Modal container stops. They can't exist there, so this PR builds nothing for them:

• A collector submission row is created only by the Hugging Face module's stage_dataset_bundle hand-over, and only when the module configuration sets collector_submissions to true (COLLECTOR_SUBMISSIONS_ENABLED_BY_DEFAULT is False).
• No interface release sets it: HuggingFaceSettings.configuration() returns only handover_database_path, and Microcosm starts the emitter only through the interface (microcosm#1410's telemetry_emitter.py passes HuggingFaceSettings()).
• Microcosm main calls stage_dataset_bundle nowhere yet. Under the decision that Modal stays compute-only, a Modal run's bundle is staged and submitted from the workstation after its outputs are copied back (tracker 6.2 · 3), with the operator's login.
• With Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34, a container's emitter gets only a telemetry token, and a session for artifacts:submit has no credential, so even a configuration that switched submissions on could not submit from Modal.

deliver also leaves graph publication jobs in the copied spool alone, and the hand-over database beside it. Their rows name files by container paths, and staging copied-back outputs is tracker 6.2 · 3.

Not in this PR: runs already local_only for a missing credential

A container that has neither a given token nor a Hugging Face credential, for example a build that opts out of obtaining a token, makes its runs local_only with missing_huggingface_credential before any collector request. deliver does not send such runs, and exits 0 if nothing else is pending. Whether deliver should return those runs to pending is decided together with Microcosm's runner PR (tracker 6.4 · 3), which decides whether that opt-out exists. Nothing is built for it here.

Environment variables

No new environment variables. deliver deliberately ignores #34's MICROCOSM_COLLECTOR_TOKEN, MICROCOSM_COLLECTOR_TOKEN_EXPIRES_AT and MICROCOSM_COLLECTOR_ORIGIN (use_given_token=False), so a build token left in the operator's shell is never used; the test sets them to an expired token to check this. It reads the operator's Hugging Face credential as every emitter does (HF_TOKEN, then HUGGINGFACE_TOKEN, then hf auth login's file). No variable chooses where it sends: the origin is PRODUCTION_COLLECTOR_URL, or a loopback --development-collector-url.

Not verified

• The deployed collector's answers are read from microcosm-collector's storage.register_run on main, not exercised against a deployment: a second registration of the same run by the same user with the same metadata gets 201, and one by another user gets 403. The fake collector in tests/support/fake_collector.py follows that rule.
• Until collector task 4.3 · 1 is deployed, no container can obtain the token this PR's copies come from (see Obtain a telemetry-only collector token for a build elsewhere, and use one the emitter is given #34).
• The test builds the unfinished write by turning off SQLite's syncing for one connection, so the journal is marked for rollback from its first write, as it is just before a commit changes the database file. A copy taken from a real interrupted commit was not tried.
• An events batch answered with 403 goes through the same _forbidden method as a registration 403, but no test reaches it. Once the collector has accepted this login's registration of a run, the login owns the run and the producer is registered, so neither of the collector's reasons for an events 403 applies.
• deliver's exit 3 for a saved login that becomes unreadable partway through delivery is not tested; it raises the same UnreadableLoginError as at the start, from the session's credential read.
• The permission check uses os.access. Run as root, it permits every file, so the permission test assumes a non-root user, as on GitHub's hosted runners.

Tests

Red, then green (Python 3.14, macOS, locally):

Behaviour	Red commit and result	Green commit and result
status --spool reads without changes or network	7615fc4: 1 failed (invalid choice: 'status')	4150297: passed
deliver --spool delivers a copy from another machine with the operator's login, without pruning; a missing or refused login leaves it unchanged	38691d8: 4 failed (invalid choice: 'deliver')	2378e22: 4 passed
deliver by another operator leaves the run pending, unchanged, and exits 8; the launching operator then delivers it	fa2b311: 1 failed, 4 passed (assert 0 == 8: the run became local_only with run_registration_rejected, with the organization-membership warning)	c18fa67: 5 passed
Both commands refuse a zero-byte file or other content (then exit 7), and leave it and its directory unchanged	ffa4baa: 2 failed, 7 passed (deliver exited 0 on the zero-byte file; on other content it left a migration lock file beside it)	67805b0: 9 passed
deliver keeps the copy's modes; both commands report an unreadable spool database as a permission error	a1ecf7d: 3 failed, 8 passed (modes became 0700 and 0600; both commands said "not a spool database")	71e0c46: 11 passed
After the merge of #34, deliver ignores a given token left in the shell (two command test modules)	85ee23d: 1 failed, 24 passed (assert 5 == 8: the session counted the given token as expired, 30 s early, and answered token_expired without sending it; the run was kept pending with collector_token_expired until the time limit)	738dd51: 25 passed
Spool file errors exit 9, apart from telemetry-token's unexpected failures (7)	e12a6a5: 6 failed, 5 passed (assert 7 == 9)	3a99f2f: 11 passed
Round 2. A login refused partway through delivery leaves the runs pending, and deliver exits 4	4c763ac: 1 failed (assert 0 == 4: the run became local_only with huggingface_credential_rejected)	536a49b: passed
Round 2. deliver names an unsendable login as such, and exits 4	00e7f82: 1 failed, 3 passed (exit 4, but it said the collector refused the credential)	b32cbf3: 4 passed
Round 2. status and deliver exit 7 for an unexpected error and print only its type; every login failure has a code and a message	41108b7: 2 failed, 4 passed (the ValueError escaped cli.main)	414a3bf: 6 passed
Round 2. A copy left with an unfinished write: status exits 9, says so and changes nothing; deliver checks a recovered copy without writing to the copy, then delivers it	104890e: 1 failed (status said "not a spool database"; deliver also exited 9, checked by hand)	3fa23f4: passed
Round 2. The emitter's source imports no database driver (test_boundaries)	3fa23f4 failed it (import sqlite3)	82685fa: passed
Round 2. The time limit: exit 5, events pending, the status JSON, and a message that names the limit rather than blaming the collector	d8cd05d: 1 failed (exit 5 and the JSON as expected; the message blamed an unavailable collector)	0f3fbe2: passed
Round 3. deliver exits 3 for a saved login it cannot read, and leaves the copy unchanged	67e461a: 1 failed, 4 passed (assert 7 == 3: it reported an unexpected PermissionError)	2f2ea22: 5 passed

a1ecf7d also strengthens two assertions that passed already. status now runs with a login and with every socket connection failing the test. deliver without a usable login now checks that the copy's directory listing and modes are unchanged, which would show the lock file left by opening the copy before the login is checked; an unreachable collector writes nothing, so the digest alone could not. With temporary edits, reverted, both assertions failed for the defect they target.

30afcac changes no behaviour: the two command test modules pass (25 passed). 0f3fbe2 also corrects --time-limit-seconds's help, which said the limit applied only while the collector was unavailable, and the exit-8 description. 8038401 updates the README and the changelog fragment. In round 3, c34ae02 tests that deliver leaves a copy and its journal byte for byte unchanged, and deletes its temporary directory, when the recovered copy fails its check (a revision this emitter does not know); it held already, and fails with the check of the recovered copy removed. 132be00 documents credential_unsendable and the exit 3, and corrects the remaining wording for an unsendable login.

At the head of this branch (132be00), locally: uv run --no-sync pytest -m "not collector" passed in full (828 passed, 9 deselected), and tools/check-quality.sh, tools/check-changelog.sh and tools/check-artifacts.sh passed. The collector job (tools/check-collector.sh, a pinned collector with PostgreSQL) was not run locally.

Follow-up

#37 tracks problems older than this PR and #34. One reaches deliver: a malformed collector answer during delivery (http.client.HTTPException) ends it with exit 7.

Release

The fragment is added. If #34 merges first and releases 0.11.0, merging this releases 0.12.0 of both distributions to PyPI.

🤖 Generated with Claude Code

[anth-volk]

Closing without merging. We decided not to deliver a Modal build's leftover telemetry after the build: that needs a collector outage at the end of the build or a rejected token, since the token outlasts the stage, and recovering it took more steps than late telemetry for a finished build is worth.