Skip to content

broker: delete the credential broker, and keep the machine bearer it hid - #244

Merged
pythonlearner1025 merged 3 commits into
mainfrom
chore/retire-broker
Sep 6, 2026
Merged

broker: delete the credential broker, and keep the machine bearer it hid#244
pythonlearner1025 merged 3 commits into
mainfrom
chore/retire-broker

Conversation

@pythonlearner1025

Copy link
Copy Markdown
Member

What part this touches

A BlitzOS box authenticates two different things, and they were tangled together
in one Go package.

  1. Machine authentication. The box holds a credential from the control
    plane. blitz-cred api-token turns it into a bearer for /agent/*. Agent
    rules and the Git credential helper call it. This is load-bearing.
  2. The credential broker. A separate VM that held members' Claude and Codex
    logins and minted them back to boxes over SSH. A box enrolled into it
    automatically at boot, through the register s6 service.

Both lived in packages/broker, so the second could not be deleted without
first rescuing the first.

The bug

The broker is not a supported authentication mode, and it overrode one that
works. blitz-cred register wrote this into the member's .codex/config.toml:

model_provider = "blitz"

[model_providers.blitz]
base_url = "https://chatgpt.com/backend-api/codex"
wire_api = "responses"

[model_providers.blitz.auth]
command = "/usr/local/bin/blitz-cred-codex"
refresh_interval_ms = 300000

With the broker watcher not running, a fresh token never arrived and every
Codex request returned HTTP 401 from
https://chatgpt.com/backend-api/codex/responses. codex login status still
reported Logged in using ChatGPT, so this did not look like a logout. Every
Lody Codex session shared that home, so the failure was workspace-wide.

plans/BROKER-RETIREMENT.md (PR #198) recorded that incident on 2026-09-02 and
planned the removal in two releases, behind a fleet-convergence gate and a
credential-custody audit.

The fix

The plan's two releases collapse into one, because its gates have nothing to
gate. Both control planes hold zero broker state, queried today:

Database broker_boxes broker_members broker_keys
prod 51bebbfa-… 0 0 0
canary 8a3458ff-… 0 0 0

Canary also reports no machine carrying a broker_box_id. No broker disk holds
a member's credential, so there is nothing to drain and nothing to audit. The
product owner confirms there are no live boxes, so no boot-time repair ships
either.

  • packages/broker is deleted, all 44 files.
  • The register and watch s6 services are deleted. Their nine dependents now
    order behind init-state, which is where register sat in the graph.
  • blitz-register, blitz-cred-claude and blitz-cred-codex are deleted.
  • The claude shim execs the native CLI and mints nothing.
  • blitz-codex-session keeps the Codex shim and loses every broker probe.
  • Cloud bootstrap no longer pokes blitz-cred register.
  • core/registry.ts and its four routes are deleted, with the broker wire
    types, BoxIdentity.isBroker, and the broker key cleanup in the destroy and
    orphan janitors.
  • /boxes/:id/feed and the constant workspace environment route go with them.
    Both were compatibility surfaces for exactly one caller, and that caller was
    the broker.
  • Migration 0053 drops broker_keys, broker_members and broker_boxes, and
    rebuilds machines and boxes without the columns that referenced them.
  • The broker CI job and the broker OCI publish are gone. A credential-helper
    job runs the new module's Go tests in their place.

blitz-cred api-token survives. It moves to packages/box/credential-helper
with the primitives it reaches, and it keeps its name, because agent rules and
the Git credential helper already call it by that name. Claude and Codex read
their own stores under HOME.

The risk trade

Migration 0053 is the point of no return. SQLite cannot drop a column that
carries a foreign key, so the migration rebuilds machines, boxes, the token
families, the credential leases and credential_events. Every row count it
touches is zero in both deployments today, which is what makes the rebuild
cheap. It is still a schema change: take the ordinary database backup first.

A box that still carries model_provider = "blitz" is not repaired. The
plan asked for a boot migration that strips the marked region. The product
owner states there are no live boxes, so that code is not written. A box found
carrying the block is fixed by editing the file, not by a release.

The rejected alternative was the plan's own two-release split. It exists to
protect enrolled brokers and deposited credentials. Both counts are zero, so the
drain window would have protected nothing and delayed the fix that closes the
incident.

packages/broker also held the machine bearer. Deleting the package
outright would have taken /agent/* authentication with it. That is why the
helper moves rather than dies, and why its tests move with it.

Tests

The moved code keeps a test for each property the plan named:

  • a token is returned only after an authenticated probe, unless the control
    plane is unreachable;
  • exactly HTTP 401 triggers refresh;
  • refresh is serialized across processes, with a lock whose inode is not the
    credential file;
  • the credential is re-read while holding the lock;
  • a rotated credential is written atomically only after the server accepts the
    single-use refresh token;
  • response bodies stay capped and are decoded strictly;
  • origins require HTTPS except for localhost;
  • api-token prints the token and one newline, and nothing else.

packages/box/credential-helper: 4 packages, all pass. Two of them —
atomicfile and filelock — carry the ownership and cross-process lock tests
the move had dropped.

test/broker-retirement-migration.test.ts applies every migration to an empty
database, applies migrations through 0052 to a second database, seeds a broker
box, a broker member, a broker key, an assigned machine, a device-code box,
token families, leases and an audit event, then applies 0053. It compares the
two sqlite_master results, checks the surviving rows, and runs
PRAGMA foreign_key_check.

That test had a hole, and it is worth naming. It asserted absence with
.not.toEqual(expect.arrayContaining([a, b, c])), which matches only when every
name is present — so it passed as soon as ONE of the three tables was gone, and
two could have survived unseen. Deleting the broker_boxes drop from the
migration did not fail it. The schema-equality check could not catch that
either: both databases run 0053, so a migration that drops nothing leaves them
equally wrong and still equal. It now asserts each table separately and fails
with broker_boxes survived the retirement migration.

agent-shims.test.ts asserted not.toContain("CLAUDE_CODE_OAUTH_TOKEN")
against the whole shim file, so a comment explaining that the shim no longer
touches that token failed the test. It now strips comments and asserts on code.

Repository gates ran in a Linux container from a clone of this branch, which is
what CI runs (node:22.20.0-bookworm, npm ci, BLITZDEV_MANAGED=1 CI=true):

  • npm run typecheck passes.
  • npm run lint:gate passes at 42 anti-slop findings, one below the previous
    baseline. lint-baseline.json is lowered in this change, never raised.
  • BLITZDEV_MANAGED=1 CI=true npm test passes, exit 0. Control plane
    64 passed (64), box guest tests 28 passed (28), webapp
    118 passed | 12 skipped (130), and every node --test suite reports
    # fail 0.

BLITZDEV_MANAGED=1 is not optional here. Deleting a core/ file needs four
hand-written edits in three files, and the third copy of the manifest lives in
blitzdev-emitter.test.ts, which only runs under that flag. A plain npm test
passed while the managed build was still 100 files against the worker's 99.

The Go module is not built in that image. CI's credential-helper job runs
go build and go test for it on Linux; both pass locally on darwin/arm64.

Deploy

Two artifacts ship this, and one database changes.

The control plane ships in the Worker bundle. The push to main runs
.github/workflows/canary.yml, which applies migration 0053 and deploys.

The box ships in the payload and the image. packages/box/rootfs and the
Dockerfile both change, so the image job derives a new base image rather than
reusing one. A running box converges on its own: blitz-payload polls every
300 s.

Client prod takes the box half at its next v* image release.

Rollback of the code:

git revert <merge-commit> && git push origin main

Rollback of the schema is not automatic. Migration 0053 drops tables, so
restoring a control plane that expects them needs the database backup taken
before the deploy. Treat every failure after 0053 applies as roll-forward.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GocywGPN9b8DrcBXR9C4id

pythonlearner1025 and others added 2 commits September 5, 2026 17:00
The broker is not a supported authentication mode. It enrolled hosted boxes
automatically and it overrode a working native Codex login: `blitz-cred
register` wrote `model_provider = "blitz"` into `.codex/config.toml`, and a
box whose watcher was not running then answered HTTP 401.

`plans/BROKER-RETIREMENT.md` (PR #198) planned this in two releases, behind a
fleet-convergence gate and a credential-custody audit. Both control planes hold
zero broker rows, verified today: prod `51bebbfa-…` and canary `8a3458ff-…`
each report broker_boxes 0, broker_members 0, broker_keys 0, and canary reports
no machine with a broker_box_id. Nothing is enrolled, so nothing needs draining
and no broker disk holds a member's credential. One release does it.

Deleted: `packages/broker` whole, the `register` and `watch` s6 services, the
`blitz-register` boot script, both token helpers, the broker mint in the
`claude` shim, the broker probes in `blitz-codex-session`, the register poke in
cloud bootstrap, `core/registry.ts` and its four routes, the broker wire types,
`BoxIdentity.isBroker`, the broker key cleanup in the destroy and orphan
janitors, the broker CI job and the OCI publish.

`/boxes/:id/feed` and the constant workspace environment route go with it. Both
were compatibility surfaces for one caller, and that caller was the broker.

Kept: `blitz-cred api-token`. Agent rules and the Git credential helper call it,
and it is machine authentication rather than a broker feature. It moves to
`packages/box/credential-helper` with the primitives it reaches, and keeps its
name. Claude and Codex now read their own stores under HOME.

Migration 0053 drops `broker_keys`, `broker_members` and `broker_boxes`, and
rebuilds `machines` and `boxes` without the columns that referenced them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GocywGPN9b8DrcBXR9C4id
Deleting a core/ file needs four hand-written edits in three files.
blitzdev-emitter.test.ts is gated behind BLITZDEV_MANAGED=1, so a plain
npm test does not reach it. Its list and its length now match the other two.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GocywGPN9b8DrcBXR9C4id
The new script shipped 0644. s6 answered `s6-applyuidgid: fatal: unable to
exec /usr/local/libexec/blitz-credential-refresh: Permission denied`, and
only the box image build saw it. Nothing else asserted the mode.

rootfs-exec-bits.test.ts now checks every script under usr/local/bin and
usr/local/libexec. s6-rc.d run and up files stay 0644 on purpose: s6 runs
them through its own launcher.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GocywGPN9b8DrcBXR9C4id
@pythonlearner1025
pythonlearner1025 merged commit a4d94d5 into main Sep 6, 2026
11 of 12 checks passed
@pythonlearner1025
pythonlearner1025 deleted the chore/retire-broker branch September 6, 2026 02:19
pythonlearner1025 added a commit that referenced this pull request Sep 6, 2026
Resolves the broker deletion (#244) against the periodic CLI updater:
the claude shim stays exec-only, the payload list keeps the
agent-cli-update service and drops the register dependency, and the
docs describe the update service as the model delivery path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01So15XYnmh9Hfx9xdNgG1Yu
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant