Node identity without a coordinator: contain a host compromise to one node #489

Activity

1. CMGS commented on Nov 7, 2026

   

   CMGS

   on Nov 7, 2026

   ContributorAuthor

   More actions

   Review against main ef58ef76. The "What a compromised node holds today" table matches the code: merge compares only the epoch (sandboxd/mesh/mesh.go:285), the adopter sends the claim token to every peer it tries (sandboxd/store/peer/adopt.go:24) and the adopter stores it (sandboxd/pool/failover.go:80), checkTemplateOwner runs on the writer (sandboxd/pool/template.go:295), and putRow writes Env unredacted (sandboxd/pool/failover.go:138). The egress paragraph also holds: the root key stays offline and nodes hold only intermediates.

   The design leaves six gaps. Gaps 1 and 2 conflict with the stated goal.

   1. Ownership still rests on rows a compromised node can write (§5, §6)

   §6 accepts a manifest when its signer is the owner of record, and the owner of record is the archive row in PostgreSQL, which every node of the cell can write. Node X can rewrite any archived claim's row to owner = X and checkpoint = <a disk X signed>, and keep token_sha256. When the victim wakes the claim, an honest node adopts it with the victim's own token, the hash matches, the signer equals the owner of record, and the claim boots X's disk. That replaces another node's checkpoint, which the design says X cannot do.

   Proposal: bind each record to the claim token, not to the row. Every archive and checkpoint record carries mac = HMAC(claim_token, record_digest), where the record names the claim id, the checkpoint digest, a sequence number and the previous record's digest. A reader verifies the MAC with the token it holds, from the client or from the handing-over owner, before it boots anything. PostgreSQL becomes an untrusted pointer. A node that never held the token can only break availability, so the rule needs no owner-of-record lookup and no liveness evidence, and it also covers adoption after the owner is gone.

   2. The tun exit trusts any node for any claim address

   The exit's vxlan0 runs with learning and arp_accept=1 (docs/advanced-network.md, single-host exit). X can send frames whose inner source is another node's claim. The exit then learns that MAC behind X's VTEP and sends the claim's return traffic to X. On a bare-metal L2 underlay, X can also spoof the exit's VTEP address toward another node's tun.listen and inject inbound frames. Either way X reaches guests it does not host.

   Proposal: run the exit with nolearning and static FDB entries that an exit-side agent programs from signed node states. Each node lists its claims' tun addresses and MACs in its signed NodeState, and the agent drops frames whose inner source a VTEP did not sign for. Bare-metal spoofing needs underlay anti-spoofing (switch port security) or an authenticated underlay between nodes and exits. If that is out of scope, the advanced network mode belongs under Residual risk.

   3. Signed state proves origin, not truth; tenant tokens remain cluster-wide bearer secrets

   X signs its own NodeState truthfully as far as signatures go, but it can inflate its warm counts, templates and volumes to draw new claims. An SDK that talks to nodes directly then hands X its tenant token, and that token is valid on every node. For example, POST /v1/checkpoints/{id}/claim lets the holder read every checkpoint of that tenant across the cell. §4 removes this only for control planes, and Residual risk does not list it.

   Proposal: give tenants a credential that never reaches a node as a bearer, for example a tenant ed25519 key whose public half sits in the tenant record, used through a client certificate or a request signature bound to the node id and a timestamp. In any case, state under Residual risk that a compromised node can attract a disproportionate share of new claims.

   4. The §3 table widens the tenant token

   The table grants tenant: own for exec, relay, ports and files. Today those accept only the sandbox token or the operator token (sandboxd/server/exec.go:47, sandboxd/pool/operator.go:91). Granting them to the tenant role makes gap 3 worse. These rows should stay no for tenant.

   5. The §5 liveness rule is absence-based

   The rule says "accept a new owner when no verified fresh state exists", and absence of evidence is what an attacker controls. X can withhold newer states or replay older ones. X can also declare Y dead at the SWIM layer, and forget() then deletes Y's state from the view (sandboxd/mesh/mesh.go:265) until Y refutes it. If the client asks X itself, the check fails open. With proposal 1 this rule is no longer needed for integrity. If it stays, the evidence should come from a direct, nonce-bound challenge to the previous owner (GET /v1/identity?nonce= returning a signed At and the nonce), never from the claimant.

   6. Delete-then-create replaces a tenant template

   The same-signer rule compares a new version against the version currently in the store. X can delete objects (accepted as availability), and after a delete no current version exists, so X's version is accepted as the first one. Either pin the signer per template name outside the store (for example, each node remembers the signer and requires a template signer to change it), or list this under Residual risk next to the "name not used yet" item.

   Not in the work breakdown

   • Adoption without broadcasting the token. The plan replaces the token with its SHA-256 on /owner. The adopter then never holds the plaintext token, while claim auth compares plaintext (sandboxd/pool/operator.go:91), so every token check would have to move to hashes. That also gives X a new move vector: copy the hash from PostgreSQL into a move's claim record. A simpler fix is a two-phase offer. Offer with the hash, and send the token only to the one peer that accepts. Refusing peers learn nothing, auth stays as it is, and the accepting node can verify the proposal-1 MACs.
   • Config validation. checkpoint_peer_heal, move_concurrency and archive_failover require mesh.cluster_key (sandboxd/config/config.go:553, :563, :838). The design makes that key optional.
   • Probe MAC. The probe MAC derives from cluster_key (sandboxd/store/peer/probe.go:150), so it has to move to node identity.

   Suggested order

   1. Independent of identity, small, can ship now: §8 env redaction, the two-phase adoption offer, and the §3 table correction.
   2. Token-bound records (proposal 1): this closes the store-integrity gap without any PKI.
   3. Node identity, signed gossip and TLS roles as designed: with §5 reduced to the challenge form or dropped.
   4. Exit hardening (proposal 2) and tenant key credentials (proposal 3): or an explicit Residual risk entry for each.
2. CMGS commented on Nov 7, 2026

   

   CMGS

   on Nov 7, 2026

   ContributorAuthor

   More actions

   All points verified against ef58ef76 and folded into the description.

   #	Finding	Verdict	Change
   1	Owner of record comes from rows any cell node can write	correct	§6 now binds every checkpoint, archive and replica record to the claim token: HMAC(claim_token, record_digest) over claim id, manifest digest, sequence and previous digest, verified before boot. PostgreSQL is an untrusted pointer. The node signature stays for attribution, sweeps and revocation only.
   2	tun exit learns MACs and trusts any VTEP (advanced-network.md:237, :242)	correct	Under Residual risk; static FDB from signed node states listed as later work.
   3	Signed state proves origin, not truth; tenant tokens stay cluster-wide bearers	correct	Under Residual risk (claim attraction and tenant token exposure to directly connected SDKs); tenant key credentials listed as later work.
   4	§3 table widened the tenant token	correct, my error (exec.go:47, operator.go:91)	exec, relay, ports and files are no for tenant.
   5	Absence-based liveness rule	correct (mesh.go:265)	Removed. Integrity no longer depends on liveness; the signed heartbeat is gone from §2.
   6	Delete-then-create resets the same-signer rule	correct	Each node remembers the last signer per template name and needs a template signer to change it; the remaining case (a node that never saw the name) is under Residual risk.
   —	Hash-only adoption breaks plaintext claim auth	correct	§5 uses the two-phase offer: hash to every peer, token only to the one that accepts.
   —	cluster_key-dependent config and probe MAC	correct (config.go:553, :563, :838; probe.go:150)	Added to the work breakdown and tasks.

   Rollout follows the suggested order: env redaction, two-phase offer and the table fix first; token-bound records second; identity, signed gossip and TLS roles third; exit hardening and tenant keys later or as residual risk.
3. CMGS commented on Nov 7, 2026

   

   CMGS

   on Nov 7, 2026 – with ChatGPT Codex Connector

   ContributorAuthor

   More actions

   Follow-up review of the revised description against main@ef58ef76d5daf02dd0a409069609d7deabad2a41.

   The architecture is feasible, but the current specification does not yet contain a host compromise to one node. The findings below use this issue's threat model: an attacker already controls host root on one admitted node. The existing residual risks for shared-store confidentiality, availability, tenant bearer exposure, and tun remain separate from these findings.

   The first five items should be treated as P1 blockers for the proposed isolation guarantee. Items 1, 2, 3, and 5 are reachable paths in the current source that the design must close; item 4 is a flaw in the proposed operator authentication protocol.

   1. S3 manifest paths can write outside the receiving node's staging directory.

      Store.Fetch → populate → materialize consumes a manifest from the shared store. materialize joins file.Path to its local root and calls MkdirAll, Create, and Truncate without rejecting paths that escape that root. A compromised node with the shared store credentials can supply a traversal path and make another node create, truncate, or overwrite files outside its download directory when that node fetches the record. A later boot failure does not undo those writes.

      Evidence: Fetch, manifest decoding, materialization. The template path also calls Fetch before checking the returned template's visibility: resolveGolden.

      Required change: confine every materialized path to the staging root, reject invalid manifests, and bound file counts and sizes. Verify the authenticated record and manifest before materialization. “Verified before boot” is too late if verification follows Fetch. Path confinement remains mandatory even for signed records: a malicious admitted node can sign malformed data for a claim whose token it legitimately holds.

   2. SDK owner discovery still broadcasts the claim token.

      Go Client.Lookup probes the entry node and then scatters requests to peers. Each ownerAt request sends the plaintext claim token. Automatic rebinding after a move invokes that same lookup. Python lookup likewise probes the entry and peers with the token.

      Evidence: Go lookup and ownerAt, automatic rebind, Python lookup.

      The two-phase offer in §5 changes the server-side adopter, but does not cover these SDK paths. TLS and a signed address prove that the recipient is an admitted node; an admitted but compromised peer can still collect a token belonging to a claim on another node.

      Required change: make owner discovery secret-free and require evidence of authority over the specific claim before sending its bearer. A node's signed assertion that it is the owner is insufficient. An existing owner can prove possession of a claim-specific secret through a nonce-bound exchange; a new adopter needs an explicit authorized transfer. The hash already readable from PostgreSQL cannot serve as the secret for that proof.

   3. The cluster-wide preview_secret remains a lateral credential.

      A compromised node holds the shared preview HMAC key. For a known claim ID on another node, it can forge a preview token containing {id, port, owner, exp}. The target validates the shared HMAC, then PreviewDial accesses the guest port using Cred{Operator: true}, without a claim token.

      Evidence: preview signing and verification, preview request handling, PreviewDial. The deployment documentation explicitly defines preview_secret as cluster-shared.

      Required change: replace the shared signing authority with a capability scoped to the claim, including verification after migration. Merely accepting signatures from any admitted node preserves the same authorization flaw. The compatibility statement that previews remain unchanged conflicts with the isolation goal.

   4. An operator OIDC bearer with a common audience can be replayed across nodes.

      Under §3, every node accepts the operator ID token with aud = "sandboxd". Once the control plane sends it to a compromised node, that node can replay it to another node until expiry and exercise operator privileges, including retrieving claim tokens. The statement in §4 that a redirect to a compromised node “harvests nothing” therefore does not hold for this credential.

      Required change: use operator mTLS for the initial implementation, or define a sender-constrained token protocol or a strict recipient-specific audience scheme that prevents cross-node replay. A short expiry only limits the attack window. This follows the replay threat and mitigations described in RFC 9700 §4.10.

   5. The shared tenant table is still a writable authentication and authorization authority.

      The tenant store uses the same configured DSN as the other shared backends, and tenant authentication directly resolves sandboxd_tenants.token_sha256. A compromised node with that database role can replace a tenant's hash with one for an attacker-controlled token. An honest node then resolves that token as the victim tenant. This permits tenant-authorized operations on the victim's claims, such as modifying claim env; it does not require the SDK to send the victim's original tenant token to the compromised node.

      Evidence: backend wiring, tenant store wiring, token resolution, tenant record writes, env authorization.

      Required change: protect tenant identity, quota, and policy records with operator authorization that a node cannot forge. Options include externally signed records verified by every reader, or a separate write authority whose credentials are absent from sandbox nodes. If node database access becomes read-only for these tables, the administrative write path must change accordingly. Adding MACs to checkpoint records does not protect the tenant table.

   The record and identity protocols also need the following decisions before implementation:

   Area	Gap and required decision
   MAC key availability for ordinary checkpoint cloning	Checkpoint.New sends the tenant/operator credential, not the source claim token. A checkpoint handle does not retain that token, and the source claim may already have been released. See checkpoint claim and handle construction. Define a checkpoint-specific capability or another explicit means to authorize and verify this operation. The archive/adoption token flow does not establish compatibility for ordinary checkpoint cloning.
   Rollback protection	A sequence number and previous-record digest authenticate a chain, but a valid old prefix still verifies. A new adopter with no trusted latest sequence/digest cannot distinguish that prefix from the current state. Retain a trusted latest-record anchor at the client or control plane, or explicitly list rollback as residual risk.
   Authenticated record coverage and verification order	Define the authenticated envelope to bind the cell, record type, claim ID, tenant, manifest, env, policy, deadlines, and security-relevant lifecycle state. The current adoption path deserializes the shared row's claim and installs it locally: adoptArchived. Verification and consistency checks must precede those state changes, not just VM boot. The four fields currently named in §6 do not specify protection for the full claim state.
   Revocation by since	A compromised signing key can sign new content with an old timestamp. A signer-supplied date cannot establish that a record predates the compromise. Without an independently trusted time/history proof, revoke the key and preserve approved historical content through exact-digest exceptions or operator countersignatures.
   Historical keys and admission continuity	A restarted node's current /identity endpoint does not establish the provenance of every old ephemeral key. A new verifier also has no previous admission state for a running peer whose original proof has expired. Retain historical public keys and their proofs independently of the writer's lifetime, define historical verification separately from live admission, and refresh the proof for a running key when needed by new verifiers. Continuity for existing verifiers alone does not solve either case.
   Node identity namespace	A bare GCE instance name is not a sufficient identity across zones, projects, and instance replacement. Use a provider-scoped identity incorporating project, zone, and instance ID; likewise namespace X.509 identities by their trust domain and enforce subject uniqueness. Google documents instance_id, project_id, and zone as the globally unique tuple to verify.

   Suggested implementation order:

   1. Fix S3 materialization confinement, SDK claim-token disclosure, and preview signing scope.
   2. Establish X.509 node identity, operator mTLS, signed gossip, strict endpoint roles, and protection for shared authentication/authorization records. Add OIDC once its replay protection is specified.
   3. Implement checkpoint capabilities, complete record authentication, the chosen rollback guarantee, historical verification materials, and revocation.
   4. Roll out the compatibility changes. Enable env redaction together with a defined secret re-injection/readiness flow for recovery. A cell must not claim the new isolation guarantee while legacy root bearers, unsigned records, plaintext credential paths, or token-broadcast fallbacks remain enabled.

   This can retain gossip and support bare metal without adding a central service. Local identity verification is feasible; ownership authority, record freshness, and revocation still require explicit trust rules.

   Validation scope: source and protocol review only. No code changes, runtime tests, or exploit reproduction were performed.
4. CMGS commented on Nov 7, 2026

   

   CMGS

   on Nov 7, 2026

   ContributorAuthor

   More actions

   Decision: the cell is the blast radius

   After both reviews, the design is narrowed to keep it simple. A host compromise on one node is bounded to its cell, ops can cut a node or a cell off without that node's help, and no new protocol or central component is added. Cell size is the operator's knob.

   Kept

   • Fix the S3 materialize path confinement now (review 2, item 1), as its own PR.
   • Scope every shared credential and backend to one cell: control-port token, mesh key, preview secret, checkpoint store location and credential, cloud identity.
   • One PostgreSQL database (or schema) per cell. sandboxd_tenants has no cell column, and the other tables share one role across cells, so a shared database would not bound a cell.
   • Cutoff: backends and node-to-node ports accept only the cell's node addresses, so blocking an address cuts a node off at once; a reloadable deny_nodes list drops a node from routing and peer calls; then rotate the cell's credentials.
   • Docs: deploy.md and security.md describe these values as shared within a cell, plus a cutoff and rotation runbook.

   Dropped
   Node attestation and ephemeral keys, signed gossip, record MAC chains, template tiers, revocation timestamps and OIDC operator tokens. The findings they addressed stay inside the cell and are accepted there; per-node roles, per-node store prefixes and mTLS remain options if a cell cannot be made small enough.

   The description above is the earlier, wider design and is kept for reference; this comment is the current scope.
5. CMGS commented on Nov 7, 2026

   

   CMGS

   on Nov 7, 2026 – with ChatGPT Codex Connector

   ContributorAuthor

   More actions

   Re-review against the cell-boundary decision, with source references at main@ef58ef76.

   The revised scope is reasonable and implementable. Treating the whole cell as potentially compromised removes the need to solve the earlier per-node identity, ownership, and record-authentication problems in this issue. The remaining work is to enforce the boundary between cells and make the cutoff and recovery procedures accurate.

   Three requirements still need to be explicit:

   1. Blocking an address does not, by itself, guarantee immediate cutoff.

      GCE and AWS stateful firewalls can retain established connections after rules change. A change that blocks new connections may leave PostgreSQL sessions, HTTP keep-alive connections, relays, and transfers already in progress usable. See GCE firewall connection tracking and AWS security group connection tracking.

      The runbook needs a containment mechanism enforced outside the compromised host, with explicit handling of established connections and database sessions. Define a measurable cutoff condition rather than assuming that a rule update has ended access.

      The same condition must cover credentials copied off the host. If an object store has a public endpoint that still accepts the stolen credentials, blocking the original node's network access does not cut off the attacker. Restrict access at the service boundary and revoke or rotate credentials as required. A private route alone does not disable public endpoint access; S3 documents this distinction.

      deny_nodes is useful for stopping placement and outbound peer calls. It is not an authorization-revocation mechanism. With signed node identity removed, a compromised member can change its self-declared node ID. Network containment must also prevent source-address spoofing where the underlay permits it.

   2. Separate database/schema and store locations must have separate permissions.

      A schema name, search_path, or S3 prefix selects a namespace; it does not restrict what a stolen credential can access. PostgreSQL explicitly notes that access across schemas depends on privileges: schema documentation.

      Make the following deployment requirements explicit:

      • Each cell has a distinct PostgreSQL role, restricted to its own database or schema and objects, with no privilege to assume another cell's role.
      • Each cell's object-store credentials are restricted to its own objects by the backend's authorization policy.
      • A node's cloud identity cannot read another cell's secrets or change the IAM, network, or firewall controls that enforce the cell boundary.
      • “Every shared credential” includes tenant bearer tokens. Reusing the same tenant bearer across cells reconnects those authorization domains once one cell obtains it.

      These are requirements for the new deployment contract, not a request to restore the dropped record-signing design.

   3. The tun exit must also respect the cell boundary.

      If multiple cells share one VXLAN learning domain, a compromised node in cell A can claim cell B's inner address/MAC and attract B's return traffic. The current exit example enables VXLAN learning and arp_accept=1. The earlier accepted network risk is confined to a cell only if the exit topology confines it there.

      The simplest deployment rule is a separate exit network domain, FDB, and neighbor table per cell, with trusted ingress controls binding node sources to the permitted overlay. Different VNIs alone are insufficient: host root can construct a packet with another VNI. Bare-metal deployments also need underlay anti-spoofing outside the compromised host. The existing documentation already requires that other hosts cannot spoof exit VTEP addresses.

   The earlier findings now have these dispositions:

   Earlier finding	Disposition under the cell scope
   S3 manifest path traversal	Still requires the independent fix already retained in the decision. The reviewed source still joins an untrusted path and creates/truncates the destination without confinement: materialize.
   SDK claim-token broadcast, shared preview signing key, writable tenant authentication table	Accepted within a cell. Cross-cell credential and backend isolation must make their consequences stop at that boundary.
   Operator OIDC replay	The proposed OIDC mode was dropped, so this no longer blocks the revised scope.
   MAC-chain freshness, historical signing keys, timestamp-based signature revocation	The associated protocols were dropped, so these no longer block the revised scope.

   Rotation and recovery need an implementation-aware runbook. At the reviewed commit, changing api_token, mesh.cluster_key, or preview_secret requires a restart; an ordinary config reload rejects those changes. See ReloadDiff and the reloadable field set. Document the restart sequence and resulting service interruption, or implement the rotation behavior explicitly.

   Because the accepted blast radius is the entire cell, rotating keys alone does not establish that its remaining hosts are trustworthy. Recovery must account for possible compromise of other nodes and tampering with that cell's templates and checkpoints. Do not automatically restore trust in those artifacts merely because the old node has been blocked.

   Suggested acceptance checks for this narrower scope:

   • Possession of all cell A credentials still does not permit access to cell B's control API, preview authority, database objects, object store, or cloud resources.
   • Cutoff ends already-established access as well as new access, within the documented bound, without cooperation from the compromised node.
   • Credentials copied to an external host cannot bypass the containment policy.
   • A cell A node cannot inject into or attract traffic from cell B's overlay through a shared exit.
   • The rotation/recovery exercise uses the actual restart requirements and a defined source of trusted recovery artifacts.

   I recommend keeping the simplified direction and incorporating these requirements into the current decision and implementation checklist. Replace the issue's active goal and task list with the cell-scoped version, or move the older design into a clearly separate historical section, so the implementation cannot accidentally follow the superseded per-node requirements.

   Validation scope: source and design review only. No code changes, runtime tests, or inspection of deployed IAM, database grants, or firewall policies were performed.
6. CMGS commented on Nov 7, 2026

   

   CMGS

   on Nov 7, 2026

   ContributorAuthor

   More actions

   Review 3 is folded into the description, which now leads with the cell scope; the per-node design sits in a collapsed history section.

   • Cutoff: host-level drop ahead of the established-connection rule, conntrack flush, and termination of PostgreSQL sessions, with a measurable done condition. Backends refuse the cell's credentials from outside the cell. Per-node PG roles and store keys are recommended so one node can be cut off without restarting the rest. deny_nodes is dropped: node ids are self-declared, and once the address is dropped SWIM removes the node from routing anyway.
   • Separate permissions: per-cell database and role, store credential scoped by backend policy, and cloud identity without access to other cells or to boundary controls; tenant tokens are not reused across cells. Also added: an egress root per cell, since guests trust the root from node config.
   • tun exit: one exit per cell with its own netns, FDB, neighbor table and underlay allowlist.
   • Rotation: confirmed restart-only in config/reload.go; the runbook restarts the whole cell at once. Rotation without downtime is listed as an undecided option.
   • Recovery and acceptance checks: added as suggested, plus a cell-isolation check for the egress intermediate.

   No further sandbox code is required for this scope; the remaining tasks are docs and the runbook.
7. CMGS commented on Nov 7, 2026

   

   CMGS

   on Nov 7, 2026

   ContributorAuthor

   More actions

   Host-only claim env at rest, split into issues (reviewed against a2a0f366): values persist only in claims.json and the archive/replica rows; never in the checkpoint store, the guest, the journals or any API read.

   • sandboxd: a missing state for host-only env entries #492 missing state for host-only entries (first)
   • sandboxd: claims.json stores no host-only env values #493 claims.json without host-only values
   • sandboxd: archive and replica rows store no host-only env values #494 archive/replica rows without host-only values
   • packaging: disable core dumps for sandboxd #495 disable core dumps
   • docs: prefer claim-supplied secrets over node-level secrets #498 docs: prefer claim-supplied secrets
   • decisions: decision: drop host-only env values when a claim hibernates #496 drop values at hibernate, decision: store claim tokens hashed in claims.json #497 hash claim tokens at rest

   Control-plane counterparts: loomhand/cocoonbox#223 (re-push on env_missing, must ship before #493/#494), #224, #225, #226.
8. CMGS commented on Nov 7, 2026

   

   CMGS

   on Nov 7, 2026

   ContributorAuthor

   More actions

   Done for the cell-scoped plan:

   • S3 manifest path confinement: sandboxd: reject S3 manifests whose paths leave the fetch dir or exceed the record bounds #490
   • cell contract, cutoff, rotation and recovery docs: docs: the cell contract, cutoff, rotation and recovery runbook #491 (+ b679e778 version fix)
   • host-only env kept off disk behind host_env_memory_only (off by default, restart only): sandboxd: a missing state for host-only env, kept off disk behind host_env_memory_only #499, closing sandboxd: a missing state for host-only env entries #492 sandboxd: claims.json stores no host-only env values #493 sandboxd: archive and replica rows store no host-only env values #494 packaging: disable core dumps for sandboxd #495 decision: drop host-only env values when a claim hibernates #496 docs: prefer claim-supplied secrets over node-level secrets #498; decision: store claim tokens hashed in claims.json #497 not planned

   Rollout order for #499: release with the flag off, deploy the control-plane re-push (loomhand/cocoonbox#228), then turn the flag on per node with a restart. The superseded per-node design stays in the collapsed section for history only.
9. CMGS

   closed this as completedon Nov 7, 2026