[samgbafa]

Scope and merge order

Combined atomic merge vehicle for approved #310 plus its recovery delta. This branch is stacked directly on #310 head 87b2a38 (parent approval); recovery head 63d11dd. Target main intentionally includes the parent changes so there is no intermediate #310-only main state. Do not merge #310 alone. #309 remains separate for the later logging window. No merge or deployment authorization is requested here; review routing stays with Amos and the same Astra reviewer.

Delta: docker.yml, reviewed quota config, two offline rollback config artifacts, a CI-only quota helper, focused tests and release documentation. No app/Rust/root-compose edits. Amos explicitly approved the separate CI helper; only docker.yml invokes it operationally, using fixed arguments and runner env, never expression interpolation or credential argv.

Deployment routing

All deploys require manual dispatch and authorize_quota_values=true. Missing authorization, missing/unknown reviewed authority, missing runner billing/admin credentials or incompatible billing URL rejects before compose/CVM/whitelist mutation.

Family	Compose selection and checks
Same-image recovery	Current main compose, exact pre-N0 tag/pinned equivalents or exact reviewed N0 bytes; no N0-only quota opt-in.
Captured pre-N0 rollback	rollback_pre_n0=true, exact captured source plus explicit approved privacy choice. No path override, symlink, edited source or forward-image substitution.
Future non-N0 image	Recorded digest/version/revision, ordinary pre-N0 main template, mandatory sealed preflight and general quota reconciliation.
N0 forward/current recovery	Original forward lock is unchanged: exact reviewed N0 compose and exact immutable node image; no arbitrary config or future-image replacement.

Raw SHA256 identities:

• Current main public pre-N0 template: 28302d6761d841b7cc5c11314aed2d377735e6bbdcaf85972a752e03bc9fdc93 (independently checked through read-only GitHub source API).
• Captured pre-N0 ops/rollback/pre-n0-compose.yaml: df239749ca0a9c09893a855ae2a8abb94090b8556515ad002105c58ac91f4c91. Saved private file was stat/hash-only, not opened; the frozen public template with only the immutable node pin reproduces these exact bytes.
• Stop-future-logs derivative: 315ceca54bf8bfefe2b395b4097fe38ed8ec847fa0688bc873220bd65efc99ec. Parsed configs prove only node and ingress logging policies change (json-file, 3×50 MB → none); other service configuration is identical.
• N0: ca110b0bae6d5a06a1002fb80df3f71cffec11659c9f9dbfa5d72624735546f6, from Add a compose-only node log sidecar #309 exact reviewed head 44eeb20, review approval. Its separately approved hosted acceptance is run 38963872225.

Rollback privacy defaults to unselected/refuse, not retention or deletion. Sam's card must explicitly choose keep-json-file or stop-future-local-logs, recorded in repository variable TC959_ROLLBACK_PRIVACY_MODE; the input must match. Only node/ingress Docker logging is changed in the latter mode. Host/SSH-gateway logging is not reconfigured. Stopping future logs does not purge old files; no truncation/deletion route exists here. Forward/current-compose promotion cannot select the rollback privacy switch.

Immutable identity and preserved sealing

Node image:
ghcr.io/tinycloudlabs/tinycloud-node@sha256:4bdae4fc232d886bb56bfbce4ec62ed4b8fb4e1369a01f6afb73d06dd68c188f

Actual OCI version 1.20.6-dstack, revision e18578f. Dispatch image_version=1.20.6; all deployment/ship/watch summaries use verified image identity, not the current crate version. Ingress remains sha256:63c1c44e2ded5b73a5f174807b91ca677826974df0f9ab3045f6ec87865a5744.

Sealed-runtime preflight runs on every selected deploy path, including old recorded-image labels. Exact N0 digest-pinned compose and original :dstack path remain supported. Secret NAME PROD_TINYCLOUD_BETTERSTACK_SOURCE_TOKEN maps only to sealed BETTERSTACK_SOURCE_TOKEN, source 2795437. Required secrets fail before deployment; .phala-prod.env stays mode 0600/umask 077, quiet compose validation, no echo/artifact, always cleanup.

Complete known env values are required before any whitelist PATCH: an existing live name absent from the sealed file fails closed. updateCvmEnvs sends complete encrypted env plus env_keys, never an allowed_envs PATCH input; no-op validates token-name presence. Async operation must have an observed transition before final running/whitelist/health gates; an unobserved fast/queued operation remains fail-closed, not assumed complete. Unexpected KMS preconditions fail closed. Existing private-logs/private-sysinfo flags remain unchanged, no tcbinfo change. Operator pre/post readback must preserve false/false/true.

General quota authority, after the final restart

Reviewed ops/quota-overrides.json explicitly selects node-override now. Values authorization is per dispatch, not N0-specific. Helper preflight validates config, credentials and HTTPS same-origin /api/quota URL shape without any service request; recovery does not require a warm/healthy node before restart.

After final deploy/allowed-env operation and required version/ingress health, before ancillary Policy probes:

1. Authenticated /admin/usage must contain each exact configured full ID once, with positive known integer usage. Missing/duplicate/null/zero/wrong-owner fails before any node PUT.
2. Read billing /api/admin/storage-comp/:space_id for BOTH targets before any node PUT. In node-override, 404 or valid HTTP200 active:false permits restore; active comp refuses with authority mismatch. Malformed/unknown bodies, other 4xx/auth, 5xx and bounded transport exhaustion fail closed. No HTTP-error retry or absent-comp fallback.
3. PUT each absolute override, then authenticated node GET verifies exact identity and BOTH override_bytes and effective_limit_bytes. Unapplied/effective mismatch fails loudly.

Later separately reviewed authority=billing never PUTs/DELETEs a node override. Active absolute per-space storage_limit_bytes, not incremental comp_bytes, must cover each reviewed floor; node GET must show override_bytes=null, otherwise stale legacy override fails. Cold cache is not failure: billing-authoritative; the node fetches on the next write (≤60 s TTL).

Any attempted CVM mutation without verified reconciliation emits fatal OVERRIDES UNVERIFIED. Quota is in the SAME deploy job, independent of watch scheduling and before Policy probes. Partial deploy/sync/health/quota failures cannot silently skip verification. This addresses loss of in-memory node overrides on restart without shadowing active billing authority.

Runner-only production-environment secret NAME PROD_TINYCLOUD_BILLING_ADMIN_SECRET is explicitly separate from node admin auth, never CVM env/allowed_envs, argv or logs. It is UNPROVISIONED: Sam's card must provision it; absence blocks before mutation. Node admin uses existing repository secret NAME PROD_TINYCLOUD_ADMIN_SECRET. No credential values, full IDs, response bodies or raw SDK messages are logged. Only classifications and the two reviewed numbers appear in quota summaries.

Exact operator commands — NOT executed

Canonical, tested one-copy-paste commands are in docs/releasing.md; rollback has two explicit privacy variants, both requiring Sam's recorded selection.

12c current-compose recovery:

gh workflow run docker.yml -R TinyCloudLabs/tinycloud-node --ref main \
  -f deploy_phala=true \
  -f authorize_quota_values=true \
  -f image_version=1.20.6 \
  -f deploy_image_digest=sha256:4bdae4fc232d886bb56bfbce4ec62ed4b8fb4e1369a01f6afb73d06dd68c188f \
  -f rollback_image_version=1.20.6 \
  -f rollback_image_revision=e18578ff9162986ddd813259c60f73866e3e60d1 \
  -f sql_identity_fence=false

TC-1302 captured rollback uses the same immutable tuple plus rollback_pre_n0=true and exactly one approved rollback_privacy_mode=keep-json-file / stop-future-local-logs; full literal commands are in the linked runbook, not an implicit production default. Other feature/audit/non-descendant inputs retain their false defaults.

No deployment on merge

A merge emits push, not workflow_dispatch. deploy-phala requires github.event_name == 'workflow_dispatch', inputs.deploy_phala, successful guards/build-or-recorded digest, and repeats main/ref validation beside production access. Version guard additionally rejects missing quota authorization before any deploy mutation. Digest promotions require refs/heads/main; explicit captured rollback requires the exact immutable tuple and approved privacy mode. Watch dispatch is downstream/manual-only and retains its own actions:write; no permission is added to the deploy job. #306 release-only floating tags are unchanged. Existing CI build/publish can still occur on configured events; that is not a CVM deploy.

Exercised evidence

Tests-first red/green: initial recovery matrix failed against the parent workflow; authority matrix failed against the old inline path. Added destination-symlink regression failed before the fixed-path guard; explicit privacy reporting failed before its implementation. Final local runs:

• recovery workflow/CLI/API boundary suite: 17 tests PASS, including parsed documented 12c + both rollback commands and actual missing-authorization rejection with zero mutation;
• quota authority: 20 tests PASS, mode/reverse-guard/identity/readback/error/leak cases;
• Docker tags 8/8, deploy refs 16/16, Policy probes 13 scenarios, actionlint v1.7.12 clean (external shellcheck/pyflakes unavailable/disabled);
• actual quota CLI subprocess smoke: no-network preflight; repeated legacy apply/readback; billing read-only/cold-cache path; canary exclusion and temporary-file cleanup observed;
• YAML interpreter smoke: only the two intended logging drivers/options change; raw artifact SHA256s above verified.

No local Docker, production/Phala/billing request, credential value read, workflow dispatch, merge or extra reviewer launch. Exact-head hosted CI and final API head/base are recorded below. Deployment stays HELD: combined review/landing, Sam's window/card/privacy decision, missing billing credential, operator pre/post privacy readback, and quota fallback ownership remain gates. Historical TC-780 commands remain explicitly retired and unchanged; operator packets must use the tested current runbook commands, not the removed N0-only authorization input.

Reviewed API contract and exact source

Billing API contract was inspected read-only at TinyCloudLabs/billing #48, head 26d9d0f7c0151c380affd4b6a621a6f16bb26c46. Its storage_limit_bytes is the absolute queried-space limit; the record's space_id can name a sibling comp intent, so node inventory/readback—not that record field—provides exact target identity proof. The separately authenticated runner credential is not assumed equal to node admin auth.

Exact source commands: 12c recovery and both explicit TC-1302 rollback privacy variants.

Final REST API: head 63d11dd67e07378acb9213ecd8e3b895c7be5931, base main at f2792bb9f2f82691fcd777bfb0701e22bfc30eff; direct parent remains frozen approved #310 87b2a38b2ad523bf6587e3c08ef1e072138b56e7. PR is open, unmerged.

Terminal exact-head hosted CI

19/19 terminal: 10 success, 9 intentionally skipped, 0 pending, 0 failure. All three workflow runs report completed/success for this exact head.

Run	Result
ci 38975186551	SUCCESS: workspace build, PostgreSQL-backed workspace tests, clippy, fmt, capabilities, m1-realdata
Docker 38975186494	SUCCESS: version guard (both new suites + tags), image build, ingress build
Release 38975186558	SUCCESS: plan

Skipped checks: promote-watch, deploy-phala, announce, publish-homebrew-formula, host, build-global-artifacts, build-local-artifacts, build-dstack, rollback-audit. In particular, neither deploy nor watch ran. Ready for Amos to route the combined delta to the same Astra reviewer; no release/deploy GO.

[samgbafa]

APPROVE: see TC-959

[samgbafa]

APPROVE: see TC-959 (reviewed exact head db5d5f7; delta from d0163d2 is a test rename only)