diff --git a/README.md b/README.md index d7cd40baf..3f431e5b5 100644 --- a/README.md +++ b/README.md @@ -77,8 +77,8 @@ sandboxd](docs/deploy.md) covers release binaries, the full config and systemd. newline-JSON frames over vsock 2048 ([reference](docs/silkd.md)) - `sandboxd/` — per-node control plane (Go): warm pools, the claim API and the relays to silkd ([deploy](docs/deploy.md), [API](docs/sandboxd-api.md)) -- `sdk/go/` — [Go SDK](docs/sdk.md); `silkd/` is its connection layer and - `silkd/silkdtest/` a test fake +- `sdk/go/` — [Go SDK](docs/sdk.md); `sdk/go/silkd/` is its connection layer + and `sdk/go/silkd/silkdtest/` a test fake - `sdk/python/` — [Python SDK](docs/sdk-python.md), stdlib-only and synchronous - `mcp/` — `sandbox-mcp`, an [MCP](docs/mcp.md) stdio server - `sdk/openai/` — `cocoonstack-sandbox-openai`, a sandbox provider for the @@ -112,7 +112,8 @@ sandboxd](docs/deploy.md) covers release binaries, the full config and systemd. changed carriers (via `build-boot.yml` / `build-silkd.yml`, `workflow_call`) then os-images, in order, so the chain is deterministic - `build-os-images.yml` — bakes base + flavors FROM the sha-pinned carriers -- `release.yml` — on a version tag, builds the release binaries and archives +- `release.yml` — on a version tag, checks the tag against silkd's version, + then builds the release binaries and archives - `publish-pypi.yml` — on an `sdk-*-v*` tag, builds and publishes the matching package via PyPI Trusted Publishing (OIDC, per-package environment) @@ -142,20 +143,12 @@ make go-lint # Go: protocol/wire + sandboxd + sdk/go + e2e + mcp, GOOS lin make go-test # Go: go test -race across the Go modules make sh-lint # shellcheck every tracked shell script make sandboxd # build dist/sandboxd -make bench # claim-tier + data-plane benchmarks on this node +make bench # claim-tier + data-plane benchmarks; removes every sbx-* VM make boot # kernel + initramfs artifact image (docker) - # KERNEL_MIRROR=… if kernel.org tarball paths 404 locally make silkd-image # silkd release binary in a scratch carrier image make images # base + python images against the local boot + silkd images ``` -The parent workspace's `go.work` includes these modules, which is why the -Makefile forces `GOWORK=off`: CI has no workspace, and a workspace build -resolves sibling checkouts instead of the pinned versions. silkd's integration -tests spawn real processes — run them in a Linux container too (`docker run -rust:1 … cargo test`) before touching platform-sensitive paths; macOS green -alone has hidden Linux-only breakage before. - ## License The server stack — sandboxd, silkd, the boot chain, the OS images, and the diff --git a/docs/cluster.md b/docs/cluster.md index fafb124e5..4abdaa3e2 100644 --- a/docs/cluster.md +++ b/docs/cluster.md @@ -175,8 +175,9 @@ An archived claim keeps only a pointer on the node; with `meta_store` with a `cell`, an `s3` `checkpoint_store` and a `mesh`), the archive also writes one row to the shared PostgreSQL, scoped by the cell: id, tenant, token SHA-256, owner node, epoch, a consumed flag, checkpoint id, -retention deadline and the claim record without its token. Only archives and -replication rounds write it, so the claim path is unchanged. +retention deadline and the claim record without its token. Archives, +replication rounds, and env writes and renews of a claim whose row must follow +it while it sleeps write it, so the claim path is unchanged. - **Adoption.** `Lookup` asks the entry node, then every peer. A node that does not hold the claim reads the row; when the token hash matches, it takes the @@ -184,8 +185,8 @@ replication rounds write it, so the claim path is unchanged. owner's [node lease](#node-lease) to be expired, unless the owner is this node (with its record gone). The node records the claim as archived and answers as the owner. Two nodes adopting at once leave one owner; the other - answers 404. A tombstone is never adopted. The next exec or file call wakes - the claim from the store as usual. An owner with a live lease answers for + answers 404. A tombstone is never adopted. The next call that reaches the + guest wakes the claim from the store as usual. An owner with a live lease answers for itself, and a node that tries to take its row is refused with a 404. A draining node never adopts, nor does a fenced node or one that lifted its fence less than 8 seconds ago (one renewal interval plus one registry query), @@ -470,8 +471,8 @@ at a node that does not hold the record runs a tier order: `Checkpoint.Delete` (`DELETE /v1/checkpoints/{id}`) removes the local record, then best-effort broadcasts the delete to every peer this node currently sees, so a replica a heal pulled earlier is cleaned up too, not -just the original. A peer that is offline or partitioned during that -broadcast keeps its copy until the checkpoint TTL ages it out. A healed +just the original. A peer that is offline, partitioned or not yet joined during +that broadcast keeps its copy until the checkpoint TTL ages it out. A healed replica carries the source checkpoint's original `CreatedAt`, so it becomes eligible for expiry at the same instant everywhere; each node then removes it on its own hourly sweep, which is independently phased and retries on @@ -482,7 +483,8 @@ sweep failure extends retention until one succeeds — the TTL is the eligibilit point, not a hard ceiling. This is why heal *requires* a nonzero, fleet-matching TTL (see [cluster-invariant config](#cluster-invariant-config)). With TTL disabled it could never close at all, so that combination is -rejected at config load. +rejected at config load. A node later run with healing off and that TTL back at +0 keeps such a replica until an explicit delete. ## State ownership diff --git a/docs/deploy.md b/docs/deploy.md index 30af7797e..0c774ccd3 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -139,12 +139,12 @@ the intersection of the two. | `preview_listen` | (off) | address for a preview HTTP server that serves guest ports under signed URLs; needs `preview_secret` | | `preview_secret` | — | cluster-shared HMAC secret signing preview tokens (all nodes share one) | | `preview_advertise` | = `preview_listen` | the browser-facing preview base URL, minted into every preview URL, so it must name a routable host (a wildcard `preview_listen` needs it set explicitly); nodes behind one TLS proxy may share it, while signed tokens route internally through each owner's `advertise_addr` | -| `checkpoint_dir` | `/checkpoints` | where checkpoints and promoted templates live. Point it at a shared FUSE mount (JuiceFS over object storage, NFS) and every node sharing the mount can branch every checkpoint — records are generation-addressed with `meta.json` as the atomic commit pointer, so no cross-node locking is required of the filesystem. A re-publish retains its superseded export generation for at least ~1h and until a following hourly sweep so an in-flight clone that resolved the old metadata can finish; budget the current generation plus every generation retained across that grace-and-sweep window. An explicit delete can make a concurrent clone fail visibly. One contract on any shared root (mount or bucket): a template key has a single writer — promotes go to the sandbox's owner node, and operators must not race promotes of one name from different nodes (checkpoint ids are node-generated and never collide) | +| `checkpoint_dir` | `/checkpoints` | where checkpoints and promoted templates live. Point it at a shared FUSE mount (JuiceFS over object storage, NFS) and every node sharing the mount can branch every checkpoint — records are generation-addressed with `meta.json` as the atomic commit pointer, so no cross-node locking is required of the filesystem. A re-publish retains its superseded export generation for at least ~1h and until a following hourly sweep so an in-flight clone that resolved the old metadata can finish; budget the current generation plus every generation retained across that grace-and-sweep window. An explicit delete can make a concurrent clone fail visibly. One contract on any shared root (mount or bucket): a template key has a single writer — promotes go to the sandbox's owner node, and operators must not race promotes of one name from different nodes (checkpoint ids are node-generated and never collide). Abandoned staging (`*.tmp`) is removed only once it is older than an hour, at startup and on the hourly generation sweep, so a publish in flight on another node of a shared root survives; the `meta.json` commit pointer is fsynced with its directory | | `checkpoint_store` | dir | checkpoint AND promoted-template backend (both live in one store root, id-namespaced ck_/tp_): `{"kind": "s3", "s3": {"bucket": "…", "prefix": "ck/", "endpoint": "…", "region": "…", "force_path_style": true}}` stores checkpoints in object storage (any node claims any checkpoint, no shared mount needed). Credentials come from the standard AWS chain (env/IAM role), never this file. An export file is stored as 16 MiB chunks keyed by SHA-256 under `chunks//`, with one `manifests//.json` per published generation naming each file's size and present chunks, and `/meta.json` uploaded last as the commit marker (the record prefix exists only while the record is live). Holes and all-zero chunks are never uploaded, and a fetch recreates the file at its full size and writes only the present chunks, so a mostly-empty disk costs its data, not its size, on the wire and on the fetching node. A lineage is the dedup scope: it defaults to the record id, and every checkpoint and archive of one claim shares the claim's sandbox id, so consecutive captures upload only the chunks that changed (the first capture after a wake uploads everything again when no other live record of that claim remains, because the wake deletes the record it restored from); a publish into a shared lineage writes a zero-byte marker `lineages//` after its manifest and before `meta.json`, and reads only the records those markers list; a crash between manifest and marker only costs a missed dedup, and Delete removes the record's marker right away. Re-publish retains the prior generation's manifest so an in-flight fetch that selected old metadata can finish. sandboxd never deletes a chunk or a manifest from the bucket: Delete removes `meta.json`, the labels and the lineage markers and re-stamps the manifests, so a fetch that already selected the record still completes, and reclaiming storage is the operator's job, see [Reclaiming S3 storage](#reclaiming-s3-storage). A publish or fetch is aborted after 30 minutes. Do not put an expiry lifecycle rule on the prefix: it would delete chunks a live record still names. Budget storage for every generation and chunk until your reclaim runs. When caching a new generation, each node prunes other cached generations of that record unused for at least an hour. The periodic generation sweep drops a node's cache of a record once the record is gone from the bucket and the cache is unused for at least an hour. Absent = the dir backend at `checkpoint_dir` | | `checkpoint_ttl_hours` | 0 (keep forever) | ages out checkpoints older than this; the sweep runs hourly and at startup. Explicit deletes never wait for it. Must be nonzero and match fleet-wide when `checkpoint_peer_heal` is on — it is the expiry eligibility point for a healed replica a delete broadcast missed, after which its next successful hourly sweep removes it; persistent sweep failure extends retention until one succeeds, so it is not a hard ceiling | | `checkpoint_peer_heal` | false | on a cluster, lets a node pull a checkpoint it lacks from a peer — found via a live probe, not gossip — rather than failing the branch; see [placement lifecycle](cluster.md#checkpoints-on-a-cluster). Three requirements, all enforced at config load: a nonempty `api_token` (the blob transfer between peers authenticates with it; without one the raw record stream would be open), `mesh.cluster_key` set (the pull presents the fleet `api_token` to an address learned from the peer probe, so the gossip layer carrying that address must itself be authenticated), and `checkpoint_ttl_hours` nonzero (a replica a delete broadcast missed becomes eligible for expiry after it, and its next successful hourly sweep removes it — so it is the finite eligibility point, not an exact ceiling). A shared checkpoint store (`checkpoint_store` kind `s3`) ignores this setting — every node already resolves every checkpoint directly, so there is nothing to heal | | `warm_max` (pool entry) | 0 (static) | turns on the demand-adaptive watermark for that pool: the warm target rises from `warm` toward `warm_max` while claims arrive faster than the measured provision lead covers, and decays back over ~a minute of silence. The pool follows the target down: warm VMs above it are destroyed once the count has stayed above it for a full decay period, so a burst does not leave the pool parked at its high-water mark | -| `warmup` (pool entry) | unset | argv run in the golden VM after readiness and before its snapshot, so the files it touches are page-cache-resident in every clone, and again in every clone before it joins the warm pool, so those pages are already faulted into the restored VM when the first command runs — e.g. `["node", "-e", "0"]` on a Node flavor. It runs under the engine's 2-minute command timeout in silkd's base environment (`PATH`, `TERM`, and the guest image's proxy variables wherever nothing routes directly — the none lane and the locked bridge egress lane — with the proxy not yet serving, since a door pre-bound at refill only starts serving at claim); a non-zero exit or a timeout fails the golden build, so the pool stays unfilled until the config is fixed. Config-owned like `egress`: `PUT /v1/pools` rejects it, and a golden built with a different warmup is rebuilt The build then waits up to 30 s for the guest's systemd to finish starting, because a snapshot taken during boot makes every clone run the queued jobs before silkd answers. | +| `warmup` (pool entry) | unset | argv run in the golden VM after readiness and before its snapshot, so the files it touches are page-cache-resident in every clone, and again in every clone before it joins the warm pool, so those pages are already faulted into the restored VM when the first command runs — e.g. `["node", "-e", "0"]` on a Node flavor. It runs under the engine's 2-minute command timeout in silkd's base environment (`PATH`, `TERM`, and the guest image's proxy variables wherever nothing routes directly — the none lane and the locked bridge egress lane — with the proxy not yet serving, since a door pre-bound at refill only starts serving at claim); a non-zero exit or a timeout fails the golden build, so the pool stays unfilled until the config is fixed. Config-owned like `egress`: `PUT /v1/pools` rejects it, and a golden built with a different warmup is rebuilt. The build then waits up to 30 s for the guest's systemd to finish starting, because a snapshot taken during boot makes every clone run the queued jobs before silkd answers. | | `capture_trim` (pool entry) | `false` | Trim the guest's copy-on-write disk before a promote or checkpoint of this pool's sandboxes (and of the clones of a template promoted from it), so the record carries only live data: blocks a build deleted — a `COPY` step's archive, a `RUN` step's caches — are no longer captured. sandboxd mounts the disk init resolved from `cocoon.cow` a second time at `/run/sandboxd-trim` through silkd as root, runs `fstrim` and unmounts, under the capture's transition lock (about 15 ms on a warm clone). A failed trim is logged and the capture proceeds, larger. A hibernated or archived sandbox is captured untrimmed. Config-owned like `egress`: `PUT /v1/pools` rejects it | | `reader` (pool entry) | `false` | Serve read-only file calls on this pool's hibernated or archived claims from a warm VM of the same pool instead of waking them; needs `warm` of at least 1 and a cocoon whose `vm disk attach` takes `--from-snapshot` and `--from-dir`. See [Reading sleeping claims](#reading-sleeping-claims-without-waking-them). Config-owned like `capture_trim`: `PUT /v1/pools` rejects it | | `egress_classes` | unset | named tenant egress layers: `[{"name": "desk", "egress": {…}, "egress_upstream_env": "DESK_UPSTREAM"}]`. A tenant names one with `egress_class` through the [tenant API](sandboxd-api.md#tenants-v1tenants) and its claims take that policy intersected with the pool's. `egress` is required; `intercept` stays a pool-rule flag. Reloadable: a class's policy reaches live claims on their next request. A class removed while tenants still name it leaves them with no egress and is logged at load and at reload | @@ -152,7 +152,7 @@ the intersection of the two. | `replicate_seconds` (pool entry) | 0 (off) | snapshot every running claim of this pool to the checkpoint store this often and point the claim's registry row at the snapshot, so another node restores the claim, memory and processes included, when this node is lost (see [running claims across node loss](cluster.md#running-claims-across-node-loss)). Each round pauses the guest for the length of a memory and disk capture. A restore resumes from the last committed replica and loses everything after it: up to the longer of the interval and one replication pass, plus the 5-second tick, plus one more pass, since a claim can run first in one pass and last in the next. A pass covers all the node's due claims of the replicating pools with bounded concurrency, so it grows with claim count and guest memory; the first round of a claim after a wake or adoption uploads the whole claim, so that pass is the long pass. The claim's row carries no retention deadline while it runs. Requires `archive_failover` and a `none`-lane pool, both checked at config load; a claim with volumes is skipped. Config-owned like `egress`: `PUT /v1/pools` rejects it. Restart only: a claim is replicated by its pool's setting from the moment it is claimed, so a reload that changes it is refused (a pool restarted with 0 starts retiring its replicas while the registry remains enabled; see [Disabling archive failover](#disabling-archive-failover)) | | `storage` (pool entry) | unset (cocoon's default, 10G) | size of the copy-on-write disk the pool's golden is cold-booted with, in cocoon's own `--storage` spelling (`"40G"`, `"40GiB"`, `"40Gi"`: Docker/Kubernetes units, all binary) and passed to it verbatim; every clone, fork, checkpoint branch and promoted template of the pool inherits it. At least cocoon's default of 10G. Equal sizes spelled differently build the same golden; a changed size rebuilds the golden on the next build, and claims already out keep their disk. Config-owned like `egress`: `PUT /v1/pools` rejects it | | `max_claims` | 0 (unlimited) | node-wide cap on live claims; claim/fork/branch requests beyond it answer 429 with the pool state unharmed (on a cluster a non-volume claim tries a warm-peer redirect first; a volume claim answers the 429 with no redirect) | -| `audit_log` | false | append every relayed frame that opens an RPC — its op + addressing fields, never payloads — to `/audit.jsonl`; continuation `data`, `data_end`, `stdin`, and `stdin_close` frames pass through unrecorded. The file is size-rotated with one `.1` backup. Records are `{t, id, op}` plus whichever addressing fields the op carries (`argv`, `path`, `dest`, `from`, `to`, `url`, `session`, `port`), plus `method` (`GET`, `CONNECT`, `SOCKS5`, …), `decision` (`allow`, `deny`, `rate_limited`), `secret` (the ref name, never its value), `upstream` (the [upstream proxy](egress.md#upstream-proxies)'s `host:port`, when one carried it), `tenant`, and on an allowed connection `src_addr` and `dst_addr` (the node-side and destination `ip:port` of the upstream TCP connection, the tunnel target through an upstream proxy) on `egress` records, which a relayed frame cannot set; preview accesses record as op `preview`, one per request; a tenant change records as op `tenants` with the `added`, `changed` and `removed` names, never a token. An opening frame whose first line exceeds 4 KiB records as op `oversized` with no addressing fields | +| `audit_log` | false | append every relayed frame that opens an RPC — its op + addressing fields, never payloads — to `/audit.jsonl`; continuation `data`, `data_end`, `stdin`, and `stdin_close` frames pass through unrecorded. The file is size-rotated with one `.1` backup. Records are `{t, id, op}` plus whichever addressing fields the op carries (`argv`, `path`, `dest`, `from`, `to`, `url`, `session`, `port`), plus `method` (`GET`, `CONNECT`, `SOCKS5`, …), `decision` (`allow`, `deny`, `rate_limited`), `secret` (the ref name, never its value), `upstream` (the [upstream proxy](egress.md#upstream-proxies)'s `host:port`, when one carried it), `tenant`, and on an allowed connection `src_addr` and `dst_addr` (the node-side and destination `ip:port` of the upstream TCP connection, the tunnel target through an upstream proxy) on `egress` records, which a relayed frame cannot set; preview accesses record as op `preview`, one per request; a tenant change records as op `tenants` with the `added`, `changed` and `removed` names, never a token; a config reload records as op `config_reload` with `changed`, and a runtime volume change as op `volumes` with `added` or `removed`. An opening frame whose first line exceeds 4 KiB records as op `oversized` with no addressing fields | | `disk_reserve` | unset | `{"free": "100G", "dirs": ["/srv/sbx/run"]}`: while the filesystem of any listed directory (default `data_dir`) has less than `free` available, checked every 2 s, the node refuses new claims, forks, checkpoint claims, wakes of hibernated or archived sandboxes, hibernates, checkpoint captures and promotes with 429, and parks refill; the idle sweep and a lease-end archive leave their sandboxes running and retry on the next tick; `/v1/info` reports it as at capacity and the mesh sees no warm VMs on it. Running sandboxes keep running. On an XFS project quota the free space is the project's. List cocoon's `run_dir` when it sits on another filesystem, since the copy-on-write disks grow there. Size `free` to at least two sandboxes' worst-case growth (copy-on-write disk plus memory file each) and keep the refusal above your disk alert's warning level, so the warning comes first: on a 640G project with 40G disks and 8G memory files, `100G` refuses at 84% used | | `idle_hibernate_seconds` | 0 (off) | node-wide idle policy for unpooled claims (template/checkpoint claims): a none-lane claim is hibernated once it has had no open data-plane connection (relay, buffered exec, preview) and no egress request in flight for this long; the clock restarts when the last connection closes, so a long command is never cut short. The next call that reaches the guest wakes it transparently. Per-pool `idle_hibernate_seconds` does the same for that pool's claims; pooled keys ignore the node-wide value, and egress pools reject it because they cannot resume safely. Opt in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice | | `archive_after_seconds` | 0 (off) | tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call that reaches the guest restores it transparently (a checkpoint restore's latency) with a fresh lease of the length the claim was granted (the server default of 5m when it asked for none); a lease moves only when a client calls [`/renew`](sandboxd-api.md#post-v1sandboxesidrenew). Keep `idle_hibernate_seconds` under the leases your claims use, or a woken sandbox that idles again is destroyed by the reap before it can hibernate, its archive already consumed by the wake; one still in use when its lease ends is destroyed the same way, unless it claimed `on_expire: archive`, which archives at the lease end whatever this setting. Requires `idle_hibernate_seconds > 0` and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool | @@ -226,7 +226,7 @@ kernel does the parsing, as it does for the claim: symlinks resolve inside the claim's filesystem, and a write fails at the read-only overlay. -The pool's image must offer `mount`, `chroot`, `env`, `mkdir` and `sync` and +The pool's image must offer `mount`, `chroot`, `env`, `mkdir`, `test` and `sync` and boot a kernel that mounts erofs, ext4 and overlayfs, which the shipped images do. The claim's image must ship `silkd` at `/usr/local/bin/silkd`, as the shipped Linux images do; the android image does not, so its claims always wake. @@ -259,24 +259,10 @@ read-only or writable mode through the and both SDKs. Changing a held entry's descriptor requires successful removal first. The [live volume contract](sandboxd-api.md#put-v1sandboxesidvolumes) defines admission policy and dirty-marker handling. -The image must already exist on the owner node. Live changes -require Cocoon v0.7.2 or newer for idempotent disk attach and detach. PUT and -PATCH reject an older release, or a runtime whose version cannot be queried, -with 503 before any intent, hold, or device change; a dev build is assumed -current. sandboxd reads the version once, so restart it after upgrading cocoon. -GET and release remain available. - -The claim journal stores confirmed applied `Volumes`, plus desired -`VolumesWant` while it differs. -Admission holds cover their union across daemon restart and teardown. Retry -an interrupted operation with the same PUT target. Release cancels convergence -and transfers holds to teardown. Capture is refused while either set is -nonempty. - -A sandbox whose VMM exits while holding volumes is release-only. Release it and -create a new sandbox. Daemon restart recovery assumes the original VMM remains -alive and does not recreate guest processes. Unfinished changes report an -interrupted diagnostic after restart and require a caller retry. +The image must already exist on the owner node. Live changes need Cocoon v0.7.2 +or newer; the journal, hold and recovery rules are in +[partial failure and recovery](sandboxd-api.md#partial-failure-and-recovery), +and a VMM exit in [live volumes](live-volumes.md). `e2e/cmd/volumesmoke -live` (`VOLUME_LIVE=1` for `scripts/sandboxd-e2e.sh`) exercises GET, repeated full-set PUT, PATCH add and remove, and clearing a new @@ -311,12 +297,9 @@ guest OS directories and cannot duplicate or nest within one claim. Volume mounts may shadow an existing populated guest directory for that claim's life. A volume claim may consume an ordinary warm Cloud Hypervisor VM. sandboxd -attaches after the warm pop or provision, waits for the device settle — a -serial match under `/sys/block`, then the `/dev/` node itself, since the -kernel publishes sysfs before devtmpfs creates it — up to 2 seconds, then -mounts the device, retrying `not a valid block device` within the 10-second -setup budget, before finalizing the claim: `mode: "ro"` (the default) -attaches and mounts read-only; `mode: "rw"` requires the catalog entry's +attaches after the warm pop or provision and mounts before finalizing the +claim, on the [device settle and mount budget](sandboxd-api.md#post-v1claim): +`mode: "ro"` (the default) attaches and mounts read-only; `mode: "rw"` requires the catalog entry's `writable: true` and attaches and mounts read-write. Setup failure destroys the VM without quiescing — the claim was never handed out, so no workload write happened — and a popped warm VM is refilled normally. @@ -630,18 +613,20 @@ reclaimed through `vm reconcile-stale-create` (cocoon ≥ v0.5.8), which refuses while a clone is in flight instead of deleting the VM out from under it; older cocoons fall back to forced removal. Any golden no pool uses is deleted. Then the refill loop builds one golden snapshot per pool (a one-time -cold boot + snapshot export, tens of seconds) and keeps each pool topped up -with claim-ready clones. +cold boot + snapshot export, seconds on bare metal, plus the pool's warmup and +the wait for the guest's systemd to finish starting) and keeps each pool topped +up with claim-ready clones. `GET /v1/info` shows `"golden": true` and `warm` at target when the node is ready to serve warm claims. A golden left on disk is adopted on start when its stamp matches the pool's -image, lane, interception, `storage` and `warmup`. The stamp does not record -the cocoon or Cloud Hypervisor version, so a golden from before an upgrade of -either is adopted. If its clones then fail, sandboxd retires it once the pool's -refills have failed 8 times running (a node-capacity failure does not count) -and builds a new golden; until it is ready, claims for that pool that find no -warm VM cold-boot. If failures continue with the new golden, it is retired again at most once every -30 s. +image, lane, interception, `storage`, `no_balloon` and `warmup`. The stamp does +not record the cocoon or Cloud Hypervisor version, so a golden from before an +upgrade of either is adopted. If its clones then fail, claims that clone from it +fail too, until 8 clones running, a claim's or a refill's, have failed (a +node-capacity failure, a claim its client canceled and a failed probe do not +count): sandboxd then retires it and builds a new golden, and until that is +ready, claims for that pool that find no warm VM cold-boot. If clones of the new +golden fail the same way, it is retired again at most once every 30 s. A minimal systemd unit (shipped as [`packaging/sandboxd.service`](https://github.com/cocoonstack/sandbox/blob/main/packaging/sandboxd.service)): @@ -688,7 +673,11 @@ never reads as a live VMM. The node then lists cocoon's VMs only every 60 s as a backstop for a dropped event, and every 10 s while the stream is down (it reconnects with backoff) or when no socket is configured. Each suspect is then checked with a single-VM `cocoon vm inspect` under its lock; a warm VM -found down is destroyed and replaced. A claim +found down is destroyed and replaced. A warm claim takes no liveness check, so +one that lands before the eviction gets the dead VM: it answers 200, then 502 +until `vmm_restart` acts on it. Measured on one host, the eviction follows the +VMM's exit by about 0.15 s with the stream and by up to 10 s without it, so set +`cocoond_socket` in production. A claim whose VMM is gone — killed, crashed, or lost with the host — is handled per `vmm_restart`. A VMM that exits because the guest powered off is treated the same way. A claim whose lease has already lapsed is left to the reap, unless it @@ -891,7 +880,8 @@ only; computing durations or prices stays with the consumer. (same nanosecond `t` and fields) land once. A line that does not decode is skipped with a warning, and a last line without its newline waits for it. A line a crash cut short stays a line of its own: the node ends it with a newline - when it opens the journal, so the first event after the restart is whole. + when it opens the journal, so the first event after the restart is whole, + and the next append ends a line a short write (a full disk) cut off. - **Order:** a cell's lands commit one at a time under a transaction-scoped advisory lock, so `cursor` grows in commit order and a reader that pages by cursor never skips a row that commits later. A node lands its lines in @@ -1167,3 +1157,14 @@ working — images built here boot with an unmodified cocoon. The kernel's config in `boot/kernel/` is a fragment. On amd64 it merges over `x86_64_defconfig` + `kvm_guest.config`; on arm64 it merges over `defconfig`, then `sandbox-arm64.config`. + +## Development + +The parent workspace's `go.work` includes the repository's Go modules, which is +why the Makefile forces `GOWORK=off`: CI has no workspace, and a workspace build +resolves sibling checkouts instead of the pinned versions. silkd's integration +tests spawn real processes; run them in a Linux container too (`docker run +rust:1 … cargo test`) before touching platform-sensitive paths, since a macOS +green alone has hidden Linux-only breakage before. `make boot` downloads the +kernel tarball from kernel.org; set `KERNEL_MIRROR=…` when those tarball paths +404 locally. diff --git a/docs/egress.md b/docs/egress.md index aea7f70dd..ea46e63f7 100644 --- a/docs/egress.md +++ b/docs/egress.md @@ -257,8 +257,9 @@ Each decision is metered as an `egress` usage event, which names the injected secrets in `secret` as the audit line does, and, when `audit_log` is on, written to `audit.jsonl` (`op:"egress"`, `dest`, `port`, `method`, `decision`, and the secret **name** in `secret`); an intercepted `CONNECT` is -recorded as a decision of its own before its inner requests. A host longer -than 253 bytes (the DNS limit) is denied, and a sandbox records at most 10 +recorded as a decision of its own before its inner requests. The HTTP and +CONNECT doors deny a host longer than 253 bytes (the DNS limit); the audit +record clips it. A sandbox records at most 10 `deny` and `rate_limited` decisions per second after a burst of 100, dropping the rest, so a guest cannot flood the journals. @@ -539,7 +540,8 @@ install is off the claim path). It is **not** re-installed on re-claim: a clone, checkpoint restore, archive wake, or reconcile adopts the guest with whatever root it was born with. Each golden's `.stamp` file records what it baked (the root fingerprint, the lane verdict, -the image id, a sized pool's disk size, and the warmup argv), and a golden is +the image id, a sized pool's disk size, `no_balloon` when set, and the warmup +argv), and a golden is adopted only when its stamp exists and matches, so a changed root rebuilds goldens; a golden from before the stamp has none and is rebuilt once. Nothing rebuilds an existing checkpoint, diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 14ac12150..a7450d5ed 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -69,7 +69,7 @@ Auth: `Authorization: Bearer ` (when configured). cold-booting an image named like the template - `env` is the claim's own environment, at most 64 entries: each name a shell identifier of at most 128 bytes, each value at most 8 KiB, a value and an - inject `query` with no control characters other than tab, and names, values and inject fields at most 64 KiB in total. An entry holds only `value`, `guest` and `inject`; any other + inject `query` or `placeholder` with no control characters other than tab, and names, values and inject fields at most 64 KiB in total. An entry holds only `value`, `guest` and `inject`; any other member, or `"guest": null`, answers 400, so a mistyped flag never lands a host-only value in the guest. `guest` (default `true`) delivers the entry into the guest; a `guest: false` entry never enters it and only feeds the node's egress @@ -225,8 +225,8 @@ name (attributed in the usage journal and counted against the tenant's `max_claims`). A catalog access list may restrict an entry to named tenants; an unknown and a forbidden volume return the same error text. -Errors: 400 unknown template axis, metadata over a bound (the message names -it), an unknown `on_expire`, invalid/duplicate volumes, +Errors: 400 unknown template axis, `claim_ref` or metadata over a bound (the +message names it), a bad `env` entry, an unknown `on_expire`, invalid/duplicate volumes, `volumes_attach_only` with no volumes, or an attach-only entry with a nonempty `mount`, `mode: "rw"` against a non-writable entry, or a volume that is unknown or forbidden (the @@ -238,8 +238,8 @@ exclude a writer), or a `ro` claim against a writable image left dirty by an unclean `rw` release (needs recovery — one `rw` claim must replay and cleanly release before `ro` claims resume; see [deploy](deploy.md#writable-dataset-volumes)); 429 node at `max_claims`, the -calling tenant at its own `max_claims`, or the node draining; 500 -provisioning failed. +calling tenant at its own `max_claims`, the node draining, or the disk under +`disk_reserve`; 500 provisioning failed. ## GET /v1/volumes @@ -298,9 +298,9 @@ The node stores metadata in `data_dir/runtime-volumes.json` independently of Malformed or conflicting metadata, including a saved path outside `volume_roots`, prevents startup. The catalog is node-local with one sandboxd writer. -Sandboxd does not create, check, or delete backings. Keep them outside its -cleanup directories: `/readers`, `/goldens`, and -`/fork-*`. +Sandboxd does not create, check, or delete backings. Keep them outside +`data_dir`: sandboxd clears its `readers`, `goldens`, `fork-*`, +`checkpoint-staging` and `template-staging` entries. ### Volume access lists @@ -482,7 +482,8 @@ independent guest processes and sessions continue. See the Auth: the sandbox's own token, or the root token for operator cleanup by id. Destroys the VM. 204 on success, 404 for an unknown id or wrong token, 500 when the VM survives removal: the claim is already gone and the node retries the -removal. +removal, 503 `archive registry down; retry` for a claim that holds a registry +row while the meta store is down; resend it. Releasing an already-gone sandbox is 404 — the SDK treats it as success. ## POST /v1/sandboxes/{id}/hibernate @@ -498,7 +499,7 @@ snapshot) at its deadline, unless it claimed `on_expire: archive` or its pool archives. When to hibernate is the caller's policy — the node only provides the transition. 204 on success, 404 unknown id or wrong token, 409 on the egress lane or when volumes are attached (neither kind of -sandbox hibernates; see [egress](egress.md)), 429 while the disk is under +sandbox hibernates; see [egress](egress.md)) or on a failed sandbox, 429 while the disk is under [`disk_reserve`](deploy.md), 503 `lease lost; retry` for a replicated claim that is not archived while its node's lease is overdue, even one already hibernated. @@ -516,7 +517,8 @@ its own disk (memory is lost) and its restart budget resets. 204 on success, 404 unknown id or wrong token, 409 a failed egress-lane or volume sandbox, or one whose VM record is gone, 403 a removed tenant's archive, 429 while the disk reserve is low, 500 when the cold boot or a restore fails, -503 `lease lost; retry` for a replicated claim while its node's lease is overdue. +503 `lease lost; retry` for a replicated claim while its node's lease is overdue, +503 `archive registry down; retry` for an archived claim under `archive_failover` while the meta store is down. ## POST /v1/sandboxes/{id}/renew @@ -535,7 +537,8 @@ body means the same as `ttl_seconds` 0. The new lease also becomes the one a wake from the archive grants again. A renew can shorten a lease as well as extend it. `on_expire` switches the expiry action to `archive` or back to `destroy`; absent keeps the current one. It is refused as on a claim for a -sandbox that cannot hibernate. +sandbox that cannot hibernate. A renew stamps no activity, so it neither restarts +the idle clock nor stops an idle hibernate. An archived sandbox is refused: off-node, its deadline is the store's retention window rather than a lease, and a TTL written over it would schedule the @@ -576,11 +579,13 @@ All-or-nothing: on error no child survived. 200 with one claim per child: ``` Children inherit the parent's tenant and count against its `max_claims`, -whoever calls. 400 invalid count or body, 401 bad api token, 404 unknown id -or wrong sandbox token, 409 egress-lane or volume parent (neither forks, -checkpoints, or promotes; see [egress](egress.md)) or an archived parent (an -exec or file call wakes it first), 429 node or the parent's -tenant at `max_claims`, or the node draining. +whoever calls. 400 invalid count or body, a `claim_ref_prefix` over its bound or +an unknown `on_expire`; 401 bad api token; 403 the parent's tenant was removed +(sandbox token); 404 unknown id or wrong sandbox token; 409 egress-lane or volume +parent (neither forks, checkpoints, or promotes; see [egress](egress.md)), a +failed parent, or an archived parent (`POST /wake` or an exec wakes it first); +429 node or the parent's tenant at `max_claims`, the node draining, or the disk +under `disk_reserve`. ## POST /v1/sandboxes/{id}/promote @@ -620,7 +625,7 @@ the export they were set on, which the new one replaces. 400 invalid name, 401 bad api token, 409 when the name collides with a configured pool, the template is owned by another tenant, or the sandbox is on the egress lane, has volumes attached (see [egress](egress.md)), or is -archived (an exec or file call wakes it first), 404 unknown id or wrong +archived (`POST /wake` or an exec wakes it first), 404 unknown id or wrong sandbox token, 429 while the disk is under [`disk_reserve`](deploy.md). ## DELETE /v1/templates?template=…&net=…&size=… @@ -744,7 +749,7 @@ kept only as its SHA-256) before it serves, and loads that file at boot. Each change is an `op:"tenants"` audit record naming the tenants added, changed and removed. On a cluster every node needs the change (see [cluster](cluster.md#tenants)). 400 a bad entry, a duplicate name or token, -a token equal to `api_token`, or a config-owned field. +a token equal to `api_token`, an unknown field, or an unknown `egress_class`. ## POST /v1/config/reload @@ -758,7 +763,7 @@ reload and logs the result. → ``` `changed` lists the settings that now apply. `ignored` lists differences the -file holds for parts another API owns: pool targets and tenant identities. +file holds for pool targets, which `PUT /v1/pools` owns. Which fields reload, and what happens to live claims and goldens, is described in [deploy](deploy.md#reloading-the-config). The status codes: @@ -796,7 +801,7 @@ The URL's life is clamped to the claim's remaining lease; an archived claim kept forever (`archive_delete_after_seconds: 0`) has no lease, so there the requested `ttl_seconds` stands unclamped and `ttl_seconds: 0` mints a one-hour URL; releasing the sandbox ends it early either way. -400 a port of 0, 501 when the node +400 a port of 0, 404 unknown id or wrong sandbox token, 501 when the node has no `preview_listen`. The signed token embeds the sandbox id, port, and owner `advertise_addr`, so any node's preview listener can serve it (forwarding to the owner's main listener) and a released sandbox's URL simply stops @@ -811,7 +816,7 @@ answers `200 {"checkpoint": {id, name, sandbox_id, key, tenant?, created_at}}` — `tenant` records the calling tenant, absent for root. 400 bad body or name, 401 bad api token, 404 unknown id or wrong sandbox token, 409 egress-lane sandbox, one with volumes attached (see -[egress](egress.md)), or an archived one (an exec or file call wakes it +[egress](egress.md)), or an archived one (`POST /wake` or an exec wakes it first), 429 while the disk is under [`disk_reserve`](deploy.md). ## POST /v1/checkpoints/{id}/claim @@ -825,29 +830,17 @@ never the source sandbox's. 400 metadata over a bound, an unknown `on_expire`, or a bad `env` entry. Checkpoints are node-local (unless the store is shared — see -[Configuration](deploy.md#configuration)), so a miss here runs a tier order: - -1. This node checks its own store first. -2. On a miss, it probes every mesh peer directly — a parallel `HEAD` to each - (authenticated on an encrypted mesh, see below) — and answers exactly like - a warm-miss - [`POST /v1/claim`](#post-v1claim): `200 {"redirect": ["10.0.0.6:7777", - "10.0.0.7:7777"]}`, retry the same body (+`no_redirect: true`) at each - candidate until one answers. The answer is capped at 3 addresses: a hint, - not an exhaustive list of every owner. The probe and the follow-up claim - are not atomic: a peer can answer the probe, then lose the record — a - delete's broadcast lands, or its own TTL sweep runs (below) — before the - retry reaches it, so a redirect can go stale between the two calls. -3. If nothing answers the probe (or `no_redirect` is set), and the node has - `checkpoint_peer_heal` enabled, it pulls the record from a probed peer - itself, validates it, publishes it locally, and serves the claim from - there — paid once per node. See the full - [placement lifecycle](cluster.md#checkpoints-on-a-cluster) for how the - three tiers fit together. +[Configuration](deploy.md#configuration)). On a miss the node probes its peers +and answers with a redirect shaped like a warm-miss +[`POST /v1/claim`](#post-v1claim), at most 3 addresses; retry the same body +(+`no_redirect: true`) at each candidate. With nothing to redirect to (or +`no_redirect` set) and `checkpoint_peer_heal` on, it pulls the record and serves +the claim. See the [placement lifecycle](cluster.md#checkpoints-on-a-cluster). 404 for an unknown checkpoint (locally, and after redirect and heal both miss), 409 for an egress-lane checkpoint (see [egress](egress.md)), 429 node -or calling tenant at `max_claims` or the node draining, 503 when the node's +or calling tenant at `max_claims`, the node draining, or the disk under +`disk_reserve`, 503 when the node's concurrent-heal cap is already full — retryable, and the response carries a `Retry-After` hint. @@ -1048,7 +1041,7 @@ tenant's claim), `children` (fork) and `ref` (the promoted template / checkpoint id, the egress host, a `vmm_failed` reason, or why reconcile ended a claim). An egress event that injected credentials names them in `secret` (`gh,claim:`, never a value). An egress event routed through an [upstream proxy](egress.md#upstream-proxies) carries `upstream` (its `host:port`), and with -`egress_usage_bytes` on, `egress_bytes` events add the connection's payload `tx`/`rx` bytes. A volume claim also carries +`egress_usage_bytes` on, `egress_bytes` events add the connection's payload `tx`/`rx` bytes. A sandbox records at most 10 denied or rate-limited egress decisions per second after a burst of 100, so `egress` events do not count every refusal. A volume claim also carries `volumes`, the applied catalog names, and — omitted when empty — `volumes_rw`, the subset of those names claimed `rw`, so billing can discriminate write access (mounts and host paths are not billing dimensions). The file rotates at @@ -1122,7 +1115,8 @@ Auth: the sandbox's own token. Requires `Upgrade: silkd` + the connection is a byte-for-byte relay to the guest's silkd, carrying RPCs back to back (see [silkd](silkd.md)). An open relay holds the sandbox's idle clock, so a client that keeps a connection warm must close it when idle for -`idle_hibernate_seconds` to apply. Frames sent right behind the request, +`idle_hibernate_seconds` to apply. A relay whose client vanished without a FIN +ends after about two minutes of failed TCP keepalives. Frames sent right behind the request, before the 101, reach silkd in order; the Go and Python SDKs send their first frames that way over plain HTTP, so a proxy in front of a plain-HTTP endpoint must forward them too (Go's `httputil.ReverseProxy` drops them; Caddy forwards @@ -1130,7 +1124,7 @@ them). 426 without the upgrade header, 403 a removed tenant's archive, 404 unknown sandbox, 409 a failed sandbox, 429 while the disk reserve is low, 500 a failed restore, 502 guest unreachable, 503 `lease lost; retry` for a replicated claim while its node's lease is overdue (see -[running claims](cluster.md#running-claims-across-node-loss)). A relay of a +[running claims](cluster.md#running-claims-across-node-loss)), 503 `archive registry down; retry` for an archived claim under `archive_failover` while the meta store is down. A relay of a claim of a replicating pool that is already open when the lease turns overdue is closed, whether or not it has a replica yet. @@ -1142,7 +1136,7 @@ reader VM, a warm VM of the sandbox's pool that mounts a copy of the sandbox's d hibernated or archived, no `wake` usage event is recorded, and the read stamps neither the idle nor the archive clock. An `info` frame pipelined ahead of the call (the SDKs' first frames) rides along to the reader; one alone, with no call -behind it for 100 ms, wakes the sandbox. The first frame on the connection that +behind it for 100 ms, wakes the sandbox. A second `info` frame before any call wakes it too. The first frame on the connection that is anything else, or any later one, wakes the sandbox and hands the same connection to its silkd with the unread bytes replayed, so a client sees one silkd and no close. When the sandbox sleeps again meanwhile (a new hibernate or @@ -1193,7 +1187,7 @@ wrong token; 409 a failed sandbox or a passive relay to a paused one; 426 withou header; 502 when the guest cannot be reached on that port — no listener, or a hibernated sandbox whose restore failed. The node log carries which; 403 a removed tenant's archive; 429 while the disk reserve is low; 503 -`lease lost; retry` for a replicated claim while its node's lease is overdue. +`lease lost; retry` for a replicated claim while its node's lease is overdue; 503 `archive registry down; retry` for an archived claim under `archive_failover` while the meta store is down. ## POST /v1/sandboxes/{id}/exec @@ -1208,9 +1202,9 @@ HTTP/2 through a TLS proxy: → `200 {"exit_code": 0, "stdout": "v22.23.2\n", "stderr": ""}`. The command runs to completion (no stdin, no streaming, no detach — use the relay for those); `timeout_seconds` 0 means no limit beyond the request itself. When -the node gives up on a started command — timeout, client gone, output cap — -it kills the child through silkd before answering, so nothing keeps running -behind a 504. Output comes back as JSON strings: bytes that are not valid +the node gives up on a started command (timeout, client gone, output cap, +buffer budget) it sends the child a `kill` through silkd, bounded to 5 s, before +answering. Output comes back as JSON strings: bytes that are not valid UTF-8 are replaced with U+FFFD, so binary output belongs on the relay. 400 empty `argv`, negative `timeout_seconds`, an unknown field, or a silkd `bad_request` — a command that is missing or not executable and a `cwd` that @@ -1220,7 +1214,7 @@ node's buffered-exec memory is exhausted (64 MiB of output across every buffered exec in flight; the command is killed); 502 guest unreachable or any other silkd error; 403 a removed tenant's archive; 409 a failed sandbox; 429 while the disk reserve is low; 500 a failed restore; 503 `lease lost; retry` for a replicated claim -while its node's lease is overdue; 504 when the command outlives `timeout_seconds`. A client that hangs up first gets no reply and the command is killed. A hibernated sandbox wakes transparently like on the relay. +while its node's lease is overdue; 503 `archive registry down; retry` for an archived claim under `archive_failover` while the meta store is down; 504 when the command outlives `timeout_seconds`. A client that hangs up first gets no reply and the command is killed. A hibernated sandbox wakes transparently like on the relay. ## GET /v1/sandboxes/{id}/owner @@ -1229,7 +1223,8 @@ this node owns the sandbox, 404 otherwise. With [`archive_failover`](cluster.md#archived-claims-across-node-loss), a node that does not hold the sandbox adopts it when its row names an owner whose node lease expired, and answers 200; 503 with `Retry-After` when the meta store is -down. A row whose owner's lease is live is never adopted: 404. A node whose +down. A row whose owner's lease is live is never adopted: 404. A draining or fenced +node, or one that lifted its fence less than 8 seconds ago, does not adopt: 404. A node whose lease is overdue answers 404 for the replicated claims it holds, so the `Lookup` scatter moves on. With `client_advertise` configured, `owner_addr` is that node's full HTTP(S) origin instead; the same contract applies @@ -1269,7 +1264,7 @@ A node cordoned via [`POST /v1/drain`](#post-v1drain) additionally reports `"draining": true`. When refill is parked because the node cannot start another VM, `"at_capacity": true` distinguishes a full node from one still filling its warm target, and `at_capacity_reason` carries the engine's capacity -reason. Both capacity fields are omitted while refill is not capacity-blocked. +reason. Both capacity fields are omitted while refill is not capacity-blocked. Each VM holds a console pty, so a node stops at `kernel.pty.max` (4096 by default) with `no space left on device`; raise it past the VM count you plan to run. `golden` reports whether the pool's snapshot exists (refill can clone); `warm` at `target` with `golden: true` means warm claims are served in diff --git a/docs/sdk-python.md b/docs/sdk-python.md index 3373e3c75..6a31b83cc 100644 --- a/docs/sdk-python.md +++ b/docs/sdk-python.md @@ -234,7 +234,11 @@ When to hibernate is your policy, unless the deployment opts into hibernates idle claims automatically with the same transparent wake. A claim with a connection live when the sweep checks it (a relay stream, a buffered exec, a preview dial, an egress request) is not swept; the idle clock -restarts when that connection ends. +restarts when that connection ends. When the sandbox's pool sets +[`reader`](deploy.md#reading-sleeping-claims-without-waking-them), `stat`, +`read_file`, `list_dir` and `pull` on a hibernated or archived handle are served +without waking it, as of the guest `sync` just before the last hibernate, and the +handle's next call that is anything else wakes the sandbox on the same connection. Data-plane calls share a handle's relay connection: after a call the SDK keeps the connection for 30 seconds (`Client(..., keep_alive=...)` tunes the @@ -320,13 +324,9 @@ records, while `client.checkpoint(id)` creates an entry-node-bound handle for an already known id without listing. `new()` follows an owner redirect and may heal a missing record locally; `delete()` acts on the handle's bound node. -`delete()` also asks every peer that node currently sees to drop any replica a -heal pulled — best-effort eventual cleanup, not a fleet-wide revocation. A peer -that misses the broadcast (offline, partitioned, or joined later) keeps serving -branches from its replica until the node's `checkpoint_ttl_hours` ages it out; -enabling peer heal requires that TTL to be set, so every healed replica has a -cleanup bound while healing stays on. A node later run with healing off and -that TTL back at 0 keeps such a replica until an explicit delete. +`delete()` also asks the peers that node sees to drop any healed replica, as +best-effort cleanup rather than a fleet-wide revocation; see +[delete is eventual](cluster.md#delete-is-eventual-not-a-fleet-wide-revocation). ## Language servers (LSP) @@ -389,6 +389,9 @@ code = sb.run(["bash", "-c", "make test"], timeout=600) # seconds; TimeoutError past it ``` +`stdin` takes `bytes`, `bytearray` or `memoryview`; a non-empty `str` raises +`TypeError`. + `timeout` on `exec` and `run` is a wall clock over the dial and the run: at its end the SDK cuts the connection and kills the reported pid, and `TimeoutError` is raised. diff --git a/docs/sdk.md b/docs/sdk.md index 311d84a44..95493fea5 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -221,8 +221,9 @@ VM and returns only after every requested disk is mounted; the finalized name, effective mount, and (for `rw` entries) mode are available in `Sandbox.Volumes`. Custom mounts must be absolute and clean, stay outside the guest OS tree, and cannot duplicate or nest. -`Sandbox.ID`, `Sandbox.Deadline`, and -`Sandbox.FromCheckpoint` (the lineage edge when branched) are exported. +`Sandbox.ID`, `Sandbox.Deadline`, `Sandbox.NetRoute` (how the guest reaches the +network at claim) and `Sandbox.FromCheckpoint` (the lineage edge when branched) +are exported. `Sandbox.TemplateDigest` is the exact content identity when the claim cloned a promoted template; it is empty for other sources. `Owner()` names the owning node, and `Token()` returns the per-sandbox bearer to persist with `ID` for a @@ -483,14 +484,9 @@ heal a missing record locally; `Checkpoint.Delete` acts on the handle's bound node. Checkpoint creation is resource-creating and takes the api token, like fork. -`Checkpoint.Delete` also asks every peer that node currently sees to drop any -replica a heal pulled — best-effort eventual cleanup, not a fleet-wide -revocation. A peer that misses the broadcast (offline, partitioned, or joined -later) keeps serving branches from its replica until the node's -`checkpoint_ttl_hours` ages it out; enabling peer heal requires that TTL to be -set, so every healed replica has a cleanup bound while healing stays on. A node -later run with healing off and that TTL back at 0 keeps such a replica until an -explicit delete. +`Checkpoint.Delete` also asks the peers that node sees to drop any healed +replica, as best-effort cleanup rather than a fleet-wide revocation; see +[delete is eventual](cluster.md#delete-is-eventual-not-a-fleet-wide-revocation). ## Language servers (LSP) diff --git a/e2e/e2e_test.go b/e2e/e2e_test.go index a3531c73d..07413a1f6 100644 --- a/e2e/e2e_test.go +++ b/e2e/e2e_test.go @@ -33,15 +33,17 @@ import ( sandbox "github.com/cocoonstack/sandbox/sdk/go" ) +const nodeToken = "node-token" + var testKey = types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeSmall} -func TestEndToEnd(t *testing.T) { - stack := startStack(t, "node-token", config.PoolSpec{PoolKey: testKey, Warm: 1}) +func TestClaimColdThenWarmThenCloseThroughTheWholeStack(t *testing.T) { + st := startStack(t, config.PoolSpec{PoolKey: testKey, Warm: 1}) var sb *sandbox.Sandbox t.Run("cold claim before any refill", func(t *testing.T) { var err error - sb, err = stack.client.New(t.Context(), "rt:24.04") + sb, err = st.client.New(t.Context(), "rt:24.04") if err != nil { t.Fatalf("New: %v", err) } @@ -55,11 +57,8 @@ func TestEndToEnd(t *testing.T) { }) t.Run("refill then warm claim", func(t *testing.T) { - waitFor(t, func() bool { - infos, _ := stack.mgr.Info() - return len(infos) == 1 && infos[0].Warm >= 1 - }) - warm, err := stack.client.New(t.Context(), "rt:24.04") + waitFor(t, st.oneWarm) + warm, err := st.client.New(t.Context(), "rt:24.04") if err != nil { t.Fatalf("New: %v", err) } @@ -87,8 +86,8 @@ func TestEndToEnd(t *testing.T) { } func TestForkEndToEnd(t *testing.T) { - stack := startStack(t, "node-token") - parent, err := stack.client.New(t.Context(), "rt:24.04") + st := startStack(t) + parent, err := st.client.New(t.Context(), "rt:24.04") if err != nil { t.Fatalf("New: %v", err) } @@ -120,8 +119,8 @@ func TestForkEndToEnd(t *testing.T) { } func TestPromoteEndToEnd(t *testing.T) { - stack := startStack(t, "node-token") - parent, err := stack.client.New(t.Context(), "rt:24.04") + st := startStack(t) + parent, err := st.client.New(t.Context(), "rt:24.04") if err != nil { t.Fatalf("New: %v", err) } @@ -146,7 +145,7 @@ func TestPromoteEndToEnd(t *testing.T) { t.Errorf("exec on promoted claim: %q, %v", out, err) } _ = child.Close() - byName, nameErr := stack.client.New(t.Context(), "e2e-tpl:1") + byName, nameErr := st.client.New(t.Context(), "e2e-tpl:1") if nameErr != nil { t.Fatalf("claim promoted template by name: %v", nameErr) } @@ -158,15 +157,15 @@ func TestPromoteEndToEnd(t *testing.T) { if err := tpl.Delete(t.Context()); err != nil { t.Fatalf("Template.Delete: %v", err) } - if err := stack.client.DeleteTemplate(t.Context(), "e2e-tpl:1"); err == nil || + if err := st.client.DeleteTemplate(t.Context(), "e2e-tpl:1"); err == nil || !strings.Contains(err.Error(), "unknown template") { t.Errorf("second delete: %v, want unknown template", err) } } func TestRenewEndToEnd(t *testing.T) { - stack := startStack(t, "node-token") - sb, err := stack.client.New(t.Context(), "rt:24.04") + st := startStack(t) + sb, err := st.client.New(t.Context(), "rt:24.04") if err != nil { t.Fatalf("New: %v", err) } @@ -182,7 +181,7 @@ func TestRenewEndToEnd(t *testing.T) { if !sb.Deadline.Equal(longer) { t.Errorf("handle deadline %v, want the grant %v", sb.Deadline, longer) } - code, raw := rawJSON(t, stack, http.MethodGet, "/v1/sandboxes/"+sb.ID, "") + code, raw := rawJSON(t, st, http.MethodGet, "/v1/sandboxes/"+sb.ID, "") var held types.Sandbox if code != http.StatusOK || json.Unmarshal(raw, &held) != nil || !held.Deadline.Equal(longer) { t.Errorf("node holds %s (HTTP %d), want deadline %v", raw, code, longer) @@ -198,8 +197,8 @@ func TestRenewEndToEnd(t *testing.T) { } func TestCheckpointEndToEnd(t *testing.T) { - stack := startStack(t, "node-token") - src, err := stack.client.New(t.Context(), "rt:24.04") + st := startStack(t) + src, err := st.client.New(t.Context(), "rt:24.04") if err != nil { t.Fatalf("New: %v", err) } @@ -228,7 +227,7 @@ func TestCheckpointEndToEnd(t *testing.T) { } _ = branch.Close() - ckpts, err := stack.client.Checkpoints(t.Context()) + ckpts, err := st.client.Checkpoints(t.Context()) if err != nil || len(ckpts) != 1 || ckpts[0].ID != ckpt.ID { t.Fatalf("Checkpoints() = %+v, %v; want the one record", ckpts, err) } @@ -241,17 +240,17 @@ func TestCheckpointEndToEnd(t *testing.T) { } } -func TestTwoTenantFlow(t *testing.T) { - tenants := []tenants.TenantSpec{ +func TestTwoTenantsKeepTheirOwnCapsCheckpointsAndNoOperatorSurface(t *testing.T) { + specs := []tenants.TenantSpec{ {Name: "acme", Token: "acme-tok", MaxClaims: 1}, {Name: "beta", Token: "beta-tok"}, } - stack := startTenantStack(t, "node-token", tenants, nil) - acme, err := sandbox.Connect(stack.addr, sandbox.WithAPIToken("acme-tok")) + st := startTenantStack(t, specs, nil) + acme, err := sandbox.Connect(st.addr, sandbox.WithAPIToken("acme-tok")) if err != nil { t.Fatalf("connect acme: %v", err) } - beta, err := sandbox.Connect(stack.addr, sandbox.WithAPIToken("beta-tok")) + beta, err := sandbox.Connect(st.addr, sandbox.WithAPIToken("beta-tok")) if err != nil { t.Fatalf("connect beta: %v", err) } @@ -283,7 +282,7 @@ func TestTwoTenantFlow(t *testing.T) { }{ {acme, []string{"acme-step"}}, {beta, []string{"beta-step"}}, - {stack.client, []string{"acme-step", "beta-step"}}, + {st.client, []string{"acme-step", "beta-step"}}, } { ckpts, err := tt.client.Checkpoints(t.Context()) if err != nil { @@ -303,13 +302,13 @@ func TestTwoTenantFlow(t *testing.T) { !strings.Contains(err.Error(), "403") { t.Errorf("tenant on operator surface: %v, want 403", err) } - if _, err := stack.client.Info(t.Context()); err != nil { + if _, err := st.client.Info(t.Context()); err != nil { t.Errorf("root on operator surface: %v", err) } } func TestRuntimeTenantEndToEnd(t *testing.T) { - st := startTenantStack(t, "node-token", nil, nil) + st := startStack(t) if err := st.client.PutTenant(t.Context(), sandbox.TenantSpec{Name: "u:42", Token: "u42-tok", MaxClaims: 1}); err != nil { t.Fatalf("PutTenant: %v", err) } @@ -352,9 +351,9 @@ func TestRuntimeTenantEndToEnd(t *testing.T) { } func TestWrongAPITokenRejected(t *testing.T) { - stack := startStack(t, "node-token") + st := startStack(t) - bad, err := sandbox.Connect(stack.addr, sandbox.WithAPIToken("wrong")) + bad, err := sandbox.Connect(st.addr, sandbox.WithAPIToken("wrong")) if err != nil { t.Fatalf("connect: %v", err) } @@ -367,19 +366,16 @@ func TestWrongAPITokenRejected(t *testing.T) { func TestVolumesEndToEnd(t *testing.T) { image := writeVolumeImage(t, "dataset.img", "dataset-bytes") scratch := writeVolumeImage(t, "scratch.img", "scratch-bytes") - stack := startTenantStack(t, "node-token", nil, + st := startTenantStack(t, nil, []config.VolumeSpec{ {Name: "dataset", Path: image, DirectIO: "off"}, {Name: "scratch", Path: scratch, Writable: true}, }, config.PoolSpec{PoolKey: testKey, Warm: 1}) - waitFor(t, func() bool { - infos, _ := stack.mgr.Info() - return len(infos) == 1 && infos[0].Warm >= 1 - }) - warmBefore := stack.mgr.Counters().ClaimsWarm + waitFor(t, st.oneWarm) + warmBefore := st.mgr.Counters().ClaimsWarm - sb, err := stack.client.New(t.Context(), "rt:24.04", + sb, err := st.client.New(t.Context(), "rt:24.04", sandbox.WithVolumes(sandbox.Volume{Name: "dataset", Mount: "/datasets/e2e"})) if err != nil { t.Fatalf("volume claim: %v", err) @@ -388,12 +384,12 @@ func TestVolumesEndToEnd(t *testing.T) { if want := []sandbox.Volume{{Name: "dataset", Mount: "/datasets/e2e"}}; !slices.Equal(sb.Volumes, want) { t.Errorf("claim volumes %+v, want %+v", sb.Volumes, want) } - if counters := stack.mgr.Counters(); counters.ClaimsWarm != warmBefore+1 { - infos, _ := stack.mgr.Info() + if counters := st.mgr.Counters(); counters.ClaimsWarm != warmBefore+1 { + infos, _ := st.mgr.Info() t.Errorf("counters=%+v pools=%+v, want warm claims %d", counters, infos, warmBefore+1) } - infos, err := stack.client.Volumes(t.Context()) + infos, err := st.client.Volumes(t.Context()) if err != nil { t.Fatalf("Volumes: %v", err) } @@ -411,7 +407,7 @@ func TestVolumesEndToEnd(t *testing.T) { var listed struct { Volumes []map[string]any `json:"volumes"` } - _, body := rawJSON(t, stack, http.MethodGet, "/v1/volumes", "") + _, body := rawJSON(t, st, http.MethodGet, "/v1/volumes", "") if err := json.Unmarshal(body, &listed); err != nil { t.Fatalf("decode catalog %s: %v", body, err) } @@ -431,10 +427,10 @@ func TestVolumesEndToEnd(t *testing.T) { func TestWritableVolumeEndToEnd(t *testing.T) { scratch := writeVolumeImage(t, "scratch.img", "scratch-bytes") - stack := startTenantStack(t, "node-token", nil, + st := startTenantStack(t, nil, []config.VolumeSpec{{Name: "scratch", Path: scratch, Writable: true}}) - sb, err := stack.client.New(t.Context(), "rt:24.04", + sb, err := st.client.New(t.Context(), "rt:24.04", sandbox.WithVolumes(sandbox.Volume{Name: "scratch", Mount: "/datasets/rw", Mode: "rw"})) if err != nil { t.Fatalf("writable claim: %v", err) @@ -443,11 +439,11 @@ func TestWritableVolumeEndToEnd(t *testing.T) { t.Errorf("claim volumes %+v, want %+v", sb.Volumes, want) } applied := []string{"attach:scratch:rw", "mount:scratch:/datasets/rw:rw"} - if got := stack.eng.volumeOpsLog(); !slices.Equal(got, applied) { + if got := st.eng.volumeOpsLog(); !slices.Equal(got, applied) { t.Errorf("engine ops %v, want %v", got, applied) } for _, requested := range []string{`{"name":"scratch"}`, `{"name":"scratch","mode":"rw"}`} { - if status, _ := rawClaim(t, stack, requested); status != http.StatusConflict { + if status, _ := rawClaim(t, st, requested); status != http.StatusConflict { t.Errorf("claim %s under a live writer: %d, want 409", requested, status) } } @@ -456,14 +452,14 @@ func TestWritableVolumeEndToEnd(t *testing.T) { t.Fatalf("Close: %v", err) } want := slices.Concat(applied, []string{"umount:/datasets/rw", "remove"}) - if got := stack.eng.volumeOpsLog(); !slices.Equal(got, want) { + if got := st.eng.volumeOpsLog(); !slices.Equal(got, want) { t.Errorf("engine ops after release %v, want %v", got, want) } } func TestVolumeModeWireShape(t *testing.T) { scratch := writeVolumeImage(t, "scratch.img", "scratch-bytes") - stack := startTenantStack(t, "node-token", nil, + st := startTenantStack(t, nil, []config.VolumeSpec{{Name: "scratch", Path: scratch, Writable: true}}) for _, tt := range []struct { name string @@ -482,14 +478,14 @@ func TestVolumeModeWireShape(t *testing.T) { }, } { t.Run(tt.name, func(t *testing.T) { - status, claimed := rawClaim(t, stack, tt.requested) + status, claimed := rawClaim(t, st, tt.requested) if status != http.StatusOK { t.Fatalf("claim: %d, want 200", status) } if len(claimed.Volumes) != 1 || !maps.Equal(claimed.Volumes[0], tt.want) { t.Errorf("reply volumes %v, want [%v]", claimed.Volumes, tt.want) } - if err := stack.client.Attach(stack.addr, claimed.ID, claimed.Token).Close(); err != nil { + if err := st.client.Attach(st.addr, claimed.ID, claimed.Token).Close(); err != nil { t.Fatalf("release: %v", err) } }) @@ -497,14 +493,14 @@ func TestVolumeModeWireShape(t *testing.T) { } func TestClaimRefRoundTrip(t *testing.T) { - stack := startStack(t, "node-token") - sb, err := stack.client.New(t.Context(), "rt:24.04", sandbox.WithClaimRef("ns/workload")) + st := startStack(t) + sb, err := st.client.New(t.Context(), "rt:24.04", sandbox.WithClaimRef("ns/workload")) if err != nil { t.Fatalf("claim: %v", err) } defer sb.Close() - list, err := stack.client.Sandboxes(t.Context()) + list, err := st.client.Sandboxes(t.Context()) if err != nil { t.Fatalf("list sandboxes: %v", err) } @@ -518,7 +514,7 @@ func TestClaimRefRoundTrip(t *testing.T) { } func TestClaimMetadataRoundTrip(t *testing.T) { - st := startStack(t, "node-token") + st := startStack(t) sb, err := st.client.New(t.Context(), "rt:24.04", sandbox.WithMetadata(map[string]string{"team": "a"})) if err != nil { t.Fatalf("claim: %v", err) @@ -558,7 +554,7 @@ func TestClaimEnvEndToEnd(t *testing.T) { _, _ = io.WriteString(w, r.Header.Get("Authorization")) })) t.Cleanup(origin.Close) - st := startConfigStack(t, "node-token", &config.Config{ + st := startConfigStack(t, &config.Config{ Secrets: []egress.SecretSpec{{Name: "GW_KEY", Header: "Authorization"}}, EgressInternalAllow: []string{"127.0.0.1/32"}, Pools: []config.PoolSpec{{PoolKey: testKey, Warm: 2, Egress: &egress.Policy{Allow: []egress.Rule{{Host: "127.0.0.1", Secret: "GW_KEY"}}}}}, @@ -590,7 +586,10 @@ func TestClaimEnvEndToEnd(t *testing.T) { t.Fatalf("egress: %v", err) } defer resp.Body.Close() - body, _ := io.ReadAll(resp.Body) + body, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatalf("egress body: %v", err) + } return string(body) } @@ -600,10 +599,10 @@ func TestClaimEnvEndToEnd(t *testing.T) { t.Fatalf("injected %q / %q, want each claim's own value", gotA, gotB) } st.eng.mu.Lock() - file := st.eng.guestEnvs[sockA] + guestEnv := st.eng.guestEnvs[sockA] st.eng.mu.Unlock() - if file != "AGENT_MODE=\"on\"\n" { - t.Errorf("guest env file %q, want only the guest entry", file) + if guestEnv != "AGENT_MODE=\"on\"\n" { + t.Errorf("guest env file %q, want only the guest entry", guestEnv) } if status, body := rawJSON(t, st, http.MethodPut, "/v1/sandboxes/"+idA+"/env", `{"env":{"GW_KEY":{"value":"Bearer rotated","guest":false}}}`); status != http.StatusNoContent { t.Fatalf("put env: %d %s", status, body) @@ -616,8 +615,8 @@ func TestClaimEnvEndToEnd(t *testing.T) { t.Errorf("read env: %d %s, want the host-only name without its value", status, body) } for _, route := range []string{"/v1/sandboxes/" + idA, "/v1/sandboxes"} { - if _, body := rawJSON(t, st, http.MethodGet, route, ""); strings.Contains(string(body), "Bearer") { - t.Errorf("GET %s serves a host-only value: %s", route, body) + if status, body := rawJSON(t, st, http.MethodGet, route, ""); status != http.StatusOK || strings.Contains(string(body), "Bearer") { + t.Errorf("GET %s: %d %s, want 200 without a host-only value", route, status, body) } } if status, _ := rawJSON(t, st, http.MethodPut, "/v1/sandboxes/"+idA+"/env", `{"env":{"BAD NAME":{"value":"x"}}}`); status != http.StatusBadRequest { @@ -626,7 +625,7 @@ func TestClaimEnvEndToEnd(t *testing.T) { } func TestClaimEnvThroughTheSDK(t *testing.T) { - st := startStack(t, "node-token", config.PoolSpec{PoolKey: testKey, Warm: 1}) + st := startStack(t, config.PoolSpec{PoolKey: testKey, Warm: 1}) guestFile := func(sb *sandbox.Sandbox) string { t.Helper() sock, err := st.mgr.AgentSocket(sb.ID, sb.Token()) @@ -688,7 +687,7 @@ func TestClaimEnvThroughTheSDK(t *testing.T) { } func TestForkClaimRefPrefixWireShape(t *testing.T) { - st := startStack(t, "node-token") + st := startStack(t) status, body := rawJSON(t, st, http.MethodPost, "/v1/claim", `{"template":"rt:24.04"}`) if status != http.StatusOK { t.Fatalf("claim: %d %s", status, body) @@ -752,10 +751,10 @@ func TestForkClaimRefPrefixWireShape(t *testing.T) { func TestAttachOnlyVolumeEndToEnd(t *testing.T) { scratch := writeVolumeImage(t, "scratch.img", "scratch-bytes") - stack := startTenantStack(t, "node-token", nil, + st := startTenantStack(t, nil, []config.VolumeSpec{{Name: "scratch", Path: scratch, Writable: true}}) - sb, err := stack.client.New(t.Context(), "rt:24.04", + sb, err := st.client.New(t.Context(), "rt:24.04", sandbox.WithVolumes(sandbox.Volume{Name: "scratch", Mode: "rw"}), sandbox.WithVolumesAttachOnly()) if err != nil { @@ -765,12 +764,12 @@ func TestAttachOnlyVolumeEndToEnd(t *testing.T) { t.Errorf("claim volumes %+v, want %+v", sb.Volumes, want) } applied := []string{"attach:scratch:rw"} - if got := stack.eng.volumeOpsLog(); !slices.Equal(got, applied) { + if got := st.eng.volumeOpsLog(); !slices.Equal(got, applied) { t.Errorf("engine ops %v, want %v", got, applied) } assertNoDirtyMarker(t, scratch, "apply") for _, requested := range []string{`{"name":"scratch"}`, `{"name":"scratch","mode":"rw"}`} { - if status, _ := rawClaim(t, stack, requested); status != http.StatusConflict { + if status, _ := rawClaim(t, st, requested); status != http.StatusConflict { t.Errorf("claim %s under a live attach-only writer: %d, want 409", requested, status) } } @@ -779,7 +778,7 @@ func TestAttachOnlyVolumeEndToEnd(t *testing.T) { t.Fatalf("Close: %v", err) } want := slices.Concat(applied, []string{"remove"}) - if got := stack.eng.volumeOpsLog(); !slices.Equal(got, want) { + if got := st.eng.volumeOpsLog(); !slices.Equal(got, want) { t.Errorf("engine ops after release %v, want %v", got, want) } assertNoDirtyMarker(t, scratch, "release") @@ -787,7 +786,7 @@ func TestAttachOnlyVolumeEndToEnd(t *testing.T) { func TestAttachOnlyVolumeWireShape(t *testing.T) { scratch := writeVolumeImage(t, "scratch.img", "scratch-bytes") - stack := startTenantStack(t, "node-token", nil, + st := startTenantStack(t, nil, []config.VolumeSpec{{Name: "scratch", Path: scratch, Writable: true}}) for _, tt := range []struct { name string @@ -806,7 +805,7 @@ func TestAttachOnlyVolumeWireShape(t *testing.T) { }, } { t.Run(tt.name, func(t *testing.T) { - status, body := rawJSON(t, stack, http.MethodPost, "/v1/claim", tt.request) + status, body := rawJSON(t, st, http.MethodPost, "/v1/claim", tt.request) if status != http.StatusOK { t.Fatalf("claim: %d %s, want 200", status, body) } @@ -829,7 +828,7 @@ func TestAttachOnlyVolumeWireShape(t *testing.T) { if err := json.Unmarshal(body, &claimed); err != nil { t.Fatalf("decode claim %s: %v", body, err) } - if err := stack.client.Attach(stack.addr, claimed.ID, claimed.Token).Close(); err != nil { + if err := st.client.Attach(st.addr, claimed.ID, claimed.Token).Close(); err != nil { t.Fatalf("release: %v", err) } }) @@ -841,13 +840,19 @@ func TestDirtyVolumeRefusesReader(t *testing.T) { if err := os.WriteFile(scratch+".dirty", nil, 0o600); err != nil { t.Fatalf("write dirty marker: %v", err) } - stack := startTenantStack(t, "node-token", nil, + st := startTenantStack(t, nil, []config.VolumeSpec{{Name: "scratch", Path: scratch, Writable: true}}) - if status, _ := rawClaim(t, stack, `{"name":"scratch"}`); status != http.StatusConflict { + if status, _ := rawClaim(t, st, `{"name":"scratch"}`); status != http.StatusConflict { t.Errorf("read-only claim on a dirty image: %d, want 409", status) } } +type rawClaimResponse struct { + ID string `json:"id"` + Token string `json:"token"` + Volumes []map[string]any `json:"volumes"` +} + type stack struct { client *sandbox.Client mgr *pool.Manager @@ -856,23 +861,28 @@ type stack struct { token string } -func startStack(t *testing.T, apiToken string, pools ...config.PoolSpec) *stack { +func (s *stack) oneWarm() bool { + infos, _ := s.mgr.Info() + return len(infos) == 1 && infos[0].Warm >= 1 +} + +func startStack(t *testing.T, pools ...config.PoolSpec) *stack { t.Helper() - return startTenantStack(t, apiToken, nil, nil, pools...) + return startTenantStack(t, nil, nil, pools...) } -func startTenantStack(t *testing.T, apiToken string, tenants []tenants.TenantSpec, volumes []config.VolumeSpec, pools ...config.PoolSpec) *stack { +func startTenantStack(t *testing.T, specs []tenants.TenantSpec, volumes []config.VolumeSpec, pools ...config.PoolSpec) *stack { t.Helper() - st := startConfigStack(t, apiToken, &config.Config{Pools: pools, Volumes: volumes}) - if len(tenants) > 0 { - if err := st.mgr.SetTenants(t.Context(), tenants); err != nil { + st := startConfigStack(t, &config.Config{Pools: pools, Volumes: volumes}) + if len(specs) > 0 { + if err := st.mgr.SetTenants(t.Context(), specs); err != nil { t.Fatalf("set tenants: %v", err) } } return st } -func startConfigStack(t *testing.T, apiToken string, cfg *config.Config) *stack { +func startConfigStack(t *testing.T, cfg *config.Config) *stack { t.Helper() dir, err := os.MkdirTemp("", "sbx") @@ -886,7 +896,7 @@ func startConfigStack(t *testing.T, apiToken string, cfg *config.Config) *stack if err != nil { t.Fatalf("secrets: %v", err) } - cfg.DataDir, cfg.APIToken = dir, apiToken + cfg.DataDir, cfg.APIToken = dir, nodeToken ckpts, err := storedir.New(filepath.Join(dir, "checkpoints"), store.CheckpointIDRe) if err != nil { t.Fatalf("checkpoint store: %v", err) @@ -895,11 +905,7 @@ func startConfigStack(t *testing.T, apiToken string, cfg *config.Config) *stack if err != nil { t.Fatalf("template store: %v", err) } - var rootSum string - if apiToken != "" { - rootSum = tenants.TokenSHA256(apiToken) - } - src, err := file.Open(t.Context(), dir, rootSum) + src, err := file.Open(t.Context(), dir, tenants.TokenSHA256(nodeToken)) if err != nil { t.Fatalf("tenant set: %v", err) } @@ -909,14 +915,14 @@ func startConfigStack(t *testing.T, apiToken string, cfg *config.Config) *stack } go mgr.Run(t.Context()) - ts := httptest.NewServer(server.New(apiToken, "", mgr, eng.real, nil, nil, nil, nil).Handler()) + ts := httptest.NewServer(server.New(nodeToken, "", mgr, eng.real, nil, nil, nil, nil).Handler()) t.Cleanup(ts.Close) addr := strings.TrimPrefix(ts.URL, "http://") - client, err := sandbox.Connect(addr, sandbox.WithAPIToken(apiToken)) + client, err := sandbox.Connect(addr, sandbox.WithAPIToken(nodeToken)) if err != nil { t.Fatalf("connect: %v", err) } - return &stack{client: client, mgr: mgr, eng: eng, addr: addr, token: apiToken} + return &stack{client: client, mgr: mgr, eng: eng, addr: addr, token: nodeToken} } func waitFor(t *testing.T, cond func() bool) { @@ -947,12 +953,6 @@ func assertNoDirtyMarker(t *testing.T, image, when string) { } } -type rawClaimResponse struct { - ID string `json:"id"` - Token string `json:"token"` - Volumes []map[string]any `json:"volumes"` -} - func rawClaim(t *testing.T, st *stack, volume string) (int, rawClaimResponse) { t.Helper() status, body := rawJSON(t, st, http.MethodPost, "/v1/claim", diff --git a/e2e/reader_test.go b/e2e/reader_test.go index 1266683da..05d29b0a1 100644 --- a/e2e/reader_test.go +++ b/e2e/reader_test.go @@ -11,7 +11,7 @@ import ( ) func TestReaderServesAHibernatedClaimUntilAWriteWakesIt(t *testing.T) { - st := startConfigStack(t, "node-token", &config.Config{ + st := startConfigStack(t, &config.Config{ Pools: []config.PoolSpec{{PoolKey: testKey, Warm: 2, Reader: true}}, }) root := t.TempDir() diff --git a/e2e/tls_test.go b/e2e/tls_test.go index 35b6e99b5..4641471f5 100644 --- a/e2e/tls_test.go +++ b/e2e/tls_test.go @@ -31,11 +31,11 @@ func TestCaddyTLSCluster(t *testing.T) { a, b := la.Addr().String(), lb.Addr().String() owner := "https://" + b entry := "https://" + a - st := startStack(t, "node-token") + st := startStack(t) ownerServer := server.New(st.token, owner, st.mgr, st.eng.real, nil, nil, nil, nil) backend := httptest.NewServer(ownerServer.Handler()) t.Cleanup(func() { backend.Close(); ownerServer.CloseRelays() }) - origin := startStack(t, "node-token") + origin := startStack(t) placer := &edgePlacer{internal: "owner.invalid:7777", public: owner} entryServer := server.New(origin.token, entry, origin.mgr, origin.eng.real, placer, nil, nil, nil) front := httptest.NewServer(entryServer.Handler()) diff --git a/e2e/volume_catalog_test.go b/e2e/volume_catalog_test.go index 22667f10c..822f9273c 100644 --- a/e2e/volume_catalog_test.go +++ b/e2e/volume_catalog_test.go @@ -13,7 +13,7 @@ import ( ) func TestRuntimeVolumeRegistrationThroughHTTP(t *testing.T) { - st := startTenantStack(t, "node-token", []tenants.TenantSpec{ + st := startTenantStack(t, []tenants.TenantSpec{ {Name: "acme", Token: "acme-token"}, {Name: "beta", Token: "beta-token"}, }, []config.VolumeSpec{{Name: "public", Path: writeVolumeImage(t, "public.img", "shared")}}) for name, body := range map[string]map[string]any{ @@ -29,12 +29,12 @@ func TestRuntimeVolumeRegistrationThroughHTTP(t *testing.T) { } } home := sandbox.Volume{Name: "home", Mount: "/mnt/home"} - for _, tc := range []struct { + for _, tt := range []struct { token string want []string private bool }{{"acme-token", []string{"home", "private", "public"}, true}, {"beta-token", []string{"home", "public"}, false}} { - client, err := sandbox.Connect(st.addr, sandbox.WithAPIToken(tc.token)) + client, err := sandbox.Connect(st.addr, sandbox.WithAPIToken(tt.token)) if err != nil { t.Fatal(err) } @@ -43,8 +43,8 @@ func TestRuntimeVolumeRegistrationThroughHTTP(t *testing.T) { for _, volume := range volumes { names = append(names, volume.Name) } - if err != nil || !slices.Equal(names, tc.want) { - t.Fatalf("catalog for %s: names=%v want=%v err=%v", tc.token, names, tc.want, err) + if err != nil || !slices.Equal(names, tt.want) { + t.Fatalf("catalog for %s: names=%v want=%v err=%v", tt.token, names, tt.want, err) } claim, err := client.New(t.Context(), "rt:24.04", sandbox.WithVolumes(home)) if err != nil || !slices.Equal(claim.Volumes, []sandbox.Volume{home}) { @@ -55,8 +55,8 @@ func TestRuntimeVolumeRegistrationThroughHTTP(t *testing.T) { t.Fatal(err) } view, err := claim.PatchVolumes(t.Context(), sandbox.VolumePatch{Add: []sandbox.Volume{home, {Name: "private"}}}) - if (err == nil) != tc.private || tc.private && !slices.Contains(view.Volumes, home) { - t.Fatalf("live attach for %s: view=%+v err=%v", tc.token, view, err) + if (err == nil) != tt.private || tt.private && !slices.Contains(view.Volumes, home) { + t.Fatalf("live attach for %s: view=%+v err=%v", tt.token, view, err) } } caller := *st @@ -75,7 +75,7 @@ func TestRuntimeVolumeRegistrationThroughHTTP(t *testing.T) { } func TestRuntimeVolumeDetachHibernateWakeThroughHTTP(t *testing.T) { - st := startTenantStack(t, "node-token", []tenants.TenantSpec{{Name: "acme", Token: "acme-token"}}, nil) + st := startTenantStack(t, []tenants.TenantSpec{{Name: "acme", Token: "acme-token"}}, nil) request, err := json.Marshal(map[string]any{"path": writeVolumeImage(t, "home.img", "external bytes"), "writable": true, "tenants": []string{"acme"}}) if err != nil { t.Fatal(err) diff --git a/e2e/volume_mutation_test.go b/e2e/volume_mutation_test.go index 1ad074dae..eb31e207d 100644 --- a/e2e/volume_mutation_test.go +++ b/e2e/volume_mutation_test.go @@ -13,8 +13,8 @@ import ( func TestLiveVolumesEndToEnd(t *testing.T) { image := writeVolumeImage(t, "dataset.img", "dataset-bytes") - stack := startTenantStack(t, "node-token", nil, []config.VolumeSpec{{Name: "dataset", Path: image}}) - sb, err := stack.client.New(t.Context(), "rt:24.04") + st := startTenantStack(t, nil, []config.VolumeSpec{{Name: "dataset", Path: image}}) + sb, err := st.client.New(t.Context(), "rt:24.04") if err != nil { t.Fatal(err) } @@ -35,15 +35,15 @@ func TestLiveVolumesEndToEnd(t *testing.T) { } } want := []string{"attach:dataset:ro", "mount:dataset:/volumes/dataset:ro", "umount:/volumes/dataset", "detach:dataset"} - if ops := stack.eng.volumeOpsLog(); !slices.Equal(ops, want) { + if ops := st.eng.volumeOpsLog(); !slices.Equal(ops, want) { t.Fatalf("device operations=%v want=%v", ops, want) } } func TestRejectedVolumeTargetExplainsRefusalWithoutChangingState(t *testing.T) { image := writeVolumeImage(t, "dataset.img", "dataset-bytes") - stack := startTenantStack(t, "node-token", nil, []config.VolumeSpec{{Name: "dataset", Path: image}}) - sb, err := stack.client.New(t.Context(), "rt:24.04") + st := startTenantStack(t, nil, []config.VolumeSpec{{Name: "dataset", Path: image}}) + sb, err := st.client.New(t.Context(), "rt:24.04") if err != nil { t.Fatal(err) } @@ -63,8 +63,8 @@ func TestRejectedVolumeTargetExplainsRefusalWithoutChangingState(t *testing.T) { func TestEmptyVolumePatchChangesNothing(t *testing.T) { image := writeVolumeImage(t, "dataset.img", "dataset-bytes") - stack := startTenantStack(t, "node-token", nil, []config.VolumeSpec{{Name: "dataset", Path: image}}) - sb, err := stack.client.New(t.Context(), "rt:24.04") + st := startTenantStack(t, nil, []config.VolumeSpec{{Name: "dataset", Path: image}}) + sb, err := st.client.New(t.Context(), "rt:24.04") if err != nil { t.Fatal(err) } @@ -91,8 +91,8 @@ func TestMixedLiveVolumeModesThroughSDK(t *testing.T) { Name: volume.Name, Path: writeVolumeImage(t, volume.Name+".img", "seed"), Writable: volume.Mode == "rw", }) } - stack := startTenantStack(t, "node-token", nil, catalog) - sb, err := stack.client.New(t.Context(), "rt:24.04") + st := startTenantStack(t, nil, catalog) + sb, err := st.client.New(t.Context(), "rt:24.04") if err != nil { t.Fatal(err) } @@ -106,7 +106,7 @@ func TestMixedLiveVolumeModesThroughSDK(t *testing.T) { if !slices.Equal(view.Volumes, want) || !slices.Equal(sb.Volumes, want) { t.Fatalf("mixed SDK attachment = %+v, handle=%+v, want %+v", view, sb.Volumes, want) } - status, body := rawJSON(t, stack, http.MethodGet, "/v1/sandboxes/"+sb.ID+"/volumes", "") + status, body := rawJSON(t, st, http.MethodGet, "/v1/sandboxes/"+sb.ID+"/volumes", "") if status != http.StatusOK || strings.Count(string(body), `"attach_only":true`) != 4 { t.Fatalf("mixed HTTP view lost explicit attach_only in desired or applied entries: status=%d body=%s", status, body) } @@ -122,7 +122,7 @@ func TestMixedLiveVolumeModesThroughSDK(t *testing.T) { if err != nil || len(view.Volumes) != 0 || len(view.VolumesWant) != 0 || view.Converging { t.Fatalf("mixed removal = %+v, err=%v", view, err) } - initial, err := stack.client.New(t.Context(), "rt:24.04", sandbox.WithVolumes(volumes...)) + initial, err := st.client.New(t.Context(), "rt:24.04", sandbox.WithVolumes(volumes...)) if err != nil { t.Fatalf("mixed initial claim: %v", err) } diff --git a/sandboxd/archives/archivestest/archivestest.go b/sandboxd/archives/archivestest/archivestest.go index 8db9782e6..f66030b46 100644 --- a/sandboxd/archives/archivestest/archivestest.go +++ b/sandboxd/archives/archivestest/archivestest.go @@ -344,8 +344,8 @@ func testTakeOwned(t *testing.T, r archives.Registry) { t.Errorf("%s moved to epoch %d, want it left at %d", id, got.Epoch, want) } } - if _, ok, _ := r.Get(ctx, "sb_none"); ok { - t.Error("take owned inserted a row") + if _, ok, getErr := r.Get(ctx, "sb_none"); ok || getErr != nil { + t.Errorf("take owned inserted a row: %v %v", ok, getErr) } if taken, err = r.TakeOwned(ctx, "a", nil); err != nil || len(taken) != 0 { t.Errorf("take owned of no ids: %v %v, want none", taken, err) @@ -466,8 +466,8 @@ func testReplace(t *testing.T, r archives.Registry) { if _, err := r.Replace(ctx, absent, 7); !errors.Is(err, archives.ErrLost) { t.Errorf("replace of an absent row at a held epoch: %v, want ErrLost", err) } - if _, ok, _ := r.Get(ctx, "sb_none"); ok { - t.Error("a refused replace inserted a row") + if _, ok, err := r.Get(ctx, "sb_none"); ok || err != nil { + t.Errorf("a refused replace inserted a row: %v %v", ok, err) } first, err := r.Replace(ctx, record("sb_1", "a", time.Time{}), 0) if err != nil || first <= 0 { diff --git a/sandboxd/archives/pg/pg.go b/sandboxd/archives/pg/pg.go index 6830bf3c4..1c0668561 100644 --- a/sandboxd/archives/pg/pg.go +++ b/sandboxd/archives/pg/pg.go @@ -83,25 +83,16 @@ func (r *Registry) TakeOwned(ctx context.Context, owner string, expect map[strin for id, epoch := range expect { ids, epochs = append(ids, id), append(epochs, epoch) } - qctx, cancel := context.WithTimeout(ctx, metastore.QueryTimeout) - defer cancel() - rows, err := r.pool.Query(qctx, `UPDATE sandboxd_archives a SET epoch = nextval('sandboxd_archives_epoch'), updated_at = now() + recs, err := r.collect(ctx, `UPDATE sandboxd_archives a SET epoch = nextval('sandboxd_archives_epoch'), updated_at = now() FROM unnest($2::text[], $3::bigint[]) AS t(id, epoch) WHERE a.cell = $1 AND a.id = t.id AND a.epoch = t.epoch AND a.owner = $4 AND NOT a.consumed - RETURNING a.id, a.epoch`, r.cell, ids, epochs, owner) + RETURNING a.id, a.tenant, a.token_sha256, a.owner, a.epoch, a.consumed, a.checkpoint, a.deadline, NULL::jsonb`, r.cell, ids, epochs, owner) if err != nil { - return nil, unavailable(err) + return nil, err } - taken := make(map[string]int64, len(expect)) - var ( - id string - epoch int64 - ) - if _, err = pgx.ForEachRow(rows, []any{&id, &epoch}, func() error { - taken[id] = epoch - return nil - }); err != nil { - return nil, unavailable(err) + taken := make(map[string]int64, len(recs)) + for _, rec := range recs { + taken[rec.ID] = rec.Epoch } return taken, nil } diff --git a/sandboxd/archives/pg/pg_test.go b/sandboxd/archives/pg/pg_test.go index b9216e48c..d00a1f3c1 100644 --- a/sandboxd/archives/pg/pg_test.go +++ b/sandboxd/archives/pg/pg_test.go @@ -42,7 +42,7 @@ func TestCellsKeepTheirOwnRows(t *testing.T) { if recs, listErr := c2.Sweepable(ctx, time.Now(), "n2"); listErr != nil || len(recs) != 0 { t.Errorf("c2 lists c1's expired row: %+v %v", recs, listErr) } - if _, err = c2.Put(ctx, archives.Record{ID: "sb_1", TokenSHA256: "cd", Owner: "n2", Checkpoint: "ck_2", Claim: []byte(`{}`)}); err != nil { + if _, err = c2.Put(ctx, sb1Record("cd", "n2", "ck_2")); err != nil { t.Errorf("put of the same id in c2 by another owner: %v, want its own row", err) } if rec, _, _ := c1.Get(ctx, "sb_1"); rec.Checkpoint != "ck_1" { @@ -63,7 +63,7 @@ func TestTwoNodesShareTheTable(t *testing.T) { dsn := metastoretest.PGSchema(t) a, b := open(t, dsn, "c1"), open(t, dsn, "c1") ctx := t.Context() - epoch, err := a.Put(ctx, archives.Record{ID: "sb_1", TokenSHA256: "ab", Owner: "a", Checkpoint: "ck_1", Claim: []byte(`{}`)}) + epoch, err := a.Put(ctx, sb1Record("ab", "a", "ck_1")) if err != nil { t.Fatalf("put on a: %v", err) } @@ -73,7 +73,7 @@ func TestTwoNodesShareTheTable(t *testing.T) { if _, err = a.Take(ctx, "sb_1", epoch, "a"); !errors.Is(err, archives.ErrLost) { t.Errorf("a's take at its old epoch: %v, want ErrLost", err) } - if _, err = a.Put(ctx, archives.Record{ID: "sb_1", TokenSHA256: "ab", Owner: "a", Checkpoint: "ck_2", Claim: []byte(`{}`)}); !errors.Is(err, archives.ErrLost) { + if _, err = a.Put(ctx, sb1Record("ab", "a", "ck_2")); !errors.Is(err, archives.ErrLost) { t.Errorf("a's put over b's row: %v, want ErrLost", err) } } @@ -118,7 +118,7 @@ func TestLeasesAreScopedToTheirCell(t *testing.T) { if err := c1.Renew(ctx, "a", time.Hour); err != nil { t.Fatalf("renew in c1: %v", err) } - epoch, err := c2.Put(ctx, archives.Record{ID: "sb_1", TokenSHA256: "ab", Owner: "a", Checkpoint: "ck_1", Claim: []byte(`{}`)}) + epoch, err := c2.Put(ctx, sb1Record("ab", "a", "ck_1")) if err != nil { t.Fatalf("put in c2: %v", err) } @@ -155,3 +155,7 @@ func open(t *testing.T, dsn, cell string) *Registry { t.Cleanup(r.Close) return r } + +func sb1Record(token, owner, checkpoint string) archives.Record { + return archives.Record{ID: "sb_1", TokenSHA256: token, Owner: owner, Checkpoint: checkpoint, Claim: []byte(`{}`)} +} diff --git a/sandboxd/config/config_test.go b/sandboxd/config/config_test.go index f1750c119..e1cccb220 100644 --- a/sandboxd/config/config_test.go +++ b/sandboxd/config/config_test.go @@ -14,10 +14,7 @@ import ( func TestLoadAppliesDefaults(t *testing.T) { path := writeConfig(t, `{"pools":[{"template":"rt:24.04","net":"none","size":"small"}]}`) - cfg, err := Load(path) - if err != nil { - t.Fatalf("Load: %v", err) - } + cfg := mustLoad(t, path) if cfg.Listen == "" || cfg.DataDir == "" || cfg.CocoonBin == "" { t.Errorf("defaults not applied: %+v", cfg) } @@ -42,10 +39,7 @@ func TestLoadAppliesDefaults(t *testing.T) { } func TestLoadReadsCaptureTrim(t *testing.T) { - cfg, err := Load(writeConfig(t, `{"pools":[{"template":"rt:24.04","capture_trim":true},{"template":"py:3.12"}]}`)) - if err != nil { - t.Fatalf("Load: %v", err) - } + cfg := mustLoad(t, writeConfig(t, `{"pools":[{"template":"rt:24.04","capture_trim":true},{"template":"py:3.12"}]}`)) if !cfg.Pools[0].CaptureTrim || cfg.Pools[1].CaptureTrim { t.Errorf("capture_trim %v/%v, want on for the pool that names it and off by default", cfg.Pools[0].CaptureTrim, cfg.Pools[1].CaptureTrim) } @@ -184,9 +178,9 @@ func TestLoadRejectsInvalid(t *testing.T) { func TestLoadMetaStore(t *testing.T) { t.Setenv("SANDBOX_TEST_DSN", "postgres://sandboxd@db/sandboxd") ms := `"meta_store":{"kind":"pg","dsn_env":"SANDBOX_TEST_DSN"}` - cfg, err := Load(writeConfig(t, `{"api_token":"r","pools":[],`+ms+`}`)) - if err != nil || cfg.MetaStore.DSNEnv != "SANDBOX_TEST_DSN" || cfg.MetaStore.Cell != "" { - t.Fatalf("Load: %+v %v, want no cell unless one is set", cfg, err) + cfg := mustLoad(t, writeConfig(t, `{"api_token":"r","pools":[],`+ms+`}`)) + if cfg.MetaStore.DSNEnv != "SANDBOX_TEST_DSN" || cfg.MetaStore.Cell != "" { + t.Fatalf("Load: %+v, want no cell unless one is set", cfg) } if _, err := Load(writeConfig(t, `{"pools":[],`+ms+`}`)); err == nil || !strings.Contains(err.Error(), "needs api_token") { t.Errorf("meta_store without api_token: %v, want a refusal", err) @@ -221,9 +215,9 @@ func TestLoadArchiveFailover(t *testing.T) { } return b + "}" } - cfg, err := Load(writeConfig(t, body(""))) - if err != nil || !cfg.ArchiveFailover || cfg.Mesh.ID() != "node1:7946" { - t.Fatalf("Load: %+v %v, want failover on with the bind as node id", cfg, err) + cfg := mustLoad(t, writeConfig(t, body(""))) + if !cfg.ArchiveFailover || cfg.Mesh.ID() != "node1:7946" { + t.Fatalf("Load: %+v, want failover on with the bind as node id", cfg) } for name := range parts { if _, err := Load(writeConfig(t, body(name))); err == nil || !strings.Contains(err.Error(), "archive_failover requires") { @@ -244,9 +238,9 @@ func TestLoadReplicateSeconds(t *testing.T) { body := func(on, pool string) string { return `{"api_token":"r","advertise_addr":"node1:7777",` + on + `"pools":[` + pool + `]}` } - cfg, err := Load(writeConfig(t, body(failover, `{"template":"rt:24.04","replicate_seconds":60}`))) - if err != nil || cfg.Pools[0].ReplicateSeconds != 60 { - t.Fatalf("Load: %+v %v, want replicate_seconds 60", cfg, err) + cfg := mustLoad(t, writeConfig(t, body(failover, `{"template":"rt:24.04","replicate_seconds":60}`))) + if cfg.Pools[0].ReplicateSeconds != 60 { + t.Fatalf("Load: %+v, want replicate_seconds 60", cfg) } for name, tt := range map[string]struct{ on, pool, want string }{ "no failover": {"", `{"template":"rt:24.04","replicate_seconds":60}`, "requires archive_failover"}, @@ -267,10 +261,7 @@ func TestLoadEgressUpstream(t *testing.T) { body := `{"egress_upstream":{"claim_env":"EGRESS_UPSTREAM","allow":["res.example","10.1.0.0/16"]},"egress_usage_bytes":true, "api_token":"r","egress_classes":[{"name":"desk","egress":{},"egress_upstream_env":"SANDBOX_TEST_DC"}], "pools":[{"template":"rt:24.04","egress_upstream_env":"SANDBOX_TEST_RES"}]}` - cfg, err := Load(writeConfig(t, body)) - if err != nil { - t.Fatalf("Load: %v", err) - } + cfg := mustLoad(t, writeConfig(t, body)) if cfg.EgressUpstream.ClaimEnv != "EGRESS_UPSTREAM" || !cfg.EgressUsageBytes || cfg.Pools[0].EgressUpstreamEnv != "SANDBOX_TEST_RES" { t.Errorf("loaded %+v / %+v", cfg.EgressUpstream, cfg.Pools[0]) } @@ -292,10 +283,7 @@ func TestLoadAcceptsPreviewAdvertiseShapes(t *testing.T) { } func TestLoadReadsReader(t *testing.T) { - cfg, err := Load(writeConfig(t, `{"pools":[{"template":"rt:24.04","reader":true},{"template":"py:3.12"}]}`)) - if err != nil { - t.Fatalf("Load: %v", err) - } + cfg := mustLoad(t, writeConfig(t, `{"pools":[{"template":"rt:24.04","reader":true},{"template":"py:3.12"}]}`)) if !cfg.Pools[0].Reader || cfg.Pools[1].Reader { t.Errorf("reader %v/%v, want on for the pool that names it and off by default", cfg.Pools[0].Reader, cfg.Pools[1].Reader) } @@ -305,10 +293,7 @@ func TestLoadAcceptsVolumes(t *testing.T) { path := writeConfig(t, `{"pools":[],"volume_roots":["/srv/models/"],"volumes":[ {"name":"imagenet","path":"/srv/datasets/imagenet.img"}, {"name":"weights-llama","path":"/srv/datasets/llama.img","directio":"on"}]}`) - cfg, err := Load(path) - if err != nil { - t.Fatalf("Load: %v", err) - } + cfg := mustLoad(t, path) if len(cfg.Volumes) != 2 || cfg.Volumes[0].DirectIO != types.DirectIOOff || cfg.Volumes[1].DirectIO != types.DirectIOOn { t.Errorf("volumes = %+v", cfg.Volumes) @@ -321,10 +306,7 @@ func TestLoadAcceptsVolumes(t *testing.T) { func TestLoadAcceptsVolumeTenantAccessList(t *testing.T) { path := writeConfig(t, `{"api_token":"root","pools":[], "volumes":[{"name":"corpus","path":"/srv/datasets/corpus.img","tenants":["acme"]}]}`) - cfg, err := Load(path) - if err != nil { - t.Fatalf("Load: %v", err) - } + cfg := mustLoad(t, path) if got := cfg.Volumes[0].Tenants; !slices.Equal(got, []string{"acme"}) { t.Errorf("volume tenants = %v, want [acme]", got) } @@ -334,10 +316,7 @@ func TestLoadAcceptsEgressPolicy(t *testing.T) { path := writeConfig(t, `{"secrets":[{"name":"gh","header":"Authorization","value_env":"GH_TOKEN"},{"name":"gw","header":"Authorization"}], "pools":[{"template":"rt:24.04","net":"none","size":"small", "egress":{"allow":[{"host":"api.github.com","methods":["GET"],"secret":"gh"},{"host":"*.googleapis.com","secret":"gw"}]}}]}`) - cfg, err := Load(path) - if err != nil { - t.Fatalf("Load: %v", err) - } + cfg := mustLoad(t, path) pol := cfg.Pools[0].Egress if pol == nil || len(pol.Allow) != 2 || pol.Allow[0].Secret != "gh" { t.Errorf("egress policy %+v", pol) @@ -411,10 +390,7 @@ func TestLoadDefaultsWarmOnlyWhenOmitted(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - cfg, err := Load(writeConfig(t, `{"pools":[`+tt.pool+`]}`)) - if err != nil { - t.Fatalf("Load: %v", err) - } + cfg := mustLoad(t, writeConfig(t, `{"pools":[`+tt.pool+`]}`)) if cfg.Pools[0].Warm != tt.want { t.Errorf("warm %d, want %d", cfg.Pools[0].Warm, tt.want) } @@ -428,10 +404,7 @@ func TestLoadDefaultsWarmOnlyWhenOmitted(t *testing.T) { func TestLoadKeepsExplicitValues(t *testing.T) { path := writeConfig(t, `{"listen":"0.0.0.0:9999","advertise_addr":"10.0.0.5:9999","max_fork_count":4, "refill_concurrency":8,"no_direct_io":true,"bridges":["br0"],"pools":[{"template":"rt:24.04","net":"egress","size":"small","warm":3}]}`) - cfg, err := Load(path) - if err != nil { - t.Fatalf("Load: %v", err) - } + cfg := mustLoad(t, path) if cfg.AdvertiseAddr != "10.0.0.5:9999" || cfg.MaxForkCount != 4 || cfg.RefillConcurrency != 8 || !cfg.NoDirectIO || cfg.Pools[0].Warm != 3 { t.Errorf("explicit values overridden: %+v", cfg) } @@ -467,10 +440,7 @@ func TestLoadAcceptsGuardedEgressOnABridgesList(t *testing.T) { func TestLoadAcceptsAShardedNetworkList(t *testing.T) { path := writeConfig(t, `{"networks":["cocoon-sbx0","cocoon-sbx1","cocoon-sbx2","cocoon-sbx3"],"pools":[]}`) - cfg, err := Load(path) - if err != nil { - t.Fatalf("Load: %v", err) - } + cfg := mustLoad(t, path) if len(cfg.Networks) != 4 || !cfg.HasEgress() { t.Errorf("Networks = %v, HasEgress = %v", cfg.Networks, cfg.HasEgress()) } @@ -484,3 +454,12 @@ func writeConfig(t *testing.T, body string) string { } return path } + +func mustLoad(t *testing.T, path string) *Config { + t.Helper() + cfg, err := Load(path) + if err != nil { + t.Fatalf("Load: %v", err) + } + return cfg +} diff --git a/sandboxd/egress/bench_test.go b/sandboxd/egress/bench_test.go index 125ee47ee..5f6d1ee75 100644 --- a/sandboxd/egress/bench_test.go +++ b/sandboxd/egress/bench_test.go @@ -12,6 +12,8 @@ import ( "testing" ) +const benchKey = "EAABsbCS1iHgBAKZBZCx9ZAZBe7ZAZCYqZBfZC0ZD" + func BenchmarkSignLeaf(b *testing.B) { ca := testCAOnly(b) for b.Loop() { @@ -99,21 +101,17 @@ func BenchmarkTunnelDecision(b *testing.B) { } func BenchmarkInjectAndScrubHeaders(b *testing.B) { - key := "EAABsbCS1iHgBAKZBZCx9ZAZBe7ZAZCYqZBfZC0ZD" - base, err := http.NewRequestWithContext(b.Context(), http.MethodPost, "https://example.com/v19.0/me", nil) - if err != nil { - b.Fatalf("build request: %v", err) - } + base := newRequest(b, http.MethodPost, "/v19.0/me", "") for _, arm := range []struct { name, query, ctype, body string cred Credential }{ {"no-credential", "a=1", "", "", Credential{}}, - {"header", "a=1", "", "", Credential{Name: "K", Header: "X-Key", Value: key, Placeholder: "@K@"}}, - {"query", "a=1&access_token=%40K%40", "", "", Credential{Name: "K", Query: "access_token", Value: key, Placeholder: "@K@"}}, - {"form-body", "", "application/x-www-form-urlencoded", "m=hi&access_token=%40K%40", Credential{Name: "K", Query: "access_token", Body: true, Value: key, Placeholder: "@K@"}}, - {"json-body", "", "application/json", `{"m":"hi","access_token":"@K@"}`, Credential{Name: "K", Query: "access_token", Body: true, Value: key, Placeholder: "@K@"}}, - {"json-body-escaped-unnamed", "", "application/json", `{"message":"` + strings.Repeat(`hello\nworld `, 150) + `"}`, Credential{Name: "K", Query: "access_token", Body: true, Value: key, Placeholder: "@K@"}}, + {"header", "a=1", "", "", Credential{Name: "K", Header: "X-Key", Value: benchKey, Placeholder: "@K@"}}, + {"query", "a=1&access_token=%40K%40", "", "", Credential{Name: "K", Query: "access_token", Value: benchKey, Placeholder: "@K@"}}, + {"form-body", "", "application/x-www-form-urlencoded", "m=hi&access_token=%40K%40", Credential{Name: "K", Query: "access_token", Body: true, Value: benchKey, Placeholder: "@K@"}}, + {"json-body", "", "application/json", `{"m":"hi","access_token":"@K@"}`, Credential{Name: "K", Query: "access_token", Body: true, Value: benchKey, Placeholder: "@K@"}}, + {"json-body-escaped-unnamed", "", "application/json", `{"message":"` + strings.Repeat(`hello\nworld `, 150) + `"}`, Credential{Name: "K", Query: "access_token", Body: true, Value: benchKey, Placeholder: "@K@"}}, } { creds := map[string][]Credential{} if arm.cred.Name != "" { @@ -130,7 +128,7 @@ func BenchmarkInjectAndScrubHeaders(b *testing.B) { out.Body, out.ContentLength = io.NopCloser(strings.NewReader(arm.body)), int64(len(arm.body)) } if _, sc := p.inject(Rule{}, out, "example.com", true); sc != nil { - sc.header(http.Header{"Location": {"/next?access_token=" + key}, "Content-Type": {"application/json"}, "Date": {"x"}}) + sc.header(http.Header{"Location": {"/next?access_token=" + benchKey}, "Content-Type": {"application/json"}, "Date": {"x"}}) } } }) @@ -138,10 +136,9 @@ func BenchmarkInjectAndScrubHeaders(b *testing.B) { } func BenchmarkScrubBody(b *testing.B) { - key := "EAABsbCS1iHgBAKZBZCx9ZAZBe7ZAZCYqZBfZC0ZD" - sc := newScrubber([]Credential{{Name: "K", Query: "access_token", Value: key, Placeholder: "@K@"}}) + sc := newScrubber([]Credential{{Name: "K", Query: "access_token", Value: benchKey, Placeholder: "@K@"}}) page := bytes.Repeat([]byte(`{"id":"1234567890","message":"hello world","created_time":"2026-10-04T00:00:00+0000"},`), 64<<10/88) - page = append(page, `"paging":{"next":"https:\/\/graph.facebook.com\/v19.0\/me\/feed?access_token=`+key+`&after=x"}`...) + page = append(page, `"paging":{"next":"https:\/\/graph.facebook.com\/v19.0\/me\/feed?access_token=`+benchKey+`&after=x"}`...) event := []byte("data: {\"choices\":[{\"delta\":{\"content\":\"hello there\"}}]}\n\n") for _, arm := range []struct { name string @@ -194,18 +191,13 @@ func benchFront(b *testing.B, mode InterceptMode) (proxyAddr string, roots *x509 if ca != nil { p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) } - front := httptest.NewServer(p) - b.Cleanup(front.Close) - return front.Listener.Addr().String(), roots + return serveFront(b, p).Listener.Addr().String(), roots } func benchRoundTrip(b *testing.B, tc *tls.Conn, br *bufio.Reader) { b.Helper() - req, err := http.NewRequestWithContext(b.Context(), http.MethodGet, "https://example.com/x", nil) - if err != nil { - b.Fatalf("build request: %v", err) - } - if err = req.Write(tc); err != nil { + req := newRequest(b, http.MethodGet, "/x", "") + if err := req.Write(tc); err != nil { b.Fatalf("write request: %v", err) } resp, err := http.ReadResponse(br, req) diff --git a/sandboxd/egress/intercept_test.go b/sandboxd/egress/intercept_test.go index cdbb239e7..e5ae4bde8 100644 --- a/sandboxd/egress/intercept_test.go +++ b/sandboxd/egress/intercept_test.go @@ -67,9 +67,7 @@ func TestInterceptLeafCacheBounded(t *testing.T) { func TestInterceptInjectsSecretIntoHTTPS(t *testing.T) { events := make(chan Event, 4) p, guestRoots, upstream := interceptProxy(t, Rule{Host: "example.com", Methods: []string{"GET"}, Secret: "gh", Intercept: InterceptAlways}, events) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) tc := connectTLS(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots) defer func() { _ = tc.Close() }() @@ -96,9 +94,7 @@ func TestInterceptInjectsSecretIntoHTTPS(t *testing.T) { func TestInterceptKeepsAClientHelloSentBeforeThe200(t *testing.T) { p, guestRoots, upstream := interceptProxy(t, Rule{Host: "example.com", Methods: []string{"GET"}, Intercept: InterceptAlways}, nil) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) raw, err := net.Dial("tcp", front.Listener.Addr().String()) if err != nil { @@ -123,9 +119,7 @@ func TestInterceptBindsInnerRequestToInterceptRule(t *testing.T) { {Host: "api.example.com", Secret: "gh", Intercept: InterceptAlways}, }} p, guestRoots, upstream := interceptProxyPolicy(t, policy, nil) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) tc := connectTLS(t, front.Listener.Addr().String(), "api.example.com:443", "api.example.com", guestRoots) defer func() { _ = tc.Close() }() @@ -159,9 +153,7 @@ func TestInterceptEnforcesRuleMethods(t *testing.T) { } { t.Run(tt.name, func(t *testing.T) { p, guestRoots, upstream := interceptProxyPolicy(t, tt.policy, nil) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) tc := connectTLS(t, front.Listener.Addr().String(), tt.host+":443", tt.host, guestRoots) defer func() { _ = tc.Close() }() @@ -177,9 +169,7 @@ func TestInterceptEnforcesRuleMethods(t *testing.T) { func TestInterceptForcesAuthorityOnForeignHost(t *testing.T) { p, guestRoots, upstream := interceptProxy(t, Rule{Host: "example.com", Intercept: InterceptAlways}, nil) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) tc := connectTLS(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots) defer func() { _ = tc.Close() }() @@ -193,9 +183,7 @@ func TestInterceptForcesAuthorityOnForeignHost(t *testing.T) { func TestCloseRevokesInterceptedSession(t *testing.T) { p, guestRoots, upstream := interceptProxy(t, Rule{Host: "example.com", Intercept: InterceptAlways}, nil) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) tc := connectTLS(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots) defer func() { _ = tc.Close() }() @@ -219,8 +207,7 @@ func TestCloseRevokesInterceptedSession(t *testing.T) { func TestInterceptRejectsUntrustedUpstream(t *testing.T) { p, guestRoots, _ := interceptProxy(t, Rule{Host: "example.com", Intercept: InterceptAlways}, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) tc := connectTLS(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots) defer func() { _ = tc.Close() }() @@ -241,9 +228,7 @@ func TestInterceptInjectsClaimCredentialsBelowThePoolSecret(t *testing.T) { {Name: "LATER", Header: "X-Api-Key", Value: "k2"}, }}} p, guestRoots, upstream := interceptProxySecrets(t, Policy{Allow: []Rule{{Host: "example.com", Secret: "gh", Intercept: InterceptAlways}}}, secrets, events) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) tc := connectTLS(t, front.Listener.Addr().String(), "Example.com:443", "example.com", guestRoots) defer func() { _ = tc.Close() }() @@ -273,9 +258,7 @@ func TestInterceptInjectsAPlaceholderCredentialOnlyWhereTheGuestSentIt(t *testin {Name: "GIT", Header: "X-Api-Key", Value: "real", Placeholder: "@GIT@"}, }}} p, guestRoots, upstream := interceptProxySecrets(t, Policy{Allow: []Rule{{Host: "example.com", Intercept: InterceptAlways}}}, secrets, nil) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) for sent, want := range map[string]string{"@GIT@": "real", "": "", "@OTHER@": "@OTHER@"} { tc := connectTLS(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots) @@ -294,9 +277,7 @@ func TestInterceptInjectsAPlaceholderCredentialOnlyWhereTheGuestSentIt(t *testin func TestAnInjectRuleInterceptsOnlyForAClaimWithACredentialForTheHost(t *testing.T) { secrets := credSecrets{creds: map[string][]Credential{"example.com": {{Name: "KEY", Header: "X-Api-Key", Value: "real"}}}} p, guestRoots, upstream := interceptProxySecrets(t, Policy{Allow: []Rule{{Host: "*", Intercept: InterceptInject}}}, secrets, nil) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) for _, tt := range []struct { target string @@ -319,7 +300,7 @@ func TestAnInjectRuleInterceptsOnlyForAClaimWithACredentialForTheHost(t *testing } } -func TestTunnelDecisionOnAnInjectRule(t *testing.T) { +func TestAnInjectRuleInterceptsOnlyAProxyDoorTunnelWithACredential(t *testing.T) { ca, _ := testCA(t) creds := credSecrets{creds: map[string][]Credential{ "empty.test": {{Name: "E", Header: "X-Key", Value: ""}}, @@ -372,9 +353,7 @@ func TestInterceptSubstitutesAQueryCredentialAndScrubsItsEcho(t *testing.T) { secrets := credSecrets{creds: map[string][]Credential{"example.com": {{Name: "SERP", Query: "api_key", Value: "real key/+", Placeholder: "@SERP@"}}}} events := make(chan Event, 8) p, guestRoots, upstream := interceptProxySecrets(t, Policy{Allow: []Rule{{Host: "example.com", Intercept: InterceptAlways}}}, secrets, events) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) for _, tt := range []struct { query, want, injected string @@ -411,9 +390,7 @@ func TestInterceptSubstitutesABodyCredentialInASmallFormOrJSONBody(t *testing.T) {Name: "URLONLY", Query: "key", Value: "other", Placeholder: "@URL@"}, }}} p, guestRoots, upstream := interceptProxySecrets(t, Policy{Allow: []Rule{{Host: "example.com", Intercept: InterceptAlways}}}, secrets, nil) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) big := "access_token=%40FB%40&pad=" + strings.Repeat("x", maxInjectBody) for _, tt := range []struct { @@ -502,9 +479,7 @@ func TestAnEncodedParameterNameIsFilled(t *testing.T) { func TestARangeCannotSplitAScrubbedQueryValue(t *testing.T) { secrets := credSecrets{creds: map[string][]Credential{"example.com": {{Name: "K", Query: "api_key", Value: "SECRETVALUE0123456789", Placeholder: "@K@"}}}} p, guestRoots, upstream := interceptProxySecrets(t, Policy{Allow: []Rule{{Host: "*", Intercept: InterceptInject}}}, secrets, nil) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) for _, byteRange := range []string{"bytes=0-9", "bytes=10-20", "bytes=0-3,10-13"} { tc := connectTLS(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots) @@ -564,11 +539,9 @@ func TestInterceptStreamsAnEventStreamAsItArrives(t *testing.T) { defer close(release) ca, _ := testCA(t) p := New(Policy{Allow: []Rule{{Host: "example.com", Intercept: InterceptAlways}}}, nil, ca, fixedDial(upstream.Listener.Addr().String()), nil, nil) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) guestRoots := x509.NewCertPool() guestRoots.AppendCertsFromPEM(ca.CertPEM()) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) tc := connectTLS(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots) defer func() { _ = tc.Close() }() @@ -582,9 +555,7 @@ func TestInterceptStreamsAnEventStreamAsItArrives(t *testing.T) { func TestInterceptResumesTheGuestsTLSSession(t *testing.T) { p, guestRoots, upstream := interceptProxy(t, Rule{Host: "example.com", Intercept: InterceptAlways}, nil) - p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) - front := httptest.NewServer(p) - defer front.Close() + front := serveTrusting(t, p, upstream) cache := tls.NewLRUClientSessionCache(4) for i := range 2 { tc := connectTLSCached(t, front.Listener.Addr().String(), "example.com:443", "example.com", guestRoots, cache) @@ -598,7 +569,62 @@ func TestInterceptResumesTheGuestsTLSSession(t *testing.T) { } } +type echoed struct { + Key, Empty, Query, Body string + Length int64 +} + +type echoReply struct { + status int + header http.Header + body string +} + +// seen decodes the upstream's /echo report; its hex survives the scrub, the query reflected after it does not. +func (r echoReply) seen(t *testing.T) (echoed, string) { + t.Helper() + report, reflected, _ := strings.Cut(r.body, "\n") + var e echoed + if raw, err := hex.DecodeString(report); err != nil || json.Unmarshal(raw, &e) != nil { + t.Fatalf("echo report %q", r.body) + } + return e, reflected +} + +type earlyConn struct { + net.Conn + preamble string + answered bool +} + +func (c *earlyConn) Write(b []byte) (int, error) { + if c.preamble == "" { + return c.Conn.Write(b) + } + if _, err := io.WriteString(c.Conn, c.preamble+string(b)); err != nil { + return 0, err + } + c.preamble = "" + return len(b), nil +} + +func (c *earlyConn) Read(b []byte) (int, error) { + if !c.answered { + var head strings.Builder + var one [1]byte + for !strings.HasSuffix(head.String(), "\r\n\r\n") { + if _, err := io.ReadFull(c.Conn, one[:]); err != nil { + return 0, err + } + head.WriteByte(one[0]) + } + c.answered = true + } + return c.Conn.Read(b) +} + func interceptProxy(t *testing.T, rule Rule, events chan Event) (*Proxy, *x509.CertPool, *httptest.Server) { + t.Helper() return interceptProxyPolicy(t, Policy{Allow: []Rule{rule}}, events) } @@ -649,6 +675,12 @@ func interceptProxySecrets(t *testing.T, policy Policy, secrets Secrets, events return p, guestRoots, upstream } +func serveTrusting(t *testing.T, p *Proxy, upstream *httptest.Server) *httptest.Server { + t.Helper() + p.pools.Load().mitm.TLSClientConfig.RootCAs = trustUpstream(upstream) + return serveFront(t, p) +} + func trustUpstream(upstream *httptest.Server) *x509.CertPool { pool := x509.NewCertPool() pool.AddCert(upstream.Certificate()) @@ -662,13 +694,7 @@ func connectTLS(tb testing.TB, proxyAddr, target, serverName string, roots *x509 func connectTLSCached(tb testing.TB, proxyAddr, target, serverName string, roots *x509.CertPool, cache tls.ClientSessionCache) *tls.Conn { tb.Helper() - conn, err := net.Dial("tcp", proxyAddr) - if err != nil { - tb.Fatalf("dial proxy: %v", err) - } - if _, err := io.WriteString(conn, "CONNECT "+target+" HTTP/1.1\r\nHost: "+target+"\r\n\r\n"); err != nil { - tb.Fatalf("write CONNECT: %v", err) - } + conn := dialConnect(tb, proxyAddr, target) if status := readPreamble(tb, conn); !strings.Contains(status, "200") { tb.Fatalf("CONNECT status = %q, want 200", status) } @@ -703,38 +729,16 @@ func roundTripTLSHost(t *testing.T, tc *tls.Conn, method, host string) *http.Res return resp } -func newRequest(t *testing.T, method, target, body string) *http.Request { - t.Helper() - req, err := http.NewRequestWithContext(t.Context(), method, "https://example.com"+target, strings.NewReader(body)) +func newRequest(tb testing.TB, method, target, body string) *http.Request { + tb.Helper() + req, err := http.NewRequestWithContext(tb.Context(), method, "https://example.com"+target, strings.NewReader(body)) if err != nil { - t.Fatalf("build request: %v", err) + tb.Fatalf("build request: %v", err) } return req } -type echoed struct { - Key, Empty, Query, Body string - Length int64 -} - -type reply struct { - status int - header http.Header - body string -} - -// seen decodes the upstream's /echo report; its hex survives the scrub, the query reflected after it does not. -func (r reply) seen(t *testing.T) (echoed, string) { - t.Helper() - report, reflected, _ := strings.Cut(r.body, "\n") - var e echoed - if raw, err := hex.DecodeString(report); err != nil || json.Unmarshal(raw, &e) != nil { - t.Fatalf("echo report %q", r.body) - } - return e, reflected -} - -func send(t *testing.T, tc *tls.Conn, req *http.Request) reply { +func send(t *testing.T, tc *tls.Conn, req *http.Request) echoReply { t.Helper() if err := req.Write(tc); err != nil { t.Fatalf("write request: %v", err) @@ -745,7 +749,7 @@ func send(t *testing.T, tc *tls.Conn, req *http.Request) reply { } body, _ := io.ReadAll(resp.Body) _ = resp.Body.Close() - return reply{status: resp.StatusCode, header: resp.Header, body: string(body)} + return echoReply{status: resp.StatusCode, header: resp.Header, body: string(body)} } func chunked(chunks ...string) io.Reader { @@ -771,35 +775,3 @@ func readPreamble(tb testing.TB, conn net.Conn) string { } return sb.String() } - -type earlyConn struct { - net.Conn - preamble string - answered bool -} - -func (c *earlyConn) Write(b []byte) (int, error) { - if c.preamble == "" { - return c.Conn.Write(b) - } - if _, err := io.WriteString(c.Conn, c.preamble+string(b)); err != nil { - return 0, err - } - c.preamble = "" - return len(b), nil -} - -func (c *earlyConn) Read(b []byte) (int, error) { - if !c.answered { - var reply strings.Builder - var one [1]byte - for !strings.HasSuffix(reply.String(), "\r\n\r\n") { - if _, err := io.ReadFull(c.Conn, one[:]); err != nil { - return 0, err - } - reply.WriteByte(one[0]) - } - c.answered = true - } - return c.Conn.Read(b) -} diff --git a/sandboxd/egress/policy_test.go b/sandboxd/egress/policy_test.go index d892c69cd..6c76b79c8 100644 --- a/sandboxd/egress/policy_test.go +++ b/sandboxd/egress/policy_test.go @@ -207,7 +207,7 @@ func TestPolicyServesSocks(t *testing.T) { } } -func TestRulePorts(t *testing.T) { +func TestARuleWithPortsAllowsOnlyThosePortsOnEveryPath(t *testing.T) { policy := Policy{Allow: []Rule{ {Host: "db.internal", Ports: []uint16{5432}}, {Host: "mail.internal", Ports: []uint16{993, 465}, Intercept: InterceptAlways}, diff --git a/sandboxd/egress/proxy_test.go b/sandboxd/egress/proxy_test.go index ca124be5c..8c52e0a98 100644 --- a/sandboxd/egress/proxy_test.go +++ b/sandboxd/egress/proxy_test.go @@ -29,19 +29,9 @@ func TestForwardAllowInjectsSecretAndOverwritesGuestHeader(t *testing.T) { secrets := fakeSecrets{"gh": {"Authorization", "Bearer SECRET"}} events := make(chan Event, 4) p := New(policy, secrets, nil, fixedDial(upstream.Listener.Addr().String()), func(ev Event) { events <- ev }, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - client := proxyClient(t, front.URL) - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://api.internal/x", nil) - if err != nil { - t.Fatalf("build request: %v", err) - } - req.Header.Set("Authorization", "Bearer GUEST") - resp, err := client.Do(req) - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, front.URL, "http://api.internal/x", func(r *http.Request) { r.Header.Set("Authorization", "Bearer GUEST") }) defer func() { _ = resp.Body.Close() }() body, _ := io.ReadAll(resp.Body) @@ -67,18 +57,9 @@ func TestForwardWithoutASecretValueInjectsNothing(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "api.internal", Secret: "gw"}}} events := make(chan Event, 4) p := New(policy, fakeSecrets{"gw": {"Authorization", ""}}, nil, fixedDial(upstream.Listener.Addr().String()), func(ev Event) { events <- ev }, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://api.internal/x", nil) - if err != nil { - t.Fatalf("build request: %v", err) - } - req.Header.Set("Authorization", "Bearer GUEST") - resp, err := proxyClient(t, front.URL).Do(req) - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, front.URL, "http://api.internal/x", func(r *http.Request) { r.Header.Set("Authorization", "Bearer GUEST") }) _ = resp.Body.Close() if resp.StatusCode != http.StatusOK || gotAuth != "Bearer GUEST" { t.Errorf("got %d with Authorization %q, want 200 and the guest's own header", resp.StatusCode, gotAuth) @@ -96,18 +77,9 @@ func TestRelayKeepsTheUpstreamConnForAClosingGuest(t *testing.T) { })) defer upstream.Close() p := New(Policy{Allow: []Rule{{Host: "api.internal"}}}, nil, nil, fixedDial(upstream.Listener.Addr().String()), nil, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://api.internal/x", nil) - if err != nil { - t.Fatalf("build request: %v", err) - } - req.Close = true - resp, err := proxyClient(t, front.URL).Do(req) - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, front.URL, "http://api.internal/x", func(r *http.Request) { r.Close = true }) _ = resp.Body.Close() if <-closeSeen { t.Error("the guest's Connection: close reached the upstream hop") @@ -129,17 +101,9 @@ func TestCloseReleasesIdleUpstreamConns(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "api.internal"}}} p := New(policy, nil, nil, fixedDial(upstream.Listener.Addr().String()), nil, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://api.internal/x", nil) - if err != nil { - t.Fatalf("build request: %v", err) - } - resp, err := proxyClient(t, front.URL).Do(req) - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, front.URL, "http://api.internal/x") _, _ = io.Copy(io.Discard, resp.Body) _ = resp.Body.Close() @@ -161,13 +125,9 @@ func TestForwardNeverInjectsInterceptSecret(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "api.internal", Secret: "gh", Intercept: InterceptAlways}}} secrets := fakeSecrets{"gh": {"Authorization", "Bearer SECRET"}} p := New(policy, secrets, nil, fixedDial(upstream.Listener.Addr().String()), nil, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - resp, err := proxyClient(t, front.URL).Get("http://api.internal/x") - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, front.URL, "http://api.internal/x") defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusForbidden { t.Fatalf("plaintext forward matching only an intercept rule got %d, want 403", resp.StatusCode) @@ -186,18 +146,9 @@ func TestForwardNeverInjectsClaimCredentials(t *testing.T) { events := make(chan Event, 1) secrets := credSecrets{creds: map[string][]Credential{"api.internal": {{Name: "API", Header: "X-Api-Key", Value: "k1"}}}} p := New(Policy{Allow: []Rule{{Host: "api.internal"}}}, secrets, nil, fixedDial(upstream.Listener.Addr().String()), func(ev Event) { events <- ev }, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://api.internal/x", nil) - if err != nil { - t.Fatalf("build request: %v", err) - } - req.Header.Set("X-Api-Key", "placeholder") - resp, err := proxyClient(t, front.URL).Do(req) - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, front.URL, "http://api.internal/x", func(r *http.Request) { r.Header.Set("X-Api-Key", "placeholder") }) _ = resp.Body.Close() if seen != "placeholder" { t.Errorf("upstream saw X-Api-Key %q, want the guest's value: a claim credential never rides a cleartext forward", seen) @@ -213,8 +164,7 @@ func TestForwardStreamsAnEventStreamAsItArrives(t *testing.T) { defer upstream.Close() defer close(release) p := New(Policy{Allow: []Rule{{Host: "sse.internal"}}}, nil, nil, fixedDial(upstream.Listener.Addr().String()), nil, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) client := proxyClient(t, front.URL) client.Timeout = 2 * time.Second @@ -241,20 +191,12 @@ func TestForwardStripsHopHeaders(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "api.internal"}}} p := New(policy, nil, nil, fixedDial(upstream.Listener.Addr().String()), nil, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - client := proxyClient(t, front.URL) - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://api.internal/x", nil) - if err != nil { - t.Fatalf("build request: %v", err) - } - req.Header.Set("Connection", "X-Guest-Hop") - req.Header.Set("X-Guest-Hop", "leak") - resp, err := client.Do(req) - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, front.URL, "http://api.internal/x", func(r *http.Request) { + r.Header.Set("Connection", "X-Guest-Hop") + r.Header.Set("X-Guest-Hop", "leak") + }) defer func() { _ = resp.Body.Close() }() if gotHop != "" || gotConn != "" { t.Errorf("upstream saw hop headers: X-Guest-Hop=%q Connection=%q", gotHop, gotConn) @@ -264,17 +206,13 @@ func TestForwardStripsHopHeaders(t *testing.T) { } } -func TestForwardDeniedIsTyped(t *testing.T) { +func TestADeniedForwardIs403AndAuditedAsDeny(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "api.internal"}}} events := make(chan Event, 4) p := New(policy, nil, nil, fixedDial("127.0.0.1:1"), func(ev Event) { events <- ev }, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - resp, err := proxyClient(t, front.URL).Get("http://blocked.internal/") - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, front.URL, "http://blocked.internal/") defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusForbidden { t.Fatalf("denied request got %d, want 403", resp.StatusCode) @@ -289,8 +227,7 @@ func TestConnectAllowTunnels(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "echo.internal"}}} events := make(chan Event, 4) p := New(policy, nil, nil, fixedDial(echo), func(ev Event) { events <- ev }, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) conn := dialConnect(t, front.Listener.Addr().String(), "echo.internal:443") defer func() { _ = conn.Close() }() @@ -318,18 +255,11 @@ func TestConnectForwardsBytesTheGuestSentBeforeThe200(t *testing.T) { p := New(Policy{Allow: []Rule{{Host: "echo.internal"}}}, nil, nil, fixedDial(echo), nil, nil) sent := make(chan int64, 1) p.OnTransfer(func(_ Event, tx, _ int64) { sent <- tx }) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - conn, err := net.Dial("tcp", front.Listener.Addr().String()) - if err != nil { - t.Fatalf("dial proxy: %v", err) - } + conn := dialProxy(t, front.Listener.Addr().String(), "CONNECT echo.internal:443 HTTP/1.1\r\nHost: echo.internal:443\r\n\r\nearly") defer func() { _ = conn.Close() }() _ = conn.SetDeadline(time.Now().Add(5 * time.Second)) - if _, err := io.WriteString(conn, "CONNECT echo.internal:443 HTTP/1.1\r\nHost: echo.internal:443\r\n\r\nearly"); err != nil { - t.Fatalf("write CONNECT with early data: %v", err) - } br := bufio.NewReader(conn) if status := readStatus(t, br); status != "HTTP/1.1 200 Connection Established" { t.Fatalf("CONNECT status = %q", status) @@ -354,17 +284,10 @@ func TestConnectCountsTheEarlyBytesAShortWriteSent(t *testing.T) { p := New(Policy{Allow: []Rule{{Host: "echo.internal"}}}, nil, nil, dial, nil, nil) sent := make(chan int64, 1) p.OnTransfer(func(_ Event, tx, _ int64) { sent <- tx }) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - conn, err := net.Dial("tcp", front.Listener.Addr().String()) - if err != nil { - t.Fatalf("dial proxy: %v", err) - } + conn := dialProxy(t, front.Listener.Addr().String(), "CONNECT echo.internal:443 HTTP/1.1\r\nHost: echo.internal:443\r\n\r\nearly") defer func() { _ = conn.Close() }() - if _, err := io.WriteString(conn, "CONNECT echo.internal:443 HTTP/1.1\r\nHost: echo.internal:443\r\n\r\nearly"); err != nil { - t.Fatalf("write CONNECT with early data: %v", err) - } select { case tx := <-sent: if tx != 2 { @@ -378,8 +301,7 @@ func TestConnectCountsTheEarlyBytesAShortWriteSent(t *testing.T) { func TestCloseEndsSplicedTunnel(t *testing.T) { echo := echoServer(t) p := New(Policy{Allow: []Rule{{Host: "echo.internal"}}}, nil, nil, fixedDial(echo), nil, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) conn := dialConnect(t, front.Listener.Addr().String(), "echo.internal:443") defer func() { _ = conn.Close() }() @@ -406,8 +328,7 @@ func TestCloseEndsSplicedTunnel(t *testing.T) { func TestCloseEndsTunnelWhoseUpstreamStaysSilent(t *testing.T) { var held holdCounter p := New(Policy{Allow: []Rule{{Host: "quiet.internal"}}}, nil, nil, fixedDial(silentServer(t)), nil, &held) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) conn := dialConnect(t, front.Listener.Addr().String(), "quiet.internal:443") defer func() { _ = conn.Close() }() @@ -423,11 +344,10 @@ func TestCloseEndsTunnelWhoseUpstreamStaysSilent(t *testing.T) { waitHolds(t, &held, 0) } -func TestConnectDeniedIsTyped(t *testing.T) { +func TestADeniedConnectIs403(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "echo.internal"}}} p := New(policy, nil, nil, fixedDial("127.0.0.1:1"), nil, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) conn := dialConnect(t, front.Listener.Addr().String(), "blocked.internal:443") defer func() { _ = conn.Close() }() @@ -441,8 +361,7 @@ func TestPortRuleGatesConnectAndForward(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "echo.internal", Ports: []uint16{443}}}} events := make(chan Event, 4) p := New(policy, nil, nil, fixedDial(echo), func(ev Event) { events <- ev }, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) conn := dialConnect(t, front.Listener.Addr().String(), "echo.internal:8443") defer func() { _ = conn.Close() }() @@ -453,14 +372,7 @@ func TestPortRuleGatesConnectAndForward(t *testing.T) { t.Errorf("audit event = %+v, want deny on port 8443", ev) } - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://echo.internal/x", nil) - if err != nil { - t.Fatalf("build request: %v", err) - } - resp, err := proxyClient(t, front.URL).Do(req) - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, front.URL, "http://echo.internal/x") _ = resp.Body.Close() if resp.StatusCode != http.StatusForbidden { t.Errorf("forward GET on port 80 = %d, want 403 with a rule that lists 443 only", resp.StatusCode) @@ -474,8 +386,7 @@ func TestUnparsablePortIsDeniedNotDefaulted(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "echo.internal", Ports: []uint16{443}}}} events := make(chan Event, 4) p := New(policy, nil, nil, fixedDial(echoServer(t)), func(ev Event) { events <- ev }, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) conn := dialConnect(t, front.Listener.Addr().String(), "echo.internal:99999") defer func() { _ = conn.Close() }() @@ -486,14 +397,7 @@ func TestUnparsablePortIsDeniedNotDefaulted(t *testing.T) { t.Errorf("audit event = %+v, want deny with no port, not the 443 default", ev) } - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://echo.internal:99999/x", nil) - if err != nil { - t.Fatalf("build request: %v", err) - } - resp, err := proxyClient(t, front.URL).Do(req) - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, front.URL, "http://echo.internal:99999/x") _ = resp.Body.Close() if resp.StatusCode != http.StatusForbidden { t.Errorf("forward GET with an out-of-range port = %d, want 403", resp.StatusCode) @@ -502,14 +406,8 @@ func TestUnparsablePortIsDeniedNotDefaulted(t *testing.T) { t.Errorf("audit event = %+v, want deny with no port", ev) } - raw, err := net.Dial("tcp", front.Listener.Addr().String()) - if err != nil { - t.Fatalf("dial proxy: %v", err) - } + raw := dialProxy(t, front.Listener.Addr().String(), "CONNECT /x HTTP/1.1\r\nHost: echo.internal:+8443\r\n\r\n") defer func() { _ = raw.Close() }() - if _, err = io.WriteString(raw, "CONNECT /x HTTP/1.1\r\nHost: echo.internal:+8443\r\n\r\n"); err != nil { - t.Fatalf("write path-form CONNECT: %v", err) - } if status := readStatus(t, bufio.NewReader(raw)); status != "HTTP/1.1 403 Forbidden" { t.Errorf("path-form CONNECT with a signed port = %q, want 403 (the dialer would have accepted +8443)", status) } @@ -517,14 +415,8 @@ func TestUnparsablePortIsDeniedNotDefaulted(t *testing.T) { t.Errorf("audit event = %+v, want deny with no port", ev) } - malformed, err := net.Dial("tcp", front.Listener.Addr().String()) - if err != nil { - t.Fatalf("dial proxy: %v", err) - } + malformed := dialProxy(t, front.Listener.Addr().String(), "CONNECT /x HTTP/1.1\r\nHost: echo.internal:443:8443\r\n\r\n") defer func() { _ = malformed.Close() }() - if _, err = io.WriteString(malformed, "CONNECT /x HTTP/1.1\r\nHost: echo.internal:443:8443\r\n\r\n"); err != nil { - t.Fatalf("write malformed CONNECT: %v", err) - } if status := readStatus(t, bufio.NewReader(malformed)); status != "HTTP/1.1 403 Forbidden" { t.Errorf("path-form CONNECT with two ports = %q, want 403, not a bare-host default", status) } @@ -537,10 +429,9 @@ func TestBracketedIPv6AuthorityIsUnwrapped(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "*"}}} events := make(chan Event, 4) p := New(policy, nil, nil, fixedDial(echoServer(t)), func(ev Event) { events <- ev }, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - for _, tc := range []struct { + for _, tt := range []struct { authority string status string decision Decision @@ -552,22 +443,16 @@ func TestBracketedIPv6AuthorityIsUnwrapped(t *testing.T) { {"echo.internal]", "HTTP/1.1 403 Forbidden", DecisionDeny}, {"", "HTTP/1.1 403 Forbidden", DecisionDeny}, } { - conn, err := net.Dial("tcp", front.Listener.Addr().String()) - if err != nil { - t.Fatalf("dial proxy: %v", err) - } - if _, err = io.WriteString(conn, "CONNECT /x HTTP/1.1\r\nHost: "+tc.authority+"\r\n\r\n"); err != nil { - t.Fatalf("write CONNECT: %v", err) - } - if status := readStatus(t, bufio.NewReader(conn)); status != tc.status { - t.Errorf("CONNECT with Host %s = %q, want %q", tc.authority, status, tc.status) + conn := dialProxy(t, front.Listener.Addr().String(), "CONNECT /x HTTP/1.1\r\nHost: "+tt.authority+"\r\n\r\n") + if status := readStatus(t, bufio.NewReader(conn)); status != tt.status { + t.Errorf("CONNECT with Host %s = %q, want %q", tt.authority, status, tt.status) } _ = conn.Close() ev := recvEvent(t, events) allowed := ev.Decision == DecisionAllow && ev.Host == "2606:4700:4700::1111" && ev.Port == 443 - denied := ev.Decision == DecisionDeny && ev.Port == 0 - if (tc.decision == DecisionAllow) != allowed || (tc.decision == DecisionDeny) != denied { - t.Errorf("audit event for %s = %+v, want %v (allow on the bare address and 443, or deny with no port)", tc.authority, ev, tc.decision) + refused := ev.Decision == DecisionDeny && ev.Port == 0 + if (tt.decision == DecisionAllow) != allowed || (tt.decision == DecisionDeny) != refused { + t.Errorf("audit event for %s = %+v, want %v (allow on the bare address and 443, or deny with no port)", tt.authority, ev, tt.decision) } } } @@ -576,10 +461,9 @@ func TestAHostPastTheDNSLimitIsRefusedAndRecordedClipped(t *testing.T) { policy := Policy{Allow: []Rule{{Host: "*"}}} events := make(chan Event, 4) p := New(policy, nil, nil, fixedDial(echoServer(t)), func(ev Event) { events <- ev }, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - for _, tc := range []struct { + for _, tt := range []struct { host string status string }{ @@ -587,9 +471,9 @@ func TestAHostPastTheDNSLimitIsRefusedAndRecordedClipped(t *testing.T) { {strings.Repeat("b", 64<<10), "HTTP/1.1 403 Forbidden"}, {strings.Repeat("c", maxHostLen) + ":443", "HTTP/1.1 200 Connection Established"}, } { - conn := dialConnect(t, front.Listener.Addr().String(), tc.host) - if status := readStatus(t, bufio.NewReader(conn)); status != tc.status { - t.Errorf("CONNECT to a %d-byte authority = %q, want %q", len(tc.host), status, tc.status) + conn := dialConnect(t, front.Listener.Addr().String(), tt.host) + if status := readStatus(t, bufio.NewReader(conn)); status != tt.status { + t.Errorf("CONNECT to a %d-byte authority = %q, want %q", len(tt.host), status, tt.status) } _ = conn.Close() if ev := recvEvent(t, events); len(ev.Host) > maxHostLen { @@ -644,25 +528,17 @@ func TestAuditNamesTheConnectionEnds(t *testing.T) { if recordEnds { p.RecordEnds() } - front := httptest.NewServer(p) - t.Cleanup(front.Close) - return front + return serveFront(t, p) } - resp, err := proxyClient(t, serve(false).URL).Get("http://api.internal/x") - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, serve(false).URL, "http://api.internal/x") _ = resp.Body.Close() if ev := recvEvent(t, events); ev.Local.IsValid() || ev.Remote.IsValid() { t.Errorf("forward event %+v carries ends without RecordEnds", ev) } front := serve(true) - resp, err = proxyClient(t, front.URL).Get("http://api.internal/y") - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp = proxiedGet(t, front.URL, "http://api.internal/y") _ = resp.Body.Close() if ev := recvEvent(t, events); ev.Remote.String() != upstream.Listener.Addr().String() || !ev.Local.IsValid() || ev.Local.Port() == 0 { t.Errorf("forward audit ends %v -> %v, want a local port and %s", ev.Local, ev.Remote, upstream.Listener.Addr()) @@ -682,13 +558,9 @@ func TestARateLimitedConnectionAnswers429AndIsAudited(t *testing.T) { events := make(chan Event, 4) dial := DialFunc(func(context.Context, string, string) (net.Conn, error) { return nil, ErrRateLimited }) p := New(Policy{Allow: []Rule{{Host: "*"}}}, nil, nil, dial, func(ev Event) { events <- ev }, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) - resp, err := proxyClient(t, front.URL).Get("http://api.internal/x") - if err != nil { - t.Fatalf("proxied GET: %v", err) - } + resp := proxiedGet(t, front.URL, "http://api.internal/x") _ = resp.Body.Close() if resp.StatusCode != http.StatusTooManyRequests { t.Errorf("forward status %d, want 429", resp.StatusCode) @@ -752,6 +624,13 @@ func (h *holdCounter) Hold() { h.Add(1) } func (h *holdCounter) Unhold() { h.Add(-1) } +type shortWriteConn struct{ net.Conn } + +func (c shortWriteConn) Write(b []byte) (int, error) { + n, _ := c.Conn.Write(b[:min(2, len(b))]) + return n, errors.New("connection reset") +} + func sseHandler(release <-chan struct{}) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "text/event-stream") @@ -777,18 +656,46 @@ func proxyClient(t *testing.T, proxyURL string) *http.Client { return &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(u)}} } -func dialConnect(t *testing.T, proxyAddr, target string) net.Conn { - t.Helper() +func dialConnect(tb testing.TB, proxyAddr, target string) net.Conn { + tb.Helper() + return dialProxy(tb, proxyAddr, "CONNECT "+target+" HTTP/1.1\r\nHost: "+target+"\r\n\r\n") +} + +func dialProxy(tb testing.TB, proxyAddr, request string) net.Conn { + tb.Helper() conn, err := net.Dial("tcp", proxyAddr) if err != nil { - t.Fatalf("dial proxy: %v", err) + tb.Fatalf("dial proxy: %v", err) } - if _, err := io.WriteString(conn, "CONNECT "+target+" HTTP/1.1\r\nHost: "+target+"\r\n\r\n"); err != nil { - t.Fatalf("write CONNECT: %v", err) + if _, err := io.WriteString(conn, request); err != nil { + tb.Fatalf("write CONNECT: %v", err) } return conn } +func serveFront(tb testing.TB, p *Proxy) *httptest.Server { + tb.Helper() + front := httptest.NewServer(p) + tb.Cleanup(front.Close) + return front +} + +func proxiedGet(t *testing.T, proxyURL, target string, edits ...func(*http.Request)) *http.Response { + t.Helper() + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, target, nil) + if err != nil { + t.Fatalf("build request: %v", err) + } + for _, edit := range edits { + edit(req) + } + resp, err := proxyClient(t, proxyURL).Do(req) + if err != nil { + t.Fatalf("proxied GET: %v", err) + } + return resp +} + func readStatus(t *testing.T, br *bufio.Reader) string { t.Helper() line, err := br.ReadString('\n') @@ -797,7 +704,10 @@ func readStatus(t *testing.T, br *bufio.Reader) string { } for { next, err := br.ReadString('\n') - if err != nil || next == "\r\n" || next == "\n" { + if err != nil { + t.Fatalf("read status headers: %v", err) + } + if next == "\r\n" || next == "\n" { break } } @@ -869,10 +779,3 @@ func waitHolds(t *testing.T, held *holdCounter, want int32) { time.Sleep(10 * time.Millisecond) } } - -type shortWriteConn struct{ net.Conn } - -func (c shortWriteConn) Write(b []byte) (int, error) { - n, _ := c.Conn.Write(b[:min(2, len(b))]) - return n, errors.New("connection reset") -} diff --git a/sandboxd/egress/socks_test.go b/sandboxd/egress/socks_test.go index 1e0bdd001..3a32bb801 100644 --- a/sandboxd/egress/socks_test.go +++ b/sandboxd/egress/socks_test.go @@ -5,7 +5,6 @@ import ( "encoding/binary" "io" "net" - "net/http/httptest" "testing" "time" ) @@ -55,8 +54,7 @@ func TestSocksDecisionMirrorsConnect(t *testing.T) { t.Run(tt.name, func(t *testing.T) { events := make(chan Event, 4) p, socksAddr := socksProxy(t, tt.policy, tt.ca, fixedDial(echoServer(t)), events, nil) - front := httptest.NewServer(p) - defer front.Close() + front := serveFront(t, p) _, reply := socksExchange(t, socksAddr, socksGreeting, socksNameRequest("echo.internal", 443)) socksOK := reply[1] == socksGranted @@ -163,8 +161,7 @@ func TestSocksPortRuleGatesTheTunnel(t *testing.T) { events := make(chan Event, 4) _, addr := socksProxy(t, Policy{Allow: []Rule{{Host: "mail.internal", Ports: []uint16{993}}}}, nil, fixedDial(echoServer(t)), events, nil) - conn, reply := socksExchange(t, addr, socksGreeting, socksNameRequest("mail.internal", 993)) - defer func() { _ = conn.Close() }() + _, reply := socksExchange(t, addr, socksGreeting, socksNameRequest("mail.internal", 993)) if reply[1] != socksGranted { t.Fatalf("listed port reply = %#x, want granted", reply[1]) } @@ -172,8 +169,7 @@ func TestSocksPortRuleGatesTheTunnel(t *testing.T) { t.Errorf("audit event = %+v, want allow on port 993", ev) } - denied, reply := socksExchange(t, addr, socksGreeting, socksNameRequest("mail.internal", 143)) - defer func() { _ = denied.Close() }() + _, reply = socksExchange(t, addr, socksGreeting, socksNameRequest("mail.internal", 143)) if reply[1] != socksDenied { t.Fatalf("unlisted port reply = %#x, want denied", reply[1]) } @@ -185,8 +181,7 @@ func TestSocksPortRuleGatesTheTunnel(t *testing.T) { func TestSocksPortZeroIsDeniedEvenByABareRule(t *testing.T) { events := make(chan Event, 4) _, addr := socksProxy(t, Policy{Allow: []Rule{{Host: "mail.internal"}}}, nil, fixedDial(echoServer(t)), events, nil) - conn, reply := socksExchange(t, addr, socksGreeting, socksNameRequest("mail.internal", 0)) - defer func() { _ = conn.Close() }() + _, reply := socksExchange(t, addr, socksGreeting, socksNameRequest("mail.internal", 0)) if reply[1] != socksDenied { t.Fatalf("port 0 reply = %#x, want denied", reply[1]) } diff --git a/sandboxd/egress/splice_linux_test.go b/sandboxd/egress/splice_linux_test.go index 4af97874f..57ec189e1 100644 --- a/sandboxd/egress/splice_linux_test.go +++ b/sandboxd/egress/splice_linux_test.go @@ -17,75 +17,76 @@ func TestSpliceCopiesThroughADoorConnInTheKernel(t *testing.T) { {"splice", false}, {"no rx splice", true}, } { - t.Run(tt.name, func(t *testing.T) { testSpliceThroughDoor(t, tt.noSplice) }) - } -} - -func testSpliceThroughDoor(t *testing.T, noSplice bool) { - const total = 16 << 20 - uln, err := net.Listen("unix", filepath.Join(t.TempDir(), "door")) - if err != nil { - t.Fatalf("listen unix: %v", err) - } - t.Cleanup(func() { _ = uln.Close() }) - tln, err := net.Listen("tcp", "127.0.0.1:0") - if err != nil { - t.Fatalf("listen tcp: %v", err) - } - t.Cleanup(func() { _ = tln.Close() }) - buffered := make(chan struct{}, 2) - go func() { - c, acceptErr := tln.Accept() - if acceptErr != nil { - return - } - pump(c, total, buffered) - _ = c.(*net.TCPConn).CloseWrite() - }() - guestDone := make(chan struct{}) - go func() { - defer close(guestDone) - c, dialErr := net.Dial("unix", uln.Addr().String()) - if dialErr != nil { - return - } - defer func() { _ = c.Close() }() - buf := make([]byte, 64<<10) - buffered <- struct{}{} - for n := 0; n < total; n += len(buf) { - _, _ = c.Write(buf) - } - _ = c.(*net.UnixConn).CloseWrite() - for { - if _, readErr := c.Read(buf); readErr != nil { - return + t.Run(tt.name, func(t *testing.T) { + const total = 16 << 20 + uln, err := net.Listen("unix", filepath.Join(t.TempDir(), "door")) + if err != nil { + t.Fatalf("listen unix: %v", err) } - } - }() - guest, err := uln.Accept() - if err != nil { - t.Fatalf("accept: %v", err) - } - upstream, err := net.Dial("tcp", tln.Addr().String()) - if err != nil { - t.Fatalf("dial: %v", err) - } - door := doorShaped{guest.(*net.UnixConn)} - defer func() { _ = door.Close(); _ = upstream.Close() }() + t.Cleanup(func() { _ = uln.Close() }) + tln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen tcp: %v", err) + } + t.Cleanup(func() { _ = tln.Close() }) + buffered := make(chan struct{}, 2) + go func() { + c, acceptErr := tln.Accept() + if acceptErr != nil { + return + } + defer func() { _ = c.Close() }() + pump(c, total, buffered) + _ = c.(*net.TCPConn).CloseWrite() + }() + guestDone := make(chan struct{}) + go func() { + defer close(guestDone) + c, dialErr := net.Dial("unix", uln.Addr().String()) + if dialErr != nil { + t.Errorf("guest dial: %v", dialErr) + _ = uln.Close() + return + } + defer func() { _ = c.Close() }() + buf := make([]byte, 64<<10) + buffered <- struct{}{} + for n := 0; n < total; n += len(buf) { + _, _ = c.Write(buf) + } + _ = c.(*net.UnixConn).CloseWrite() + for { + if _, readErr := c.Read(buf); readErr != nil { + return + } + } + }() + guest, err := uln.Accept() + if err != nil { + t.Fatalf("accept: %v", err) + } + upstream, err := net.Dial("tcp", tln.Addr().String()) + if err != nil { + t.Fatalf("dial: %v", err) + } + door := doorShaped{guest.(*net.UnixConn)} + defer func() { _ = door.Close(); _ = upstream.Close() }() - <-buffered - <-buffered - runtime.GC() - var before, after runtime.MemStats - runtime.ReadMemStats(&before) - up, down := splice(door, upstream, noSplice) - runtime.ReadMemStats(&after) - <-guestDone - if up != total || down != total { - t.Fatalf("moved %d up and %d down, want %d each", up, down, total) - } - if alloc := after.TotalAlloc - before.TotalAlloc; !noSplice && alloc >= 32<<10 { - t.Errorf("splice allocated %d bytes: a door conn fell back to buffered copies", alloc) + <-buffered + <-buffered + runtime.GC() + var before, after runtime.MemStats + runtime.ReadMemStats(&before) + up, down := splice(door, upstream, tt.noSplice) + runtime.ReadMemStats(&after) + <-guestDone + if up != total || down != total { + t.Fatalf("moved %d up and %d down, want %d each", up, down, total) + } + if alloc := after.TotalAlloc - before.TotalAlloc; !tt.noSplice && alloc >= 32<<10 { + t.Errorf("splice allocated %d bytes: a door conn fell back to buffered copies", alloc) + } + }) } } diff --git a/sandboxd/egress/upstream_test.go b/sandboxd/egress/upstream_test.go index 6ad555cd9..dd3a3ddd5 100644 --- a/sandboxd/egress/upstream_test.go +++ b/sandboxd/egress/upstream_test.go @@ -11,6 +11,7 @@ import ( "net" "net/http" "net/http/httptest" + "net/url" "slices" "strings" "sync" @@ -66,7 +67,7 @@ func TestUpstreamAllow(t *testing.T) { func TestDialUpstreamThroughHTTPConnect(t *testing.T) { const greeting = "SSH-2.0-test\r\n" addr, seen := fakeConnectProxy(t, http.StatusOK, greeting) - u, _ := ParseUpstream("http://alice:pw@" + addr) + u := mustUpstream(t, "http://alice:pw@"+addr) conn, err := DialUpstream(t.Context(), u, "203.0.113.9:443", plainDial) if err != nil { t.Fatalf("DialUpstream: %v", err) @@ -77,7 +78,7 @@ func TestDialUpstreamThroughHTTPConnect(t *testing.T) { t.Fatalf("greeting %q, %v", buf, err) } assertEcho(t, conn) - if err := conn.(interface{ CloseWrite() error }).CloseWrite(); err != nil { + if err := conn.(closeWriter).CloseWrite(); err != nil { t.Fatalf("half-close: %v", err) } if tail, err := io.ReadAll(conn); err != nil || len(tail) != 0 { @@ -93,7 +94,7 @@ func TestDialUpstreamThroughHTTPConnect(t *testing.T) { func TestUpstreamTunnelKeepsTheZeroCopyPathAndBufferedBytes(t *testing.T) { const greeting = "220 ready\r\n" addr, _ := fakeConnectProxy(t, http.StatusOK, greeting) - u, _ := ParseUpstream("http://" + addr) + u := mustUpstream(t, "http://"+addr) conn, err := DialUpstream(t.Context(), u, "203.0.113.9:25", plainDial) if err != nil { t.Fatalf("DialUpstream: %v", err) @@ -107,7 +108,7 @@ func TestUpstreamTunnelKeepsTheZeroCopyPathAndBufferedBytes(t *testing.T) { if _, err := rf.ReadFrom(strings.NewReader("ping")); err != nil { t.Fatalf("ReadFrom: %v", err) } - if err := conn.(interface{ CloseWrite() error }).CloseWrite(); err != nil { + if err := conn.(closeWriter).CloseWrite(); err != nil { t.Fatalf("half-close: %v", err) } var got strings.Builder @@ -118,7 +119,7 @@ func TestUpstreamTunnelKeepsTheZeroCopyPathAndBufferedBytes(t *testing.T) { func TestDialUpstreamRefusedNamesNoCredentials(t *testing.T) { addr, _ := fakeConnectProxy(t, http.StatusProxyAuthRequired, "") - u, _ := ParseUpstream("http://alice:pw-secret@" + addr) + u := mustUpstream(t, "http://alice:pw-secret@"+addr) _, err := DialUpstream(t.Context(), u, "203.0.113.9:443", plainDial) if err == nil || !strings.Contains(err.Error(), "407") || strings.Contains(err.Error(), "pw-secret") { t.Fatalf("refused tunnel: %v, want a 407 error without the password", err) @@ -127,7 +128,7 @@ func TestDialUpstreamRefusedNamesNoCredentials(t *testing.T) { func TestDialUpstreamThroughSOCKS5WithAuth(t *testing.T) { addr, seen := fakeSOCKS5(t) - u, _ := ParseUpstream("socks5://bob:hunter2@" + addr) + u := mustUpstream(t, "socks5://bob:hunter2@"+addr) conn, err := DialUpstream(t.Context(), u, "203.0.113.9:443", plainDial) if err != nil { t.Fatalf("DialUpstream: %v", err) @@ -137,7 +138,7 @@ func TestDialUpstreamThroughSOCKS5WithAuth(t *testing.T) { if got, want := <-seen, "bob:hunter2 203.0.113.9:443"; got != want { t.Errorf("socks5 upstream saw %q, want %q", got, want) } - if _, ok := conn.(interface{ CloseWrite() error }); !ok { + if _, ok := conn.(closeWriter); !ok { t.Errorf("tunnel %T lost CloseWrite, so splice cannot half-close it", conn) } } @@ -146,7 +147,7 @@ func TestDialUpstreamTimesOutAndCloses(t *testing.T) { for _, stage := range []string{"dial", "http", "socks5"} { t.Run(stage, func(t *testing.T) { synctest.Test(t, func(t *testing.T) { - u, _ := ParseUpstream("http://proxy.example:80") + u := mustUpstream(t, "http://proxy.example:80") if stage == "socks5" { u.Scheme = stage } @@ -246,8 +247,7 @@ func TestTransferCountsTunnelAndRequestBytes(t *testing.T) { p := New(Policy{Allow: []Rule{{Host: "*"}}}, nil, nil, router, nil, nil) t.Cleanup(p.Close) p.OnTransfer(func(_ Event, sent, received int64) { got <- transfer{sent, received} }) - srv := httptest.NewServer(p) - t.Cleanup(srv.Close) + srv := serveFront(t, p) conn := dialConnect(t, srv.Listener.Addr().String(), "tunnel.example:7") br := bufio.NewReader(conn) @@ -298,8 +298,7 @@ func TestADeadUpstreamIsNamedInTheAudit(t *testing.T) { events := make(chan Event, 4) p := New(Policy{Allow: []Rule{{Host: "*"}}}, nil, nil, upstreamRouter{route: "http://u:pw@" + dead}, func(ev Event) { events <- ev }, nil) t.Cleanup(p.Close) - srv := httptest.NewServer(p) - t.Cleanup(srv.Close) + srv := serveFront(t, p) conn := dialConnect(t, srv.Listener.Addr().String(), "203.0.113.9:443") if status := readStatus(t, bufio.NewReader(conn)); !strings.Contains(status, "502") { @@ -309,10 +308,7 @@ func TestADeadUpstreamIsNamedInTheAudit(t *testing.T) { if ev := recvEvent(t, events); ev.Upstream != dead { t.Errorf("CONNECT audit names upstream %q, want %q", ev.Upstream, dead) } - resp, err := proxyClient(t, srv.URL).Get("http://203.0.113.9/") - if err != nil { - t.Fatalf("forward: %v", err) - } + resp := proxiedGet(t, srv.URL, "http://203.0.113.9/") _ = resp.Body.Close() if resp.StatusCode != http.StatusBadGateway { t.Errorf("forward through a dead upstream: %d, want 502", resp.StatusCode) @@ -373,6 +369,19 @@ func (r *switchRouter) dialed() []string { return slices.Clone(r.dials) } +type closeWriter interface { + CloseWrite() error +} + +func mustUpstream(t *testing.T, raw string) *url.URL { + t.Helper() + u, err := ParseUpstream(raw) + if err != nil { + t.Fatalf("ParseUpstream(%q): %v", raw, err) + } + return u +} + func plainDial(ctx context.Context, network, addr string) (net.Conn, error) { var d net.Dialer return d.DialContext(ctx, network, addr) diff --git a/sandboxd/engine/capacity_test.go b/sandboxd/engine/capacity_test.go index e9373f341..2d64c4e98 100644 --- a/sandboxd/engine/capacity_test.go +++ b/sandboxd/engine/capacity_test.go @@ -7,7 +7,7 @@ import ( ) func TestOnlyNodeWideFailuresCarryACapacitySignature(t *testing.T) { - for _, tc := range []struct { + for _, tt := range []struct { name string err error want bool @@ -19,9 +19,9 @@ func TestOnlyNodeWideFailuresCarryACapacitySignature(t *testing.T) { {"guest command out of disk", fmt.Errorf("warmup: %w", &guestExitError{code: 1, output: "dd: no space left on device"}), false}, {"no error", nil, false}, } { - t.Run(tc.name, func(t *testing.T) { - if got := CapacitySignature(tc.err) != ""; got != tc.want { - t.Errorf("CapacitySignature(%v) carries a signature: %v, want %v", tc.err, got, tc.want) + t.Run(tt.name, func(t *testing.T) { + if got := CapacitySignature(tt.err) != ""; got != tt.want { + t.Errorf("CapacitySignature(%v) carries a signature: %v, want %v", tt.err, got, tt.want) } }) } diff --git a/sandboxd/engine/cloneargs_test.go b/sandboxd/engine/cloneargs_test.go index 3c531cded..14868be7c 100644 --- a/sandboxd/engine/cloneargs_test.go +++ b/sandboxd/engine/cloneargs_test.go @@ -21,20 +21,20 @@ func TestCloneArgsRestoreMode(t *testing.T) { {"egress lane carries the flag", types.RestoreMmap, egressKey, true}, {"unset mode adds nothing", "", noneKey, false}, } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - e := New("cocoon", []string{"br0"}, nil, false, false, tc.mode) + for _, tt := range cases { + t.Run(tt.name, func(t *testing.T) { + e := New("cocoon", []string{"br0"}, nil, false, false, tt.mode) for _, args := range [][]string{ - e.cloneArgs("/goldens/g1", "sbx-1", tc.key), - e.cloneSnapArgs("ck_1", "sbx-1", tc.key), + e.cloneArgs("/goldens/g1", "sbx-1", tt.key), + e.cloneSnapArgs("ck_1", "sbx-1", tt.key), e.restoreCmdArgs("sbx-1", "sbx-hib-1"), } { i := slices.Index(args, "--restore-mode") - if got := i >= 0; got != tc.want { - t.Fatalf("args %v: restore-mode present=%v, want %v", args, got, tc.want) + if got := i >= 0; got != tt.want { + t.Fatalf("args %v: restore-mode present=%v, want %v", args, got, tt.want) } - if tc.want && args[i+1] != string(tc.mode) { - t.Fatalf("args %v: mode %q, want %q", args, args[i+1], tc.mode) + if tt.want && args[i+1] != string(tt.mode) { + t.Fatalf("args %v: mode %q, want %q", args, args[i+1], tt.mode) } } }) @@ -57,7 +57,7 @@ func TestLifecycleArgsApplyDirectIOPolicy(t *testing.T) { t.Errorf("args %v missing %q", args, want) } } - if !slices.Contains(cold, "--nics") || !slices.Contains(cold, "0") { + if i := slices.Index(cold, "--nics"); i < 0 || i+1 == len(cold) || cold[i+1] != "0" { t.Errorf("none-lane cold args %v missing --nics 0", cold) } }) @@ -67,7 +67,7 @@ func TestLifecycleArgsApplyDirectIOPolicy(t *testing.T) { func TestEgressVMsSpreadOverEveryConfiguredShard(t *testing.T) { shards := []string{"sbx0", "sbx1", "sbx2", "sbx3"} key := types.PoolKey{Template: "rt:24.04", Net: types.NetEgress, Size: types.SizeMedium} - for name, tc := range map[string]struct { + for name, tt := range map[string]struct { e *Engine flag string }{ @@ -77,10 +77,10 @@ func TestEgressVMsSpreadOverEveryConfiguredShard(t *testing.T) { t.Run(name, func(t *testing.T) { counts := map[string]int{} for i := range 4000 { - args := tc.e.cloneArgs("/goldens/g1", "sbx-pool1-"+strconv.Itoa(i), key) - j := slices.Index(args, tc.flag) + args := tt.e.cloneArgs("/goldens/g1", "sbx-pool1-"+strconv.Itoa(i), key) + j := slices.Index(args, tt.flag) if j < 0 { - t.Fatalf("args %v carry no %s", args, tc.flag) + t.Fatalf("args %v carry no %s", args, tt.flag) } counts[args[j+1]]++ } @@ -135,7 +135,7 @@ func TestNoBalloonReachesColdBootsOnly(t *testing.T) { func TestColdArgsSizeTheDiskOnlyWhenAsked(t *testing.T) { key := types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeSmall} - e := New("cocoon", nil, nil, false, false, "") + e := newTestEngine() if args := e.runColdArgs("sbx-1", key, ""); slices.Contains(args, "--storage") { t.Errorf("unsized cold args %v carry --storage, want cocoon's default disk", args) } diff --git a/sandboxd/engine/cocoond_test.go b/sandboxd/engine/cocoond_test.go index 053dc205e..24f44cd73 100644 --- a/sandboxd/engine/cocoond_test.go +++ b/sandboxd/engine/cocoond_test.go @@ -18,7 +18,7 @@ func TestVMEventsDeliversTheSnapshotThenChanges(t *testing.T) { "event: change\ndata: {\"type\":\"MODIFIED\",\"vm\":{\"name\":\"sbx-1\",\"state\":\"stopped\",\"live\":false,\"last_transition_reason\":\"unexpected-exit\"}}\n\n") var synced []VMStatus var changes []VMChange - err := New("cocoon", nil, nil, false, false, "").VMEvents(t.Context(), socket, + err := newTestEngine().VMEvents(t.Context(), socket, func(vms []VMStatus) { synced = vms }, func(c VMChange) { changes = append(changes, c) }) if err == nil { @@ -35,7 +35,7 @@ func TestVMEventsDeliversTheSnapshotThenChanges(t *testing.T) { func TestVMEventsFailsWithoutADaemon(t *testing.T) { ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second) defer cancel() - err := New("cocoon", nil, nil, false, false, "").VMEvents(ctx, sockPath(t), func([]VMStatus) {}, func(VMChange) {}) + err := newTestEngine().VMEvents(ctx, sockPath(t), func([]VMStatus) {}, func(VMChange) {}) if err == nil { t.Fatal("VMEvents succeeded with no socket") } @@ -46,7 +46,7 @@ func TestInspectReadsOneVMAndReportsAMissingOne(t *testing.T) { "if [ \"$3\" = sbx-1 ]; then echo '{\"config\":{\"name\":\"sbx-1\"},\"state\":\"running\","+ "\"attached_devices\":{\"disks\":[{\"id\":\"disk-a\",\"name\":\"imagenet\",\"path\":\"/v/a.img\",\"readonly\":true},{\"id\":\"disk-b\",\"name\":\"scratch\",\"path\":\"/v/b.img\"}]}}'; exit 0; fi\n"+ "echo \"Error: inspect: vm $3: vm not found\" >&2; exit 1\n") - e := New("cocoon", nil, nil, false, false, "") + e := newTestEngine() rec, ok, err := e.Inspect(t.Context(), "sbx-1") wantDisks := []types.VMDisk{{Name: "imagenet", ReadOnly: true}, {Name: "scratch"}} if err != nil || !ok || rec.State != "running" || rec.Config.Name != "sbx-1" || !slices.Equal(rec.AttachedDevices.Disks, wantDisks) { diff --git a/sandboxd/engine/engine.go b/sandboxd/engine/engine.go index 66f8c8f3c..611f9f329 100644 --- a/sandboxd/engine/engine.go +++ b/sandboxd/engine/engine.go @@ -140,7 +140,7 @@ func (e *Engine) RunCold(ctx context.Context, name string, key types.PoolKey, st // Remove force-deletes a VM; a refused delete gets a force stop and one more delete. func (e *Engine) Remove(ctx context.Context, name string) error { _, err := e.run(ctx, "vm", "rm", "--force", name) - if err == nil || strings.Contains(err.Error(), vmNotFound) { + if err == nil || isVMNotFound(err) { return err } _ = e.Stop(ctx, name) @@ -261,7 +261,7 @@ func (e *Engine) List(ctx context.Context) ([]types.VMRecord, error) { func (e *Engine) Inspect(ctx context.Context, name string) (rec types.VMRecord, ok bool, err error) { out, err := e.run(ctx, "vm", "inspect", name) if err != nil { - if strings.Contains(err.Error(), vmNotFound) { + if isVMNotFound(err) { return types.VMRecord{}, false, nil } return types.VMRecord{}, false, err @@ -516,6 +516,10 @@ func shardOf(shards []string, name string) string { return shards[int(h.Sum32()>>1)%len(shards)] } +func isVMNotFound(err error) bool { + return strings.Contains(err.Error(), vmNotFound) +} + func respFail(resp wire.Response) string { if errResp, ok := resp.(*wire.ErrorResp); ok { return errResp.Error() diff --git a/sandboxd/engine/engine_test.go b/sandboxd/engine/engine_test.go index f64283d09..c7ebdace5 100644 --- a/sandboxd/engine/engine_test.go +++ b/sandboxd/engine/engine_test.go @@ -25,7 +25,7 @@ func TestDialSilkdConsumesOnlyHandshake(t *testing.T) { listenMuxer(t, path, "OK 2048\nX") - conn, err := New("cocoon", nil, nil, false, false, "").DialSilkd(t.Context(), path) + conn, err := newTestEngine().DialSilkd(t.Context(), path) if err != nil { t.Fatalf("DialSilkd: %v", err) } @@ -43,7 +43,7 @@ func TestDialSilkdRejectedHandshake(t *testing.T) { path := sockPath(t) listenMuxer(t, path, "ERR no guest listener\n") - _, err := New("cocoon", nil, nil, false, false, "").DialSilkd(t.Context(), path) + _, err := newTestEngine().DialSilkd(t.Context(), path) if err == nil || !strings.Contains(err.Error(), "no guest listener") { t.Errorf("got %v, want handshake rejection", err) } @@ -53,7 +53,7 @@ func TestDialSilkdReturnsRawConnWithoutOverRead(t *testing.T) { path := sockPath(t) listenMuxer(t, path, "OK 2048\n") - conn, err := New("cocoon", nil, nil, false, false, "").DialSilkd(t.Context(), path) + conn, err := newTestEngine().DialSilkd(t.Context(), path) if err != nil { t.Fatalf("DialSilkd: %v", err) } @@ -67,16 +67,16 @@ func TestDialSilkdRejectsOverlongReply(t *testing.T) { path := sockPath(t) listenMuxer(t, path, "OK "+strings.Repeat("0", connectMax)+"\n") - if _, err := New("cocoon", nil, nil, false, false, "").DialSilkd(t.Context(), path); !errors.Is(err, bufio.ErrBufferFull) { + if _, err := newTestEngine().DialSilkd(t.Context(), path); !errors.Is(err, bufio.ErrBufferFull) { t.Errorf("got %v, want bufio.ErrBufferFull", err) } } -func TestProbeSucceeds(t *testing.T) { +func TestProbeSucceedsOnAnInfoFrame(t *testing.T) { path := sockPath(t) listenMuxer(t, path, "OK 2048\n", infoFrame) - if err := New("cocoon", nil, nil, false, false, "").Probe(t.Context(), path, 2*time.Second); err != nil { + if err := newTestEngine().Probe(t.Context(), path, 2*time.Second); err != nil { t.Errorf("Probe: %v", err) } } @@ -85,7 +85,7 @@ func TestInfoRoundTripRejectsErrorFrame(t *testing.T) { path := sockPath(t) listenMuxer(t, path, "OK 2048\n", errFrame) - err := New("cocoon", nil, nil, false, false, "").infoRoundTrip(t.Context(), path, silkdPort) + err := newTestEngine().infoRoundTrip(t.Context(), path, silkdPort) if err == nil || !strings.Contains(err.Error(), `info reply type "error"`) { t.Errorf("got %v, want error-frame rejection", err) } @@ -95,7 +95,7 @@ func TestProbeRetriesPastFailures(t *testing.T) { path := sockPath(t) listenMuxer(t, path, "OK 2048\n", errFrame, errFrame, infoFrame) - if err := New("cocoon", nil, nil, false, false, "").Probe(t.Context(), path, 2*time.Second); err != nil { + if err := newTestEngine().Probe(t.Context(), path, 2*time.Second); err != nil { t.Errorf("Probe: %v", err) } } @@ -104,7 +104,7 @@ func TestProbeRetriesUntilListenerAppears(t *testing.T) { path := sockPath(t) done := make(chan error, 1) go func() { - done <- New("cocoon", nil, nil, false, false, "").Probe(t.Context(), path, 2*time.Second) + done <- newTestEngine().Probe(t.Context(), path, 2*time.Second) }() time.Sleep(60 * time.Millisecond) @@ -142,14 +142,14 @@ func TestDialGuestPortCtxCancel(t *testing.T) { ctx, cancel := context.WithTimeout(t.Context(), 150*time.Millisecond) defer cancel() - if _, err := New("cocoon", nil, nil, false, false, "").DialGuestPort(ctx, path, 8080); !errors.Is(err, context.DeadlineExceeded) { + if _, err := newTestEngine().DialGuestPort(ctx, path, 8080); !errors.Is(err, context.DeadlineExceeded) { t.Errorf("got %v, want context.DeadlineExceeded", err) } } func TestSnapshotListReadsAnEmptyStoreAndRejectsProse(t *testing.T) { fakeCocoon(t, "#!/bin/sh\nif [ \"$COCOON_TEST_PROSE\" = 1 ]; then echo 'No snapshots found.'; else echo '[]'; fi\n") - e := New("cocoon", nil, nil, false, false, "") + e := newTestEngine() t.Setenv("COCOON_TEST_PROSE", "0") if snaps, err := e.SnapshotList(t.Context()); err != nil || len(snaps) != 0 { @@ -163,7 +163,7 @@ func TestSnapshotListReadsAnEmptyStoreAndRejectsProse(t *testing.T) { func TestListPassesNoPositionalArgsToCocoon(t *testing.T) { fakeCocoon(t, "#!/bin/sh\n[ \"$#\" = 4 ] || { echo \"unknown command $4 for cocoon vm list\" >&2; exit 1; }\necho '[{\"config\":{\"name\":\"sbx-1\"},\"state\":\"running\"}]'\n") - vms, err := New("cocoon", nil, nil, false, false, "").List(t.Context()) + vms, err := newTestEngine().List(t.Context()) if err != nil || len(vms) != 1 || vms[0].Config.Name != "sbx-1" { t.Fatalf("List = %+v, %v; want the one VM cocoon printed", vms, err) } @@ -172,7 +172,7 @@ func TestListPassesNoPositionalArgsToCocoon(t *testing.T) { func TestImageIDsMapsNameToID(t *testing.T) { fakeCocoon(t, "#!/bin/sh\n[ \"$*\" = \"image list --format json\" ] || { echo \"unexpected args $*\" >&2; exit 1; }\n"+ "echo '[{\"id\":\"sha256:afe6\",\"name\":\"ghcr.io/cocoonstack/sandbox/rt:24.04\",\"type\":\"oci\",\"size\":1}]'\n") - ids, err := New("cocoon", nil, nil, false, false, "").ImageIDs(t.Context()) + ids, err := newTestEngine().ImageIDs(t.Context()) if err != nil || len(ids) != 1 || ids["ghcr.io/cocoonstack/sandbox/rt:24.04"] != "sha256:afe6" { t.Fatalf("ImageIDs = %v, %v; want the one image cocoon printed", ids, err) } @@ -203,7 +203,7 @@ func TestRemoveStopsAVMThatRefusesDeletionThenRemovesItAgain(t *testing.T) { if tt.env != "" { t.Setenv(tt.env, "1") } - err := New("cocoon", nil, nil, false, false, "").Remove(t.Context(), "sbx-1") + err := newTestEngine().Remove(t.Context(), "sbx-1") data, _ := os.ReadFile(trace) if calls := strings.Split(strings.TrimSpace(string(data)), "\n"); !slices.Equal(calls, tt.calls) || (err != nil) != tt.fails { t.Errorf("Remove = %v with cocoon calls %q, want %q and failure %v", err, calls, tt.calls, tt.fails) @@ -216,7 +216,7 @@ func TestMemoryDumpsRunAtMostEightAtOnce(t *testing.T) { trace := filepath.Join(t.TempDir(), "trace") t.Setenv("COCOON_TEST_TRACE", trace) fakeCocoon(t, "#!/bin/sh\necho + >> \"$COCOON_TEST_TRACE\"\nsleep 0.1\necho - >> \"$COCOON_TEST_TRACE\"\n") - e := New("cocoon", nil, nil, false, false, "") + e := newTestEngine() var wg sync.WaitGroup for i := range 40 { wg.Go(func() { @@ -246,10 +246,10 @@ func TestMemoryDumpsRunAtMostEightAtOnce(t *testing.T) { } } -func TestProbeTimeout(t *testing.T) { +func TestProbeOfAMissingSocketTimesOut(t *testing.T) { path := sockPath(t) - err := New("cocoon", nil, nil, false, false, "").Probe(t.Context(), path, 150*time.Millisecond) + err := newTestEngine().Probe(t.Context(), path, 150*time.Millisecond) if err == nil || !strings.Contains(err.Error(), "silkd probe") { t.Errorf("got %v, want probe timeout", err) } @@ -292,6 +292,10 @@ func serveConn(conn net.Conn, handshake, frame string) { _, _ = io.WriteString(conn, frame) } +func newTestEngine() *Engine { + return New("cocoon", nil, nil, false, false, "") +} + func sockPath(t *testing.T) string { t.Helper() dir, err := os.MkdirTemp("", "sbx") diff --git a/sandboxd/engine/fakesilkd_test.go b/sandboxd/engine/fakesilkd_test.go new file mode 100644 index 000000000..720d6b04b --- /dev/null +++ b/sandboxd/engine/fakesilkd_test.go @@ -0,0 +1,287 @@ +package engine + +import ( + "bufio" + "bytes" + "cmp" + "encoding/json/v2" + "io" + "net" + "slices" + "strings" + "sync" + "testing" + + "github.com/cocoonstack/sandbox/protocol/wire" +) + +type fakeEdit func(*fakeSilkd) + +type fakeSilkd struct { + mu sync.Mutex + + connects []string + + writePath string + writeMode uint32 + writeData []byte + writeErr string + writeKind string + + execArgv []string + execCalls [][]string + execDetach []bool + execEnv map[string]string + execCode int32 + execFailAt int + execFailFrom int + execHook func([]string) + execErr string + execDrop bool + notReady int + realpath map[string]string + noRealpath bool + + block []wire.DirEntry + listErr string + listCalls int + + serial map[string][]byte + readMisses map[string]int + blankReads map[string]int + readErr map[string]string + readCalls []string + + stat map[string]bool + statMisses map[string]int + statErr map[string]string + statCalls []string +} + +func serveFakeSilkd(t *testing.T, path string) *fakeSilkd { + t.Helper() + ln, err := net.Listen("unix", path) + if err != nil { + t.Fatalf("listen: %v", err) + } + t.Cleanup(func() { _ = ln.Close() }) + f := &fakeSilkd{ + serial: make(map[string][]byte), + readMisses: make(map[string]int), + blankReads: make(map[string]int), + readErr: make(map[string]string), + stat: make(map[string]bool), + statMisses: make(map[string]int), + statErr: make(map[string]string), + } + go func() { + for { + conn, err := ln.Accept() + if err != nil { + return + } + go f.serve(conn) + } + }() + return f +} + +func (f *fakeSilkd) set(fn fakeEdit) { + f.mu.Lock() + defer f.mu.Unlock() + fn(f) +} + +func (f *fakeSilkd) serve(conn net.Conn) { + defer func() { _ = conn.Close() }() + r := bufio.NewReader(conn) + connect, err := r.ReadString('\n') + if err != nil { + return + } + f.mu.Lock() + f.connects = append(f.connects, strings.TrimSpace(connect)) + f.mu.Unlock() + if _, err = io.WriteString(conn, "OK 2048\n"); err != nil { + return + } + line, err := r.ReadString('\n') + if err != nil { + return + } + var req struct { + Op string `json:"op"` + Path string `json:"path"` + Mode uint32 `json:"mode"` + Argv []string `json:"argv"` + Env map[string]string `json:"env"` + Detach bool `json:"detach"` + } + if json.Unmarshal([]byte(line), &req) != nil { + return + } + if req.Op == "exec" { + f.mu.Lock() + f.execDetach = append(f.execDetach, req.Detach) + f.mu.Unlock() + } + switch req.Op { + case "info": + writeFakeSilkdResponse(conn, &wire.InfoResp{Version: "test", Proto: wire.KeepAliveProto}) + case "fs_write": + f.handleWrite(conn, r, req.Path, req.Mode) + case "fs_list": + f.handleList(conn, req.Path) + case "fs_read": + f.handleRead(conn, req.Path) + case "fs_stat": + f.handleStat(conn, req.Path) + case "exec": + f.handleExec(conn, req.Argv, req.Env) + } +} + +func (f *fakeSilkd) handleWrite(conn net.Conn, r *bufio.Reader, path string, mode uint32) { + var data []byte + for { + line, err := r.ReadString('\n') + if err != nil { + return + } + var frame struct { + Op string `json:"op"` + Data []byte `json:"data"` + } + if json.Unmarshal([]byte(line), &frame) != nil { + return + } + if frame.Op == "data_end" { + break + } + data = append(data, frame.Data...) + } + f.mu.Lock() + f.writePath, f.writeMode, f.writeData = path, mode, data + reply, kind := f.writeErr, cmp.Or(f.writeKind, wire.KindInternal) + f.mu.Unlock() + if reply != "" { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: kind, Message: reply}) + return + } + writeFakeSilkdResponse(conn, &wire.Done{}) +} + +func (f *fakeSilkd) handleExec(conn net.Conn, argv []string, env map[string]string) { + f.mu.Lock() + f.execArgv = argv + f.execCalls = append(f.execCalls, argv) + f.execEnv = env + code, reply, disconnect := f.execCode, f.execErr, f.execDrop + if argv[0] == "realpath" { + code, reply, disconnect = 0, "", false + if f.noRealpath { + code = 1 + } + } + if f.execFailAt > 0 && len(f.execCalls) != f.execFailAt || len(f.execCalls) < f.execFailFrom { + code, reply = 0, "" + } + stderr := "" + if argv[0] == "mount" && f.notReady > 0 { + f.notReady-- + code, stderr = 32, "mount: /data: /dev/vdc is not a valid block device." + } + if code == 0 && reply == "" && f.execHook != nil { + f.execHook(argv) + } + resolved := f.realpath[argv[len(argv)-1]] + f.mu.Unlock() + if disconnect { + return + } + if reply != "" { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindInternal, Message: reply}) + return + } + writeFakeSilkdResponse(conn, &wire.Started{PID: 1}) + if stderr != "" { + writeFakeSilkdResponse(conn, &wire.Stderr{Data: []byte(stderr)}) + } + if argv[0] == "realpath" && code == 0 { + writeFakeSilkdResponse(conn, &wire.Stdout{Data: []byte(cmp.Or(resolved, argv[len(argv)-1]) + "\n")}) + } + writeFakeSilkdResponse(conn, &wire.Exit{Code: code}) +} + +func (f *fakeSilkd) handleList(conn net.Conn, path string) { + f.mu.Lock() + f.listCalls++ + entries, reply := slices.Clone(f.block), f.listErr + f.mu.Unlock() + if reply != "" { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindInternal, Message: reply}) + return + } + if path != "/sys/block" { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindNotFound, Message: path}) + return + } + writeFakeSilkdResponse(conn, &wire.Entries{Entries: entries}) + writeFakeSilkdResponse(conn, &wire.Done{}) +} + +func (f *fakeSilkd) handleRead(conn net.Conn, path string) { + f.mu.Lock() + f.readCalls = append(f.readCalls, path) + data, ok := f.serial[path] + reply := f.readErr[path] + if f.readMisses[path] > 0 { + f.readMisses[path]-- + ok = false + } + if f.blankReads[path] > 0 { + f.blankReads[path]-- + data = []byte("\n") + } + data = bytes.Clone(data) + f.mu.Unlock() + if reply != "" { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindInternal, Message: reply}) + return + } + if !ok { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindNotFound, Message: path}) + return + } + writeFakeSilkdResponse(conn, &wire.DataResp{Data: data}) + writeFakeSilkdResponse(conn, &wire.Done{}) +} + +func (f *fakeSilkd) handleStat(conn net.Conn, path string) { + f.mu.Lock() + f.statCalls = append(f.statCalls, path) + ok := f.stat[path] + reply := f.statErr[path] + if f.statMisses[path] > 0 { + f.statMisses[path]-- + ok = false + } + f.mu.Unlock() + if reply != "" { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindInternal, Message: reply}) + return + } + if !ok { + writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindNotFound, Message: path}) + return + } + writeFakeSilkdResponse(conn, &wire.Stat{Info: wire.FileInfo{Kind: wire.FileKindFile}}) +} + +func writeFakeSilkdResponse(conn net.Conn, resp wire.Response) { + buf, err := wire.EncodeResponse(resp) + if err != nil { + return + } + _, _ = conn.Write(append(buf, '\n')) +} diff --git a/sandboxd/engine/imds_test.go b/sandboxd/engine/imds_test.go index a7a47f5ef..e41cfd271 100644 --- a/sandboxd/engine/imds_test.go +++ b/sandboxd/engine/imds_test.go @@ -11,7 +11,7 @@ func TestWriteInstanceMetadataReplacesTheGuestDocument(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) doc := []byte(`{"region":"local"}`) - if err := New("cocoon", nil, nil, false, false, "").WriteInstanceMetadata(t.Context(), path, doc); err != nil { + if err := newTestEngine().WriteInstanceMetadata(t.Context(), path, doc); err != nil { t.Fatalf("WriteInstanceMetadata: %v", err) } fake.mu.Lock() @@ -24,14 +24,12 @@ func TestWriteInstanceMetadataReplacesTheGuestDocument(t *testing.T) { func TestWriteInstanceMetadataReportsAGuestWithoutAResponder(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) - fake.writeErr, fake.writeKind = "create: No such file or directory", wire.KindNotFound - e := New("cocoon", nil, nil, false, false, "") + fake.set(func(f *fakeSilkd) { f.writeErr, f.writeKind = "create: No such file or directory", wire.KindNotFound }) + e := newTestEngine() if err := e.WriteInstanceMetadata(t.Context(), path, []byte("{}")); !errors.Is(err, ErrNoInstanceMetadata) { t.Errorf("got %v, want ErrNoInstanceMetadata", err) } - fake.mu.Lock() - fake.writeKind = wire.KindInternal - fake.mu.Unlock() + fake.set(func(f *fakeSilkd) { f.writeKind = wire.KindInternal }) if err := e.WriteInstanceMetadata(t.Context(), path, []byte("{}")); err == nil || errors.Is(err, ErrNoInstanceMetadata) { t.Errorf("internal error: got %v, want a plain failure", err) } diff --git a/sandboxd/engine/installca_test.go b/sandboxd/engine/installca_test.go index a890f591e..84e6f945d 100644 --- a/sandboxd/engine/installca_test.go +++ b/sandboxd/engine/installca_test.go @@ -1,25 +1,14 @@ package engine import ( - "bufio" - "bytes" - "cmp" - "encoding/json/v2" - "fmt" - "io" - "net" - "slices" "strings" - "sync" "testing" - - "github.com/cocoonstack/sandbox/protocol/wire" ) func TestInstallCACertWritesCertAndUpdates(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) - if err := New("cocoon", nil, nil, false, false, "").InstallCACert(t.Context(), path, []byte("CERT-PEM")); err != nil { + if err := newTestEngine().InstallCACert(t.Context(), path, []byte("CERT-PEM")); err != nil { t.Fatalf("InstallCACert: %v", err) } fake.mu.Lock() @@ -44,7 +33,7 @@ func TestInstallCACertWritesCertAndUpdates(t *testing.T) { t.Errorf("exec %q does not sync the cert and bundle, so a cold boot inside the writeback window loses them", cmd) } if fake.execEnv["PATH"] == "" { - t.Errorf("exec env has no PATH; the guest command is run with an empty env") + t.Error("exec env has no PATH; the guest command is run with an empty env") } } @@ -52,7 +41,7 @@ func TestInstallCACertNonzeroExitFails(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) fake.execCode = 3 - err := New("cocoon", nil, nil, false, false, "").InstallCACert(t.Context(), path, []byte("x")) + err := newTestEngine().InstallCACert(t.Context(), path, []byte("x")) if err == nil || !strings.Contains(err.Error(), "exit code 3") { t.Errorf("got %v, want exit code 3 failure", err) } @@ -62,266 +51,8 @@ func TestInstallCACertWriteErrorFrameFails(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) fake.writeErr = "disk full" - err := New("cocoon", nil, nil, false, false, "").InstallCACert(t.Context(), path, []byte("x")) + err := newTestEngine().InstallCACert(t.Context(), path, []byte("x")) if err == nil || !strings.Contains(err.Error(), "disk full") { t.Errorf("got %v, want fs_write error-frame failure", err) } } - -type fakeSilkd struct { - mu sync.Mutex - writePath string - writeMode uint32 - writeData []byte - execArgv []string - execCalls [][]string - execEnv map[string]string - execCode int32 - execFailAt int - notReady int - execHook func([]string) - realpath map[string]string - noRealpath bool - writeErr string - writeKind string - execErr string - execDrop bool - block []wire.DirEntry - serial map[string][]byte - readMisses map[string]int - blankReads map[string]int - listErr string - readErr map[string]string - listCalls int - readCalls []string - stat map[string]bool - statMisses map[string]int - statErr map[string]string - statCalls []string - connects []string - execDetach []bool - - execFailFrom int -} - -func serveFakeSilkd(t *testing.T, path string) *fakeSilkd { - t.Helper() - ln, err := net.Listen("unix", path) - if err != nil { - t.Fatalf("listen: %v", err) - } - t.Cleanup(func() { _ = ln.Close() }) - f := &fakeSilkd{ - serial: make(map[string][]byte), - readMisses: make(map[string]int), - blankReads: make(map[string]int), - readErr: make(map[string]string), - stat: make(map[string]bool), - statMisses: make(map[string]int), - statErr: make(map[string]string), - } - go func() { - for { - conn, err := ln.Accept() - if err != nil { - return - } - go f.serve(conn) - } - }() - return f -} - -func (f *fakeSilkd) serve(conn net.Conn) { - defer func() { _ = conn.Close() }() - r := bufio.NewReader(conn) - connect, err := r.ReadString('\n') - if err != nil { - return - } - f.mu.Lock() - f.connects = append(f.connects, strings.TrimSpace(connect)) - f.mu.Unlock() - if _, err = io.WriteString(conn, "OK 2048\n"); err != nil { - return - } - line, err := r.ReadString('\n') - if err != nil { - return - } - var req struct { - Op string `json:"op"` - Path string `json:"path"` - Mode uint32 `json:"mode"` - Argv []string `json:"argv"` - Env map[string]string `json:"env"` - Detach bool `json:"detach"` - } - if json.Unmarshal([]byte(line), &req) != nil { - return - } - if req.Op == "exec" { - f.mu.Lock() - f.execDetach = append(f.execDetach, req.Detach) - f.mu.Unlock() - } - switch req.Op { - case "info": - writeFakeSilkdResponse(conn, &wire.InfoResp{Version: "test", Proto: wire.KeepAliveProto}) - case "fs_write": - f.handleWrite(conn, r, req.Path, req.Mode) - case "fs_list": - f.handleList(conn, req.Path) - case "fs_read": - f.handleRead(conn, req.Path) - case "fs_stat": - f.handleStat(conn, req.Path) - case "exec": - f.handleExec(conn, req.Argv, req.Env) - } -} - -func (f *fakeSilkd) handleWrite(conn net.Conn, r *bufio.Reader, path string, mode uint32) { - var data []byte - for { - line, err := r.ReadString('\n') - if err != nil { - return - } - var frame struct { - Op string `json:"op"` - Data []byte `json:"data"` - } - if json.Unmarshal([]byte(line), &frame) != nil { - return - } - if frame.Op == "data_end" { - break - } - data = append(data, frame.Data...) - } - f.mu.Lock() - f.writePath, f.writeMode, f.writeData = path, mode, data - reply, kind := f.writeErr, cmp.Or(f.writeKind, wire.KindInternal) - f.mu.Unlock() - if reply != "" { - writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: kind, Message: reply}) - return - } - _, _ = io.WriteString(conn, `{"type":"done"}`+"\n") -} - -func (f *fakeSilkd) handleExec(conn net.Conn, argv []string, env map[string]string) { - f.mu.Lock() - f.execArgv = argv - f.execCalls = append(f.execCalls, argv) - f.execEnv = env - code, reply, disconnect := f.execCode, f.execErr, f.execDrop - if argv[0] == "realpath" { - code, reply, disconnect = 0, "", false - if f.noRealpath { - code = 1 - } - } - if f.execFailAt > 0 && len(f.execCalls) != f.execFailAt || len(f.execCalls) < f.execFailFrom { - code, reply = 0, "" - } - stderr := "" - if argv[0] == "mount" && f.notReady > 0 { - f.notReady-- - code, stderr = 32, "mount: /data: /dev/vdc is not a valid block device." - } - if code == 0 && reply == "" && f.execHook != nil { - f.execHook(argv) - } - resolved := f.realpath[argv[len(argv)-1]] - f.mu.Unlock() - if disconnect { - return - } - if reply != "" { - _, _ = io.WriteString(conn, `{"type":"error","kind":"internal","message":"`+reply+`"}`+"\n") - return - } - _, _ = io.WriteString(conn, `{"type":"started","pid":1}`+"\n") - if stderr != "" { - writeFakeSilkdResponse(conn, &wire.Stderr{Data: []byte(stderr)}) - } - if argv[0] == "realpath" && code == 0 { - writeFakeSilkdResponse(conn, &wire.Stdout{Data: []byte(cmp.Or(resolved, argv[len(argv)-1]) + "\n")}) - } - _, _ = fmt.Fprintf(conn, `{"type":"exit","code":%d}`+"\n", code) -} - -func (f *fakeSilkd) handleList(conn net.Conn, path string) { - f.mu.Lock() - f.listCalls++ - entries, reply := slices.Clone(f.block), f.listErr - f.mu.Unlock() - if reply != "" { - writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindInternal, Message: reply}) - return - } - if path != "/sys/block" { - writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindNotFound, Message: path}) - return - } - writeFakeSilkdResponse(conn, &wire.Entries{Entries: entries}) - writeFakeSilkdResponse(conn, &wire.Done{}) -} - -func (f *fakeSilkd) handleRead(conn net.Conn, path string) { - f.mu.Lock() - f.readCalls = append(f.readCalls, path) - data, ok := f.serial[path] - reply := f.readErr[path] - if f.readMisses[path] > 0 { - f.readMisses[path]-- - ok = false - } - if f.blankReads[path] > 0 { - f.blankReads[path]-- - data = []byte("\n") - } - data = bytes.Clone(data) - f.mu.Unlock() - if reply != "" { - writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindInternal, Message: reply}) - return - } - if !ok { - writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindNotFound, Message: path}) - return - } - writeFakeSilkdResponse(conn, &wire.DataResp{Data: data}) - writeFakeSilkdResponse(conn, &wire.Done{}) -} - -func (f *fakeSilkd) handleStat(conn net.Conn, path string) { - f.mu.Lock() - f.statCalls = append(f.statCalls, path) - ok := f.stat[path] - reply := f.statErr[path] - if f.statMisses[path] > 0 { - f.statMisses[path]-- - ok = false - } - f.mu.Unlock() - if reply != "" { - writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindInternal, Message: reply}) - return - } - if !ok { - writeFakeSilkdResponse(conn, &wire.ErrorResp{Kind: wire.KindNotFound, Message: path}) - return - } - writeFakeSilkdResponse(conn, &wire.Stat{Info: wire.FileInfo{Kind: wire.FileKindFile}}) -} - -func writeFakeSilkdResponse(conn net.Conn, resp wire.Response) { - buf, err := wire.EncodeResponse(resp) - if err != nil { - return - } - _, _ = conn.Write(append(buf, '\n')) -} diff --git a/sandboxd/engine/portconn_test.go b/sandboxd/engine/portconn_test.go index 8e0d80e3e..6d4936e37 100644 --- a/sandboxd/engine/portconn_test.go +++ b/sandboxd/engine/portconn_test.go @@ -16,7 +16,7 @@ import ( "github.com/cocoonstack/sandbox/protocol/wire" ) -func TestGuestPortConnFramesBothWays(t *testing.T) { +func TestGuestPortConnReadsDataUntilDone(t *testing.T) { silk, gc := newTestGuestPortConn(t) go func() { @@ -35,14 +35,14 @@ func TestGuestPortConnFramesBothWays(t *testing.T) { } func TestGuestPortConnCutBeforeDoneIsUnexpectedEOF(t *testing.T) { - for name, tail := range map[string]string{"between frames": "", "mid frame": `{"type":"da`} { + for name, trailer := range map[string]string{"between frames": "", "mid frame": `{"type":"da`} { t.Run(name, func(t *testing.T) { silk, gc := newTestGuestPortConn(t) go func() { frame, _ := json.Marshal(map[string]string{"type": "data", "data": base64.StdEncoding.EncodeToString([]byte("partial"))}, json.Deterministic(true)) _, _ = silk.Write(append(frame, '\n')) - _, _ = silk.Write([]byte(tail)) + _, _ = silk.Write([]byte(trailer)) _ = silk.Close() }() got := make([]byte, 64) @@ -107,9 +107,10 @@ func TestGuestPortConnRoundTripsFixtures(t *testing.T) { silk, gc := newTestGuestPortConn(t) + respData, respDone := fixture("resp_data.json"), fixture("resp_done.json") go func() { - _, _ = silk.Write(fixture("resp_data.json")) - _, _ = silk.Write(fixture("resp_done.json")) + _, _ = silk.Write(respData) + _, _ = silk.Write(respDone) }() buf := make([]byte, 64) n, err := gc.Read(buf) @@ -123,7 +124,8 @@ func TestGuestPortConnRoundTripsFixtures(t *testing.T) { client2, silk2 := net.Pipe() t.Cleanup(func() { _ = client2.Close(); _ = silk2.Close() }) gc2 := newGuestPortConn(client2, bufio.NewReaderSize(client2, portReadBuf)) - go func() { _, _ = silk2.Write(fixture("resp_error.json")) }() + respError := fixture("resp_error.json") + go func() { _, _ = silk2.Write(respError) }() if _, err := gc2.Read(buf); err == nil || errors.Is(err, io.EOF) { t.Errorf("fixture error frame: %v, want a typed error", err) } @@ -175,12 +177,12 @@ func TestGuestPortConnReadsSmallFramesThenBulkFrames(t *testing.T) { } func TestGuestPortConnRefusesAFrameOverMaxFrame(t *testing.T) { - for name, tail := range map[string]string{"unterminated": "", "terminated": "\n"} { + for name, trailer := range map[string]string{"unterminated": "", "terminated": "\n"} { t.Run(name, func(t *testing.T) { silk, gc := newTestGuestPortConn(t) go func() { - _, _ = silk.Write(append(bytes.Repeat([]byte("a"), wire.MaxFrame+1), tail...)) + _, _ = silk.Write(append(bytes.Repeat([]byte("a"), wire.MaxFrame+1), trailer...)) _ = silk.Close() }() if _, err := gc.Read(make([]byte, 64)); err == nil || !strings.Contains(err.Error(), "exceeds") { @@ -198,6 +200,7 @@ func newTestGuestPortConn(t *testing.T) (net.Conn, *guestPortConn) { } func writeDataFrames(t *testing.T, silk net.Conn, chunks ...[]byte) { + t.Helper() var stream []byte for _, chunk := range chunks { frame, err := wire.EncodeResponse(wire.DataResp{Data: chunk}) diff --git a/sandboxd/engine/reader_test.go b/sandboxd/engine/reader_test.go index 6c2d94ba7..c299f8714 100644 --- a/sandboxd/engine/reader_test.go +++ b/sandboxd/engine/reader_test.go @@ -12,7 +12,7 @@ func TestStartReaderOverlaysTheAttachedDiskOnTheGuestsOwnLayers(t *testing.T) { sock := sockPath(t) fake := serveFakeSilkd(t, sock) configureReaderDevices(fake) - e := New("cocoon", nil, nil, false, false, "") + e := newTestEngine() if err := e.StartReader(t.Context(), sock, "snap-ab12cd34"); err != nil { t.Fatalf("StartReader: %v", err) @@ -53,11 +53,9 @@ func TestStartReaderFailsFastWhenTheClaimImageHasNoSilkd(t *testing.T) { sock := sockPath(t) fake := serveFakeSilkd(t, sock) configureReaderDevices(fake) - fake.mu.Lock() - fake.execFailAt, fake.execCode = 9, 1 - fake.mu.Unlock() + fake.set(func(f *fakeSilkd) { f.execFailAt, f.execCode = 9, 1 }) - err := New("cocoon", nil, nil, false, false, "").StartReader(t.Context(), sock, "snap-ab12cd34") + err := newTestEngine().StartReader(t.Context(), sock, "snap-ab12cd34") if err == nil || !strings.Contains(err.Error(), "no silkd") { t.Fatalf("StartReader = %v, want the missing silkd reported", err) } @@ -72,11 +70,9 @@ func TestStartReaderStopsAtTheFirstFailedMount(t *testing.T) { sock := sockPath(t) fake := serveFakeSilkd(t, sock) configureReaderDevices(fake) - fake.mu.Lock() - fake.execFailFrom, fake.execCode = 6, 32 - fake.mu.Unlock() + fake.set(func(f *fakeSilkd) { f.execFailFrom, f.execCode = 6, 32 }) - err := New("cocoon", nil, nil, false, false, "").StartReader(t.Context(), sock, "snap-ab12cd34") + err := newTestEngine().StartReader(t.Context(), sock, "snap-ab12cd34") if err == nil || !strings.Contains(err.Error(), "mount layer 1") { t.Fatalf("StartReader = %v, want the layer 1 mount failure", err) } @@ -91,11 +87,9 @@ func TestStartReaderRefusesAGuestWithoutLayers(t *testing.T) { sock := sockPath(t) fake := serveFakeSilkd(t, sock) configureReaderDevices(fake) - fake.mu.Lock() - fake.serial["/proc/cmdline"] = []byte("console=hvc0 cocoon.cow=cow\n") - fake.mu.Unlock() + fake.set(func(f *fakeSilkd) { f.serial["/proc/cmdline"] = []byte("console=hvc0 cocoon.cow=cow\n") }) - err := New("cocoon", nil, nil, false, false, "").StartReader(t.Context(), sock, "snap-ab12cd34") + err := newTestEngine().StartReader(t.Context(), sock, "snap-ab12cd34") if err == nil || !strings.Contains(err.Error(), "cocoon.layers=") { t.Fatalf("StartReader = %v, want the missing layers reported", err) } @@ -128,7 +122,7 @@ func TestCmdlineLayersReadsTheLastParamInOrder(t *testing.T) { func TestDialReaderConnectsToTheViewPort(t *testing.T) { sock := sockPath(t) fake := serveFakeSilkd(t, sock) - conn, err := New("cocoon", nil, nil, false, false, "").DialReader(t.Context(), sock) + conn, err := newTestEngine().DialReader(t.Context(), sock) if err != nil { t.Fatalf("DialReader: %v", err) } diff --git a/sandboxd/engine/version_test.go b/sandboxd/engine/version_test.go index 0969b28d6..9e6174dc7 100644 --- a/sandboxd/engine/version_test.go +++ b/sandboxd/engine/version_test.go @@ -7,7 +7,7 @@ import ( "testing" ) -func TestBelowFloor(t *testing.T) { +func TestBelowFloorComparesReleasesAndLeavesOthersUnknown(t *testing.T) { tests := []struct { v string below bool @@ -35,7 +35,7 @@ func TestBelowFloor(t *testing.T) { } } -func TestCheckLiveVolumeSupport(t *testing.T) { +func TestLiveVolumeSupportFollowsTheCocoonVersion(t *testing.T) { for _, tt := range []struct { version string fail, allowed bool @@ -58,7 +58,7 @@ if [ "$#" != 1 ] || [ "$1" != version ]; then exit 2; fi if [ -n "$COCOON_TEST_VERSION_FAIL" ]; then echo /private/cocoon >&2; exit 1; fi printf 'Version: %s\n' "$COCOON_TEST_VERSION" `) - err := New("cocoon", nil, nil, false, false, "").CheckLiveVolumeSupport(t.Context()) + err := newTestEngine().CheckLiveVolumeSupport(t.Context()) if tt.allowed { if err != nil { t.Fatal(err) @@ -78,7 +78,7 @@ func TestVersionQueriesCocoonOnce(t *testing.T) { if [ -n "$COCOON_TEST_VERSION_FAIL" ]; then exit 1; fi printf 'Version: v0.7.2\n' `) - eng := New("cocoon", nil, nil, false, false, "") + eng := newTestEngine() if _, err := eng.Version(t.Context()); err != nil { t.Fatal(err) } diff --git a/sandboxd/engine/volume.go b/sandboxd/engine/volume.go index b0f24840b..27f6612ad 100644 --- a/sandboxd/engine/volume.go +++ b/sandboxd/engine/volume.go @@ -14,11 +14,10 @@ import ( ) const ( - // VolumeCallTimeout bounds one guest teardown exec. - VolumeCallTimeout = 2 * time.Second // VolumeMutationTimeout bounds one live attach or detach, guest steps included. VolumeMutationTimeout = 30 * time.Second + syncTimeout = 2 * time.Second volumePollInterval = 10 * time.Millisecond volumeProbeTimeout = 2 * time.Second volumeSetupTimeout = 10 * time.Second @@ -73,7 +72,7 @@ func (e *Engine) MountVolume(ctx context.Context, vsockSocket, name, mount strin // SyncGuest flushes the guest page cache, the only flush left for a failed unmount. func (e *Engine) SyncGuest(ctx context.Context, vsockSocket string) error { - ctx, cancel := context.WithTimeout(ctx, VolumeCallTimeout) + ctx, cancel := context.WithTimeout(ctx, syncTimeout) defer cancel() if err := e.silkdExec(ctx, vsockSocket, "sync"); err != nil { return fmt.Errorf("sync guest: %w", err) diff --git a/sandboxd/engine/volume_live_test.go b/sandboxd/engine/volume_live_test.go index 57209babe..3ad5635d7 100644 --- a/sandboxd/engine/volume_live_test.go +++ b/sandboxd/engine/volume_live_test.go @@ -25,27 +25,22 @@ func TestLiveVolumeConvergesMountAndDetach(t *testing.T) { {name: "unresolvable path", mount: "/missing/data", kernelPath: "/missing/data", noRealpath: true, lookups: 1}, } { t.Run(tt.name, func(t *testing.T) { - sock := sockPath(t) - fake := serveFakeSilkd(t, sock) - configureVolumeDevices(fake) - spec := liveDiskCLI(t, "") + sock, fake, spec := newLiveFake(t, "", "") if err := os.Remove(os.Getenv("COCOON_TEST_ATTACHED")); err != nil { t.Fatal(err) } - fake.mu.Lock() - fake.realpath, fake.noRealpath = tt.realpath, tt.noRealpath - fake.serial["/sys/block/vdc/dev"] = []byte("8:32\n") - fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo) - fake.execHook = func(argv []string) { - switch argv[0] { - case "mount": - fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + "2 1 8:32 / " + tt.kernelPath + " ro - ext4 /dev/vdc ro\n") - case "umount": - fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo) + fake.set(func(f *fakeSilkd) { + f.realpath, f.noRealpath = tt.realpath, tt.noRealpath + f.execHook = func(argv []string) { + switch argv[0] { + case "mount": + f.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + "2 1 8:32 / " + tt.kernelPath + " ro - ext4 /dev/vdc ro\n") + case "umount": + f.serial["/proc/self/mountinfo"] = []byte(rootMountInfo) + } } - } - fake.mu.Unlock() - e := New("cocoon", nil, nil, false, false, "") + }) + e := newTestEngine() for range 2 { if err := e.DiskAttach(t.Context(), "sbx-1", spec); err != nil { t.Fatal(err) @@ -75,16 +70,9 @@ func TestLiveVolumeConvergesMountAndDetach(t *testing.T) { } func TestLiveMountRefusesAnOccupiedPathBehindASymlink(t *testing.T) { - sock := sockPath(t) - fake := serveFakeSilkd(t, sock) - configureVolumeDevices(fake) - spec := liveDiskCLI(t, "") - fake.mu.Lock() - fake.serial["/sys/block/vdc/dev"] = []byte("8:32\n") - fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + "2 1 8:48 / /home/u/ws/data ro - ext4 /dev/vdd ro\n") - fake.realpath = map[string]string{"/workspace/data": "/home/u/ws/data"} - fake.mu.Unlock() - e := New("cocoon", nil, nil, false, false, "") + sock, fake, spec := newLiveFake(t, "", "2 1 8:48 / /home/u/ws/data ro - ext4 /dev/vdd ro\n") + fake.set(func(f *fakeSilkd) { f.realpath = map[string]string{"/workspace/data": "/home/u/ws/data"} }) + e := newTestEngine() if err := e.DiskAttach(t.Context(), "sbx-1", spec); err != nil { t.Fatal(err) } @@ -100,19 +88,12 @@ func TestLiveMountRefusesAnOccupiedPathBehindASymlink(t *testing.T) { } func TestLiveMountReadsMountinfoBeyondTheSysfsCap(t *testing.T) { - sock := sockPath(t) - fake := serveFakeSilkd(t, sock) - configureVolumeDevices(fake) - spec := liveDiskCLI(t, "") overlays := strings.Repeat("9 1 0:99 / /var/lib/docker/overlay2/layer/merged rw - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/ABCDEFGHIJKLMNOPQRSTUVWXYZ\n", 2000) - fake.mu.Lock() - fake.serial["/sys/block/vdc/dev"] = []byte("8:32\n") - fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + overlays) - fake.mu.Unlock() + sock, _, spec := newLiveFake(t, "", overlays) if len(rootMountInfo+overlays) <= silkdReadMax { t.Fatalf("fixture of %d bytes does not exceed the sysfs cap", len(rootMountInfo+overlays)) } - e := New("cocoon", nil, nil, false, false, "") + e := newTestEngine() if err := e.DiskAttach(t.Context(), "sbx-1", spec); err != nil { t.Fatal(err) } @@ -129,15 +110,8 @@ func TestLiveAttachRejectsForeignMounts(t *testing.T) { "malformed\n", } { t.Run(strings.TrimSpace(row), func(t *testing.T) { - sock := sockPath(t) - fake := serveFakeSilkd(t, sock) - configureVolumeDevices(fake) - spec := liveDiskCLI(t, "") - fake.mu.Lock() - fake.serial["/sys/block/vdc/dev"] = []byte("8:32\n") - fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + row) - fake.mu.Unlock() - err := New("cocoon", nil, nil, false, false, "").MountLiveVolume(t.Context(), sock, spec.Name, "/data", false) + sock, fake, spec := newLiveFake(t, "", row) + err := newTestEngine().MountLiveVolume(t.Context(), sock, spec.Name, "/data", false) if err == nil { t.Fatal("accepted a foreign mount") } @@ -160,17 +134,12 @@ func TestLiveAttachRejectsForeignMounts(t *testing.T) { func TestLiveDetachKeepsBusyAndUnobservableDevices(t *testing.T) { for _, mode := range []string{"busy", "second mount", "probe error", "guest error", "transport disconnect", "host error", "stacked mount", "stacked bind"} { t.Run(mode, func(t *testing.T) { - sock := sockPath(t) - fake := serveFakeSilkd(t, sock) - configureVolumeDevices(fake) hostErr := "" if mode == "host error" { hostErr = "device removal initiated but guest has not ejected it" } - spec := liveDiskCLI(t, hostErr) + sock, fake, spec := newLiveFake(t, hostErr, "") fake.mu.Lock() - fake.serial["/sys/block/vdc/dev"] = []byte("8:32\n") - fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo) switch mode { case "busy": fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + "2 1 8:32 / /data ro - ext4 /dev/vdc ro\n") @@ -191,7 +160,7 @@ func TestLiveDetachKeepsBusyAndUnobservableDevices(t *testing.T) { fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + "2 1 8:32 / /data ro - ext4 /dev/vdc ro\n3 2 8:32 /sub /data ro - ext4 /dev/vdc ro\n") } fake.mu.Unlock() - _, err := New("cocoon", nil, nil, false, false, "").DetachVolume(t.Context(), "sbx-1", sock, spec.Name, "/data") + _, err := newTestEngine().DetachVolume(t.Context(), "sbx-1", sock, spec.Name, "/data") if err == nil { t.Fatal("detach must fail closed") } @@ -222,7 +191,7 @@ func TestLiveDetachMissingGuestStillChecksHost(t *testing.T) { sock := sockPath(t) serveFakeSilkd(t, sock) spec := liveDiskCLI(t, hostErr) - _, err := New("cocoon", nil, nil, false, false, "").DetachVolume(t.Context(), "sbx-1", sock, spec.Name, "/data") + _, err := newTestEngine().DetachVolume(t.Context(), "sbx-1", sock, spec.Name, "/data") wantErr := hostErr != "" if (err != nil) != wantErr { t.Fatalf("got %v, want error=%v", err, wantErr) @@ -232,15 +201,8 @@ func TestLiveDetachMissingGuestStillChecksHost(t *testing.T) { } func TestLiveDetachIgnoresAnotherDeviceAtThePath(t *testing.T) { - sock := sockPath(t) - fake := serveFakeSilkd(t, sock) - configureVolumeDevices(fake) - spec := liveDiskCLI(t, "") - fake.mu.Lock() - fake.serial["/sys/block/vdc/dev"] = []byte("8:32\n") - fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + "2 1 8:48 / /data ro - ext4 /dev/vdd ro\n") - fake.mu.Unlock() - if _, err := New("cocoon", nil, nil, false, false, "").DetachVolume(t.Context(), "sbx-1", sock, spec.Name, "/data"); err != nil { + sock, fake, spec := newLiveFake(t, "", "2 1 8:48 / /data ro - ext4 /dev/vdd ro\n") + if _, err := newTestEngine().DetachVolume(t.Context(), "sbx-1", sock, spec.Name, "/data"); err != nil { t.Fatalf("detach of an unmounted device blocked by another device's mount: %v", err) } fake.mu.Lock() @@ -251,7 +213,7 @@ func TestLiveDetachIgnoresAnotherDeviceAtThePath(t *testing.T) { } func TestLiveDetachCleanEvidenceRequiresWritableUnmountAndHostSuccess(t *testing.T) { - for _, tc := range []struct { + for _, tt := range []struct { name, options, superOptions, mount, hostErr string missing, unmounted, busy, residual, wantClean, wantErr bool }{ @@ -269,35 +231,35 @@ func TestLiveDetachCleanEvidenceRequiresWritableUnmountAndHostSuccess(t *testing {name: "residual mount", options: "rw", superOptions: "rw", mount: "/data", residual: true, wantErr: true}, {name: "attach-only mounted", options: "rw", superOptions: "rw", wantErr: true}, } { - t.Run(tc.name, func(t *testing.T) { + t.Run(tt.name, func(t *testing.T) { sock := sockPath(t) fake := serveFakeSilkd(t, sock) - if !tc.missing { + if !tt.missing { configureVolumeDevices(fake) } - spec := liveDiskCLI(t, tc.hostErr) + spec := liveDiskCLI(t, tt.hostErr) fake.mu.Lock() fake.serial["/sys/block/vdc/dev"] = []byte("8:32\n") base := rootMountInfo + "501 84 0:62 / /empty-source rw,relatime - tmpfs rw,size=1024k,inode64\n" raw := base - if !tc.unmounted { - raw += "2 1 8:32 / /data " + tc.options + " shared:4 - ext4 /dev/vdc " + tc.superOptions + "\n" + if !tt.unmounted { + raw += "2 1 8:32 / /data " + tt.options + " shared:4 - ext4 /dev/vdc " + tt.superOptions + "\n" } fake.serial["/proc/self/mountinfo"] = []byte(raw) - if tc.busy { + if tt.busy { fake.execCode = 32 } fake.execHook = func(argv []string) { - if argv[0] == "umount" && !tc.residual && !tc.busy { + if argv[0] == "umount" && !tt.residual && !tt.busy { fake.serial["/proc/self/mountinfo"] = []byte(base) } } fake.mu.Unlock() - clean, err := New("cocoon", nil, nil, false, false, "").DetachVolume(t.Context(), "sbx-1", sock, spec.Name, tc.mount) - if (err != nil) != tc.wantErr || clean != tc.wantClean { + clean, err := newTestEngine().DetachVolume(t.Context(), "sbx-1", sock, spec.Name, tt.mount) + if (err != nil) != tt.wantErr || clean != tt.wantClean { t.Fatalf("clean=%t err=%v", clean, err) } - if tc.mount == "" { + if tt.mount == "" { fake.mu.Lock() defer fake.mu.Unlock() if len(guestChanges(fake.execCalls)) != 0 { @@ -310,7 +272,7 @@ func TestLiveDetachCleanEvidenceRequiresWritableUnmountAndHostSuccess(t *testing func TestUnmountVolumeReportsCleanWritableUnmount(t *testing.T) { const entry = "2 1 8:32 / /data rw,relatime shared:4 - ext4 /dev/vdc " - for _, tc := range []struct { + for _, tt := range []struct { name, mountinfo string busy, wantClean bool wantErr bool @@ -324,23 +286,36 @@ func TestUnmountVolumeReportsCleanWritableUnmount(t *testing.T) { {name: "not mounted"}, {name: "busy", mountinfo: entry + "rw\n", busy: true, wantErr: true}, } { - t.Run(tc.name, func(t *testing.T) { + t.Run(tt.name, func(t *testing.T) { sock := sockPath(t) fake := serveFakeSilkd(t, sock) fake.mu.Lock() - fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + tc.mountinfo) - if tc.busy { + fake.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + tt.mountinfo) + if tt.busy { fake.execCode = 32 } fake.mu.Unlock() - clean, err := New("cocoon", nil, nil, false, false, "").UnmountVolume(t.Context(), sock, "/data") - if (err != nil) != tc.wantErr || clean != tc.wantClean { + clean, err := newTestEngine().UnmountVolume(t.Context(), sock, "/data") + if (err != nil) != tt.wantErr || clean != tt.wantClean { t.Fatalf("clean=%t err=%v", clean, err) } }) } } +func newLiveFake(t *testing.T, detachErr, mountinfo string) (string, *fakeSilkd, VolumeSpec) { + t.Helper() + sock := sockPath(t) + fake := serveFakeSilkd(t, sock) + configureVolumeDevices(fake) + spec := liveDiskCLI(t, detachErr) + fake.set(func(f *fakeSilkd) { + f.serial["/sys/block/vdc/dev"] = []byte("8:32\n") + f.serial["/proc/self/mountinfo"] = []byte(rootMountInfo + mountinfo) + }) + return sock, fake, spec +} + func liveDiskCLI(t *testing.T, detachErr string) VolumeSpec { t.Helper() dir := t.TempDir() diff --git a/sandboxd/engine/volume_test.go b/sandboxd/engine/volume_test.go index 8a9f71c46..d6230cae8 100644 --- a/sandboxd/engine/volume_test.go +++ b/sandboxd/engine/volume_test.go @@ -26,7 +26,7 @@ func TestDiskAttachArgsModeAndDirectIO(t *testing.T) { {"writable", "", true, "off"}, } { t.Run(tt.name, func(t *testing.T) { - e := New("cocoon", nil, nil, false, false, "") + e := newTestEngine() args, err := e.diskAttachArgs("sbx-1", VolumeSpec{ Name: "imagenet", Path: "/srv/datasets/imagenet.img", DirectIO: tt.directIO, RW: tt.rw, }) @@ -50,7 +50,7 @@ func TestDiskAttachArgsModeAndDirectIO(t *testing.T) { } func TestDiskAttachArgsRejectBadOptions(t *testing.T) { - _, err := New("cocoon", nil, nil, false, false, "").diskAttachArgs("sbx-1", VolumeSpec{DirectIO: "maybe"}) + _, err := newTestEngine().diskAttachArgs("sbx-1", VolumeSpec{DirectIO: "maybe"}) if err == nil || !strings.Contains(err.Error(), "on, off, or auto") { t.Errorf("got %v, want directio validation error", err) } @@ -73,7 +73,7 @@ func TestDiskAttachCopyPassesTheSourceAndReturnsTheGuestSerial(t *testing.T) { args := filepath.Join(t.TempDir(), "args") t.Setenv("COCOON_TEST_ARGS", args) fakeCocoon(t, "#!/bin/sh\necho \"$*\" > \"$COCOON_TEST_ARGS\"\necho '{\"vm\":\"sbx-1\",\"name\":\"snap-ab12cd34\",\"id\":\"d1\",\"format\":\"raw\"}'\n") - name, err := tt.attach(New("cocoon", nil, nil, false, false, "")) + name, err := tt.attach(newTestEngine()) if err != nil || name != "snap-ab12cd34" { t.Fatalf("attach = %q, %v, want the serial cocoon reported", name, err) } @@ -87,7 +87,7 @@ func TestDiskAttachCopyPassesTheSourceAndReturnsTheGuestSerial(t *testing.T) { func TestDiskAttachCopyRejectsAnUnnamedResult(t *testing.T) { for _, body := range []string{"not json", `{"vm":"sbx-1"}`} { fakeCocoon(t, "#!/bin/sh\necho '"+body+"'\n") - if name, err := New("cocoon", nil, nil, false, false, "").DiskAttachSnapshot(t.Context(), "sbx-1", "snap-1"); err == nil { + if name, err := newTestEngine().DiskAttachSnapshot(t.Context(), "sbx-1", "snap-1"); err == nil { t.Errorf("DiskAttachSnapshot over %q = %q, want an error", body, name) } } @@ -106,9 +106,7 @@ func TestMountVolumeUsesSysfsAndRequestedMode(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) configureVolumeDevices(fake) - if err := New("cocoon", nil, nil, false, false, "").MountVolume( - t.Context(), path, "imagenet", "/datasets/training", tt.rw, - ); err != nil { + if err := newTestEngine().MountVolume(t.Context(), path, "imagenet", "/datasets/training", tt.rw); err != nil { t.Fatalf("MountVolume: %v", err) } @@ -139,7 +137,7 @@ func TestMountVolumeUsesSysfsAndRequestedMode(t *testing.T) { func TestSettleGuestWaitsForInitAndTakesAnyExitAsFinished(t *testing.T) { for _, tt := range []struct { name string - prepare func(*fakeSilkd) + prepare fakeEdit wantErr bool }{ {"running", func(*fakeSilkd) {}, false}, @@ -149,10 +147,8 @@ func TestSettleGuestWaitsForInitAndTakesAnyExitAsFinished(t *testing.T) { t.Run(tt.name, func(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) - fake.mu.Lock() - tt.prepare(fake) - fake.mu.Unlock() - err := New("cocoon", nil, nil, false, false, "").SettleGuest(t.Context(), path) + fake.set(tt.prepare) + err := newTestEngine().SettleGuest(t.Context(), path) if (err != nil) != tt.wantErr { t.Fatalf("SettleGuest: %v, want error %t", err, tt.wantErr) } @@ -168,7 +164,7 @@ func TestSettleGuestWaitsForInitAndTakesAnyExitAsFinished(t *testing.T) { func TestUmountVolumeExecsUmount(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) - e := New("cocoon", nil, nil, false, false, "") + e := newTestEngine() if err := e.umountVolume(t.Context(), path, "/datasets/training"); err != nil { t.Fatalf("umountVolume: %v", err) } @@ -191,18 +187,18 @@ func TestUmountVolumeWaitsOutAFlushPastTheCallTimeout(t *testing.T) { fake := serveFakeSilkd(t, path) fake.execHook = func(argv []string) { if argv[0] == "umount" { - time.Sleep(VolumeCallTimeout + 250*time.Millisecond) + time.Sleep(syncTimeout + 250*time.Millisecond) } } - if err := New("cocoon", nil, nil, false, false, "").umountVolume(t.Context(), path, "/data"); err != nil { - t.Errorf("umount whose flush outlasts %s: %v", VolumeCallTimeout, err) + if err := newTestEngine().umountVolume(t.Context(), path, "/data"); err != nil { + t.Errorf("umount whose flush outlasts %s: %v", syncTimeout, err) } } func TestSyncGuestExecsPlainSync(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) - e := New("cocoon", nil, nil, false, false, "") + e := newTestEngine() if err := e.SyncGuest(t.Context(), path); err != nil { t.Fatalf("SyncGuest: %v", err) } @@ -223,12 +219,8 @@ func TestMountVolumeWaitsForDelayedSysfsSerial(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) configureVolumeDevices(fake) - fake.mu.Lock() - fake.readMisses["/sys/block/vdc/serial"] = 1 - fake.mu.Unlock() - if err := New("cocoon", nil, nil, false, false, "").MountVolume( - t.Context(), path, "imagenet", "/datasets/training", false, - ); err != nil { + fake.set(func(f *fakeSilkd) { f.readMisses["/sys/block/vdc/serial"] = 1 }) + if err := newTestEngine().MountVolume(t.Context(), path, "imagenet", "/datasets/training", false); err != nil { t.Fatalf("MountVolume: %v", err) } @@ -246,12 +238,8 @@ func TestMountVolumeWaitsForDelayedDevNode(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) configureVolumeDevices(fake) - fake.mu.Lock() - fake.statMisses["/dev/vdc"] = 3 - fake.mu.Unlock() - if err := New("cocoon", nil, nil, false, false, "").MountVolume( - t.Context(), path, "imagenet", "/datasets/training", false, - ); err != nil { + fake.set(func(f *fakeSilkd) { f.statMisses["/dev/vdc"] = 3 }) + if err := newTestEngine().MountVolume(t.Context(), path, "imagenet", "/datasets/training", false); err != nil { t.Fatalf("MountVolume: %v", err) } @@ -269,10 +257,8 @@ func TestMountVolumeFallsBackToTheProbeForAnotherFilesystem(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) configureVolumeDevices(fake) - fake.mu.Lock() - fake.execCode, fake.execFailAt = 32, 2 - fake.mu.Unlock() - if err := New("cocoon", nil, nil, false, false, "").MountVolume(t.Context(), path, "imagenet", "/datasets/training", false); err != nil { + fake.set(func(f *fakeSilkd) { f.execCode, f.execFailAt = 32, 2 }) + if err := newTestEngine().MountVolume(t.Context(), path, "imagenet", "/datasets/training", false); err != nil { t.Fatalf("MountVolume: %v", err) } fake.mu.Lock() @@ -291,10 +277,8 @@ func TestMountVolumeReadsEachOtherSerialOnce(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) configureVolumeDevices(fake) - fake.mu.Lock() - fake.blankReads["/sys/block/vdc/serial"] = 3 - fake.mu.Unlock() - if err := New("cocoon", nil, nil, false, false, "").MountVolume(t.Context(), path, "imagenet", "/datasets/training", false); err != nil { + fake.set(func(f *fakeSilkd) { f.blankReads["/sys/block/vdc/serial"] = 3 }) + if err := newTestEngine().MountVolume(t.Context(), path, "imagenet", "/datasets/training", false); err != nil { t.Fatalf("MountVolume: %v", err) } fake.mu.Lock() @@ -314,10 +298,8 @@ func TestMountVolumeRetriesUntilTheDeviceOpens(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) configureVolumeDevices(fake) - fake.mu.Lock() - fake.notReady = 2 - fake.mu.Unlock() - if err := New("cocoon", nil, nil, false, false, "").MountVolume(t.Context(), path, "imagenet", "/datasets/training", false); err != nil { + fake.set(func(f *fakeSilkd) { f.notReady = 2 }) + if err := newTestEngine().MountVolume(t.Context(), path, "imagenet", "/datasets/training", false); err != nil { t.Fatalf("MountVolume: %v", err) } fake.mu.Lock() @@ -334,12 +316,10 @@ func TestMountVolumeNotReadyUntilDeadlineReportsTheTimeout(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) configureVolumeDevices(fake) - fake.mu.Lock() - fake.notReady = 1 << 20 - fake.mu.Unlock() + fake.set(func(f *fakeSilkd) { f.notReady = 1 << 20 }) ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond) defer cancel() - err := New("cocoon", nil, nil, false, false, "").MountVolume(ctx, path, "imagenet", "/datasets/training", false) + err := newTestEngine().MountVolume(ctx, path, "imagenet", "/datasets/training", false) if !errors.Is(err, context.DeadlineExceeded) || !strings.Contains(err.Error(), "mount volume imagenet") { t.Fatalf("got %v, want the mount deadline", err) } @@ -348,7 +328,7 @@ func TestMountVolumeNotReadyUntilDeadlineReportsTheTimeout(t *testing.T) { func TestMountVolumeStopsAtFailedStage(t *testing.T) { for _, tt := range []struct { name string - prepare func(*fakeSilkd) + prepare fakeEdit wantErr string wantExec int }{ @@ -362,12 +342,8 @@ func TestMountVolumeStopsAtFailedStage(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) configureVolumeDevices(fake) - fake.mu.Lock() - tt.prepare(fake) - fake.mu.Unlock() - err := New("cocoon", nil, nil, false, false, "").MountVolume( - t.Context(), path, "imagenet", "/datasets/training", false, - ) + fake.set(tt.prepare) + err := newTestEngine().MountVolume(t.Context(), path, "imagenet", "/datasets/training", false) if err == nil || !strings.Contains(err.Error(), tt.wantErr) { t.Errorf("got %v, want %q failure", err, tt.wantErr) } @@ -389,9 +365,7 @@ func TestMountVolumeDeviceProbeIsBoundedAndCancelable(t *testing.T) { configureVolumeDevices(fake) ctx, cancel := context.WithCancel(t.Context()) cancel() - err := New("cocoon", nil, nil, false, false, "").MountVolume( - ctx, path, "missing", "/datasets/training", false, - ) + err := newTestEngine().MountVolume(ctx, path, "missing", "/datasets/training", false) if !errors.Is(err, context.Canceled) { t.Errorf("error = %v, want context cancellation", err) } @@ -401,12 +375,8 @@ func TestMountVolumeDevNodeNeverAppearsTimesOut(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) configureVolumeDevices(fake) - fake.mu.Lock() - delete(fake.stat, "/dev/vdc") - fake.mu.Unlock() - err := New("cocoon", nil, nil, false, false, "").MountVolume( - t.Context(), path, "imagenet", "/datasets/training", false, - ) + fake.set(func(f *fakeSilkd) { delete(f.stat, "/dev/vdc") }) + err := newTestEngine().MountVolume(t.Context(), path, "imagenet", "/datasets/training", false) if !errors.Is(err, context.DeadlineExceeded) || !strings.Contains(err.Error(), "wait for volume device imagenet") { t.Errorf("got %v, want a bounded wait-for-device timeout", err) } @@ -415,11 +385,11 @@ func TestMountVolumeDevNodeNeverAppearsTimesOut(t *testing.T) { func TestSilkdListAndReadAreBounded(t *testing.T) { path := sockPath(t) fake := serveFakeSilkd(t, path) - fake.mu.Lock() - fake.block = make([]wire.DirEntry, silkdListMax+1) - fake.serial["/sys/block/vda/serial"] = make([]byte, silkdReadMax+1) - fake.mu.Unlock() - e := New("cocoon", nil, nil, false, false, "") + fake.set(func(f *fakeSilkd) { + f.block = make([]wire.DirEntry, silkdListMax+1) + f.serial["/sys/block/vda/serial"] = make([]byte, silkdReadMax+1) + }) + e := newTestEngine() if _, err := e.silkdList(t.Context(), path, "/sys/block"); err == nil || !strings.Contains(err.Error(), "exceeds") { t.Errorf("list of %d entries: %v, want a bound error", silkdListMax+1, err) } diff --git a/sandboxd/outbound/claim_test.go b/sandboxd/outbound/claim_test.go index 8844200f5..db5406fd9 100644 --- a/sandboxd/outbound/claim_test.go +++ b/sandboxd/outbound/claim_test.go @@ -29,7 +29,7 @@ func TestRouteTakesTheClaimThenClassThenPoolUpstream(t *testing.T) { Pools: []config.PoolSpec{{PoolKey: testKey, EgressUpstreamEnv: "POOL_UPSTREAM"}}, EgressClasses: []config.EgressClass{{Name: "desk", EgressUpstreamEnv: "CLASS_UPSTREAM"}}, }, nil) - for _, tc := range []struct { + for _, tt := range []struct { name, class string env types.Env want string @@ -39,10 +39,10 @@ func TestRouteTakesTheClaimThenClassThenPoolUpstream(t *testing.T) { {"claim value", "desk", types.Env{"EGRESS_UPSTREAM": {Value: "socks5://127.0.0.1:3003", Guest: hostOnly}}, "socks5://127.0.0.1:3003"}, {"claim direct", "desk", types.Env{"EGRESS_UPSTREAM": {Value: "direct", Guest: hostOnly}}, ""}, } { - sb := &types.Sandbox{ID: "sb_route", Key: testKey, Tenant: "acme", EgressClass: tc.class} - sb.SetEnv(tc.env) - if got := newClaim(h, sb, nil, nil).Route(); got != tc.want { - t.Errorf("%s: route %q, want %q", tc.name, got, tc.want) + sb := &types.Sandbox{ID: "sb_route", Key: testKey, Tenant: "acme", EgressClass: tt.class} + sb.SetEnv(tt.env) + if got := newClaim(h, sb, nil, nil).Route(); got != tt.want { + t.Errorf("%s: route %q, want %q", tt.name, got, tt.want) } } } diff --git a/sandboxd/outbound/dial_test.go b/sandboxd/outbound/dial_test.go index d681243d0..711e574dd 100644 --- a/sandboxd/outbound/dial_test.go +++ b/sandboxd/outbound/dial_test.go @@ -46,7 +46,7 @@ func TestEgressDialerAdmitsOnlyNamedInternalPrefixes(t *testing.T) { check := func(addr string) error { return d.Control("tcp", addr, nil) } - for name, tc := range map[string]struct { + for name, tt := range map[string]struct { addr string allowed bool }{ @@ -63,12 +63,12 @@ func TestEgressDialerAdmitsOnlyNamedInternalPrefixes(t *testing.T) { "CGNAT metadata": {"100.100.100.200:80", false}, } { t.Run(name, func(t *testing.T) { - err := check(tc.addr) - if tc.allowed && err != nil { - t.Errorf("%s was blocked: %v", tc.addr, err) + err := check(tt.addr) + if tt.allowed && err != nil { + t.Errorf("%s was blocked: %v", tt.addr, err) } - if !tc.allowed && err == nil { - t.Errorf("%s was allowed; it must not be reachable from a guest", tc.addr) + if !tt.allowed && err == nil { + t.Errorf("%s was allowed; it must not be reachable from a guest", tt.addr) } }) } @@ -92,7 +92,7 @@ func TestEgressDialerWildcardAllowsEverything(t *testing.T) { func TestEgressDialerScopesInternalAllowToPorts(t *testing.T) { d := newDialer(verdictOf([]string{"10.8.0.1/32:18090,9000", "fdc8::/16:443", "10.9.0.0/16"})) - for name, tc := range map[string]struct { + for name, tt := range map[string]struct { addr string allowed bool }{ @@ -109,12 +109,12 @@ func TestEgressDialerScopesInternalAllowToPorts(t *testing.T) { "outside the prefix": {"10.8.0.2:18090", false}, } { t.Run(name, func(t *testing.T) { - err := d.Control("tcp", tc.addr, nil) - if tc.allowed && err != nil { - t.Errorf("%s was blocked: %v", tc.addr, err) + err := d.Control("tcp", tt.addr, nil) + if tt.allowed && err != nil { + t.Errorf("%s was blocked: %v", tt.addr, err) } - if !tc.allowed && (err == nil || !strings.Contains(err.Error(), "blocked internal address")) { - t.Errorf("%s: %v, want blocked internal address", tc.addr, err) + if !tt.allowed && (err == nil || !strings.Contains(err.Error(), "blocked internal address")) { + t.Errorf("%s: %v, want blocked internal address", tt.addr, err) } }) } @@ -122,7 +122,7 @@ func TestEgressDialerScopesInternalAllowToPorts(t *testing.T) { func TestEgressDialerBlocksTheNodesOwnAddresses(t *testing.T) { d := newDialer(verdictOf([]string{"93.184.216.34/32:80,443"}, netip.MustParseAddr("93.184.216.34"), netip.MustParseAddr("2606:2800:220:1:248:1893:25c8:1946"))) - for name, tc := range map[string]struct { + for name, tt := range map[string]struct { addr string allowed bool }{ @@ -136,12 +136,12 @@ func TestEgressDialerBlocksTheNodesOwnAddresses(t *testing.T) { "another public host": {"93.184.216.35:22", true}, } { t.Run(name, func(t *testing.T) { - err := d.Control("tcp", tc.addr, nil) - if tc.allowed && err != nil { - t.Errorf("%s was blocked: %v", tc.addr, err) + err := d.Control("tcp", tt.addr, nil) + if tt.allowed && err != nil { + t.Errorf("%s was blocked: %v", tt.addr, err) } - if !tc.allowed && (err == nil || !strings.Contains(err.Error(), "blocked internal address")) { - t.Errorf("%s: %v, want blocked internal address", tc.addr, err) + if !tt.allowed && (err == nil || !strings.Contains(err.Error(), "blocked internal address")) { + t.Errorf("%s: %v, want blocked internal address", tt.addr, err) } }) } diff --git a/sandboxd/outbound/doors_test.go b/sandboxd/outbound/doors_test.go index 48f304f63..5f52f9317 100644 --- a/sandboxd/outbound/doors_test.go +++ b/sandboxd/outbound/doors_test.go @@ -30,15 +30,8 @@ func TestEgressDoorHalfCloseReachesTheGuest(t *testing.T) { _, _ = io.WriteString(conn, "hello") _ = conn.Close() }() - pol := &egress.Policy{Allow: []egress.Rule{{Host: "127.0.0.1"}}} - h := newHost(t, &fakeEngine{}, poolConfig(testKey, pol), func(o *Options) { o.Dial = (&net.Dialer{}).DialContext }) - sb := outboundtest.VsockSandbox(t, "sb_halfclose", testKey) - if err := h.Arm(t.Context(), sb); err != nil { - t.Fatalf("arm egress: %v", err) - } - t.Cleanup(func() { h.Disarm(sb.ID, true) }) - - _, tunnel := outboundtest.ConnectDoor(t, engine.EgressSocketPath(sb.VsockSocket), origin.Addr().String()) + _, path := armDialingDoor(t, "sb_halfclose", &egress.Policy{Allow: []egress.Rule{{Host: "127.0.0.1"}}}) + _, tunnel := outboundtest.ConnectDoor(t, path, origin.Addr().String()) body, bodyErr := io.ReadAll(tunnel) if bodyErr != nil { t.Fatalf("tunnel did not end with EOF after the origin closed: %v", bodyErr) @@ -82,19 +75,13 @@ func TestEgressDoorCapsConcurrentConnections(t *testing.T) { _, _ = io.WriteString(w, "ok") })) t.Cleanup(origin.Close) - pol := &egress.Policy{Allow: []egress.Rule{{Host: outboundtest.Hostname(t, origin.URL)}}} - h := newHost(t, &fakeEngine{}, poolConfig(testKey, pol), func(o *Options) { o.Dial = (&net.Dialer{}).DialContext }) - sb := outboundtest.VsockSandbox(t, "sb_cap", testKey) - if err := h.Arm(t.Context(), sb); err != nil { - t.Fatalf("arm egress: %v", err) - } - t.Cleanup(func() { h.Disarm(sb.ID, true) }) - path := engine.EgressSocketPath(sb.VsockSocket) + h, path := armDialingDoor(t, "sb_cap", &egress.Policy{Allow: []egress.Rule{{Host: outboundtest.Hostname(t, origin.URL)}}}) idle := make([]net.Conn, 0, doorConns) + deadline := time.Now().Add(10 * time.Second) for len(idle) < doorConns { conn, err := net.Dial("unix", path) - if errors.Is(err, syscall.ECONNREFUSED) || errors.Is(err, syscall.EAGAIN) { + if (errors.Is(err, syscall.ECONNREFUSED) || errors.Is(err, syscall.EAGAIN)) && time.Now().Before(deadline) { time.Sleep(10 * time.Millisecond) continue } @@ -109,7 +96,7 @@ func TestEgressDoorCapsConcurrentConnections(t *testing.T) { } }) - waitHeld(t, h, sb.ID, doorConns) + waitHeld(t, h, "sb_cap", doorConns) client := outboundtest.EgressClient(path) client.Timeout = 500 * time.Millisecond @@ -158,6 +145,17 @@ func TestReachedSeesAGuestInTheBacklog(t *testing.T) { } } +func armDialingDoor(t *testing.T, id string, pol *egress.Policy) (*Host, string) { + t.Helper() + h := newHost(t, &fakeEngine{}, poolConfig(testKey, pol), func(o *Options) { o.Dial = (&net.Dialer{}).DialContext }) + sb := outboundtest.VsockSandbox(t, id, testKey) + if err := h.Arm(t.Context(), sb); err != nil { + t.Fatalf("arm egress: %v", err) + } + t.Cleanup(func() { h.Disarm(sb.ID, true) }) + return h, engine.EgressSocketPath(sb.VsockSocket) +} + func waitHeld(t *testing.T, h *Host, id string, want int) { t.Helper() h.mu.Lock() diff --git a/sandboxd/outbound/outbound_test.go b/sandboxd/outbound/outbound_test.go index 33e76c6e6..f51160870 100644 --- a/sandboxd/outbound/outbound_test.go +++ b/sandboxd/outbound/outbound_test.go @@ -185,7 +185,7 @@ func TestNetRouteFollowsTheLaneAndTheDoors(t *testing.T) { h := newHost(t, &fakeEngine{}, &config.Config{}, nil) armed := &types.Sandbox{ID: "sb_armed", Key: testKey} h.doors[armed.ID] = &doors{} - for _, tc := range []struct { + for _, tt := range []struct { lock bool sb *types.Sandbox want types.NetRoute @@ -195,9 +195,9 @@ func TestNetRouteFollowsTheLaneAndTheDoors(t *testing.T) { {false, &types.Sandbox{ID: "sb_nic", Key: egKey}, types.NetRouteDirect}, {true, &types.Sandbox{ID: "sb_locked", Key: egKey}, types.NetRouteNone}, } { - h.o.LockNIC = tc.lock - if got := h.Route(tc.sb); got != tc.want { - t.Errorf("%s (lock %v): route %q, want %q", tc.sb.ID, tc.lock, got, tc.want) + h.o.LockNIC = tt.lock + if got := h.Route(tt.sb); got != tt.want { + t.Errorf("%s (lock %v): route %q, want %q", tt.sb.ID, tt.lock, got, tt.want) } } } diff --git a/sandboxd/pool/archive_test.go b/sandboxd/pool/archive_test.go index 59064b8bd..aacb7f742 100644 --- a/sandboxd/pool/archive_test.go +++ b/sandboxd/pool/archive_test.go @@ -26,9 +26,7 @@ func TestArchivePublishWindowPinsCheckpoint(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, archivePool(3600)) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) stall := &stallingStore{Store: m.ckpts, published: make(chan string, 1), release: make(chan struct{})} m.ckpts = stall @@ -55,9 +53,7 @@ func TestArchivePublishesUnderTheSandboxLineage(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, archivePool(3600)) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) rec := &lineageStore{Store: m.ckpts} m.ckpts = rec if err := m.archive(t.Context(), sb); err != nil { @@ -72,9 +68,7 @@ func TestArchiveWakeRehibernateWindowAborts(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, archivePool(3600)) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) stall := &stallingStore{Store: m.ckpts, published: make(chan string, 1), release: make(chan struct{})} m.ckpts = stall @@ -85,9 +79,7 @@ func TestArchiveWakeRehibernateWindowAborts(t *testing.T) { if _, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("wake mid-publish: %v", err) } - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("re-hibernate mid-publish: %v", err) - } + hibernateClaim(t, m, sb) m.mu.Lock() newSnap := sb.HibernateSnap m.mu.Unlock() @@ -155,9 +147,7 @@ func TestReconcileReclaimsOrphanArchiveCk(t *testing.T) { claims := map[string]*types.Sandbox{ "sb_mine": {ID: "sb_mine", VMName: "sbx-gone-1", Key: testKey, Token: "tok"}, } - if err := newClaimStore(dataDir, false).save(claims); err != nil { - t.Fatalf("setup: %v", err) - } + seedClaims(t, dataDir, claims) m := newTestManagerAt(t, eng, dataDir) seedCk := func(id, sandboxID string) { t.Helper() @@ -341,10 +331,7 @@ func TestArchiveRetentionPurge(t *testing.T) { synctest.Test(t, func(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, archivePool(3600)) - sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("claim: %v", err) - } + sb := claimTenant(t, m, "acme") mustArchive(t, m, sb) ck := sb.ArchiveCk @@ -755,9 +742,7 @@ func TestArchivePersistFailureRollsBack(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, archivePool(3600)) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) snap, vm := sb.HibernateSnap, sb.VMName breakStore(t, m) @@ -859,9 +844,7 @@ func TestArchiveOnceSkipsInFlight(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, archivePool(3600)) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) backdate(m, sb, time.Hour) m.mu.Lock() @@ -884,9 +867,7 @@ func TestArchiveWakeEvictsRecLock(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, archivePool(3600)) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) base := lockCount(m) if err := m.archive(t.Context(), sb); err != nil { t.Fatalf("archive: %v", err) @@ -971,50 +952,11 @@ func (s *failingDeleteStore) Delete(_ context.Context, id string) error { return errors.New("delete failed") } -func breakStore(t *testing.T, m *Manager) { - t.Helper() - m.store.path = filepath.Join(t.TempDir(), "gone", "claims.json") -} - -func healStore(t *testing.T, m *Manager) { - t.Helper() - if err := os.MkdirAll(filepath.Dir(m.store.path), 0o750); err != nil { - t.Fatalf("heal store: %v", err) - } - waitFor(t, m.store.synced) -} - func archivedCount(m *Manager) int { _, g := m.Info() return g.Archived } -func archivePool(deleteAfter int) config.PoolSpec { - return config.PoolSpec{ - PoolKey: testKey, - Warm: 1, - IdleHibernateSeconds: 1, - ArchiveAfterSeconds: 2, - ArchiveDeleteAfterSeconds: deleteAfter, - } -} - -func mustArchive(t *testing.T, m *Manager, sb *types.Sandbox) { - t.Helper() - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } - if err := m.archive(t.Context(), sb); err != nil { - t.Fatalf("archive: %v", err) - } -} - -func ckExists(t *testing.T, m *Manager, ck string) bool { - t.Helper() - _, _, _, err := m.ckpts.Fetch(t.Context(), ck) //nolint:dogsled - return err == nil -} - func archiveCkMarked(m *Manager, ck string) bool { _, err := os.Stat(filepath.Join(m.dataDir, archiveDeleteDir, ck)) return err == nil diff --git a/sandboxd/pool/checkpoint.go b/sandboxd/pool/checkpoint.go index e5f6eab58..87c470382 100644 --- a/sandboxd/pool/checkpoint.go +++ b/sandboxd/pool/checkpoint.go @@ -47,7 +47,7 @@ func (m *Manager) Checkpoint(ctx context.Context, id string, cred Cred, name, te if err := m.diskErr(); err != nil { return types.Checkpoint{}, fmt.Errorf("checkpoint %s: %w", sb.ID, err) } - // See Hibernate: a started capture must finish even if the caller hangs up. + // A started capture must finish even if the caller hangs up. ctx = context.WithoutCancel(ctx) ckpt, _, err := m.publishCheckpoint(ctx, sb, store.CheckpointID(randHex(8)), name, tenant, false) if err != nil { diff --git a/sandboxd/pool/checkpoint_cleanup_test.go b/sandboxd/pool/checkpoint_cleanup_test.go index c94c71795..cd21b008d 100644 --- a/sandboxd/pool/checkpoint_cleanup_test.go +++ b/sandboxd/pool/checkpoint_cleanup_test.go @@ -25,7 +25,7 @@ func TestCheckpointCleanupWaitsForTheClaimWrite(t *testing.T) { t.Fatalf("sweep erased an unacknowledged retirement: %t, %v", ok, err) } } - for _, tc := range []struct { + for _, tt := range []struct { name string setup func(t *testing.T, m *Manager, c *failoverCluster, sb *types.Sandbox) string act func(t *testing.T, m *Manager, c *failoverCluster, sb *types.Sandbox) @@ -60,7 +60,7 @@ func TestCheckpointCleanupWaitsForTheClaimWrite(t *testing.T) { setup: func(t *testing.T, m *Manager, c *failoverCluster, sb *types.Sandbox) string { m.archives = &hookRegistry{Registry: c.reg, failPut: true} if err := m.replicate(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { - t.Fatal(err) + t.Fatalf("replicate with the registry down: %v", err) } m.archives = c.reg return sb.ReplicaPending @@ -79,7 +79,7 @@ func TestCheckpointCleanupWaitsForTheClaimWrite(t *testing.T) { previous := sb.ReplicaCk m.archives = &hookRegistry{Registry: c.reg, losePutReply: true} if err := m.replicate(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { - t.Fatal(err) + t.Fatalf("replicate with the put reply lost: %v", err) } m.archives = c.reg return previous @@ -96,13 +96,13 @@ func TestCheckpointCleanupWaitsForTheClaimWrite(t *testing.T) { }, } { for _, restart := range []bool{false, true} { - t.Run(fmt.Sprintf("%s/restart=%t", tc.name, restart), func(t *testing.T) { + t.Run(fmt.Sprintf("%s/restart=%t", tt.name, restart), func(t *testing.T) { c := newReplicaCluster(t) m := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, m, testKey) - ck := tc.setup(t, m, c, sb) + ck := tt.setup(t, m, c, sb) restoreWrites := failClaimWrites(t, m.store) - tc.act(t, m, c, sb) + tt.act(t, m, c, sb) m.retryArchiveDeletes(t.Context()) if !ckExists(t, m, ck) || !m.deleteMarked(ck) { t.Fatal("cleanup dropped the checkpoint or its delete marker before the claim write landed") @@ -111,7 +111,7 @@ func TestCheckpointCleanupWaitsForTheClaimWrite(t *testing.T) { if restart { m = c.restart(t, m) if err := m.Reconcile(t.Context()); err != nil { - t.Fatal(err) + t.Fatalf("reconcile: %v", err) } var ok bool if sb, ok = m.byID(sb.ID); !ok { @@ -119,7 +119,7 @@ func TestCheckpointCleanupWaitsForTheClaimWrite(t *testing.T) { } m.retireReplica(t.Context(), sb) } else if err := m.store.commit(m.store.mark()); err != nil { - t.Fatal(err) + t.Fatalf("commit the journal: %v", err) } other := mustClaim(t, m, testKey) m.mu.Lock() @@ -160,17 +160,15 @@ func TestFailoverDisabledExitKeepsReplicaCheckpoint(t *testing.T) { {"pending replica", func(t *testing.T, m *Manager, c *failoverCluster, sb *types.Sandbox) string { m.archives = &hookRegistry{Registry: c.reg, failPut: true} if err := m.replicate(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { - t.Fatal(err) + t.Fatalf("replicate with the registry down: %v", err) } return sb.ReplicaPending }}, {"archive intent", func(t *testing.T, m *Manager, c *failoverCluster, sb *types.Sandbox) string { - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatal(err) - } + hibernateClaim(t, m, sb) m.archives = &hookRegistry{Registry: c.reg, failPut: true} if err := m.archive(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { - t.Fatal(err) + t.Fatalf("archive with the registry down: %v", err) } return sb.ArchivePending }}, @@ -181,7 +179,7 @@ func TestFailoverDisabledExitKeepsReplicaCheckpoint(t *testing.T) { }{ {"release", func(t *testing.T, m *Manager, sb *types.Sandbox) { if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatal(err) + t.Fatalf("release: %v", err) } }}, {"expire", func(t *testing.T, m *Manager, sb *types.Sandbox) { @@ -205,10 +203,10 @@ func TestFailoverDisabledExitKeepsReplicaCheckpoint(t *testing.T) { } m, err := NewManager(t.Context(), cfg, old.eng, testSecrets(t), testBackends(t, cfg)) if err != nil { - t.Fatal(err) + t.Fatalf("manager: %v", err) } if err = m.Reconcile(t.Context()); err != nil { - t.Fatal(err) + t.Fatalf("reconcile: %v", err) } restored, ok := m.byID(sb.ID) if !ok || checkpointRefs(restored) != checkpointRefs(sb) { diff --git a/sandboxd/pool/checkpoint_heal_test.go b/sandboxd/pool/checkpoint_heal_test.go index d85cde2f4..45030a5be 100644 --- a/sandboxd/pool/checkpoint_heal_test.go +++ b/sandboxd/pool/checkpoint_heal_test.go @@ -16,9 +16,10 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/types" ) +const healID = "ck_00000000000000bb" + func TestClaimCheckpointNeverPullsOnMiss(t *testing.T) { - ckpt := types.Checkpoint{ID: "ck_00000000000000bb", Key: testKey, CreatedAt: time.Now()} - m, puller := newHealManager(t, ckpt, []string{"peer-a:7777"}) + m, puller := newHealManager(t) if _, err := m.ClaimCheckpoint(t.Context(), "ck_00000000000000cc", ClaimOptions{TTL: time.Hour}); !errors.Is(err, ErrUnknownCheckpoint) { t.Errorf("claim: %v, want ErrUnknownCheckpoint", err) @@ -29,34 +30,30 @@ func TestClaimCheckpointNeverPullsOnMiss(t *testing.T) { } func TestClaimCheckpointHealPullsOnce(t *testing.T) { - id := "ck_00000000000000bb" - ckpt := types.Checkpoint{ID: id, Key: testKey, CreatedAt: time.Now()} - m, puller := newHealManager(t, ckpt, []string{"peer-a:7777"}) + m, puller := newHealManager(t) - sb, err := m.ClaimCheckpointHeal(t.Context(), id, ClaimOptions{TTL: time.Hour, Metadata: types.Metadata{"role": "branch"}}) + sb, err := m.ClaimCheckpointHeal(t.Context(), healID, ClaimOptions{TTL: time.Hour, Metadata: types.Metadata{"role": "branch"}}) if err != nil { t.Fatalf("ClaimCheckpointHeal: %v", err) } - if sb.FromCheckpoint != id || sb.Metadata["role"] != "branch" { - t.Errorf("FromCheckpoint = %q, metadata %v, want %q and role=branch", sb.FromCheckpoint, sb.Metadata, id) + if sb.FromCheckpoint != healID || sb.Metadata["role"] != "branch" { + t.Errorf("FromCheckpoint = %q, metadata %v, want %q and role=branch", sb.FromCheckpoint, sb.Metadata, healID) } if got := puller.count(); got != 1 { t.Errorf("puller called %d times, want 1", got) } - if !m.HasCheckpoint(t.Context(), id) { - t.Errorf("HasCheckpoint(%s) = false, want true after heal", id) + if !m.HasCheckpoint(t.Context(), healID) { + t.Errorf("HasCheckpoint(%s) = false, want true after heal", healID) } } func TestClaimCheckpointAfterHealStaysLocal(t *testing.T) { - id := "ck_00000000000000bb" - ckpt := types.Checkpoint{ID: id, Key: testKey, CreatedAt: time.Now()} - m, puller := newHealManager(t, ckpt, []string{"peer-a:7777"}) + m, puller := newHealManager(t) - if _, err := m.ClaimCheckpointHeal(t.Context(), id, ClaimOptions{TTL: time.Hour}); err != nil { + if _, err := m.ClaimCheckpointHeal(t.Context(), healID, ClaimOptions{TTL: time.Hour}); err != nil { t.Fatalf("ClaimCheckpointHeal: %v", err) } - if _, err := m.ClaimCheckpoint(t.Context(), id, ClaimOptions{TTL: time.Hour}); err != nil { + if _, err := m.ClaimCheckpoint(t.Context(), healID, ClaimOptions{TTL: time.Hour}); err != nil { t.Fatalf("ClaimCheckpoint after heal: %v", err) } if got := puller.count(); got != 1 { @@ -66,9 +63,7 @@ func TestClaimCheckpointAfterHealStaysLocal(t *testing.T) { func TestClaimCheckpointHealDedupsConcurrentPulls(t *testing.T) { synctest.Test(t, func(t *testing.T) { - id := "ck_00000000000000bb" - ckpt := types.Checkpoint{ID: id, Key: testKey, CreatedAt: time.Now()} - m, puller := newHealManager(t, ckpt, []string{"peer-a:7777"}) + m, puller := newHealManager(t) puller.release = make(chan struct{}) var wg sync.WaitGroup @@ -76,7 +71,7 @@ func TestClaimCheckpointHealDedupsConcurrentPulls(t *testing.T) { errs := make([]error, 2) for i := range 2 { wg.Go(func() { - sbs[i], errs[i] = m.ClaimCheckpointHeal(t.Context(), id, ClaimOptions{TTL: time.Hour}) + sbs[i], errs[i] = m.ClaimCheckpointHeal(t.Context(), healID, ClaimOptions{TTL: time.Hour}) }) } waitFor(t, func() bool { return puller.count() >= 1 }) @@ -89,8 +84,8 @@ func TestClaimCheckpointHealDedupsConcurrentPulls(t *testing.T) { t.Errorf("goroutine %d: %v", i, err) continue } - if sbs[i] == nil || sbs[i].FromCheckpoint != id { - t.Errorf("goroutine %d: sandbox = %+v, want a valid branch of %s", i, sbs[i], id) + if sbs[i] == nil || sbs[i].FromCheckpoint != healID { + t.Errorf("goroutine %d: sandbox = %+v, want a valid branch of %s", i, sbs[i], healID) } } if got := puller.count(); got != 1 { @@ -101,19 +96,17 @@ func TestClaimCheckpointHealDedupsConcurrentPulls(t *testing.T) { func TestDeleteVetoesInFlightHeal(t *testing.T) { synctest.Test(t, func(t *testing.T) { - id := "ck_00000000000000bb" - ckpt := types.Checkpoint{ID: id, Key: testKey, CreatedAt: time.Now()} - m, puller := newHealManager(t, ckpt, []string{"peer-a:7777"}) + m, puller := newHealManager(t) puller.release = make(chan struct{}) healDone := make(chan error, 1) go func() { - _, err := m.ClaimCheckpointHeal(t.Context(), id, ClaimOptions{TTL: time.Hour}) + _, err := m.ClaimCheckpointHeal(t.Context(), healID, ClaimOptions{TTL: time.Hour}) healDone <- err }() waitFor(t, func() bool { return puller.count() >= 1 }) - if err := m.DeleteCheckpoint(t.Context(), id, "", DeleteLocal); !errors.Is(err, ErrUnknownCheckpoint) { + if err := m.DeleteCheckpoint(t.Context(), healID, "", DeleteLocal); !errors.Is(err, ErrUnknownCheckpoint) { t.Fatalf("delete while heal is staging: %v, want ErrUnknownCheckpoint (nothing local yet, and it must not block)", err) } @@ -121,7 +114,7 @@ func TestDeleteVetoesInFlightHeal(t *testing.T) { if err := <-healDone; !errors.Is(err, ErrUnknownCheckpoint) { t.Errorf("heal after a concurrent delete: %v, want ErrUnknownCheckpoint (vetoed, must not publish)", err) } - if m.HasCheckpoint(t.Context(), id) { + if m.HasCheckpoint(t.Context(), healID) { t.Error("checkpoint present after a delete vetoed the pending heal — resurrection") } }) @@ -129,27 +122,25 @@ func TestDeleteVetoesInFlightHeal(t *testing.T) { func TestDeleteOfAPublishedRecordVetoesInFlightHeal(t *testing.T) { synctest.Test(t, func(t *testing.T) { - id := "ck_00000000000000bb" - ckpt := types.Checkpoint{ID: id, Key: testKey, CreatedAt: time.Now()} - m, puller := newHealManager(t, ckpt, []string{"peer-a:7777"}) + m, puller := newHealManager(t) puller.release = make(chan struct{}) healDone := make(chan error, 1) go func() { - _, err := m.ClaimCheckpointHeal(t.Context(), id, ClaimOptions{TTL: time.Hour}) + _, err := m.ClaimCheckpointHeal(t.Context(), healID, ClaimOptions{TTL: time.Hour}) healDone <- err }() waitFor(t, func() bool { return puller.count() >= 1 }) - staging, err := m.ckpts.Stage(id) + staging, err := m.ckpts.Stage(healID) if err != nil { t.Fatalf("stage: %v", err) } - plantHealedRecord(t, staging, ckpt) - if err := m.ckpts.Publish(t.Context(), staging, id, ""); err != nil { + plantHealedRecord(t, staging, puller.ckpt) + if err := m.ckpts.Publish(t.Context(), staging, healID, ""); err != nil { t.Fatalf("publish: %v", err) } - if err := m.DeleteCheckpoint(t.Context(), id, "", DeleteLocal); err != nil { + if err := m.DeleteCheckpoint(t.Context(), healID, "", DeleteLocal); err != nil { t.Fatalf("delete a published record while a heal is staging: %v", err) } @@ -157,7 +148,7 @@ func TestDeleteOfAPublishedRecordVetoesInFlightHeal(t *testing.T) { if err := <-healDone; !errors.Is(err, ErrUnknownCheckpoint) { t.Errorf("heal after a delete: %v, want ErrUnknownCheckpoint", err) } - if m.HasCheckpoint(t.Context(), id) { + if m.HasCheckpoint(t.Context(), healID) { t.Error("a heal republished a checkpoint deleted with 204") } }) @@ -165,15 +156,13 @@ func TestDeleteOfAPublishedRecordVetoesInFlightHeal(t *testing.T) { func TestClaimCheckpointHealCtxCancelReturnsPromptly(t *testing.T) { synctest.Test(t, func(t *testing.T) { - id := "ck_00000000000000bb" - ckpt := types.Checkpoint{ID: id, Key: testKey, CreatedAt: time.Now()} - m, puller := newHealManager(t, ckpt, []string{"peer-a:7777"}) + m, puller := newHealManager(t) puller.release = make(chan struct{}) ctx, cancel := context.WithCancel(t.Context()) done := make(chan error, 1) go func() { - _, err := m.ClaimCheckpointHeal(ctx, id, ClaimOptions{TTL: time.Hour}) + _, err := m.ClaimCheckpointHeal(ctx, healID, ClaimOptions{TTL: time.Hour}) done <- err }() waitFor(t, func() bool { return puller.count() >= 1 }) @@ -189,7 +178,7 @@ func TestClaimCheckpointHealCtxCancelReturnsPromptly(t *testing.T) { } close(puller.release) - waitFor(t, func() bool { return m.HasCheckpoint(t.Context(), id) }) + waitFor(t, func() bool { return m.HasCheckpoint(t.Context(), healID) }) }) } @@ -221,31 +210,10 @@ func TestRecLockEvictionWaitsForAllHolders(t *testing.T) { } } -func TestRecLockEvictDeferredToLastHolder(t *testing.T) { - m := newTestManager(t, newFakeEngine()) - const id = "ck_00000000000000aa" - - l1 := m.recLock(id) - l2 := m.recLock(id) - if l2 != l1 { - t.Fatal("recLock diverged for the same id") - } - m.recDone(id) - if !hasRecLock(m, id) { - t.Fatal("entry evicted while an ordinary holder still references it") - } - m.recDone(id) - if hasRecLock(m, id) { - t.Error("recLocks entry leaked: a delete's deferred eviction was lost when a plain recDone dropped the last reference") - } -} - func TestFetchCheckpointNeverPulls(t *testing.T) { - id := "ck_00000000000000bb" - ckpt := types.Checkpoint{ID: id, Key: testKey, CreatedAt: time.Now()} - m, puller := newHealManager(t, ckpt, []string{"peer-a:7777"}) + m, puller := newHealManager(t) - if _, _, _, err := m.FetchCheckpoint(t.Context(), id); !errors.Is(err, ErrUnknownCheckpoint) { + if _, _, _, err := m.FetchCheckpoint(t.Context(), healID); !errors.Is(err, ErrUnknownCheckpoint) { t.Errorf("FetchCheckpoint: %v, want ErrUnknownCheckpoint", err) } if got := puller.count(); got != 0 { @@ -253,7 +221,7 @@ func TestFetchCheckpointNeverPulls(t *testing.T) { } } -func TestHasCheckpoint(t *testing.T) { +func TestHasCheckpointAnswersOnlyForABranchableRecord(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, archivePool(3600)) src := mustClaim(t, m, testKey) @@ -295,12 +263,10 @@ func TestClaimCheckpointQuotaDoesNotMaskMiss(t *testing.T) { } func TestClaimCheckpointHealQuotaBeforePull(t *testing.T) { - id := "ck_00000000000000bb" - ckpt := types.Checkpoint{ID: id, Key: testKey, CreatedAt: time.Now()} - m, puller := newHealManager(t, ckpt, []string{"peer-a:7777"}) + m, puller := newHealManager(t) m.draining = true - if _, err := m.ClaimCheckpointHeal(t.Context(), id, ClaimOptions{TTL: time.Hour}); !errors.Is(err, ErrQuota) { + if _, err := m.ClaimCheckpointHeal(t.Context(), healID, ClaimOptions{TTL: time.Hour}); !errors.Is(err, ErrQuota) { t.Errorf("heal while draining: %v, want ErrQuota", err) } if got := puller.count(); got != 0 { @@ -311,7 +277,8 @@ func TestClaimCheckpointHealQuotaBeforePull(t *testing.T) { func TestClaimCheckpointHealConcurrencyCapRejectsExtra(t *testing.T) { m := newTestManager(t, newFakeEngine()) m.healSem = make(chan struct{}, 1) - puller := &blockingPuller{started: make(chan struct{}), release: make(chan struct{})} + started := make(chan struct{}) + puller := &blockingPuller{markStarted: sync.OnceFunc(func() { close(started) }), release: make(chan struct{})} m.healer = peer.NewHealer(func(string) []string { return []string{"peer-a:7777"} }, puller) done := make(chan error, 1) @@ -319,7 +286,7 @@ func TestClaimCheckpointHealConcurrencyCapRejectsExtra(t *testing.T) { _, err := m.ClaimCheckpointHeal(t.Context(), "ck_00000000000000aa", ClaimOptions{TTL: time.Hour}) done <- err }() - <-puller.started + <-started if _, err := m.ClaimCheckpointHeal(t.Context(), "ck_00000000000000bb", ClaimOptions{TTL: time.Hour}); !errors.Is(err, ErrHealBusy) { t.Errorf("second heal while one is in flight: %v, want ErrHealBusy", err) @@ -333,8 +300,6 @@ func TestClaimCheckpointHealConcurrencyCapRejectsExtra(t *testing.T) { func TestValidateHealedCheckpoint(t *testing.T) { const id = "ck_00000000000000aa" valid := types.Checkpoint{ID: id, Key: testKey} - egress := testKey - egress.Net = types.NetEgress tests := []struct { name string plant func(t *testing.T, dir string) @@ -349,7 +314,7 @@ func TestValidateHealedCheckpoint(t *testing.T) { {"rejects an invalid key", func(t *testing.T, dir string) { plantHealedRecord(t, dir, types.Checkpoint{ID: id, Key: types.PoolKey{Net: "bogus"}}) }, false}, - {"rejects an egress key", func(t *testing.T, dir string) { plantHealedRecord(t, dir, types.Checkpoint{ID: id, Key: egress}) }, false}, + {"rejects an egress key", func(t *testing.T, dir string) { plantHealedRecord(t, dir, types.Checkpoint{ID: id, Key: egKey}) }, false}, {"rejects a contentless export", func(t *testing.T, dir string) { plantExportDir(t, dir, "sub"); plantMeta(t, dir, valid) }, false}, } for _, tt := range tests { @@ -422,14 +387,6 @@ func TestDeleteCheckpointSharedStoreSkipsBroadcast(t *testing.T) { } } -func newHealManager(t *testing.T, ckpt types.Checkpoint, addrs []string) (*Manager, *healPuller) { - t.Helper() - m := newTestManager(t, newFakeEngine()) - puller := &healPuller{ckpt: ckpt} - m.healer = peer.NewHealer(func(string) []string { return addrs }, puller) - return m, puller -} - type healPuller struct { mu sync.Mutex calls int @@ -454,17 +411,24 @@ func (p *healPuller) count() int { } type blockingPuller struct { - started chan struct{} - release chan struct{} - once sync.Once + markStarted func() + release chan struct{} } func (p *blockingPuller) Pull(_ context.Context, _, id, dst string) error { - p.once.Do(func() { close(p.started) }) + p.markStarted() <-p.release return writeHealedRecord(dst, types.Checkpoint{ID: id, Key: testKey, CreatedAt: time.Now()}) } +func newHealManager(t *testing.T) (*Manager, *healPuller) { + t.Helper() + m := newTestManager(t, newFakeEngine()) + puller := &healPuller{ckpt: types.Checkpoint{ID: healID, Key: testKey, CreatedAt: time.Now()}} + m.healer = peer.NewHealer(func(string) []string { return []string{"peer-a:7777"} }, puller) + return m, puller +} + func plantHealedRecord(t *testing.T, dir string, ckpt types.Checkpoint) { t.Helper() if ckpt.Key == (types.PoolKey{}) { diff --git a/sandboxd/pool/checkpoint_references_test.go b/sandboxd/pool/checkpoint_references_test.go index 3125c68a8..0c4d63073 100644 --- a/sandboxd/pool/checkpoint_references_test.go +++ b/sandboxd/pool/checkpoint_references_test.go @@ -22,7 +22,7 @@ const ( func TestClaimCheckpointReferencesOutliveUnacknowledgedWrites(t *testing.T) { recreate := func(s *claimStore) { s.set(&types.Sandbox{ID: "owner", ReplicaCk: "recreated"}) } - for _, tc := range []struct { + for _, tt := range []struct { name string fault int acked, next *types.Sandbox @@ -86,31 +86,31 @@ func TestClaimCheckpointReferencesOutliveUnacknowledgedWrites(t *testing.T) { afterCommit: []string{"other-replica"}, }, } { - t.Run(tc.name, func(t *testing.T) { - if tc.fault == failAfterRename && os.Geteuid() == 0 { + t.Run(tt.name, func(t *testing.T) { + if tt.fault == failAfterRename && os.Geteuid() == 0 { t.Skip("requires non-root: root can read the write-only directory") } dir := t.TempDir() - s := newClaimStore(dir, tc.fault == failAfterRename) - if tc.acked != nil { - if err := s.commit(s.set(tc.acked)); err != nil { - t.Fatal(err) + s := newClaimStore(dir, tt.fault == failAfterRename) + if tt.acked != nil { + if err := s.commit(s.set(tt.acked)); err != nil { + t.Fatalf("commit: %v", err) } } var snap claimSnapshot - if tc.startup != nil { + if tt.startup != nil { s = newClaimStore(dir, false) claims, err := s.load() if err != nil { - t.Fatal(err) + t.Fatalf("load: %v", err) } - tc.startup(claims) + tt.startup(claims) snap = s.reset(claims) } else { - snap = s.set(tc.next) + snap = s.set(tt.next) } var finish func() - switch tc.fault { + switch tt.fault { case failWrite: finish = failClaimWrites(t, s) if err := s.commit(snap); err == nil { @@ -118,7 +118,7 @@ func TestClaimCheckpointReferencesOutliveUnacknowledgedWrites(t *testing.T) { } case failAfterRename: if err := os.Chmod(dir, 0o300); err != nil { - t.Fatal(err) + t.Fatalf("Chmod: %v", err) } finish = func() { _ = os.Chmod(dir, 0o700) } t.Cleanup(finish) @@ -127,7 +127,7 @@ func TestClaimCheckpointReferencesOutliveUnacknowledgedWrites(t *testing.T) { } case captureWrite: if err := syscall.Mkfifo(s.path+".tmp", 0o600); err != nil { - t.Fatal(err) + t.Fatalf("Mkfifo: %v", err) } done := make(chan error, 1) go func() { done <- s.commit(snap) }() @@ -139,42 +139,42 @@ func TestClaimCheckpointReferencesOutliveUnacknowledgedWrites(t *testing.T) { finish = func() { r, err := os.Open(s.path + ".tmp") if err != nil { - t.Fatal(err) + t.Fatalf("open the blocked write: %v", err) } raw, err := io.ReadAll(r) _ = r.Close() if err = errors.Join(err, <-done); err != nil { - t.Fatal(err) + t.Fatalf("read the blocked write: %v", err) } if err = os.Remove(s.path); err != nil { - t.Fatal(err) + t.Fatalf("Remove: %v", err) } if err = os.WriteFile(s.path, raw, 0o600); err != nil { - t.Fatal(err) + t.Fatalf("WriteFile: %v", err) } } } - if tc.mutate != nil { - tc.mutate(s) + if tt.mutate != nil { + tt.mutate(s) } finish() loaded, err := newClaimStore(dir, false).load() if err != nil { - t.Fatal(err) + t.Fatalf("load journal: %v", err) } var onDisk []string for _, sb := range loaded { onDisk = appendRefs(onDisk, checkpointRefs(sb)) } slices.Sort(onDisk) - if pinned := slices.Sorted(maps.Keys(s.checkpointReferences())); !slices.Equal(onDisk, tc.onDisk) || !slices.Equal(pinned, tc.afterFault) { - t.Fatalf("after the fault: on disk %v, pinned %v; want %v, %v", onDisk, pinned, tc.onDisk, tc.afterFault) + if pinned := slices.Sorted(maps.Keys(s.checkpointReferences())); !slices.Equal(onDisk, tt.onDisk) || !slices.Equal(pinned, tt.afterFault) { + t.Fatalf("after the fault: on disk %v, pinned %v; want %v, %v", onDisk, pinned, tt.onDisk, tt.afterFault) } if err = s.commit(s.mark()); err != nil { - t.Fatal(err) + t.Fatalf("commit: %v", err) } - if pinned := slices.Sorted(maps.Keys(s.checkpointReferences())); !slices.Equal(pinned, tc.afterCommit) { - t.Fatalf("after the next commit: pinned %v, want %v", pinned, tc.afterCommit) + if pinned := slices.Sorted(maps.Keys(s.checkpointReferences())); !slices.Equal(pinned, tt.afterCommit) { + t.Fatalf("after the next commit: pinned %v, want %v", pinned, tt.afterCommit) } }) } @@ -183,7 +183,7 @@ func TestClaimCheckpointReferencesOutliveUnacknowledgedWrites(t *testing.T) { func TestClaimSharedCheckpointReferencesRetirePerOwner(t *testing.T) { s := newClaimStore(t.TempDir(), false) if err := s.commit(s.set(&types.Sandbox{ID: "first", ReplicaCk: "shared", ArchivePending: "shared"}, &types.Sandbox{ID: "second", ArchiveCk: "shared"})); err != nil { - t.Fatal(err) + t.Fatalf("commit: %v", err) } if err := s.commit(s.del("first")); err != nil || !s.referencesCheckpoint("shared") { t.Fatalf("one owner's retirement released another owner's checkpoint: %v", err) @@ -197,7 +197,7 @@ func failClaimWrites(t *testing.T, s *claimStore) func() { t.Helper() tmp := s.path + ".tmp" if err := os.Mkdir(tmp, 0o700); err != nil { - t.Fatal(err) + t.Fatalf("Mkdir: %v", err) } s.recommitting.Store(true) restore := sync.OnceFunc(func() { diff --git a/sandboxd/pool/checkpoint_test.go b/sandboxd/pool/checkpoint_test.go index e89f3a675..ad4f70249 100644 --- a/sandboxd/pool/checkpoint_test.go +++ b/sandboxd/pool/checkpoint_test.go @@ -2,7 +2,6 @@ package pool import ( "context" - "encoding/json/v2" "errors" "os" "path/filepath" @@ -68,11 +67,7 @@ func TestCheckpointExportLetsADialReachTheRunningSource(t *testing.T) { _, err := m.Checkpoint(context.WithoutCancel(t.Context()), src.ID, Cred{Token: src.Token}, "", "") done <- err }() - waitFor(t, func() bool { - eng.mu.Lock() - defer eng.mu.Unlock() - return len(eng.exports) == 1 - }) + waitFor(t, func() bool { return eng.exportCount() == 1 }) dialed := make(chan error, 1) go func() { _, _, release, err := m.WakeAgentSocket(t.Context(), src.ID, src.Token) @@ -138,7 +133,7 @@ func TestCheckpointBranchTakesItsOwnMetadata(t *testing.T) { } } -func TestCheckpointValidation(t *testing.T) { +func TestCheckpointRejectsABadTokenNameOrUnknownID(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) src := mustClaim(t, m, testKey) @@ -191,10 +186,7 @@ func TestRetentionSweepsExpiredCheckpoints(t *testing.T) { if err := os.MkdirAll(dir, 0o750); err != nil { t.Fatalf("setup: %v", err) } - meta, _ := json.Marshal(types.Checkpoint{ID: id, CreatedAt: time.Now().Add(-age)}) - if err := os.WriteFile(filepath.Join(dir, "meta.json"), meta, 0o600); err != nil { - t.Fatalf("setup: %v", err) - } + plantMeta(t, dir, types.Checkpoint{ID: id, CreatedAt: time.Now().Add(-age)}) } plant("ck_000000000000000a", 2*time.Hour) plant("ck_000000000000000b", time.Minute) @@ -228,11 +220,7 @@ func TestRunRefillsWhileTheRetentionSweepBlocks(t *testing.T) { }() <-stall.entered - waitFor(t, func() bool { - m.mu.Lock() - defer m.mu.Unlock() - return len(m.pools[testKey].warm) > 0 - }) + waitFor(t, func() bool { return warmLen(m, testKey) > 0 }) if got := eng.cloneCount(); got != 1 { t.Errorf("clones=%d, want one warm VM spawned while the store listing is stuck", got) } @@ -261,25 +249,15 @@ func TestRunRefillsWhileTheGenerationSweepBlocks(t *testing.T) { <-stall.entered mustClaim(t, m, testKey) - waitFor(t, func() bool { - m.mu.Lock() - defer m.mu.Unlock() - return len(m.pools[testKey].warm) > 0 - }) + waitFor(t, func() bool { return warmLen(m, testKey) > 0 }) }) } func TestCheckpointTenantIsolation(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - srcA, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("acme claim: %v", err) - } - srcB, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "beta"}) - if err != nil { - t.Fatalf("beta claim: %v", err) - } + srcA := claimTenant(t, m, "acme") + srcB := claimTenant(t, m, "beta") ckA, err := m.Checkpoint(t.Context(), srcA.ID, Cred{Token: srcA.Token}, "a-step", "acme") if err != nil { t.Fatalf("acme checkpoint: %v", err) diff --git a/sandboxd/pool/claims_test.go b/sandboxd/pool/claims_test.go index 8a6db5a1e..2d793a70b 100644 --- a/sandboxd/pool/claims_test.go +++ b/sandboxd/pool/claims_test.go @@ -77,7 +77,7 @@ func TestRecommitDrainsAChangeItWasTooBusyToSee(t *testing.T) { waitFor(t, func() bool { return !m.store.recommitting.Load() && !m.store.pending(m.store.mark()) }) } -func TestStoreRoundTrip(t *testing.T) { +func TestClaimStoreRoundTripsWithoutLeavingATempFile(t *testing.T) { dataset := []types.Volume{{Name: "dataset-a", Mount: "/datasets/a"}} claims := map[string]*types.Sandbox{ "sb_a": {ID: "sb_a", VMName: "sbx-1", Key: testKey, Token: "t1", Deadline: time.Now().Add(time.Minute).UTC(), ClaimedAt: time.Now().Add(-time.Minute).UTC(), ClaimRef: "ns/workload", Metadata: types.Metadata{"team": "a"}, OnExpire: types.ExpireArchive, Volumes: dataset, VolumesWant: dataset, VsockSocket: "/v/1"}, @@ -138,7 +138,7 @@ func TestClaimRowOmitsUnsetFlags(t *testing.T) { } } -func TestStoreLoadMissingFile(t *testing.T) { +func TestClaimStoreLoadsAMissingFileAsEmpty(t *testing.T) { got, err := newClaimStore(t.TempDir(), false).load() if err != nil { t.Fatalf("load: %v", err) @@ -148,7 +148,7 @@ func TestStoreLoadMissingFile(t *testing.T) { } } -func TestStoreLoadCorruptFile(t *testing.T) { +func TestClaimStoreRefusesACorruptFile(t *testing.T) { dir := t.TempDir() if err := os.WriteFile(filepath.Join(dir, "claims.json"), []byte("{oops"), 0o600); err != nil { t.Fatalf("setup: %v", err) diff --git a/sandboxd/pool/digest_test.go b/sandboxd/pool/digest_test.go index e68760a92..af8ce7e64 100644 --- a/sandboxd/pool/digest_test.go +++ b/sandboxd/pool/digest_test.go @@ -10,7 +10,7 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/types" ) -func TestClusterDigest(t *testing.T) { +func TestClusterDigestCoversSharedConfigAndOnlyKeyedSecrets(t *testing.T) { acme := []tenants.TenantSpec{{Name: "acme", Token: "t1"}} rotated := []tenants.TenantSpec{{Name: "acme", Token: "rotated"}} base := &config.Config{APIToken: "tok", PreviewSecret: "ps"} @@ -124,8 +124,8 @@ func TestClusterDigestCoversTheEgressLayerInAnyOrder(t *testing.T) { func tenantRecords(specs []tenants.TenantSpec) []tenants.TenantRecord { out := make([]tenants.TenantRecord, len(specs)) - for i, t := range specs { - out[i] = tenants.TenantRecord{Name: t.Name, TokenSHA256: tenants.TokenSHA256(t.Token), MaxClaims: t.MaxClaims} + for i, spec := range specs { + out[i] = tenants.TenantRecord{Name: spec.Name, TokenSHA256: tenants.TokenSHA256(spec.Token), MaxClaims: spec.MaxClaims} } return out } diff --git a/sandboxd/pool/disk_test.go b/sandboxd/pool/disk_test.go index 4def4c8c5..b64a52594 100644 --- a/sandboxd/pool/disk_test.go +++ b/sandboxd/pool/disk_test.go @@ -10,20 +10,12 @@ import ( ) func TestDiskUnderReserveRefusesClaimsAndWakesUntilItRecovers(t *testing.T) { - eng := newFakeEngine() - dir := t.TempDir() - cfg := &config.Config{DataDir: dir, DiskReserve: &config.DiskReserve{Free: "1000P"}, Pools: []config.PoolSpec{{PoolKey: testKey}}} - m, err := NewManager(t.Context(), cfg, eng, testSecrets(t), testBackends(t, cfg)) - if err != nil { - t.Fatalf("setup manager: %v", err) - } + m := diskReserveManager(t, "1000P") sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, sb) m.checkDisk(t.Context()) - if _, err := claimAny(t.Context(), m, testKey, time.Hour); !errors.Is(err, ErrQuota) { + if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{TTL: time.Hour}); !errors.Is(err, ErrQuota) { t.Errorf("claim under the reserve: %v, want ErrQuota", err) } if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); !errors.Is(err, ErrQuota) { @@ -33,23 +25,19 @@ func TestDiskUnderReserveRefusesClaimsAndWakesUntilItRecovers(t *testing.T) { t.Errorf("gauges %+v, want at capacity for the disk", g) } - next := &config.Config{DataDir: dir, DiskReserve: &config.DiskReserve{Free: "1K"}, Pools: []config.PoolSpec{{PoolKey: testKey}}} + next := &config.Config{DataDir: m.dataDir, DiskReserve: &config.DiskReserve{Free: "1K"}, Pools: []config.PoolSpec{{PoolKey: testKey}}} m.view.Store(newConfigView(next, testSecrets(t))) m.checkDisk(t.Context()) if err := m.Wake(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { t.Errorf("wake above the reserve: %v", err) } - if _, err := claimAny(t.Context(), m, testKey, time.Hour); err != nil { + if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{TTL: time.Hour}); err != nil { t.Errorf("claim above the reserve: %v", err) } } func TestDiskUnderReserveAdvertisesNoWarm(t *testing.T) { - cfg := &config.Config{DataDir: t.TempDir(), DiskReserve: &config.DiskReserve{Free: "1000P"}, Pools: []config.PoolSpec{{PoolKey: testKey}}} - m, err := NewManager(t.Context(), cfg, newFakeEngine(), testSecrets(t), testBackends(t, cfg)) - if err != nil { - t.Fatalf("setup manager: %v", err) - } + m := diskReserveManager(t, "1000P") p := m.pools[testKey] p.warm = append(p.warm, &types.Sandbox{VMName: "sbx-warm", Key: testKey, VsockSocket: "/vsock/warm"}) if n := m.WarmCounts()[p.hash]; n != 1 { @@ -62,12 +50,7 @@ func TestDiskUnderReserveAdvertisesNoWarm(t *testing.T) { } func TestDiskUnderReserveHoldsHibernatesAndCaptures(t *testing.T) { - dir := t.TempDir() - cfg := &config.Config{DataDir: dir, DiskReserve: &config.DiskReserve{Free: "1000P"}, Pools: []config.PoolSpec{{PoolKey: testKey}}} - m, err := NewManager(t.Context(), cfg, newFakeEngine(), testSecrets(t), testBackends(t, cfg)) - if err != nil { - t.Fatalf("setup manager: %v", err) - } + m := diskReserveManager(t, "1000P") sb := mustClaim(t, m, testKey) cred := Cred{Token: sb.Token} m.checkDisk(t.Context()) @@ -85,7 +68,7 @@ func TestDiskUnderReserveHoldsHibernatesAndCaptures(t *testing.T) { t.Errorf("promote under the reserve: %v, want ErrQuota", err) } - m.view.Store(newConfigView(&config.Config{DataDir: dir, DiskReserve: &config.DiskReserve{Free: "1K"}, Pools: []config.PoolSpec{{PoolKey: testKey}}}, testSecrets(t))) + m.view.Store(newConfigView(&config.Config{DataDir: m.dataDir, DiskReserve: &config.DiskReserve{Free: "1K"}, Pools: []config.PoolSpec{{PoolKey: testKey}}}, testSecrets(t))) m.checkDisk(t.Context()) if err := m.Hibernate(t.Context(), sb.ID, cred); err != nil { t.Errorf("hibernate above the reserve: %v", err) @@ -93,17 +76,22 @@ func TestDiskUnderReserveHoldsHibernatesAndCaptures(t *testing.T) { } func TestAFailedStatfsKeepsTheDiskLowState(t *testing.T) { - dir := t.TempDir() - cfg := &config.Config{DataDir: dir, DiskReserve: &config.DiskReserve{Free: "1000P"}, Pools: []config.PoolSpec{{PoolKey: testKey}}} - m, err := NewManager(t.Context(), cfg, newFakeEngine(), testSecrets(t), testBackends(t, cfg)) - if err != nil { - t.Fatalf("setup manager: %v", err) - } + m := diskReserveManager(t, "1000P") m.checkDisk(t.Context()) - unreadable := &config.Config{DataDir: dir, DiskReserve: &config.DiskReserve{Free: "1K", Dirs: []string{dir + "/missing"}}, Pools: []config.PoolSpec{{PoolKey: testKey}}} + unreadable := &config.Config{DataDir: m.dataDir, DiskReserve: &config.DiskReserve{Free: "1K", Dirs: []string{m.dataDir + "/missing"}}, Pools: []config.PoolSpec{{PoolKey: testKey}}} m.view.Store(newConfigView(unreadable, testSecrets(t))) m.checkDisk(t.Context()) - if _, err := claimAny(t.Context(), m, testKey, time.Hour); !errors.Is(err, ErrQuota) { + if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{TTL: time.Hour}); !errors.Is(err, ErrQuota) { t.Errorf("claim after a failed statfs: %v, want ErrQuota while the disk was last seen low", err) } } + +func diskReserveManager(t *testing.T, free string) *Manager { + t.Helper() + cfg := &config.Config{DataDir: t.TempDir(), DiskReserve: &config.DiskReserve{Free: free}, Pools: []config.PoolSpec{{PoolKey: testKey}}} + m, err := NewManager(t.Context(), cfg, newFakeEngine(), testSecrets(t), testBackends(t, cfg)) + if err != nil { + t.Fatalf("setup manager: %v", err) + } + return m +} diff --git a/sandboxd/pool/drain_test.go b/sandboxd/pool/drain_test.go index 3efbebe55..908e2cceb 100644 --- a/sandboxd/pool/drain_test.go +++ b/sandboxd/pool/drain_test.go @@ -12,7 +12,7 @@ func TestDrainRefusesClaimsTrimsWarmAndUncordonRefills(t *testing.T) { synctest.Test(t, func(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - seedGolden(t, m, "") + seedGolden(t, m, testKey, "") if err := m.SetPools(t.Context(), []config.PoolSpec{{PoolKey: testKey, Warm: 2}}); err != nil { t.Fatalf("SetPools: %v", err) } @@ -43,10 +43,10 @@ func TestDrainRefusesClaimsTrimsWarmAndUncordonRefills(t *testing.T) { m.Uncordon(t.Context()) waitFor(t, func() bool { - infos, g := m.Info() - return !g.Draining && len(infos) == 1 && infos[0].Warm == 2 + now, gauges := m.Info() + return !gauges.Draining && len(now) == 1 && now[0].Warm == 2 }) - if _, err := claimAny(t.Context(), m, testKey, 0); err != nil { + if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{}); err != nil { t.Fatalf("claim after uncordon: %v", err) } }) diff --git a/sandboxd/pool/egress_test.go b/sandboxd/pool/egress_test.go index 528eb2090..cebd64000 100644 --- a/sandboxd/pool/egress_test.go +++ b/sandboxd/pool/egress_test.go @@ -20,11 +20,6 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/usage" ) -var ( - egKey = types.PoolKey{Template: "rt:24.04", Net: types.NetEgress, Size: types.SizeSmall} - egPolicy = &egress.Policy{Allow: []egress.Rule{{Host: "example.com", Secret: "gh"}}} -) - func TestLaneVerdictReachesGuestOnColdBoots(t *testing.T) { tests := []struct { name string @@ -75,9 +70,7 @@ func TestGoldenStampGatesAdoptionOnLane(t *testing.T) { if p.goldenDir != "" { t.Error("adopted an egress-lane golden that never marked its guest; want rebuild") } - if err := os.WriteFile(g+goldenStampSuffix, []byte(m.goldenStamp(m.view.Load(), egKey, "")), 0o644); err != nil { - t.Fatalf("write stamp: %v", err) - } + seedGolden(t, m, egKey, "") m.adoptGolden(p, "") if p.goldenDir != g { t.Error("rejected an egress-lane golden whose stamp says the guest was marked") @@ -104,9 +97,9 @@ func TestEgressLaneWakeFailsClosed(t *testing.T) { {"hibernated", &types.Sandbox{ID: "sb_h", Key: egKey, VMName: "sbx-h", HibernateSnap: "sbx-hib-x"}}, {"archived", &types.Sandbox{ID: "sb_a", Key: egKey, ArchiveCk: "ck_x"}}, } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - if _, err := m.wakeResolved(t.Context(), tc.sb); err == nil { + for _, tt := range cases { + t.Run(tt.name, func(t *testing.T) { + if _, err := m.wakeResolved(t.Context(), tt.sb); err == nil { t.Fatal("egress-lane wake must fail closed, not resume unguarded") } }) @@ -134,7 +127,7 @@ func TestClaimRecordsItsPolicyLayer(t *testing.T) { if sb := mustClaim(t, m, testKey); sb.Layer != types.LayerPooled { t.Errorf("pooled claim layer %q, want %q", sb.Layer, types.LayerPooled) } - unpooled := types.PoolKey{Template: "promoted-name", Net: types.NetNone, Size: types.SizeSmall} + unpooled := tplKey("promoted-name") sb, err := m.ClaimProvision(t.Context(), unpooled, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) if err != nil { t.Fatalf("claim: %v", err) @@ -144,7 +137,7 @@ func TestClaimRecordsItsPolicyLayer(t *testing.T) { } } -func TestEffectivePolicyComposition(t *testing.T) { +func TestEffectivePolicyIntersectsPoolAndTenantPolicies(t *testing.T) { m := newTestManager(t, newFakeEngine()) both := &egress.Policy{Allow: []egress.Rule{{Host: "a.test"}, {Host: "b.test"}}} tenantOnly := &egress.Policy{Allow: []egress.Rule{{Host: "b.test"}, {Host: "c.test"}}} @@ -168,35 +161,35 @@ func TestEffectivePolicyComposition(t *testing.T) { {"root on a promoted template", "", false, false, nil, nil, "", "", false}, {"neither", "acme", false, false, nil, nil, "", "", false}, } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { + for _, tt := range cases { + t.Run(tt.name, func(t *testing.T) { m.pools = map[types.PoolKey]*pool{} - if tc.pooled { + if tt.pooled { m.pools[testKey] = newPool(testKey) - m.pools[testKey].removed = tc.removed + m.pools[testKey].removed = tt.removed } editEgress(t, m, func(cfg *config.Config) { - cfg.Pools = []config.PoolSpec{{PoolKey: testKey, Egress: tc.pool}} - if tc.tnPol != nil { - cfg.EgressClasses = []config.EgressClass{{Name: "desk", Egress: tc.tnPol}} + cfg.Pools = []config.PoolSpec{{PoolKey: testKey, Egress: tt.pool}} + if tt.tnPol != nil { + cfg.EgressClasses = []config.EgressClass{{Name: "desk", Egress: tt.tnPol}} } }) - sb := &types.Sandbox{Key: testKey, Tenant: tc.tenant} - if tc.tenant != "" { + sb := &types.Sandbox{Key: testKey, Tenant: tt.tenant} + if tt.tenant != "" { sb.EgressClass = "desk" } eval, ok := m.view.Load().out.Resolve(sb, m.claimPooled) - if ok != tc.wantArmed { - t.Fatalf("armed=%v, want %v", ok, tc.wantArmed) + if ok != tt.wantArmed { + t.Fatalf("armed=%v, want %v", ok, tt.wantArmed) } if !ok { return } - if _, d := eval.Eval(tc.allow, "GET", 443); d != egress.DecisionAllow { - t.Errorf("%s should allow", tc.allow) + if _, d := eval.Eval(tt.allow, "GET", 443); d != egress.DecisionAllow { + t.Errorf("%s should allow", tt.allow) } - if _, d := eval.Eval(tc.deny, "GET", 443); d != egress.DecisionDeny { - t.Errorf("%s should deny", tc.deny) + if _, d := eval.Eval(tt.deny, "GET", 443); d != egress.DecisionDeny { + t.Errorf("%s should deny", tt.deny) } }) } @@ -337,13 +330,7 @@ func TestQuarantineFailedRemoveStaysUnswept(t *testing.T) { func TestSetPoolsPreservesEgressPolicy(t *testing.T) { m := egressManager(t, newFakeEngine(), config.PoolSpec{PoolKey: egKey, Egress: egPolicy}) - gd := filepath.Join(m.goldensDir(), egKey.Hash()) - if err := os.MkdirAll(gd, 0o750); err != nil { - t.Fatalf("golden dir: %v", err) - } - if err := os.WriteFile(gd+goldenStampSuffix, []byte(m.goldenStamp(m.view.Load(), egKey, "")), 0o644); err != nil { - t.Fatalf("golden stamp: %v", err) - } + gd := seedGolden(t, m, egKey, "") m.mu.Lock() m.pools[egKey].goldenDir = gd m.mu.Unlock() @@ -519,11 +506,11 @@ func TestAPromotedTemplatesCloneInterceptsAsItsSourcePool(t *testing.T) { t.Fatalf("claim the template: %v", err) } conn, _ := outboundtest.ConnectDoor(t, engine.EgressSocketPath(clone.VsockSocket), "api.github.com:443") - tc := tls.Client(conn, &tls.Config{ServerName: "api.github.com", InsecureSkipVerify: true}) - if err = tc.Handshake(); err != nil { + tlsConn := tls.Client(conn, &tls.Config{ServerName: "api.github.com", InsecureSkipVerify: true}) + if err = tlsConn.Handshake(); err != nil { t.Fatalf("handshake through the door: %v", err) } - if issuer := tc.ConnectionState().PeerCertificates[0].Issuer.CommonName; !strings.HasSuffix(issuer, "node-test") { + if issuer := tlsConn.ConnectionState().PeerCertificates[0].Issuer.CommonName; !strings.HasSuffix(issuer, "node-test") { t.Errorf("leaf issued by %q, want the node intermediate: the clone must intercept as its source pool does", issuer) } } @@ -565,7 +552,7 @@ func TestAClaimWithoutEgressGetsNoDoorsWhateverItsPolicy(t *testing.T) { func TestARepromoteCarriesItsNewSourcePool(t *testing.T) { eng := newFakeEngine() eng.sockRoot = outboundtest.SockRoot(t) - other := types.PoolKey{Template: "py:3.12", Net: testKey.Net, Size: testKey.Size} + other := tplKey("py:3.12") m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: &egress.Policy{Allow: []egress.Rule{{Host: "a.test"}}}}, config.PoolSpec{PoolKey: other, Warm: 1, Egress: &egress.Policy{Allow: []egress.Rule{{Host: "b.test"}}}}) @@ -637,30 +624,6 @@ func servesPinned(t *testing.T, sb *types.Sandbox, pins []string) bool { return true } -func refillWarmVM(t *testing.T, m *Manager) *types.Sandbox { - t.Helper() - seedGolden(t, m, "") - m.mu.Lock() - m.adoptGolden(m.pools[testKey], "") - m.mu.Unlock() - m.refillOnce(t.Context()) - var warm *types.Sandbox - waitFor(t, func() bool { - m.mu.Lock() - defer m.mu.Unlock() - if p := m.pools[testKey]; p != nil && len(p.warm) > 0 { - warm = p.warm[0] - } - return warm != nil - }) - return warm -} - -func egressManager(t testing.TB, eng *fakeEngine, pools ...config.PoolSpec) *Manager { - t.Helper() - return egressManagerAt(t, eng, t.TempDir(), pools...) -} - func egressManagerAt(t testing.TB, eng *fakeEngine, dataDir string, pools ...config.PoolSpec) *Manager { t.Helper() t.Setenv("GH_TOKEN", "s3cr3t") @@ -696,14 +659,3 @@ func writeTestEgressCA(t testing.TB) *config.EgressCAConfig { } return ca } - -func withOutbound(t *testing.T, m *Manager, edit func(*outbound.Options)) { - t.Helper() - ca, err := outbound.LoadCA(m.cfg) - if err != nil { - t.Fatalf("load egress ca: %v", err) - } - o := m.outboundOptions(ca) - edit(&o) - m.out = outbound.New(o) -} diff --git a/sandboxd/pool/env.go b/sandboxd/pool/env.go index 5af30ac65..6314d4917 100644 --- a/sandboxd/pool/env.go +++ b/sandboxd/pool/env.go @@ -1,6 +1,7 @@ package pool import ( + "cmp" "context" "errors" "fmt" @@ -120,13 +121,7 @@ func (m *Manager) rewriteArchiveRow(ctx context.Context, sb *types.Sandbox, lock if !rowed { return nil } - var epoch int64 - var err error - if ck != "" { - epoch, err = m.replaceArchive(ctx, sb, ck) - } else { - epoch, err = m.replaceReplica(ctx, sb, replica) - } + epoch, err := m.putRow(ctx, sb, cmp.Or(ck, replica), ck == "", true) if err == nil { _, _ = m.commitIfLive(ctx, sb, func() { sb.ArchiveEpoch = epoch }) return nil diff --git a/sandboxd/pool/env_test.go b/sandboxd/pool/env_test.go index c4d57766b..ff1132509 100644 --- a/sandboxd/pool/env_test.go +++ b/sandboxd/pool/env_test.go @@ -22,8 +22,6 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/types" ) -var hostOnly = new(false) - func TestClaimEnvFeedsTheLiveProxysSecrets(t *testing.T) { origin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = io.WriteString(w, r.Header.Get("Authorization")) @@ -298,9 +296,7 @@ func TestAGuestEnvChangeNeedsARunningGuest(t *testing.T) { if err != nil { t.Fatalf("claim: %v", err) } - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, sb) if err := m.SetEnv(t.Context(), sb.ID, types.Env{"G": {Value: "2"}}, ""); !errors.Is(err, ErrPaused) { t.Errorf("guest change while hibernated: %v, want ErrPaused", err) } @@ -443,9 +439,7 @@ func TestPatchEnvMergesAndRepairsTheGuest(t *testing.T) { t.Errorf("another tenant's patch: %v, want ErrUnknownSandbox", err) } - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, sb) if err := patch(types.EnvPatch{"MODE": {Value: "b"}}); !errors.Is(err, ErrPaused) { t.Errorf("guest patch while hibernated: %v, want ErrPaused", err) } @@ -505,8 +499,7 @@ func TestAClaimWithAnUninterceptedInjectIsRefused(t *testing.T) { eng := newFakeEngine() eng.sockRoot = outboundtest.SockRoot(t) m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: interceptPolicy()}) - env := types.Env{"KEY": {Value: "k", Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"other.test"}, Header: "X-Api-Key"}}} - if _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: env}); !errors.Is(err, ErrBadEnv) { + if _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: injectEnv("k", "other.test")}); !errors.Is(err, ErrBadEnv) { t.Fatalf("claim: %v, want ErrBadEnv", err) } m.mu.Lock() @@ -523,8 +516,7 @@ func TestEachClaimInjectsItsOwnCredentialAndAHostOnlyChangeNeedsNoGuest(t *testi m := egressManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: interceptPolicy()}) claim := func(value string) *types.Sandbox { t.Helper() - env := types.Env{"KEY": {Value: value, Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.github.com"}, Header: "X-Api-Key"}}} - sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: env}) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: injectEnv(value, "api.github.com")}) if err != nil { t.Fatalf("claim: %v", err) } @@ -552,14 +544,14 @@ func TestInjectsSurviveARestartAndAReloadThatDropsTheInterceptStopsThem(t *testi eng := newFakeEngine() eng.sockRoot = outboundtest.SockRoot(t) dataDir := t.TempDir() - pool := config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com", Intercept: egress.InterceptAlways}}}} - m := egressManagerAt(t, eng, dataDir, pool) - env := types.Env{"KEY": {Value: "k", Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.github.com"}, Header: "X-Api-Key"}}} + spec := config.PoolSpec{PoolKey: testKey, Warm: 1, Egress: &egress.Policy{Allow: []egress.Rule{{Host: "api.github.com", Intercept: egress.InterceptAlways}}}} + m := egressManagerAt(t, eng, dataDir, spec) + env := injectEnv("k", "api.github.com") sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: env}) if err != nil { t.Fatalf("claim: %v", err) } - m2 := egressManagerAt(t, eng, dataDir, pool) + m2 := egressManagerAt(t, eng, dataDir, spec) if err = m2.Reconcile(t.Context()); err != nil { t.Fatalf("Reconcile: %v", err) } @@ -631,6 +623,10 @@ func TestALostInterceptRefusesOnlyAChangedInject(t *testing.T) { } } +func injectEnv(value string, hosts ...string) types.Env { + return types.Env{"KEY": {Value: value, Guest: hostOnly, Inject: &types.EnvInject{Hosts: hosts, Header: "X-Api-Key"}}} +} + func injectPatch(header string, hosts ...string) types.EnvPatch { return types.EnvPatch{"KEY": {Value: "k", Guest: hostOnly, Inject: &types.EnvInject{Hosts: hosts, Header: header}}} } diff --git a/sandboxd/pool/expiry_test.go b/sandboxd/pool/expiry_test.go index a0bbee2fa..bbac80d7e 100644 --- a/sandboxd/pool/expiry_test.go +++ b/sandboxd/pool/expiry_test.go @@ -50,9 +50,7 @@ func TestExpiryArchiveArchivesAHibernatedClaimWithoutPoolArchiving(t *testing.T) if err != nil { t.Fatalf("claim: %v", err) } - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) expire(m, sb) m.reapOnce(t.Context()) waitFor(t, func() bool { return archivedCount(m) == 1 }) @@ -172,9 +170,7 @@ func TestExpiryArchiveWakeBeforeTheExportGrantsAFreshLease(t *testing.T) { t.Fatalf("claim: %v", err) } expire(m, sb) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) if _, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("wake: %v", err) } @@ -258,16 +254,15 @@ func TestCheckpointBranchTakesItsOwnExpireAction(t *testing.T) { } func TestArchivableRefusesClaimsThatCannotHibernate(t *testing.T) { - egress := types.PoolKey{Template: testKey.Template, Net: types.NetEgress, Size: types.SizeSmall} for _, tt := range []struct { key types.PoolKey volumes bool action types.ExpireAction want error }{ - {egress, false, types.ExpireArchive, ErrNoEgressHibernate}, + {egKey, false, types.ExpireArchive, ErrNoEgressHibernate}, {testKey, true, types.ExpireArchive, ErrVolumeCapture}, - {egress, true, types.ExpireDestroy, nil}, + {egKey, true, types.ExpireDestroy, nil}, {testKey, false, types.ExpireArchive, nil}, } { if err := archivable(tt.key, tt.volumes, tt.action); !errors.Is(err, tt.want) { @@ -300,12 +295,6 @@ func TestArchiveOnExpireRefusesAVolumeClaimOnEveryPath(t *testing.T) { } } -func expire(m *Manager, sb *types.Sandbox) { - m.mu.Lock() - sb.Deadline = time.Now().Add(-time.Second) - m.mu.Unlock() -} - type failingPublishStore struct { store.Store fail atomic.Bool diff --git a/sandboxd/pool/failover_test.go b/sandboxd/pool/failover_test.go index 9bb3e593e..55abae069 100644 --- a/sandboxd/pool/failover_test.go +++ b/sandboxd/pool/failover_test.go @@ -21,8 +21,6 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/types" ) -const keepAnHour = 3600 - func TestFailoverArchiveWritesTheRow(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) @@ -50,9 +48,7 @@ func TestFailoverRowWriteFailureKeepsTheClaimHibernated(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) c.reg.SetDown(true) if err := a.archive(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { t.Fatalf("archive with the registry down: %v, want ErrArchiveStoreDown", err) @@ -75,9 +71,7 @@ func TestFailoverRowWriteFailureKeepsTheClaimHibernated(t *testing.T) { func TestFailoverEnvRotationReachesTheAdopter(t *testing.T) { c := newFailoverCluster(t) a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) - sb := mustClaim(t, a, testKey) - patchHostOnly(t, a, sb, "old") - mustArchive(t, a, sb) + sb := archivedWithEnv(t, a) if rowEnv(t, c, sb) != "old" { t.Fatalf("archive row env %q, want old", rowEnv(t, c, sb)) } @@ -95,10 +89,7 @@ func TestFailoverEnvRotationReachesTheAdopter(t *testing.T) { } c.expire("a") - adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token) - if err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") held, ok := b.byID(sb.ID) if !ok { t.Fatal("adopter has no claim") @@ -111,11 +102,9 @@ func TestFailoverEnvRotationReachesTheAdopter(t *testing.T) { func TestFailoverEnvRotationWithTheRegistryDownIsLoudAndARetryRepairs(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) - sb := mustClaim(t, a, testKey) - patchHostOnly(t, a, sb, "old") - mustArchive(t, a, sb) + sb := archivedWithEnv(t, a) - reg := &replaceFailRegistry{Registry: c.reg, fail: true} + reg := &hookRegistry{Registry: c.reg, failPut: true} a.archives = reg patch := types.EnvPatch{"GH_TOKEN": {Value: "new", Guest: hostOnly}} if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); !errors.Is(err, archives.ErrUnavailable) { @@ -124,7 +113,7 @@ func TestFailoverEnvRotationWithTheRegistryDownIsLoudAndARetryRepairs(t *testing if got := rowEnv(t, c, sb); got != "old" { t.Errorf("row env %q, want old until the write lands", got) } - reg.fail = false + reg.failPut = false if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); err != nil { t.Fatalf("resend: %v", err) } @@ -136,11 +125,9 @@ func TestFailoverEnvRotationWithTheRegistryDownIsLoudAndARetryRepairs(t *testing func TestFailoverEnvRotationSurvivesALostReply(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) - sb := mustClaim(t, a, testKey) - patchHostOnly(t, a, sb, "old") - mustArchive(t, a, sb) + sb := archivedWithEnv(t, a) - reg := &replaceFailRegistry{Registry: c.reg, loseReply: true} + reg := &hookRegistry{Registry: c.reg, losePutReply: true} a.archives = reg patch := types.EnvPatch{"GH_TOKEN": {Value: "new", Guest: hostOnly}} if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); !errors.Is(err, archives.ErrUnavailable) { @@ -162,13 +149,9 @@ func TestFailoverEnvRotationSurvivesALostReply(t *testing.T) { func TestFailoverEnvRotationOfAClaimAdoptedElsewhereDropsIt(t *testing.T) { c := newFailoverCluster(t) a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) - sb := mustClaim(t, a, testKey) - patchHostOnly(t, a, sb, "old") - mustArchive(t, a, sb) + sb := archivedWithEnv(t, a) c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") patch := types.EnvPatch{"GH_TOKEN": {Value: "new", Guest: hostOnly}} if err := a.PatchEnv(t.Context(), sb.ID, patch, ""); !errors.Is(err, ErrUnknownSandbox) { @@ -185,9 +168,7 @@ func TestFailoverEnvRotationOfAClaimAdoptedElsewhereDropsIt(t *testing.T) { func TestFailoverEnvRotationWaitsForATransitionInFlight(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) - sb := mustClaim(t, a, testKey) - patchHostOnly(t, a, sb, "old") - mustArchive(t, a, sb) + sb := archivedWithEnv(t, a) sb.Transition.Lock() done := make(chan error, 1) @@ -211,11 +192,11 @@ func TestFailoverEnvWriteOfARunningClaimLeavesTheRegistryAlone(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - reg := &replaceFailRegistry{Registry: c.reg, fail: true} + reg := &hookRegistry{Registry: c.reg, failPut: true} a.archives = reg patchHostOnly(t, a, sb, "new") - if reg.calls != 0 { - t.Errorf("%d registry replaces for a claim with no row, want 0", reg.calls) + if reg.replaces != 0 { + t.Errorf("%d registry replaces for a claim with no row, want 0", reg.replaces) } } @@ -266,9 +247,7 @@ func TestFailoverAdoptionMovesTheClaimAndTheOldOwnerLetsGo(t *testing.T) { ck := sb.ArchiveCk c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v, want adopted", adopted, err) - } + mustAdopt(t, b, sb, "adopt") if rec, _, _ := c.reg.Get(t.Context(), sb.ID); rec.Owner != "b" || rec.Epoch != 2 { t.Errorf("row %s@%d, want b@2", rec.Owner, rec.Epoch) } @@ -296,9 +275,7 @@ func TestFailoverNoAdoptionWhileTheOwnerIsAlive(t *testing.T) { sb := mustClaim(t, a, testKey) mustArchive(t, a, sb) - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { - t.Errorf("adopt from a live owner: %v %v, want refused", adopted, err) - } + assertNotAdopted(t, b, sb, "adopt from a live owner") if rec, _, _ := c.reg.Get(t.Context(), sb.ID); rec.Owner != "a" || rec.Epoch != 1 { t.Errorf("row %s@%d, want a@1", rec.Owner, rec.Epoch) } @@ -312,18 +289,14 @@ func TestFailoverADrainingNodeDoesNotAdopt(t *testing.T) { c.expire("a") b.Drain(t.Context()) - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { - t.Errorf("adopt on a draining node: %v %v, want refused", adopted, err) - } + assertNotAdopted(t, b, sb, "adopt on a draining node") if rec, _, _ := c.reg.Get(t.Context(), sb.ID); rec.Owner != "a" || rec.Epoch != 1 { t.Errorf("row %s@%d, want a@1 left for another node", rec.Owner, rec.Epoch) } b.mu.Lock() b.draining = false b.mu.Unlock() - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Errorf("adopt once the drain lifts: %v %v, want adopted", adopted, err) - } + mustAdopt(t, b, sb, "adopt once the drain lifts") } func TestFailoverAdoptionNeedsTheToken(t *testing.T) { @@ -387,9 +360,7 @@ func TestFailoverRestartDropsAClaimAdoptedElsewhere(t *testing.T) { sb := mustClaim(t, a, testKey) mustArchive(t, a, sb) c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") if _, _, _, err := b.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("adopter's wake: %v", err) } @@ -419,9 +390,7 @@ func TestFailoverRestartReclaimsAnInterruptedArchive(t *testing.T) { claims := map[string]*types.Sandbox{ "sb_x": {ID: "sb_x", VMName: "sbx-gone-1", Key: testKey, Token: "tok", HibernateSnap: "sbx-hib-x"}, } - if err := newClaimStore(dirA, false).save(claims); err != nil { - t.Fatalf("setup: %v", err) - } + seedClaims(t, dirA, claims) a := c.node(t, "a", dirA, keepAnHour) const ck = "ck_00000000000000aa" seedArchiveCk(t, a, ck, "sb_x") @@ -442,9 +411,7 @@ func TestFailoverRestartReclaimSparesACheckpointAdoptedMidReconcile(t *testing.T claims := map[string]*types.Sandbox{ "sb_x": {ID: "sb_x", VMName: "sbx-gone-1", Key: testKey, Token: "tok", HibernateSnap: "sbx-hib-x"}, } - if err := newClaimStore(dirA, false).save(claims); err != nil { - t.Fatalf("setup: %v", err) - } + seedClaims(t, dirA, claims) a := c.node(t, "a", dirA, keepAnHour) const ck = "ck_00000000000000cc" seedArchiveCk(t, a, ck, "sb_x") @@ -467,9 +434,7 @@ func TestFailoverRestartReclaimsOnlyTheCheckpointsItPublished(t *testing.T) { claims := map[string]*types.Sandbox{ "sb_x": {ID: "sb_x", VMName: "sbx-gone-1", Key: testKey, Token: "tok", HibernateSnap: "sbx-hib-x"}, } - if err := newClaimStore(dirA, false).save(claims); err != nil { - t.Fatalf("setup: %v", err) - } + seedClaims(t, dirA, claims) a := c.node(t, "a", dirA, keepAnHour) const own, foreign = "ck_00000000000000e1", "ck_00000000000000e2" seedArchiveCkBy(t, a, "a", own, "sb_x") @@ -489,9 +454,7 @@ func TestFailoverRestartDropsAClaimWhoseInterruptedArchiveWasAdopted(t *testing. c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) if err := a.FlushClaims(); err != nil { t.Fatalf("flush: %v", err) } @@ -511,7 +474,7 @@ func TestFailoverRestartDropsAClaimWhoseInterruptedArchiveWasAdopted(t *testing. if _, held := restarted.byID(sb.ID); held { t.Error("the restarted node kept a claim whose archive another node adopted") } - eng := a.eng.(*fakeEngine) + eng := fakeOf(a) eng.mu.Lock() snapDropped := slices.Contains(eng.snapRemoves, snap) eng.mu.Unlock() @@ -572,9 +535,7 @@ func TestFailoverReleaseAfterAdoptionKeepsTheCheckpoint(t *testing.T) { sb := mustClaim(t, a, testKey) mustArchive(t, a, sb) c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") if err := a.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); !errors.Is(err, ErrUnknownSandbox) { t.Errorf("old owner's release: %v, want ErrUnknownSandbox", err) } @@ -658,7 +619,7 @@ func TestFailoverAdoptionMidWakeFencesTheOwnerOut(t *testing.T) { if _, _, _, err := a.WakeAgentSocket(t.Context(), sb.ID, sb.Token); !errors.Is(err, ErrUnknownSandbox) { t.Fatalf("owner's wake after a mid-wake adoption: %v, want ErrUnknownSandbox", err) } - eng := a.eng.(*fakeEngine) + eng := fakeOf(a) eng.mu.Lock() built := eng.clones[len(eng.clones)-1] eng.mu.Unlock() @@ -763,9 +724,7 @@ func TestFailoverArchivePutsTheRowUnderTransition(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) wakeable := false a.archives = &hookRegistry{Registry: c.reg, beforePut: func() { if sb.Transition.TryLock() { @@ -808,27 +767,22 @@ func TestFailoverRestartAfterAConsumeWithoutCommitKeepsTheClaim(t *testing.T) { a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) mustArchive(t, a, sb) - epoch, err := c.reg.Take(t.Context(), sb.ID, 1, "a") - if err != nil { - t.Fatalf("take: %v", err) - } + epoch := c.take(t, sb.ID, "a") _, _ = a.commitIfLive(t.Context(), sb, func() { sb.ArchiveEpoch = epoch }) - if _, err = c.reg.Consume(t.Context(), sb.ID, epoch); err != nil { - t.Fatalf("consume: %v", err) - } - if err = a.FlushClaims(); err != nil { + c.consume(t, sb.ID, epoch) + if err := a.FlushClaims(); err != nil { t.Fatalf("flush: %v", err) } restarted := c.restart(t, a) - if err = restarted.Reconcile(t.Context()); err != nil { + if err := restarted.Reconcile(t.Context()); err != nil { t.Fatalf("reconcile: %v", err) } held, _ := restarted.byID(sb.ID) if rec, ok, _ := c.reg.Get(t.Context(), sb.ID); !ok || rec.Owner != "a" || rec.Consumed || held == nil || held.ArchiveEpoch != rec.Epoch { t.Fatalf("row %+v present %v, held %v, want the tombstone re-armed at the claim's epoch", rec, ok, held != nil) } - if _, _, _, err = restarted.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { + if _, _, _, err := restarted.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Errorf("wake after the restart: %v", err) } } @@ -843,9 +797,7 @@ func TestFailoverAnArchiveWokenElsewhereIsDropped(t *testing.T) { t.Fatalf("flush: %v", err) } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") if _, _, _, err := b.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("adopter's wake: %v", err) } @@ -876,9 +828,7 @@ func TestFailoverALeftoverDeleteMarkerSparesTheAdoptersCheckpoint(t *testing.T) t.Fatalf("mark: %v", err) } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") c.renew(t, "a") if _, _, _, err := a.WakeAgentSocket(t.Context(), sb.ID, sb.Token); !errors.Is(err, ErrUnknownSandbox) { t.Fatalf("old owner's wake: %v, want ErrUnknownSandbox", err) @@ -902,9 +852,7 @@ func TestFailoverRestartClearsTheMarkerOfAnAdoptedClaim(t *testing.T) { t.Fatalf("flush: %v", err) } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") c.renew(t, "a") restarted := c.restart(t, a) @@ -924,7 +872,7 @@ func TestFailoverARepeatedAdoptionOnOneNodeKeepsOneClaim(t *testing.T) { t.Fatalf("put tenant: %v", err) } } - sb, err := claimAnyAs(t, a, "acme") + sb, err := claimAny(t.Context(), a, testKey, ClaimOptions{Tenant: "acme"}) if err != nil { t.Fatalf("claim: %v", err) } @@ -940,9 +888,7 @@ func TestFailoverARepeatedAdoptionOnOneNodeKeepsOneClaim(t *testing.T) { }) } wg.Wait() - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("repeated adopt: %v %v, want adopted", adopted, err) - } + mustAdopt(t, b, sb, "repeated adopt") if n := b.TenantClaims()["acme"]; n != 1 { t.Errorf("tenant acme counts %d claims on b, want 1", n) } @@ -969,9 +915,7 @@ func TestFailoverRestartInsideAnAdoptersWakeDropsTheClaim(t *testing.T) { } ck := sb.ArchiveCk c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") b.ckpts = &failingDeleteStore{Store: b.ckpts, attempts: make(chan string, 8)} if _, _, _, err := b.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("adopter's wake: %v", err) @@ -1007,9 +951,7 @@ func TestFailoverRestartAfterAnAdopterReleasedDropsTheClaim(t *testing.T) { t.Fatalf("flush: %v", err) } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") b.ckpts = &failingDeleteStore{Store: b.ckpts, attempts: make(chan string, 8)} if err := b.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { t.Fatalf("adopter's release: %v", err) @@ -1035,9 +977,7 @@ func TestFailoverReleaseDuringAnArchiveLeavesNoRow(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) released := make(chan error, 1) a.archives = &hookRegistry{Registry: c.reg, beforePut: func() { go func() { released <- a.Release(t.Context(), sb.ID, Cred{Token: sb.Token}) }() @@ -1068,9 +1008,7 @@ func TestFailoverRestartDropsAClaimAdoptedWhileItsInterruptedArchiveSettles(t *t c := newFailoverCluster(t) a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) if err := a.FlushClaims(); err != nil { t.Fatalf("flush: %v", err) } @@ -1102,9 +1040,7 @@ func TestFailoverRestartKeepsAClaimWhoseInterruptedArchiveRowVanished(t *testing c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) if err := a.FlushClaims(); err != nil { t.Fatalf("flush: %v", err) } @@ -1166,9 +1102,7 @@ func TestFailoverSweepLeavesADepartedOwnersTombstoneUntilItReturns(t *testing.T) mustArchive(t, a, sb) ck := sb.ArchiveCk c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") healthy := b.ckpts b.ckpts = &failingDeleteStore{Store: healthy, attempts: make(chan string, 8)} if _, _, _, err := b.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { @@ -1260,9 +1194,7 @@ func TestFailoverATombstoneIsNotAdopted(t *testing.T) { counted := &countingRegistry{Registry: c.reg} d.archives = counted - if adopted, err := d.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { - t.Errorf("adopt of a tombstone: %v %v, want refused", adopted, err) - } + assertNotAdopted(t, d, sb, "adopt of a tombstone") if n := counted.takes.Load(); n != 0 { t.Errorf("adoption tried %d takes on a tombstone, want none", n) } @@ -1279,9 +1211,7 @@ func TestFailoverADropClearsTheMarkerBeforeTheCheckpointIsUnpinned(t *testing.T) t.Fatalf("mark: %v", err) } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") done := make(chan struct{}) var wg sync.WaitGroup wg.Go(func() { @@ -1349,18 +1279,14 @@ func TestFailoverARepeatedAdoptionOfAReleasedClaimIsRefused(t *testing.T) { sb := mustClaim(t, a, testKey) mustArchive(t, a, sb) c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") cur, _ := b.byID(sb.ID) ck := cur.ArchiveCk if err := b.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { t.Fatalf("release: %v", err) } - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { - t.Errorf("repeated adopt of a released claim: %v %v, want refused", adopted, err) - } + assertNotAdopted(t, b, sb, "repeated adopt of a released claim") if rec, ok, _ := c.reg.Get(t.Context(), sb.ID); ok { t.Errorf("row %+v outlived the released claim", rec) } @@ -1373,13 +1299,8 @@ func TestFailoverSweepDropsTheRowOfAnArchiveThatNeverCommitted(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } - a.archives = &hookRegistry{Registry: c.reg, losePutReply: true} - if err := a.archive(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { - t.Fatalf("archive whose put reply was lost: %v, want ErrArchiveStoreDown", err) - } + hibernateClaim(t, a, sb) + c.archiveLosingPutReply(t, a, sb) stale, _, _ := c.reg.Get(t.Context(), sb.ID) a.sweepOrphanArchives(t.Context()) @@ -1413,9 +1334,7 @@ func TestFailoverSweepRearmsATombstoneItsArchivedClaimHolds(t *testing.T) { t.Fatalf("row %+v, local epoch %d, want the tombstone re-armed at the claim's epoch", rec, epoch) } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Errorf("adopt of the re-armed archive: %v %v, want adopted", adopted, err) - } + mustAdopt(t, b, sb, "adopt of the re-armed archive") } func TestFailoverSweepLeavesADepartedOwnersWakeTombstone(t *testing.T) { @@ -1423,13 +1342,8 @@ func TestFailoverSweepLeavesADepartedOwnersWakeTombstone(t *testing.T) { a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) mustArchive(t, a, sb) - epoch, err := c.reg.Take(t.Context(), sb.ID, 1, "a") - if err != nil { - t.Fatalf("take: %v", err) - } - if _, err = c.reg.Consume(t.Context(), sb.ID, epoch); err != nil { - t.Fatalf("consume: %v", err) - } + epoch := c.take(t, sb.ID, "a") + c.consume(t, sb.ID, epoch) c.expire("a") b.sweepOrphanArchives(t.Context()) @@ -1443,15 +1357,10 @@ func TestFailoverRestartAfterAConsumeWhileAPeerSweptKeepsTheClaim(t *testing.T) a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) mustArchive(t, a, sb) - epoch, err := c.reg.Take(t.Context(), sb.ID, 1, "a") - if err != nil { - t.Fatalf("take: %v", err) - } + epoch := c.take(t, sb.ID, "a") _, _ = a.commitIfLive(t.Context(), sb, func() { sb.ArchiveEpoch = epoch }) - if _, err = c.reg.Consume(t.Context(), sb.ID, epoch); err != nil { - t.Fatalf("consume: %v", err) - } - if err = a.FlushClaims(); err != nil { + c.consume(t, sb.ID, epoch) + if err := a.FlushClaims(); err != nil { t.Fatalf("flush: %v", err) } c.expire("a") @@ -1459,13 +1368,13 @@ func TestFailoverRestartAfterAConsumeWhileAPeerSweptKeepsTheClaim(t *testing.T) c.renew(t, "a") restarted := c.restart(t, a) - if err = restarted.Reconcile(t.Context()); err != nil { + if err := restarted.Reconcile(t.Context()); err != nil { t.Fatalf("reconcile: %v", err) } if _, held := restarted.byID(sb.ID); !held { t.Fatal("the restart dropped a claim whose only copy a peer swept") } - if _, _, _, err = restarted.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { + if _, _, _, err := restarted.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Errorf("wake after the restart: %v", err) } } @@ -1474,9 +1383,7 @@ func TestFailoverArchiveSweepExportsNothingWhileTheRegistryIsDown(t *testing.T) c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) c.reg.SetDown(true) for range 3 { backdate(a, sb, time.Minute) @@ -1495,9 +1402,7 @@ func TestFailoverArchiveExportsNothingForARowAnotherNodeHolds(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) if _, err := c.reg.Put(t.Context(), archives.Record{ID: sb.ID, Owner: "b", Checkpoint: "ck_00000000000000ee", Claim: []byte(`{}`)}); err != nil { t.Fatalf("put: %v", err) } @@ -1518,9 +1423,7 @@ func TestFailoverAWakeKeepsItsFenceAgainstThisNodesOwnAdoption(t *testing.T) { mustArchive(t, a, sb) ck := sb.ArchiveCk c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") b.archives = &hookRegistry{Registry: c.reg, beforeConsume: func() { taken, _, _ := c.reg.Get(t.Context(), sb.ID) if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { @@ -1554,7 +1457,7 @@ func TestFailoverConcurrentAdoptionsOnOneNodeAllAnswerForIt(t *testing.T) { t.Fatalf("put tenant: %v", err) } } - sb, err := claimAnyAs(t, a, "acme") + sb, err := claimAny(t.Context(), a, testKey, ClaimOptions{Tenant: "acme"}) if err != nil { t.Fatalf("claim: %v", err) } @@ -1588,38 +1491,36 @@ func TestFailoverConcurrentAdoptionsOnOneNodeAllAnswerForIt(t *testing.T) { } func TestFailoverACoalescedAdoptionStillNeedsTheToken(t *testing.T) { - c := newFailoverCluster(t) - a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) - sb := mustClaim(t, a, testKey) - mustArchive(t, a, sb) - c.expire("a") - wrong := make(chan bool, 1) - b.archives = &hookRegistry{Registry: c.reg, beforeTake: func() { - go func() { - ok, _ := b.AdoptArchived(t.Context(), sb.ID, "wrong") - wrong <- ok - }() - select { - case ok := <-wrong: - wrong <- ok - case <-time.After(200 * time.Millisecond): + synctest.Test(t, func(t *testing.T) { + c := newFailoverCluster(t) + a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + mustArchive(t, a, sb) + c.expire("a") + wrong := make(chan bool, 1) + b.archives = &hookRegistry{Registry: c.reg, beforeTake: func() { + go func() { + ok, _ := b.AdoptArchived(t.Context(), sb.ID, "wrong") + wrong <- ok + }() + select { + case ok := <-wrong: + wrong <- ok + case <-time.After(200 * time.Millisecond): + } + }} + mustAdopt(t, b, sb, "adopt") + if <-wrong { + t.Error("an adoption with a wrong token answered for the claim") } - }} - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } - if <-wrong { - t.Error("an adoption with a wrong token answered for the claim") - } + }) } func TestFailoverArchivePersistsItsIntentBeforeThePut(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) var intent string a.archives = &hookRegistry{Registry: c.reg, beforePut: func() { stored, err := newClaimStore(a.dataDir, false).load() @@ -1645,17 +1546,13 @@ func TestFailoverRestartDropsAClaimWhoseInterruptedArchiveWasConsumedElsewhere(t c := newFailoverCluster(t) a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) a.mu.Lock() vm := sb.VMName a.mu.Unlock() ck := interruptArchive(t, a, sb, true) c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") if _, _, _, err := b.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("adopter's wake: %v", err) } @@ -1671,7 +1568,7 @@ func TestFailoverRestartDropsAClaimWhoseInterruptedArchiveWasConsumedElsewhere(t if _, held := restarted.byID(sb.ID); held { t.Error("the restarted node kept a claim another node adopted and woke") } - if eng := a.eng.(*fakeEngine); !eng.removed(vm) { + if eng := fakeOf(a); !eng.removed(vm) { t.Errorf("the dropped claim's VM %s was not swept", vm) } } @@ -1680,9 +1577,7 @@ func TestFailoverRestartKeepsAClaimWhoseArchivePutNeverCommitted(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) ck := interruptArchive(t, a, sb, false) restarted := c.restart(t, a) @@ -1702,9 +1597,7 @@ func TestFailoverRestartKeepsAnIntentWhoseCheckpointItCannotRead(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) ck := interruptArchive(t, a, sb, false) restarted := c.restart(t, a) @@ -1725,9 +1618,7 @@ func TestFailoverRestartAfterAnAbandonClearedItsIntentReclaimsTheExport(t *testi c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) ck := interruptArchive(t, a, sb, false) if !a.clearArchiveIntent(t.Context(), sb, ck) { t.Fatal("setup: clear intent") @@ -1746,9 +1637,7 @@ func TestFailoverAnAbandonKeepsTheExportWhileItsIntentIsDurable(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) ck := interruptArchive(t, a, sb, true) rec, _, _ := c.reg.Get(t.Context(), sb.ID) breakStore(t, a) @@ -1796,9 +1685,7 @@ func TestFailoverRestartDropsAClaimConsumedWhileItSettlesItsInterruptedArchive(t c := newFailoverCluster(t) a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) ck := interruptArchive(t, a, sb, true) restarted := c.restart(t, a) @@ -1829,9 +1716,7 @@ func TestFailoverRestartUndoesItsOwnInterruptedArchiveAndItsIntent(t *testing.T) c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) ck := interruptArchive(t, a, sb, true) restarted := c.restart(t, a) @@ -1851,13 +1736,8 @@ func TestFailoverReleaseAfterALostArchivePutReplyLeavesNothingAdoptable(t *testi c := newFailoverCluster(t) a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } - a.archives = &hookRegistry{Registry: c.reg, losePutReply: true} - if err := a.archive(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { - t.Fatalf("archive whose put reply was lost: %v, want ErrArchiveStoreDown", err) - } + hibernateClaim(t, a, sb) + c.archiveLosingPutReply(t, a, sb) if err := a.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { t.Fatalf("release: %v", err) } @@ -1871,13 +1751,8 @@ func TestFailoverTheNextArchiveSettlesALostPutReplyFirst(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } - a.archives = &hookRegistry{Registry: c.reg, losePutReply: true} - if err := a.archive(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { - t.Fatalf("archive whose put reply was lost: %v, want ErrArchiveStoreDown", err) - } + hibernateClaim(t, a, sb) + c.archiveLosingPutReply(t, a, sb) a.archives = c.reg if err := a.archive(t.Context(), sb); err != nil { t.Fatalf("next archive: %v", err) @@ -1893,13 +1768,8 @@ func TestFailoverSweepSettlesALostArchivePutReply(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } - a.archives = &hookRegistry{Registry: c.reg, losePutReply: true} - if err := a.archive(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { - t.Fatalf("archive whose put reply was lost: %v, want ErrArchiveStoreDown", err) - } + hibernateClaim(t, a, sb) + c.archiveLosingPutReply(t, a, sb) a.mu.Lock() pending := sb.ArchivePending a.mu.Unlock() @@ -1925,13 +1795,8 @@ func TestFailoverAWakeSettlesALostArchivePutReplyFirst(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } - a.archives = &hookRegistry{Registry: c.reg, losePutReply: true} - if err := a.archive(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { - t.Fatalf("archive whose put reply was lost: %v, want ErrArchiveStoreDown", err) - } + hibernateClaim(t, a, sb) + c.archiveLosingPutReply(t, a, sb) a.archives = c.reg if _, _, _, err := a.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("wake: %v", err) @@ -1954,13 +1819,8 @@ func TestFailoverAWakeWithoutARegistryKeepsItsPendingIntent(t *testing.T) { c := newFailoverCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } - a.archives = &hookRegistry{Registry: c.reg, losePutReply: true} - if err := a.archive(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { - t.Fatalf("archive whose put reply was lost: %v, want ErrArchiveStoreDown", err) - } + hibernateClaim(t, a, sb) + c.archiveLosingPutReply(t, a, sb) a.archives = nil if _, _, _, err := a.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("wake with failover off: %v", err) @@ -1974,20 +1834,13 @@ func TestFailoverAReleaseFindsItsUnsettledArchiveTakenElsewhere(t *testing.T) { c := newFailoverCluster(t) a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) a.mu.Lock() vm := sb.VMName a.mu.Unlock() - a.archives = &hookRegistry{Registry: c.reg, losePutReply: true} - if err := a.archive(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { - t.Fatalf("archive whose put reply was lost: %v, want ErrArchiveStoreDown", err) - } + c.archiveLosingPutReply(t, a, sb) c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") if _, _, _, err := b.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("adopter's wake: %v", err) } @@ -1996,7 +1849,7 @@ func TestFailoverAReleaseFindsItsUnsettledArchiveTakenElsewhere(t *testing.T) { if err := a.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); !errors.Is(err, ErrUnknownSandbox) { t.Fatalf("old owner's release: %v, want ErrUnknownSandbox", err) } - if _, held := a.byID(sb.ID); held || !a.eng.(*fakeEngine).removed(vm) { + if _, held := a.byID(sb.ID); held || !fakeOf(a).removed(vm) { t.Error("the old owner kept the claim or its VM") } if held, ok := b.byID(sb.ID); !ok || held.ArchiveCk != "" { @@ -2026,6 +1879,30 @@ func (c *failoverCluster) renew(t *testing.T, node string) { } } +func (c *failoverCluster) take(t *testing.T, id, owner string) int64 { + t.Helper() + epoch, err := c.reg.Take(t.Context(), id, 1, owner) + if err != nil { + t.Fatalf("take: %v", err) + } + return epoch +} + +func (c *failoverCluster) consume(t *testing.T, id string, epoch int64) { + t.Helper() + if _, err := c.reg.Consume(t.Context(), id, epoch); err != nil { + t.Fatalf("consume: %v", err) + } +} + +func (c *failoverCluster) archiveLosingPutReply(t *testing.T, m *Manager, sb *types.Sandbox) { + t.Helper() + m.archives = &hookRegistry{Registry: c.reg, losePutReply: true} + if err := m.archive(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { + t.Fatalf("archive whose put reply was lost: %v, want ErrArchiveStoreDown", err) + } +} + func (c *failoverCluster) node(t *testing.T, name, dataDir string, deleteAfter int) *Manager { t.Helper() m := c.nodeOn(t, newFakeEngine(), name, dataDir, deleteAfter) @@ -2035,7 +1912,7 @@ func (c *failoverCluster) node(t *testing.T, name, dataDir string, deleteAfter i func (c *failoverCluster) restart(t *testing.T, old *Manager) *Manager { t.Helper() - return c.nodeOn(t, old.eng.(*fakeEngine), old.nodeID, old.dataDir, keepAnHour) + return c.nodeOn(t, fakeOf(old), old.nodeID, old.dataDir, keepAnHour) } func (c *failoverCluster) nodeOn(t *testing.T, eng *fakeEngine, name, dataDir string, deleteAfter int) *Manager { @@ -2056,116 +1933,6 @@ func (c *failoverCluster) nodeOn(t *testing.T, eng *fakeEngine, name, dataDir st return m } -func assertReleasedForGood(t *testing.T, c *failoverCluster, sb *types.Sandbox, nodes ...*Manager) { - t.Helper() - if rec, ok, _ := c.reg.Get(t.Context(), sb.ID); ok && !rec.Consumed { - t.Errorf("a live row %+v outlived the release", rec) - } - c.expire("a") - for _, m := range nodes { - if adopted, err := m.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { - t.Errorf("node %s adopts the released claim: %v %v, want a 404", m.nodeID, adopted, err) - } - } -} - -func claimAnyAs(t *testing.T, m *Manager, tenant string) (*types.Sandbox, error) { - t.Helper() - sb, err := m.ClaimWarm(t.Context(), testKey, ClaimOptions{Tenant: tenant}) - if errors.Is(err, ErrNoWarm) { - return m.ClaimProvision(t.Context(), testKey, ClaimOptions{Tenant: tenant}) - } - return sb, err -} - -func interruptArchive(t *testing.T, m *Manager, sb *types.Sandbox, putCommitted bool) string { - t.Helper() - ck, _, err := m.publishCheckpoint(t.Context(), sb, store.CheckpointID(randHex(8)), "archive", sb.Tenant, true) - if err != nil { - t.Fatalf("publish: %v", err) - } - m.mu.Lock() - sb.ArchivePending = ck.ID - js := m.store.set(sb) - m.mu.Unlock() - if err = m.store.commit(js); err != nil { - t.Fatalf("commit intent: %v", err) - } - if putCommitted { - if _, err = m.putArchive(t.Context(), sb, ck.ID); err != nil { - t.Fatalf("put: %v", err) - } - } - return ck.ID -} - -func seedArchiveCk(t *testing.T, m *Manager, id, sandboxID string) { - t.Helper() - seedArchiveCkBy(t, m, m.nodeID, id, sandboxID) -} - -func seedArchiveCkBy(t *testing.T, m *Manager, node, id, sandboxID string) { - t.Helper() - staging, err := m.ckpts.Stage(id) - if err != nil { - t.Fatalf("stage: %v", err) - } - if err := os.MkdirAll(filepath.Join(staging, store.ExportDir), 0o750); err != nil { - t.Fatalf("mkdir: %v", err) - } - meta := `{"id":"` + id + `","sandbox_id":"` + sandboxID + `","node":"` + node + `","archive":true}` - if err := os.WriteFile(filepath.Join(staging, store.MetaFile), []byte(meta), 0o600); err != nil { - t.Fatalf("meta: %v", err) - } - if err := m.ckpts.Publish(t.Context(), staging, id, ""); err != nil { - t.Fatalf("publish: %v", err) - } -} - -func patchHostOnly(t *testing.T, m *Manager, sb *types.Sandbox, value string) { - t.Helper() - if err := m.PatchEnv(t.Context(), sb.ID, types.EnvPatch{"GH_TOKEN": {Value: value, Guest: hostOnly}}, ""); err != nil { - t.Fatalf("patch env: %v", err) - } -} - -func rowEnv(t *testing.T, c *failoverCluster, sb *types.Sandbox) string { - t.Helper() - return rowClaim(t, c, sb).Env["GH_TOKEN"].Value -} - -func rowClaim(t *testing.T, c *failoverCluster, sb *types.Sandbox) *types.Sandbox { - t.Helper() - rec, ok, err := c.reg.Get(t.Context(), sb.ID) - if err != nil || !ok { - t.Fatalf("row: %v %v", ok, err) - } - claim := &types.Sandbox{} - if err = json.Unmarshal(rec.Claim, claim); err != nil { - t.Fatalf("decode row claim: %v", err) - } - return claim -} - -type replaceFailRegistry struct { - archives.Registry - fail, loseReply bool - calls int -} - -func (r *replaceFailRegistry) Replace(ctx context.Context, rec archives.Record, expect int64) (int64, error) { - r.calls++ - if r.fail { - return 0, archives.ErrUnavailable - } - epoch, err := r.Registry.Replace(ctx, rec, expect) - if r.loseReply && err == nil { - r.loseReply = false - return 0, archives.ErrUnavailable - } - return epoch, err -} - type hookRegistry struct { archives.Registry beforePut func() @@ -2177,6 +1944,7 @@ type hookRegistry struct { loseTakeReply bool losePutReply bool failPut bool + replaces int } func (r *hookRegistry) Put(ctx context.Context, rec archives.Record) (int64, error) { @@ -2196,6 +1964,7 @@ func (r *hookRegistry) Put(ctx context.Context, rec archives.Record) (int64, err } func (r *hookRegistry) Replace(ctx context.Context, rec archives.Record, expect int64) (int64, error) { + r.replaces++ if r.failPut { return 0, archives.ErrUnavailable } @@ -2299,3 +2068,93 @@ func (r *countingRegistry) Sweepable(ctx context.Context, now time.Time, owner s r.sweepable.Add(1) return r.Registry.Sweepable(ctx, now, owner) } + +func assertReleasedForGood(t *testing.T, c *failoverCluster, sb *types.Sandbox, nodes ...*Manager) { + t.Helper() + if rec, ok, _ := c.reg.Get(t.Context(), sb.ID); ok && !rec.Consumed { + t.Errorf("a live row %+v outlived the release", rec) + } + c.expire("a") + for _, m := range nodes { + if adopted, err := m.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { + t.Errorf("node %s adopts the released claim: %v %v, want a 404", m.nodeID, adopted, err) + } + } +} + +func interruptArchive(t *testing.T, m *Manager, sb *types.Sandbox, putCommitted bool) string { + t.Helper() + ck, _, err := m.publishCheckpoint(t.Context(), sb, store.CheckpointID(randHex(8)), "archive", sb.Tenant, true) + if err != nil { + t.Fatalf("publish: %v", err) + } + m.mu.Lock() + sb.ArchivePending = ck.ID + js := m.store.set(sb) + m.mu.Unlock() + if err = m.store.commit(js); err != nil { + t.Fatalf("commit intent: %v", err) + } + if putCommitted { + if _, err = m.putArchive(t.Context(), sb, ck.ID); err != nil { + t.Fatalf("put: %v", err) + } + } + return ck.ID +} + +func seedArchiveCk(t *testing.T, m *Manager, id, sandboxID string) { + t.Helper() + seedArchiveCkBy(t, m, m.nodeID, id, sandboxID) +} + +func seedArchiveCkBy(t *testing.T, m *Manager, node, id, sandboxID string) { + t.Helper() + staging, err := m.ckpts.Stage(id) + if err != nil { + t.Fatalf("stage: %v", err) + } + if err := os.MkdirAll(filepath.Join(staging, store.ExportDir), 0o750); err != nil { + t.Fatalf("mkdir: %v", err) + } + meta := `{"id":"` + id + `","sandbox_id":"` + sandboxID + `","node":"` + node + `","archive":true}` + if err := os.WriteFile(filepath.Join(staging, store.MetaFile), []byte(meta), 0o600); err != nil { + t.Fatalf("meta: %v", err) + } + if err := m.ckpts.Publish(t.Context(), staging, id, ""); err != nil { + t.Fatalf("publish: %v", err) + } +} + +func archivedWithEnv(t *testing.T, m *Manager) *types.Sandbox { + t.Helper() + sb := mustClaim(t, m, testKey) + patchHostOnly(t, m, sb, "old") + mustArchive(t, m, sb) + return sb +} + +func patchHostOnly(t *testing.T, m *Manager, sb *types.Sandbox, value string) { + t.Helper() + if err := m.PatchEnv(t.Context(), sb.ID, types.EnvPatch{"GH_TOKEN": {Value: value, Guest: hostOnly}}, ""); err != nil { + t.Fatalf("patch env: %v", err) + } +} + +func rowEnv(t *testing.T, c *failoverCluster, sb *types.Sandbox) string { + t.Helper() + return rowClaim(t, c, sb).Env["GH_TOKEN"].Value +} + +func rowClaim(t *testing.T, c *failoverCluster, sb *types.Sandbox) *types.Sandbox { + t.Helper() + rec, ok, err := c.reg.Get(t.Context(), sb.ID) + if err != nil || !ok { + t.Fatalf("row: %v %v", ok, err) + } + claim := &types.Sandbox{} + if err = json.Unmarshal(rec.Claim, claim); err != nil { + t.Fatalf("decode row claim: %v", err) + } + return claim +} diff --git a/sandboxd/pool/fork.go b/sandboxd/pool/fork.go index 5197d70c4..defcf4edf 100644 --- a/sandboxd/pool/fork.go +++ b/sandboxd/pool/fork.go @@ -31,7 +31,7 @@ func (m *Manager) Fork(ctx context.Context, id string, cred Cred, count int, ttl if err := m.overQuota(count, sb.Tenant); err != nil { return nil, err } - // See Hibernate: a started fork must finish even if the caller hangs up. + // A started fork must finish even if the caller hangs up. ctx = context.WithoutCancel(ctx) children, inherited, err := m.forkClones(ctx, sb, count) diff --git a/sandboxd/pool/fork_test.go b/sandboxd/pool/fork_test.go index fe79a8707..8e5785d04 100644 --- a/sandboxd/pool/fork_test.go +++ b/sandboxd/pool/fork_test.go @@ -24,11 +24,7 @@ func TestForkRacingHibernateUsesPrivateSource(t *testing.T) { t.Errorf("Hibernate: %v", err) } }) - waitFor(t, func() bool { - eng.mu.Lock() - defer eng.mu.Unlock() - return len(eng.hibernates) == 1 - }) + waitFor(t, func() bool { return eng.hibernateCount() == 1 }) var children []string wg.Go(func() { sbs, err := m.Fork(t.Context(), parent.ID, Cred{Token: parent.Token}, 2, time.Hour, "", "") @@ -57,7 +53,7 @@ func TestForkRacingHibernateUsesPrivateSource(t *testing.T) { } } -func TestForkFromRunning(t *testing.T) { +func TestForkOfARunningClaimClonesOneTransientSnapshot(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) parent := mustClaim(t, m, testKey) @@ -95,7 +91,7 @@ func TestForkFromRunning(t *testing.T) { t.Errorf("child deadline off requested TTL by %v", d) } } - if got, _ := newClaimStore(m.dataDir, false).load(); len(got) != 4 { + if got := journaled(t, m); len(got) != 4 { t.Errorf("persisted %d claims, want parent + 3 children", len(got)) } leftovers, _ := filepath.Glob(filepath.Join(m.dataDir, "fork-*")) @@ -108,9 +104,7 @@ func TestForkFromHibernatedUsesWakeImage(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) parent := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), parent.ID, Cred{Token: parent.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, parent) children, err := m.Fork(t.Context(), parent.ID, Cred{Token: parent.Token}, 2, 0, "", "") if err != nil { @@ -135,7 +129,7 @@ func TestForkFromHibernatedUsesWakeImage(t *testing.T) { } } -func TestForkCountValidation(t *testing.T) { +func TestForkRefusesAnInvalidCountBeforeASnapshot(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) parent := mustClaim(t, m, testKey) @@ -149,7 +143,7 @@ func TestForkCountValidation(t *testing.T) { } } -func TestForkUnknownSandbox(t *testing.T) { +func TestForkWithAWrongTokenIsAnUnknownSandbox(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) parent := mustClaim(t, m, testKey) @@ -161,7 +155,7 @@ func TestForkUnknownSandbox(t *testing.T) { func TestForkAllOrNothing(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - parent, err := claimAny(t.Context(), m, testKey, 0) + parent, err := claimAny(t.Context(), m, testKey, ClaimOptions{}) if err != nil { t.Fatalf("Claim: %v", err) } @@ -174,7 +168,7 @@ func TestForkAllOrNothing(t *testing.T) { if _, g := m.Info(); g.Claimed != 1 { t.Errorf("claimed=%d, want only the parent", g.Claimed) } - if got, _ := newClaimStore(m.dataDir, false).load(); len(got) != 1 { + if got := journaled(t, m); len(got) != 1 { t.Errorf("persisted %d claims, want only the parent", len(got)) } eng.mu.Lock() @@ -189,9 +183,7 @@ func TestReconcileSweepsOrphanSnapshots(t *testing.T) { dataDir := t.TempDir() m := newTestManagerAt(t, eng, dataDir) parent := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), parent.ID, Cred{Token: parent.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, parent) referenced := eng.hibernates[0] eng.mu.Lock() eng.snapshots = []string{ @@ -273,10 +265,7 @@ func TestForkChildrenInheritTenantAndQuota(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) setTenantCaps(t, m, map[string]int{"acme": 3}) - parent, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("claim: %v", err) - } + parent := claimTenant(t, m, "acme") children, err := m.Fork(t.Context(), parent.ID, Cred{Token: parent.Token}, 2, 0, "", "") if err != nil { diff --git a/sandboxd/pool/hibernate.go b/sandboxd/pool/hibernate.go index 97507a2f3..58748368d 100644 --- a/sandboxd/pool/hibernate.go +++ b/sandboxd/pool/hibernate.go @@ -4,7 +4,6 @@ import ( "context" "errors" "fmt" - "maps" "slices" "strings" "time" @@ -183,7 +182,7 @@ func (m *Manager) wakeOrRead(ctx context.Context, sb *types.Sandbox, readerOK bo if err := m.diskErr(); err != nil { return "", fmt.Errorf("wake %s: %w", sb.ID, err) } - // See Hibernate: a half-restored VM is worse than a wasted wake. + // A half-restored VM is worse than a wasted wake. ctx = context.WithoutCancel(ctx) if err := m.resolveIntent(ctx, sb); err != nil { return "", fmt.Errorf("wake %s: %w", sb.ID, err) @@ -245,7 +244,11 @@ func (m *Manager) idleOnce(ctx context.Context) { type victim struct{ id, token string } var victims []victim m.mu.Lock() - if m.idleDefault <= 0 && !slices.ContainsFunc(slices.Collect(maps.Values(m.pools)), func(p *pool) bool { return p.idle > 0 }) { + idlePolicy := m.idleDefault > 0 + for _, p := range m.pools { + idlePolicy = idlePolicy || p.idle > 0 + } + if !idlePolicy { m.mu.Unlock() m.idleSweep.Store(false) return diff --git a/sandboxd/pool/hibernate_test.go b/sandboxd/pool/hibernate_test.go index ffc4dd9e3..9ff72a479 100644 --- a/sandboxd/pool/hibernate_test.go +++ b/sandboxd/pool/hibernate_test.go @@ -17,8 +17,7 @@ func TestHibernatePersistFailureSurfacesAndConverges(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - broken := filepath.Join(t.TempDir(), "gone", "claims.json") - m.store.path = broken + breakStore(t, m) if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { t.Fatal("Hibernate reported success despite a persist failure") @@ -29,13 +28,13 @@ func TestHibernatePersistFailureSurfacesAndConverges(t *testing.T) { if len(eng.hibernates) != 0 { t.Fatalf("engine hibernated %d times before the intent could persist, want 0", len(eng.hibernates)) } - if err := os.MkdirAll(filepath.Dir(broken), 0o750); err != nil { + if err := os.MkdirAll(filepath.Dir(m.store.path), 0o750); err != nil { t.Fatalf("heal store dir: %v", err) } if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { t.Fatalf("Hibernate retry after heal: %v", err) } - got, err := newClaimStore(filepath.Dir(broken), false).load() + got, err := newClaimStore(filepath.Dir(m.store.path), false).load() if err != nil || got[sb.ID] == nil || got[sb.ID].HibernateSnap == "" || got[sb.ID].PendingSnap != "" { t.Fatalf("journal after healed retry: %+v, %v", got[sb.ID], err) } @@ -46,12 +45,9 @@ func TestWakePersistFailureSurfaces(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, sb) snap := eng.hibernates[0] - broken := filepath.Join(t.TempDir(), "gone", "claims.json") - m.store.path = broken + breakStore(t, m) if _, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err == nil { t.Fatal("wake reported success despite a persist failure") @@ -65,7 +61,7 @@ func TestWakePersistFailureSurfaces(t *testing.T) { if eng.snapRemoved(snap) { t.Error("fast path dropped the snapshot while the journal still lags") } - if err := os.MkdirAll(filepath.Dir(broken), 0o750); err != nil { + if err := os.MkdirAll(filepath.Dir(m.store.path), 0o750); err != nil { t.Fatalf("heal store dir: %v", err) } @@ -75,7 +71,7 @@ func TestWakePersistFailureSurfaces(t *testing.T) { } return eng.snapRemoved(snap) }) - got, loadErr := newClaimStore(filepath.Dir(broken), false).load() + got, loadErr := newClaimStore(filepath.Dir(m.store.path), false).load() if loadErr != nil || got[sb.ID] == nil || got[sb.ID].HibernateSnap != "" { t.Fatalf("journal after converge: %+v, %v", got[sb.ID], loadErr) } @@ -95,7 +91,7 @@ func TestHibernateAmbiguousErrorTreatedAsDone(t *testing.T) { if _, g := m.Info(); g.Hibernated != 1 { t.Errorf("hibernated %d, want 1", g.Hibernated) } - got, _ := newClaimStore(m.dataDir, false).load() + got := journaled(t, m) if got[sb.ID] == nil || got[sb.ID].HibernateSnap == "" || got[sb.ID].PendingSnap != "" { t.Fatalf("journal %+v, want HibernateSnap set + PendingSnap clear", got[sb.ID]) } @@ -113,7 +109,7 @@ func TestHibernateVerifiedFailureClearsIntent(t *testing.T) { if _, g := m.Info(); g.Hibernated != 0 { t.Errorf("hibernated %d, want 0 (still running)", g.Hibernated) } - got, _ := newClaimStore(m.dataDir, false).load() + got := journaled(t, m) if got[sb.ID] == nil || got[sb.ID].HibernateSnap != "" || got[sb.ID].PendingSnap != "" { t.Fatalf("journal %+v, want running (no snaps)", got[sb.ID]) } @@ -136,44 +132,52 @@ func TestHibernateUnverifiableErrorKeepsIntent(t *testing.T) { if _, g := m.Info(); g.Hibernated != 0 { t.Errorf("hibernated %d, want 0 (unconfirmed, not committed)", g.Hibernated) } - got, _ := newClaimStore(m.dataDir, false).load() + got := journaled(t, m) if got[sb.ID] == nil || got[sb.ID].HibernateSnap != "" || got[sb.ID].PendingSnap == "" { t.Fatalf("journal %+v, want the intent kept for Reconcile", got[sb.ID]) } } -func TestWakeRestoreErrorStopsAndRetries(t *testing.T) { - eng := newFakeEngine() - m := newTestManager(t, eng) - sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } - snap := eng.hibernates[0] - eng.restoreErr = errors.New("restore timeout") - - if _, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err == nil { - t.Fatal("wake reported success despite a restore error") - } - if !slices.Contains(eng.stops, sb.VMName) || !eng.stopped[sb.VMName] { - t.Errorf("stops=%v, want the possibly-booted VM stopped", eng.stops) - } - if eng.removed(sb.VMName) { - t.Error("restore error removed the VM the retry restores into") - } - if eng.snapRemoved(snap) { - t.Error("restore error dropped the snapshot needed to retry") - } - if got, _ := newClaimStore(m.dataDir, false).load(); got[sb.ID] == nil || got[sb.ID].HibernateSnap != snap { - t.Fatalf("journal %+v, want HibernateSnap=%q kept for the retry", got[sb.ID], snap) - } - eng.restoreErr = nil - sock, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token) - if err != nil || sock == "" { - t.Fatalf("wake retry after restore recovered: %q, %v", sock, err) - } - if len(eng.restores) != 2 { - t.Errorf("restores=%d, want 2 (failed attempt + retry)", len(eng.restores)) +func TestWakeFailureStopsTheVMAndRetries(t *testing.T) { + for _, tt := range []struct { + name string + knob func(*fakeEngine) *error + }{ + {"restore error", func(eng *fakeEngine) *error { return &eng.restoreErr }}, + {"probe timeout", func(eng *fakeEngine) *error { return &eng.probeErr }}, + } { + t.Run(tt.name, func(t *testing.T) { + eng := newFakeEngine() + m := newTestManager(t, eng) + sb := mustClaim(t, m, testKey) + hibernateClaim(t, m, sb) + snap := eng.hibernates[0] + *tt.knob(eng) = errors.New(tt.name) + + if _, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err == nil { + t.Fatal("wake reported success despite the failure") + } + if !slices.Contains(eng.stops, sb.VMName) || !eng.stopped[sb.VMName] { + t.Errorf("stops=%v, want the VM stopped", eng.stops) + } + if eng.removed(sb.VMName) { + t.Error("the failure removed the VM the retry restores into") + } + if eng.snapRemoved(snap) { + t.Error("the failure dropped the snapshot needed to retry") + } + if got := journaled(t, m); got[sb.ID] == nil || got[sb.ID].HibernateSnap != snap { + t.Fatalf("journal %+v, want HibernateSnap=%q kept for the retry", got[sb.ID], snap) + } + *tt.knob(eng) = nil + sock, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token) + if err != nil || sock == "" { + t.Fatalf("wake retry after recovery: %q, %v", sock, err) + } + if len(eng.restores) != 2 { + t.Errorf("restores=%d, want 2 (failed attempt + retry)", len(eng.restores)) + } + }) } } @@ -182,14 +186,9 @@ func TestRetryResolvesDanglingIntent(t *testing.T) { m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - eng.hibernateErr = errors.New("cli timeout") - eng.hibernateErrCompletes = true - eng.snapListErr = errors.New("cocoon unreachable") - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { - t.Fatal("Hibernate reported success on an unconfirmed transition") - } + leaveDanglingIntent(t, m, eng, sb) realSnap := eng.hibernates[0] - if got, _ := newClaimStore(m.dataDir, false).load(); got[sb.ID].PendingSnap != realSnap { + if got := journaled(t, m); got[sb.ID].PendingSnap != realSnap { t.Fatalf("intent %q, want the real snapshot %q kept", got[sb.ID].PendingSnap, realSnap) } @@ -201,7 +200,7 @@ func TestRetryResolvesDanglingIntent(t *testing.T) { if len(eng.hibernates) != 1 { t.Errorf("engine hibernated %d times, want 1 (retry adopted the existing snapshot)", len(eng.hibernates)) } - got, _ := newClaimStore(m.dataDir, false).load() + got := journaled(t, m) if got[sb.ID].HibernateSnap != realSnap || got[sb.ID].PendingSnap != "" { t.Fatalf("journal %+v, want HibernateSnap=%q + no pending", got[sb.ID], realSnap) } @@ -220,12 +219,7 @@ func TestWakeResolvesDanglingIntent(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - eng.hibernateErr = errors.New("cli timeout") - eng.hibernateErrCompletes = true - eng.snapListErr = errors.New("cocoon unreachable") - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { - t.Fatal("Hibernate reported success on an unconfirmed transition") - } + leaveDanglingIntent(t, m, eng, sb) eng.hibernateErr = nil eng.snapListErr = nil @@ -250,12 +244,7 @@ func TestResolvePendingSnapReleaseRaceDropsOrphan(t *testing.T) { m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - eng.hibernateErr = errors.New("cli timeout") - eng.hibernateErrCompletes = true - eng.snapListErr = errors.New("cocoon unreachable") - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { - t.Fatal("Hibernate reported success on an unconfirmed transition") - } + leaveDanglingIntent(t, m, eng, sb) realSnap := eng.hibernates[0] eng.snapListErr = nil @@ -284,16 +273,10 @@ func TestResolveAdoptBillsWhenPersistFails(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - eng.hibernateErr = errors.New("cli timeout") - eng.hibernateErrCompletes = true - eng.snapListErr = errors.New("cocoon unreachable") - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { - t.Fatal("Hibernate reported success on an unconfirmed transition") - } + leaveDanglingIntent(t, m, eng, sb) eng.snapListErr = nil - broken := filepath.Join(t.TempDir(), "gone", "claims.json") - m.store.path = broken + breakStore(t, m) if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { t.Fatal("Hibernate reported success despite a persist failure") } @@ -301,85 +284,45 @@ func TestResolveAdoptBillsWhenPersistFails(t *testing.T) { t.Errorf("hibernates billed=%d, want 1 (billed even while the persist lags)", n) } - if err := os.MkdirAll(filepath.Dir(broken), 0o750); err != nil { + if err := os.MkdirAll(filepath.Dir(m.store.path), 0o750); err != nil { t.Fatalf("heal store dir: %v", err) } waitFor(t, func() bool { - got, err := newClaimStore(filepath.Dir(broken), false).load() + got, err := newClaimStore(filepath.Dir(m.store.path), false).load() return err == nil && got[sb.ID] != nil && got[sb.ID].HibernateSnap != "" }) }) } -func TestWakeProbeFailureStopsAndRetries(t *testing.T) { - eng := newFakeEngine() - m := newTestManager(t, eng) - sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } - snap := eng.hibernates[0] - eng.probeErr = errors.New("probe timeout") - - if _, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err == nil { - t.Fatal("wake reported success despite a probe timeout") - } - if !slices.Contains(eng.stops, sb.VMName) || !eng.stopped[sb.VMName] { - t.Errorf("stops=%v, want the restored VM stopped", eng.stops) - } - if eng.removed(sb.VMName) { - t.Error("probe failure removed the VM the retry restores into") - } - if eng.snapRemoved(snap) { - t.Error("probe failure dropped the snapshot needed to retry") - } - if got, _ := newClaimStore(m.dataDir, false).load(); got[sb.ID] == nil || got[sb.ID].HibernateSnap != snap { - t.Fatalf("journal %+v, want HibernateSnap=%q kept for the retry", got[sb.ID], snap) - } - eng.probeErr = nil - sock, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token) - if err != nil || sock == "" { - t.Fatalf("wake retry after the probe recovered: %q, %v", sock, err) - } - if len(eng.restores) != 2 { - t.Errorf("restores=%d, want 2 (failed attempt + retry)", len(eng.restores)) - } -} - func TestFlushClaimsClosesShutdownWindow(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } - broken := filepath.Join(t.TempDir(), "gone", "claims.json") - m.store.path = broken + hibernateClaim(t, m, sb) + breakStore(t, m) if _, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err == nil { t.Fatal("wake reported success despite a persist failure") } - if err := os.MkdirAll(filepath.Dir(broken), 0o750); err != nil { + if err := os.MkdirAll(filepath.Dir(m.store.path), 0o750); err != nil { t.Fatalf("heal store dir: %v", err) } if err := m.FlushClaims(); err != nil { t.Fatalf("FlushClaims: %v", err) } - got, err := newClaimStore(filepath.Dir(broken), false).load() + got, err := newClaimStore(filepath.Dir(m.store.path), false).load() if err != nil || got[sb.ID] == nil || got[sb.ID].HibernateSnap != "" { t.Fatalf("journal after flush: %+v, %v", got[sb.ID], err) } } -func TestHibernateWakeCycle(t *testing.T) { +func TestHibernateIsIdempotentAndAWakeRestoresOnce(t *testing.T) { synctest.Test(t, func(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, sb) if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { t.Fatalf("repeat Hibernate: %v", err) } @@ -389,7 +332,7 @@ func TestHibernateWakeCycle(t *testing.T) { if _, g := m.Info(); g.Hibernated != 1 { t.Errorf("hibernated count %d, want 1", g.Hibernated) } - if got, _ := newClaimStore(m.dataDir, false).load(); got[sb.ID] == nil || got[sb.ID].HibernateSnap == "" { + if got := journaled(t, m); got[sb.ID] == nil || got[sb.ID].HibernateSnap == "" { t.Error("hibernation flag not persisted") } hibSnap := eng.hibernates[0] @@ -426,9 +369,7 @@ func TestWakeDropsSnapshotOffReturnPath(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) eng.snapRemoveStall = make(chan struct{}) sock, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token) @@ -450,9 +391,7 @@ func TestWakeFailureKeepsHibernated(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, sb) eng.restoreErr = errors.New("restore boom") if _, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err == nil { @@ -470,9 +409,7 @@ func TestReleaseHibernatedDropsSnapshot(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, sb) if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { t.Fatalf("Release: %v", err) @@ -494,11 +431,7 @@ func TestReleaseMidHibernateDropsOrphanSnapshot(t *testing.T) { hibErr := make(chan error, 1) go func() { hibErr <- m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}) }() - waitFor(t, func() bool { - eng.mu.Lock() - defer eng.mu.Unlock() - return len(eng.hibernates) == 1 - }) + waitFor(t, func() bool { return eng.hibernateCount() == 1 }) if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { t.Fatalf("Release: %v", err) @@ -528,9 +461,7 @@ func TestReconcileAdoptsJournaledIntent(t *testing.T) { VsockSocket: "/vsock/lagged", PendingSnap: hibernatePrefix + "lagged-1-abc123", }, } - if err := newClaimStore(dataDir, false).save(claims); err != nil { - t.Fatalf("setup: %v", err) - } + seedClaims(t, dataDir, claims) m := newTestManagerAt(t, eng, dataDir) if err := m.Reconcile(t.Context()); err != nil { @@ -557,9 +488,7 @@ func TestReconcileIgnoresStaleOrphanSnapshot(t *testing.T) { claims := map[string]*types.Sandbox{ "sb_orp": {ID: "sb_orp", VMName: "sbx-orphan-1", Key: testKey, Token: "tok", VsockSocket: "/vsock/orphan"}, } - if err := newClaimStore(dataDir, false).save(claims); err != nil { - t.Fatalf("setup: %v", err) - } + seedClaims(t, dataDir, claims) m := newTestManagerAt(t, eng, dataDir) if err := m.Reconcile(t.Context()); err != nil { @@ -583,9 +512,7 @@ func TestReconcileKeepsIntentWithoutImageAsRunning(t *testing.T) { VsockSocket: "/vsock/noimg", PendingSnap: hibernatePrefix + "noimg-1-abc123", }, } - if err := newClaimStore(dataDir, false).save(claims); err != nil { - t.Fatalf("setup: %v", err) - } + seedClaims(t, dataDir, claims) m := newTestManagerAt(t, eng, dataDir) if err := m.Reconcile(t.Context()); err != nil { @@ -613,9 +540,7 @@ func TestReconcileAdoptsHibernated(t *testing.T) { HibernateSnap: hibernatePrefix + "hibernated-1", }, } - if err := newClaimStore(dataDir, false).save(claims); err != nil { - t.Fatalf("setup: %v", err) - } + seedClaims(t, dataDir, claims) m := newTestManagerAt(t, eng, dataDir, config.PoolSpec{PoolKey: testKey, Warm: 1}) if err := m.Reconcile(t.Context()); err != nil { @@ -634,3 +559,13 @@ func TestReconcileAdoptsHibernated(t *testing.T) { t.Errorf("restores=%v, want the adopted claim woken", eng.restores) } } + +func leaveDanglingIntent(t *testing.T, m *Manager, eng *fakeEngine, sb *types.Sandbox) { + t.Helper() + eng.hibernateErr = errors.New("cli timeout") + eng.hibernateErrCompletes = true + eng.snapListErr = errors.New("cocoon unreachable") + if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { + t.Fatal("Hibernate reported success on an unconfirmed transition") + } +} diff --git a/sandboxd/pool/idle_test.go b/sandboxd/pool/idle_test.go index f4727195d..1ca0731b2 100644 --- a/sandboxd/pool/idle_test.go +++ b/sandboxd/pool/idle_test.go @@ -15,7 +15,6 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/engine" "github.com/cocoonstack/sandbox/sandboxd/outbound" "github.com/cocoonstack/sandbox/sandboxd/outbound/outboundtest" - "github.com/cocoonstack/sandbox/sandboxd/types" ) func TestIdleOnceHibernatesPastThreshold(t *testing.T) { @@ -42,7 +41,7 @@ func TestIdleOnceHibernatesPastThreshold(t *testing.T) { }) } -func TestIdleOncePolicyScope(t *testing.T) { +func TestIdleHibernateDefaultAppliesOnlyToUnpooledKeys(t *testing.T) { synctest.Test(t, func(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) @@ -55,7 +54,7 @@ func TestIdleOncePolicyScope(t *testing.T) { t.Fatal("pooled key without the policy was idle-hibernated by the node default") } - unpooled := types.PoolKey{Template: "tpl:v1", Net: types.NetNone, Size: types.SizeSmall} + unpooled := tplKey("tpl:v1") sb2, err := m.ClaimProvision(t.Context(), unpooled, ClaimOptions{TTL: time.Hour}) if err != nil { t.Fatalf("provision: %v", err) @@ -184,20 +183,3 @@ func TestEgressRequestInFlightBlocksIdleSweep(t *testing.T) { t.Fatal("idle claim was not hibernated") } } - -func sweepIdle(t *testing.T, m *Manager) { - t.Helper() - m.idleOnce(t.Context()) - waitFor(t, func() bool { return !m.idleSweep.Load() }) -} - -func backdate(m *Manager, sb *types.Sandbox, by time.Duration) { - m.mu.Lock() - sb.TouchAt(time.Now().Add(-by)) - m.mu.Unlock() -} - -func hibernated(m *Manager) int { - _, g := m.Info() - return g.Hibernated -} diff --git a/sandboxd/pool/imagecheck_test.go b/sandboxd/pool/imagecheck_test.go index 0a74bac54..d7ffd0aa4 100644 --- a/sandboxd/pool/imagecheck_test.go +++ b/sandboxd/pool/imagecheck_test.go @@ -20,10 +20,7 @@ func TestGoldenStampCarriesImageID(t *testing.T) { eng := newFakeEngine() eng.imageIDs = map[string]string{testKey.Template: imageA} m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) - final := filepath.Join(m.goldensDir(), testKey.Hash()) - if err := m.buildGoldenSteps(t.Context(), m.view.Load(), testKey, "sbx-gb", "snap", final, imageA); err != nil { - t.Fatalf("buildGoldenSteps: %v", err) - } + final := buildGoldenAt(t, m, testKey, imageA) stamp, err := os.ReadFile(final + goldenStampSuffix) if err != nil || string(stamp) != m.goldenStamp(m.view.Load(), testKey, imageA) { t.Fatalf("stamp = %q (%v), want one carrying %s", stamp, err, imageA) @@ -38,7 +35,7 @@ func TestBuildGoldenAdoptsMatchingStampWithoutBoot(t *testing.T) { eng := newFakeEngine() eng.imageIDs = map[string]string{testKey.Template: imageA} m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) - g := seedGolden(t, m, imageA) + g := seedGolden(t, m, testKey, imageA) m.refillOnce(t.Context()) waitFor(t, func() bool { @@ -61,12 +58,9 @@ func TestImageChangeDropsGoldenAndRebuildsOnce(t *testing.T) { eng := newFakeEngine() eng.imageIDs = map[string]string{testKey.Template: imageA} m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 2}) - seedGolden(t, m, imageA) + seedGolden(t, m, testKey, imageA) m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 2 - }) + waitWarm(t, m, 2) eng.mu.Lock() eng.imageIDs = map[string]string{testKey.Template: imageB} @@ -86,10 +80,7 @@ func TestImageChangeDropsGoldenAndRebuildsOnce(t *testing.T) { return infos[0].Golden }) m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 2 - }) + waitWarm(t, m, 2) removedBefore := len(eng.removedNames()) m.checkImagesOnce(t.Context()) m.refillOnce(t.Context()) @@ -112,12 +103,9 @@ func TestImageListFailureChangesNothing(t *testing.T) { eng := newFakeEngine() eng.imageIDs = map[string]string{testKey.Template: imageA} m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 2}) - g := seedGolden(t, m, imageA) + g := seedGolden(t, m, testKey, imageA) m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 2 - }) + waitWarm(t, m, 2) eng.mu.Lock() eng.imageErr = errors.New("cocoon image list: exit status 1") diff --git a/sandboxd/pool/imds_test.go b/sandboxd/pool/imds_test.go index 5cca29d28..e6e946fb8 100644 --- a/sandboxd/pool/imds_test.go +++ b/sandboxd/pool/imds_test.go @@ -32,15 +32,13 @@ func TestSetInstanceMetadataRefusesAPausedSandboxWithoutWaking(t *testing.T) { m := newTestManager(t, eng) doc := []byte(`{}`) - hibernated := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), hibernated.ID, Cred{Token: hibernated.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + paused := mustClaim(t, m, testKey) + hibernateClaim(t, m, paused) restores := len(eng.restores) - if err := m.SetInstanceMetadata(t.Context(), hibernated.ID, doc); !errors.Is(err, ErrPaused) { + if err := m.SetInstanceMetadata(t.Context(), paused.ID, doc); !errors.Is(err, ErrPaused) { t.Errorf("hibernated: %v, want ErrPaused", err) } - if len(eng.restores) != restores || hibernated.HibernateSnap == "" { + if len(eng.restores) != restores || paused.HibernateSnap == "" { t.Error("SetInstanceMetadata restored a hibernated sandbox") } diff --git a/sandboxd/pool/intercept_test.go b/sandboxd/pool/intercept_test.go index 3ffb6470b..770014f59 100644 --- a/sandboxd/pool/intercept_test.go +++ b/sandboxd/pool/intercept_test.go @@ -18,10 +18,7 @@ func TestGoldenBuildInstallsCAForInterceptPool(t *testing.T) { if m.out.CAFingerprint() == "" { t.Fatal("egress CA not loaded for an intercept pool") } - final := filepath.Join(m.goldensDir(), testKey.Hash()) - if err := m.buildGoldenSteps(t.Context(), m.view.Load(), testKey, "sbx-gb", "snap", final, ""); err != nil { - t.Fatalf("buildGoldenSteps: %v", err) - } + final := buildGoldenAt(t, m, testKey, "") if len(eng.caInstalls) != 1 { t.Fatalf("InstallCACert calls = %d, want 1", len(eng.caInstalls)) } @@ -41,10 +38,7 @@ func TestGoldenBuildSkipsCAForPlainPool(t *testing.T) { if m.out.CAFingerprint() != "" { t.Error("egress CA loaded though no pool intercepts") } - final := filepath.Join(m.goldensDir(), testKey.Hash()) - if err := m.buildGoldenSteps(t.Context(), m.view.Load(), testKey, "sbx-gb", "snap", final, ""); err != nil { - t.Fatalf("buildGoldenSteps: %v", err) - } + final := buildGoldenAt(t, m, testKey, "") if len(eng.caInstalls) != 0 { t.Errorf("InstallCACert called %d times for a plain pool", len(eng.caInstalls)) } diff --git a/sandboxd/pool/journal_test.go b/sandboxd/pool/journal_test.go index 61105e1be..d167726cb 100644 --- a/sandboxd/pool/journal_test.go +++ b/sandboxd/pool/journal_test.go @@ -15,18 +15,9 @@ import ( func TestJournalRotateFailureDegradesAndRecovers(t *testing.T) { path := filepath.Join(t.TempDir(), "usage.jsonl") - if err := os.WriteFile(path, nil, 0o600); err != nil { - t.Fatalf("setup: %v", err) - } - if err := os.Truncate(path, journalMaxBytes); err != nil { - t.Fatalf("setup: %v", err) - } - j, err := newJournal(path) + j := fullJournal(t, path) + err := os.Mkdir(path+".1", 0o750) if err != nil { - t.Fatalf("new journal: %v", err) - } - - if err = os.Mkdir(path+".1", 0o750); err != nil { t.Fatalf("setup: %v", err) } if err = j.append(map[string]string{"ev": "a"}); err == nil { @@ -73,21 +64,12 @@ func TestJournalKeepsAnEventWithInvalidUTF8(t *testing.T) { func TestJournalRotateReopenFailureKeepsWriting(t *testing.T) { dir := t.TempDir() path := filepath.Join(dir, "usage.jsonl") - if err := os.WriteFile(path, nil, 0o600); err != nil { - t.Fatalf("setup: %v", err) - } - if err := os.Truncate(path, journalMaxBytes); err != nil { - t.Fatalf("setup: %v", err) - } - j, err := newJournal(path) - if err != nil { - t.Fatalf("new journal: %v", err) - } + j := fullJournal(t, path) j.path = filepath.Join(dir, "gone", "usage.jsonl") - if err = j.append(map[string]string{"ev": "a"}); err == nil { + if err := j.append(map[string]string{"ev": "a"}); err == nil { t.Fatal("append across a failed reopen must surface the error") } - if err = j.append(map[string]string{"ev": "b"}); err == nil { + if err := j.append(map[string]string{"ev": "b"}); err == nil { t.Fatal("append while the reopen stays blocked must keep surfacing the error") } if got := tail(t, path); !strings.Contains(got, `"ev":"a"`) || !strings.Contains(got, `"ev":"b"`) { @@ -107,18 +89,7 @@ func TestJournalKeepsTheFirstAppendAfterATornLine(t *testing.T) { if err = j.append(usage.Event{Event: "claim", ID: "sb_b"}); err != nil { t.Fatalf("append: %v", err) } - raw, err := os.ReadFile(path) - if err != nil { - t.Fatalf("read: %v", err) - } - lines := strings.Split(strings.TrimSuffix(string(raw), "\n"), "\n") - var ev usage.Event - if len(lines) != 3 || json.Unmarshal([]byte(lines[2]), &ev) != nil || ev.ID != "sb_b" { - t.Errorf("journal after a torn line %q, want the append on a line of its own", raw) - } - if st, _ := os.Stat(path); j.size != st.Size() { - t.Errorf("journal size %d, file %d", j.size, st.Size()) - } + assertLastAppendOnItsOwnLine(t, j, path, "a torn line") } func TestJournalEndsALineAShortWriteTore(t *testing.T) { @@ -149,18 +120,7 @@ func TestJournalEndsALineAShortWriteTore(t *testing.T) { if err = j.append(usage.Event{Event: "claim", ID: "sb_b"}); err != nil { t.Fatalf("append after the short write: %v", err) } - raw, err := os.ReadFile(path) - if err != nil { - t.Fatalf("read: %v", err) - } - lines := strings.Split(strings.TrimSuffix(string(raw), "\n"), "\n") - var ev usage.Event - if len(lines) != 3 || json.Unmarshal([]byte(lines[2]), &ev) != nil || ev.ID != "sb_b" { - t.Errorf("journal after a short write %q, want the next append on a line of its own", raw) - } - if st, _ := os.Stat(path); j.size != st.Size() { - t.Errorf("journal size %d, file %d", j.size, st.Size()) - } + assertLastAppendOnItsOwnLine(t, j, path, "a short write") } func TestJournalLeavesAWholeLastLine(t *testing.T) { @@ -177,6 +137,37 @@ func TestJournalLeavesAWholeLastLine(t *testing.T) { } } +func fullJournal(t *testing.T, path string) *journal { + t.Helper() + if err := os.WriteFile(path, nil, 0o600); err != nil { + t.Fatalf("setup: %v", err) + } + if err := os.Truncate(path, journalMaxBytes); err != nil { + t.Fatalf("setup: %v", err) + } + j, err := newJournal(path) + if err != nil { + t.Fatalf("new journal: %v", err) + } + return j +} + +func assertLastAppendOnItsOwnLine(t *testing.T, j *journal, path, after string) { + t.Helper() + raw, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read: %v", err) + } + lines := strings.Split(strings.TrimSuffix(string(raw), "\n"), "\n") + var ev usage.Event + if len(lines) != 3 || json.Unmarshal([]byte(lines[2]), &ev) != nil || ev.ID != "sb_b" { + t.Errorf("journal after %s %q, want the append sb_b on a line of its own", after, raw) + } + if st, _ := os.Stat(path); j.size != st.Size() { + t.Errorf("journal size %d, file %d", j.size, st.Size()) + } +} + func tail(t *testing.T, path string) string { t.Helper() f, err := os.Open(path) diff --git a/sandboxd/pool/lease.go b/sandboxd/pool/lease.go index 5f86a2b6d..791da7dae 100644 --- a/sandboxd/pool/lease.go +++ b/sandboxd/pool/lease.go @@ -23,25 +23,9 @@ type fenceState struct { lifted time.Time } -// CloseOnFence closes c when fenced closes; stop ends the watch. -func CloseOnFence(fenced <-chan struct{}, c io.Closer) (stop func()) { - if fenced == nil { - return func() {} - } - stopped := make(chan struct{}) - go func() { - select { - case <-fenced: - _ = c.Close() - case <-stopped: - } - }() - return sync.OnceFunc(func() { close(stopped) }) -} - // keepLease renews this node's lease until ctx ends and fences its replicated claims when a renewal is overdue. func (m *Manager) keepLease(ctx context.Context) { - fence := time.AfterFunc(leaseTTL-leaseMargin, func() { m.trip(ctx) }) + fence := time.AfterFunc(leaseWindow, func() { m.trip(ctx) }) defer fence.Stop() tick := time.Tick(leaseRenew) for { @@ -60,7 +44,7 @@ func (m *Manager) renewLease(ctx context.Context, fence *time.Timer) { log.WithFunc("pool.renewLease").Warnf(ctx, "renew node lease: %v", err) return } - fence.Reset(leaseTTL - leaseMargin - time.Since(start)) + fence.Reset(leaseWindow - time.Since(start)) if m.isFenced() && m.unfencing.CompareAndSwap(false, true) { go func() { defer m.unfencing.Store(false) @@ -88,7 +72,7 @@ func (m *Manager) trip(ctx context.Context) { continue } close(old.ch) - log.WithFunc("pool.fence").Warnf(ctx, "node lease not renewed for %s; replicated claims answer 503 until it is", leaseTTL-leaseMargin) + log.WithFunc("pool.trip").Warnf(ctx, "node lease not renewed for %s; replicated claims answer 503 until it is", leaseWindow) return } } @@ -146,7 +130,7 @@ func (m *Manager) lift(ctx context.Context, fenced *fenceState, start time.Time) if !m.fence.CompareAndSwap(fenced, &fenceState{ch: make(chan struct{}), lifted: time.Now()}) { return } - if time.Since(start) > leaseTTL-leaseMargin { + if time.Since(start) > leaseWindow { log.WithFunc("pool.unfence").Warn(ctx, "replicated claims stay fenced: confirming ownership outlasted the lease window") m.trip(ctx) return @@ -229,3 +213,19 @@ func (m *Manager) fenceWatch(sb *types.Sandbox) (<-chan struct{}, error) { } return st.ch, nil } + +// CloseOnFence closes c when fenced closes; stop ends the watch. +func CloseOnFence(fenced <-chan struct{}, c io.Closer) (stop func()) { + if fenced == nil { + return func() {} + } + stopped := make(chan struct{}) + go func() { + select { + case <-fenced: + _ = c.Close() + case <-stopped: + } + }() + return sync.OnceFunc(func() { close(stopped) }) +} diff --git a/sandboxd/pool/lease_test.go b/sandboxd/pool/lease_test.go index 8615bde57..81145a46e 100644 --- a/sandboxd/pool/lease_test.go +++ b/sandboxd/pool/lease_test.go @@ -15,8 +15,6 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/types" ) -var plainKey = types.PoolKey{Template: "plain:24.04", Net: types.NetNone, Size: testKey.Size} - func TestLeaseRenewLoopKeepsTheNodeLeasedUntilItStops(t *testing.T) { synctest.Test(t, func(t *testing.T) { c := newReplicaCluster(t) @@ -50,17 +48,14 @@ func TestLeaseOverdueRenewalFencesReplicatedClaimsOnly(t *testing.T) { a.archives = cut replicated, plain := mustClaim(t, a, testKey), mustClaim(t, a, plainKey) replicateNow(t, a, replicated) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - - go a.keepLease(ctx) + ctx := startLease(t, a) synctest.Wait() time.Sleep(4*leaseRenew + 2*time.Second) synctest.Wait() assertServed(t, a, replicated.ID, replicated.Token) cut.cut.Store(true) - time.Sleep(leaseTTL - leaseMargin - 3*time.Second) + time.Sleep(leaseWindow - 3*time.Second) synctest.Wait() assertServed(t, a, replicated.ID, replicated.Token) time.Sleep(2 * time.Second) @@ -75,7 +70,7 @@ func TestLeaseOverdueRenewalFencesReplicatedClaimsOnly(t *testing.T) { t.Errorf("preview dial past the fence: %v, want ErrLeaseLost", err) } assertServed(t, a, plain.ID, plain.Token) - if _, held := a.byID(replicated.ID); !held || a.eng.(*fakeEngine).removed(lockedVMName(replicated)) { + if _, held := a.byID(replicated.ID); !held || fakeOf(a).removed(lockedVMName(replicated)) { t.Error("the fence dropped the claim or removed its VM") } @@ -94,13 +89,11 @@ func TestLeaseFenceClosesOpenStreamsOfReplicatedClaimsOnly(t *testing.T) { a.archives = cut replicated, plain, early := mustClaim(t, a, testKey), mustClaim(t, a, plainKey), mustClaim(t, a, testKey) replicateNow(t, a, replicated) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go a.keepLease(ctx) + ctx := startLease(t, a) synctest.Wait() replicatedCred, plainCred := Cred{Token: replicated.Token}, Cred{Token: plain.Token} - a.eng.(*fakeEngine).guestDial = func() (net.Conn, error) { + fakeOf(a).guestDial = func() (net.Conn, error) { host, _ := net.Pipe() return host, nil } @@ -135,7 +128,7 @@ func TestLeaseFenceClosesOpenStreamsOfReplicatedClaimsOnly(t *testing.T) { defer plainDone() cut.cut.Store(true) - time.Sleep(leaseTTL - leaseMargin + time.Second) + time.Sleep(leaseWindow + time.Second) synctest.Wait() select { case <-relayFenced: @@ -184,9 +177,7 @@ func TestLeaseSlowUnfenceStaysFencedAndKeepsRenewing(t *testing.T) { a.archives = cut sb := mustClaim(t, a, testKey) replicateNow(t, a, sb) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go a.keepLease(ctx) + ctx := startLease(t, a) synctest.Wait() cut.cut.Store(true) @@ -233,22 +224,18 @@ func TestLeaseFailedConfirmKeepsTheTeardownOfAForgottenClaim(t *testing.T) { a.archives = cut replicateNow(t, a, moved) replicateNow(t, a, kept) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go a.keepLease(ctx) + startLease(t, a) synctest.Wait() cut.cut.Store(true) time.Sleep(leaseTTL) synctest.Wait() c.expire("a") - if adopted, err := b.AdoptArchived(ctx, moved.ID, moved.Token); err != nil || !adopted { - t.Fatalf("adopt while the owner is fenced: %v %v", adopted, err) - } + mustAdopt(t, b, moved, "adopt while the owner is fenced") cut.cut.Store(false) time.Sleep(leaseRenew) synctest.Wait() - if exercised && !a.eng.(*fakeEngine).removed(vm) { + if exercised && !fakeOf(a).removed(vm) { t.Error("a failed confirmation dropped the teardown of a claim the owner had already forgotten") } }) @@ -268,9 +255,7 @@ func TestLeaseAdoptionDuringUnfenceDropsTheClaimWithoutServingIt(t *testing.T) { sb := mustClaim(t, a, testKey) vm := lockedVMName(sb) replicateNow(t, a, sb) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go a.keepLease(ctx) + ctx := startLease(t, a) synctest.Wait() cut.cut.Store(true) @@ -294,7 +279,7 @@ func TestLeaseAdoptionDuringUnfenceDropsTheClaimWithoutServingIt(t *testing.T) { if _, _, _, err := a.WakeAgentSocket(ctx, sb.ID, sb.Token); !errors.Is(err, ErrUnknownSandbox) { t.Errorf("the owner after losing the row: %v, want ErrUnknownSandbox", err) } - if !a.eng.(*fakeEngine).removed(vm) { + if !fakeOf(a).removed(vm) { t.Error("the owner kept its VM of the adopted claim") } if _, _, _, err := b.WakeAgentSocket(ctx, sb.ID, sb.Token); err != nil { @@ -315,33 +300,25 @@ func TestLeaseANodeLiftedAfterAnOutageAdoptsOnlyOnceEveryOwnerCouldRenew(t *test c.expire("a") c.expire("b") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { - t.Fatalf("adoption by a fenced node: %v %v, want refused", adopted, err) - } + mustNotAdopt(t, b, sb, "adoption by a fenced node") c.renew(t, "b") b.unfence(t.Context(), time.Now()) if b.isFenced() { t.Fatal("the node stayed fenced after its renewal") } - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { - t.Fatalf("adoption right after the lift: %v %v, want refused", adopted, err) - } + mustNotAdopt(t, b, sb, "adoption right after the lift") c.renew(t, "a") a.unfence(t.Context(), time.Now()) synctest.Wait() - if _, held := a.byID(sb.ID); !held || a.eng.(*fakeEngine).removed(vm) { - t.Fatalf("the healthy owner lost its claim: held=%v VM removed=%v", held, a.eng.(*fakeEngine).removed(vm)) + if _, held := a.byID(sb.ID); !held || fakeOf(a).removed(vm) { + t.Fatalf("the healthy owner lost its claim: held=%v VM removed=%v", held, fakeOf(a).removed(vm)) } c.expire("a") time.Sleep(adoptGrace - time.Millisecond) - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { - t.Fatalf("adoption inside the grace: %v %v, want refused", adopted, err) - } + mustNotAdopt(t, b, sb, "adoption inside the grace") time.Sleep(time.Millisecond) - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Errorf("adoption once the grace passed: %v %v, want adopted", adopted, err) - } + mustAdopt(t, b, sb, "adoption once the grace passed") }) } @@ -368,7 +345,7 @@ func TestLeaseLiftAfterTheWindowClosedFencesAgain(t *testing.T) { c := newReplicaCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) a.tripFence() - late := time.Now().Add(-(leaseTTL - leaseMargin + time.Second)) + late := time.Now().Add(-(leaseWindow + time.Second)) a.lift(t.Context(), a.fence.Load(), late) if !a.isFenced() { @@ -412,8 +389,8 @@ func TestLeaseConfirmThatKeepsLosingOnItsOwnRowStaysFencedAndKeepsTheClaim(t *te if !a.isFenced() { t.Error("the node lifted its fence without confirming a claim it owns") } - if _, held := a.byID(sb.ID); !held || a.eng.(*fakeEngine).removed(vm) { - t.Errorf("claim held %v, VM removed %v, want the claim and VM kept: its row is still this node's", held, a.eng.(*fakeEngine).removed(vm)) + if _, held := a.byID(sb.ID); !held || fakeOf(a).removed(vm) { + t.Errorf("claim held %v, VM removed %v, want the claim and VM kept: its row is still this node's", held, fakeOf(a).removed(vm)) } } @@ -451,25 +428,19 @@ func TestLeaseAdoptionWaitsOutTheFenceAndTheOldOwnerDropsItsCopy(t *testing.T) { sb := mustClaim(t, a, testKey) vm := lockedVMName(sb) replicateNow(t, a, sb) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go a.keepLease(ctx) + ctx := startLease(t, a) synctest.Wait() cut.cut.Store(true) - time.Sleep(leaseTTL - leaseMargin + time.Second) + time.Sleep(leaseWindow + time.Second) synctest.Wait() if _, _, _, err := a.WakeAgentSocket(ctx, sb.ID, sb.Token); !errors.Is(err, ErrLeaseLost) { t.Fatalf("the cut-off owner past its fence: %v, want ErrLeaseLost", err) } - if adopted, err := b.AdoptArchived(ctx, sb.ID, sb.Token); err != nil || adopted { - t.Fatalf("adoption while the fenced owner's lease is still live: %v %v, want refused", adopted, err) - } + mustNotAdopt(t, b, sb, "adoption while the fenced owner's lease is still live") time.Sleep(leaseMargin) - if adopted, err := b.AdoptArchived(ctx, sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adoption after the lease expired: %v %v, want adopted", adopted, err) - } + mustAdopt(t, b, sb, "adoption after the lease expired") if tt.wakeBefore { if _, _, _, err := b.WakeAgentSocket(ctx, sb.ID, sb.Token); err != nil { t.Fatalf("adopter's wake: %v", err) @@ -479,8 +450,8 @@ func TestLeaseAdoptionWaitsOutTheFenceAndTheOldOwnerDropsItsCopy(t *testing.T) { cut.cut.Store(false) time.Sleep(leaseRenew) synctest.Wait() - if _, held := a.byID(sb.ID); held || !a.eng.(*fakeEngine).removed(vm) { - t.Errorf("old owner holds the claim %v, VM removed %v, want its copy dropped and its VM removed", held, a.eng.(*fakeEngine).removed(vm)) + if _, held := a.byID(sb.ID); held || !fakeOf(a).removed(vm) { + t.Errorf("old owner holds the claim %v, VM removed %v, want its copy dropped and its VM removed", held, fakeOf(a).removed(vm)) } if a.isFenced() { t.Error("the old owner stays fenced after its lease renewed") @@ -515,7 +486,7 @@ func TestLeaseUnfenceOfManyClaimsLiftsInsideTheWindowOverASlowRegistry(t *testin a.unfence(t.Context(), time.Now()) if a.isFenced() { - t.Fatalf("%d claims at a 100 ms round trip outlasted the %s window", len(sbs), leaseTTL-leaseMargin) + t.Fatalf("%d claims at a 100 ms round trip outlasted the %s window", len(sbs), leaseWindow) } for _, sb := range sbs { if rec, _, _ := c.reg.Get(t.Context(), sb.ID); rec.Owner != "a" || rec.Epoch != a.epochOf(sb) { @@ -599,22 +570,6 @@ func (r *cutRegistry) Rows(ctx context.Context, ids []string) (map[string]archiv return r.Registry.Rows(ctx, ids) } -func (m *Manager) epochOf(sb *types.Sandbox) int64 { - m.mu.Lock() - defer m.mu.Unlock() - return sb.ArchiveEpoch -} - -func (m *Manager) tripFence() { - ch := make(chan struct{}) - close(ch) - m.fence.Store(&fenceState{ch: ch, fenced: true}) -} - -func (m *Manager) liftFence() { - m.fence.Store(&fenceState{ch: make(chan struct{})}) -} - func assertServed(t *testing.T, m *Manager, id, token string) { t.Helper() _, _, done, err := m.WakeAgentSocket(t.Context(), id, token) diff --git a/sandboxd/pool/liveness.go b/sandboxd/pool/liveness.go index 490f94d0e..fd92a6123 100644 --- a/sandboxd/pool/liveness.go +++ b/sandboxd/pool/liveness.go @@ -145,7 +145,7 @@ func (m *Manager) evictWarm(ctx context.Context, suspect liveSuspect) { vm, found, err := m.eng.Inspect(ctx, suspect.vmName) switch { case err != nil: - logger.Warnf(ctx, "list %s: %v", suspect.vmName, err) + logger.Warnf(ctx, "inspect %s: %v", suspect.vmName, err) return case found && vm.State == vmStateRunning: return @@ -185,7 +185,7 @@ func (m *Manager) recoverVMM(ctx context.Context, sb *types.Sandbox) { vm, found, err := m.eng.Inspect(ctx, sb.VMName) switch { case err != nil: - logger.Warnf(ctx, "list %s: %v", sb.VMName, err) + logger.Warnf(ctx, "inspect %s: %v", sb.VMName, err) return case found && vm.State == vmStateRunning: return diff --git a/sandboxd/pool/liveness_test.go b/sandboxd/pool/liveness_test.go index 3f45ed85d..4f8c38d34 100644 --- a/sandboxd/pool/liveness_test.go +++ b/sandboxd/pool/liveness_test.go @@ -1,7 +1,6 @@ package pool import ( - "context" "errors" "net" "os" @@ -140,9 +139,7 @@ func TestLivenessLeavesLiveAndPausedClaims(t *testing.T) { m := newTestManager(t, eng) live := mustClaim(t, m, testKey) paused := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), paused.ID, Cred{Token: paused.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, paused) runLiveness(t, m) m.recoverVMM(t.Context(), live) @@ -315,9 +312,7 @@ func TestDaemonExitEventRecoversWithoutAList(t *testing.T) { eng.vmEvents = make(chan fakeVMEvent) m := newTestManager(t, eng) sb, other := mustClaim(t, m, testKey), mustClaim(t, m, testKey) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go m.watchVMMs(ctx) + startWatch(t, m) eng.vmEvents <- fakeVMEvent{sync: []engine.VMStatus{{Name: sb.VMName, Live: true}, {Name: other.VMName, Live: true}}} waitFor(t, m.vmmEvents.Load) lists := eng.listCalls() @@ -342,9 +337,7 @@ func TestDaemonSyncRecoversAClaimTheSnapshotShowsDown(t *testing.T) { m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) killVMM(eng, sb.VMName) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go m.watchVMMs(ctx) + startWatch(t, m) eng.vmEvents <- fakeVMEvent{sync: []engine.VMStatus{{Name: sb.VMName}}} @@ -356,9 +349,7 @@ func TestPollBackstopsTheDaemonStream(t *testing.T) { eng.vmEvents = make(chan fakeVMEvent) m := newTestManager(t, eng) mustClaim(t, m, testKey) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go m.watchVMMs(ctx) + startWatch(t, m) eng.vmEvents <- fakeVMEvent{sync: []engine.VMStatus{}} waitFor(t, m.vmmEvents.Load) lists := eng.listCalls() @@ -384,8 +375,8 @@ func TestLivenessReplacesADeadWarmVM(t *testing.T) { runLiveness(t, m) - if infos, _ := m.Info(); infos[0].Warm != 0 { - t.Fatalf("warm %d after its VMM exited, want 0", infos[0].Warm) + if n := warmLen(m, testKey); n != 0 { + t.Fatalf("warm %d after its VMM exited, want 0", n) } if removed := eng.removedNames(); !slices.Equal(removed, []string{dead}) { t.Errorf("removed %v, want %s", removed, dead) @@ -407,16 +398,14 @@ func TestDaemonExitEventReplacesADeadWarmVM(t *testing.T) { eng.vmEvents = make(chan fakeVMEvent) m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) dead := fillOneWarm(t, m) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go m.watchVMMs(ctx) + startWatch(t, m) eng.vmEvents <- fakeVMEvent{sync: []engine.VMStatus{{Name: dead, Live: true}}} waitFor(t, m.vmmEvents.Load) killVMM(eng, dead) eng.vmEvents <- fakeVMEvent{change: engine.VMChange{Kind: "MODIFIED", VM: engine.VMStatus{Name: dead}}} - waitFor(t, func() bool { infos, _ := m.Info(); return infos[0].Warm == 0 }) + waitFor(t, func() bool { return warmLen(m, testKey) == 0 }) waitFor(t, func() bool { return slices.Equal(eng.removedNames(), []string{dead}) }) } @@ -425,9 +414,7 @@ func TestAWarmVMClaimedDuringItsEvictionIsRecoveredWithoutAPoll(t *testing.T) { eng.vmEvents = make(chan fakeVMEvent) m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) dead := fillOneWarm(t, m) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - go m.watchVMMs(ctx) + startWatch(t, m) eng.vmEvents <- fakeVMEvent{sync: []engine.VMStatus{{Name: dead, Live: true}}} waitFor(t, m.vmmEvents.Load) lists := eng.listCalls() @@ -445,7 +432,12 @@ func TestAWarmVMClaimedDuringItsEvictionIsRecoveredWithoutAPoll(t *testing.T) { eng.vmEvents <- fakeVMEvent{change: engine.VMChange{Kind: "MODIFIED", VM: engine.VMStatus{Name: dead}}} - sb := <-claimed + var sb *types.Sandbox + select { + case sb = <-claimed: + case <-time.After(3 * time.Second): + t.Fatal("evictWarm's Inspect never ran, so the claim never landed") + } if sb == nil { t.Fatal("no claim during the eviction") } @@ -469,8 +461,8 @@ func TestStaleReportsKeepALiveWarmVM(t *testing.T) { m.evictDeadWarm(t.Context(), map[string]bool{}).Wait() m.evictDeadWarm(t.Context(), map[string]bool{name: false}).Wait() - if infos, _ := m.Info(); infos[0].Warm != 1 { - t.Errorf("warm %d, want the running VM kept", infos[0].Warm) + if n := warmLen(m, testKey); n != 1 { + t.Errorf("warm %d, want the running VM kept", n) } if removed := eng.removedNames(); len(removed) != 0 { t.Errorf("removed %v, want none", removed) @@ -527,9 +519,7 @@ func TestResyncRecordsADownEgressLaneTap(t *testing.T) { } } -func TestColdBootBlocker(t *testing.T) { - egressKey := testKey - egressKey.Net = types.NetEgress +func TestColdBootBlockerNamesWhyAVMCannotColdBoot(t *testing.T) { tests := []struct { name string sb *types.Sandbox @@ -538,7 +528,7 @@ func TestColdBootBlocker(t *testing.T) { }{ {"plain", &types.Sandbox{Key: testKey}, true, ""}, {"no record", &types.Sandbox{Key: testKey}, false, "vm record gone"}, - {"egress lane", &types.Sandbox{Key: egressKey}, true, "cannot cold-boot"}, + {"egress lane", &types.Sandbox{Key: egKey}, true, "cannot cold-boot"}, {"volumes", &types.Sandbox{Key: testKey, Volumes: []types.Volume{{Name: "data"}}}, true, "volume mounts"}, } for _, tt := range tests { @@ -590,9 +580,7 @@ func TestReconcileKeepsRunningClaimsTheHostLost(t *testing.T) { claims := map[string]*types.Sandbox{ "sb_lost": {ID: "sb_lost", VMName: "sbx-lost-1", Key: testKey, Token: "tok", VsockSocket: "/vsock/lost"}, } - if err := newClaimStore(dataDir, false).save(claims); err != nil { - t.Fatalf("setup: %v", err) - } + seedClaims(t, dataDir, claims) cfg := &config.Config{DataDir: dataDir, VMMRestart: tt.policy} m, err := NewManager(t.Context(), cfg, eng, testSecrets(t), testBackends(t, cfg)) if err != nil { @@ -615,11 +603,6 @@ func TestReconcileKeepsRunningClaimsTheHostLost(t *testing.T) { } } -type fakeVMEvent struct { - sync []engine.VMStatus - change engine.VMChange -} - func assertRestartFailsUntilBudget(t *testing.T, m *Manager, eng *fakeEngine, sb *types.Sandbox) { t.Helper() for attempt := range vmmRestartBudget { @@ -661,7 +644,7 @@ func fillOneWarm(t *testing.T, m *Manager) string { m.pools[testKey].goldenDir = "/goldens/x" m.mu.Unlock() m.refillOnce(t.Context()) - waitFor(t, func() bool { infos, _ := m.Info(); return infos[0].Warm == 1 && infos[0].Refilling == 0 }) + waitWarm(t, m, 1) select { case <-m.refillKick: default: diff --git a/sandboxd/pool/pool.go b/sandboxd/pool/pool.go index 8d37a2bf9..9d14c6bc7 100644 --- a/sandboxd/pool/pool.go +++ b/sandboxd/pool/pool.go @@ -57,6 +57,7 @@ const ( leaseTTL = 15 * time.Second leaseRenew = 5 * time.Second leaseMargin = 3 * time.Second + leaseWindow = leaseTTL - leaseMargin adoptGrace = leaseRenew + leaseMargin // One failed boot is ordinary; only an unbroken run predicts the next failure refillFailStreak = 8 @@ -110,7 +111,7 @@ var ( // Replaying a journal takes a writable mount, so readers stay out. ErrVolumeNeedsRecovery = errors.New("volume needs recovery by a writable claim") - ErrArchived = errors.New("sandbox is archived; an exec or file call wakes it first") + ErrArchived = errors.New("sandbox is archived; a wake or an exec wakes it first") ErrLeaseLost = errors.New("node lease lost; retry") ErrPaused = errors.New("sandbox is paused; an operator call does not wake it") ErrFailed = errors.New("sandbox failed: its VMM is down; release it, or wake it to cold-boot its disk") @@ -215,6 +216,18 @@ type Gauges struct { AtCapacityReason string } +type Backends struct { + Checkpoints store.Store + Templates store.Store + SharedCheckpoints bool + Tenants tenants.Source + CellPools poolset.Shared + Archives archives.Registry +} + +// PeerDeleteFunc broadcasts a checkpoint delete to every peer. +type PeerDeleteFunc func(ctx context.Context, id string) + type pendingRemoval struct { sandboxID string tap string @@ -328,18 +341,6 @@ func (p *pool) trimWarm(target int) []string { return trim } -type Backends struct { - Checkpoints store.Store - Templates store.Store - SharedCheckpoints bool - Tenants tenants.Source - CellPools poolset.Shared - Archives archives.Registry -} - -// PeerDeleteFunc broadcasts a checkpoint delete to every peer. -type PeerDeleteFunc func(ctx context.Context, id string) - // Manager owns the node's pools, claims, and their persistence. type Manager struct { eng Engine diff --git a/sandboxd/pool/pool_test.go b/sandboxd/pool/pool_test.go index 4d6a5f123..c581d0ebb 100644 --- a/sandboxd/pool/pool_test.go +++ b/sandboxd/pool/pool_test.go @@ -4,6 +4,7 @@ import ( "cmp" "context" "encoding/hex" + "encoding/json/v2" "errors" "fmt" "io" @@ -22,14 +23,26 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/config" "github.com/cocoonstack/sandbox/sandboxd/egress" "github.com/cocoonstack/sandbox/sandboxd/engine" + "github.com/cocoonstack/sandbox/sandboxd/outbound" "github.com/cocoonstack/sandbox/sandboxd/store" "github.com/cocoonstack/sandbox/sandboxd/store/dir" "github.com/cocoonstack/sandbox/sandboxd/tenants" "github.com/cocoonstack/sandbox/sandboxd/tenants/file" "github.com/cocoonstack/sandbox/sandboxd/types" + "github.com/cocoonstack/sandbox/sandboxd/usage" ) -var testKey = types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeSmall} +const keepAnHour = 3600 + +var ( + testKey = types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeSmall} + egKey = types.PoolKey{Template: "rt:24.04", Net: types.NetEgress, Size: types.SizeSmall} + otherKey = types.PoolKey{Template: "other:1", Net: types.NetNone, Size: types.SizeSmall} + plainKey = tplKey("plain:24.04") + + egPolicy = &egress.Policy{Allow: []egress.Rule{{Host: "example.com", Secret: "gh"}}} + hostOnly = new(false) +) func TestClaimWarmHitTransfersOwnership(t *testing.T) { eng := newFakeEngine() @@ -51,7 +64,7 @@ func TestClaimWarmHitTransfersOwnership(t *testing.T) { if n := len(eng.clones) + len(eng.colds); n != 0 { t.Errorf("warm hit ran %d VM operations, want 0", n) } - if got, _ := newClaimStore(m.dataDir, false).load(); len(got) != 1 { + if got := journaled(t, m); len(got) != 1 { t.Errorf("claim not persisted: %d entries", len(got)) } } @@ -60,7 +73,7 @@ func TestClaimClampsTTL(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) - sb, err := claimAny(t.Context(), m, testKey, 48*time.Hour) + sb, err := claimAny(t.Context(), m, testKey, ClaimOptions{TTL: 48 * time.Hour}) if err != nil { t.Fatalf("Claim: %v", err) } @@ -78,11 +91,10 @@ func TestClaimMissClonesFromGolden(t *testing.T) { if len(eng.clones) != 1 || len(eng.colds) != 0 { t.Fatalf("got clones=%v colds=%v, want one clone", eng.clones, eng.colds) } - prefix := vmPrefix + testKey.Hash() + "-" - if !strings.HasPrefix(sb.VMName, prefix) { + suffix, ok := strings.CutPrefix(sb.VMName, vmPrefix+testKey.Hash()+"-") + if !ok { t.Fatalf("vm name %q missing pool prefix", sb.VMName) } - suffix := strings.TrimPrefix(sb.VMName, prefix) if len(suffix) != 12 { t.Errorf("vm name suffix %q has %d chars, want 12", suffix, len(suffix)) } @@ -101,7 +113,7 @@ func TestClaimUnpooledKeyColdBoots(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - if _, err := claimAny(t.Context(), m, testKey, 0); err != nil { + if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{}); err != nil { t.Fatalf("Claim: %v", err) } if len(eng.colds) != 1 || len(eng.clones) != 0 { @@ -117,7 +129,7 @@ func TestClaimProbeFailureDestroysVM(t *testing.T) { eng.probeErr = errors.New("never ready") m := newTestManager(t, eng) - if _, err := claimAny(t.Context(), m, testKey, 0); err == nil { + if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{}); err == nil { t.Fatal("Claim succeeded with failing probe") } if len(eng.removes) != 1 { @@ -133,7 +145,7 @@ func TestClaimCreateFailureDestroysVM(t *testing.T) { eng.runColdErr = errors.New("cli killed by timeout") m := newTestManager(t, eng) - if _, err := claimAny(t.Context(), m, testKey, 0); err == nil { + if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{}); err == nil { t.Fatal("Claim succeeded with failing create") } if len(eng.removes) != 1 { @@ -148,7 +160,7 @@ func TestClaimCanceledRequestStillDestroys(t *testing.T) { ctx, cancel := context.WithCancel(t.Context()) cancel() - if _, err := claimAny(ctx, m, testKey, 0); err == nil { + if _, err := claimAny(ctx, m, testKey, ClaimOptions{}); err == nil { t.Fatal("Claim succeeded with failing probe") } if len(eng.removes) != 1 { @@ -159,7 +171,7 @@ func TestClaimCanceledRequestStillDestroys(t *testing.T) { func TestClaimEgressNeedsAttachment(t *testing.T) { m := newTestManager(t, newFakeEngine()) - if _, err := claimAny(t.Context(), m, egKey, 0); !errors.Is(err, ErrNoEgress) { + if _, err := claimAny(t.Context(), m, egKey, ClaimOptions{}); !errors.Is(err, ErrNoEgress) { t.Errorf("got %v, want ErrNoEgress", err) } } @@ -168,7 +180,7 @@ func TestClaimRejectsBadKey(t *testing.T) { m := newTestManager(t, newFakeEngine()) key := types.PoolKey{Template: "rt:24.04", Net: "lan", Size: types.SizeSmall} - if _, err := claimAny(t.Context(), m, key, 0); !errors.Is(err, ErrBadKey) { + if _, err := claimAny(t.Context(), m, key, ClaimOptions{}); !errors.Is(err, ErrBadKey) { t.Errorf("got %v, want ErrBadKey", err) } } @@ -190,7 +202,7 @@ func TestReleaseValidatesToken(t *testing.T) { if !slices.Contains(eng.removes, sb.VMName) { t.Errorf("removes=%v, want %s", eng.removes, sb.VMName) } - if got, _ := newClaimStore(m.dataDir, false).load(); len(got) != 0 { + if got := journaled(t, m); len(got) != 0 { t.Errorf("release not persisted: %d entries", len(got)) } } @@ -261,7 +273,7 @@ func TestReleaseByOperatorCred(t *testing.T) { if !slices.Contains(eng.removes, sb.VMName) { t.Errorf("removes=%v, want %s", eng.removes, sb.VMName) } - if got, _ := newClaimStore(m.dataDir, false).load(); len(got) != 0 { + if got := journaled(t, m); len(got) != 0 { t.Errorf("release not persisted: %d entries", len(got)) } } @@ -307,13 +319,13 @@ func TestReapDestroysExpiredClaims(t *testing.T) { if _, g := m.Info(); g.Claimed != 0 { t.Errorf("claimed=%d, want 0", g.Claimed) } - if got, _ := newClaimStore(m.dataDir, false).load(); len(got) != 0 { + if got := journaled(t, m); len(got) != 0 { t.Errorf("reap not persisted: %d entries", len(got)) } }) } -func TestReconcile(t *testing.T) { +func TestReconcileAdoptsLiveClaimsAndDestroysTheRest(t *testing.T) { eng := newFakeEngine() eng.vms["sbx-live-1"] = "/vsock/live-new" eng.vms["sbx-stale-1"] = "/vsock/stale" @@ -323,11 +335,9 @@ func TestReconcile(t *testing.T) { "sb_live": {ID: "sb_live", VMName: "sbx-live-1", Key: testKey, Token: "tok", VsockSocket: "/vsock/live-old"}, "sb_dead": {ID: "sb_dead", VMName: "sbx-dead-1", Key: testKey, Token: "tok"}, } - if err := newClaimStore(dataDir, false).save(claims); err != nil { - t.Fatalf("setup: %v", err) - } + seedClaims(t, dataDir, claims) m := newTestManagerAt(t, eng, dataDir, config.PoolSpec{PoolKey: testKey, Warm: 1}) - goldenDir := seedGolden(t, m, "") + goldenDir := seedGolden(t, m, testKey, "") if err := m.Reconcile(t.Context()); err != nil { t.Fatalf("Reconcile: %v", err) @@ -360,10 +370,7 @@ func TestRefillTopsUpToTarget(t *testing.T) { m.pools[testKey].goldenDir = "/goldens/x" m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 2 && infos[0].Refilling == 0 - }) + waitWarm(t, m, 2) if n := eng.cloneCount(); n != 2 { t.Errorf("clones=%d, want 2", n) } @@ -374,7 +381,7 @@ func TestSetPoolsGrowShrinkAndDrain(t *testing.T) { synctest.Test(t, func(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - seedGolden(t, m, "") + seedGolden(t, m, testKey, "") if err := m.SetPools(t.Context(), []config.PoolSpec{{PoolKey: testKey, Warm: 2}}); err != nil { t.Fatalf("SetPools grow: %v", err) @@ -465,7 +472,7 @@ func TestSetPoolsRemovalShedsArchivePolicy(t *testing.T) { func TestSetPoolsRemovalDeletesTheIdlePoolsGolden(t *testing.T) { m := newTestManager(t, newFakeEngine(), config.PoolSpec{PoolKey: testKey, Warm: 0}) - g := seedGolden(t, m, "") + g := seedGolden(t, m, testKey, "") if err := os.WriteFile(filepath.Join(g, "memory"), []byte("x"), 0o600); err != nil { t.Fatalf("setup: %v", err) } @@ -519,7 +526,7 @@ func TestARemovedPoolKeepsItsGoldenUntilItSettles(t *testing.T) { } { t.Run(tt.name, func(t *testing.T) { m := newTestManager(t, newFakeEngine(), config.PoolSpec{PoolKey: testKey, Warm: 0}) - g := seedGolden(t, m, "") + g := seedGolden(t, m, testKey, "") m.mu.Lock() p := m.pools[testKey] tt.busy(p) @@ -549,7 +556,7 @@ func TestAClaimCloneHoldsTheGoldenOfItsRemovedPool(t *testing.T) { eng := newFakeEngine() eng.cloneStall = make(chan struct{}) m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 0}) - g := seedGolden(t, m, "") + g := seedGolden(t, m, testKey, "") m.mu.Lock() m.pools[testKey].goldenDir = g m.mu.Unlock() @@ -585,7 +592,7 @@ func TestAClaimCloneHoldsTheGoldenOfItsRemovedPool(t *testing.T) { func TestSetPoolsWarmZeroKeepsTheGolden(t *testing.T) { m := newTestManager(t, newFakeEngine(), config.PoolSpec{PoolKey: testKey, Warm: 1}) - g := seedGolden(t, m, "") + g := seedGolden(t, m, testKey, "") marker := filepath.Join(g, "memory") if err := os.WriteFile(marker, []byte("x"), 0o600); err != nil { t.Fatalf("setup: %v", err) @@ -654,10 +661,7 @@ func TestRefillRespectsSemaphore(t *testing.T) { } close(eng.probeStall) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 7 && infos[0].Refilling == 0 - }) + waitWarm(t, m, 7) }) } @@ -677,10 +681,7 @@ func TestRefillCloneStageIsBounded(t *testing.T) { t.Errorf("clones=%d while clone stage stalled, want 2", got) } close(eng.cloneStall) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 5 && infos[0].Refilling == 0 - }) + waitWarm(t, m, 5) }) } @@ -700,10 +701,7 @@ func TestRefillProbeFailureCleansUp(t *testing.T) { eng.probeErr = nil eng.mu.Unlock() m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 1 && infos[0].Refilling == 0 - }) + waitWarm(t, m, 1) }) } @@ -738,7 +736,7 @@ func TestDirectClaimSkipsProbeGate(t *testing.T) { <-m.probeSem } }() - if _, err := claimAny(t.Context(), m, testKey, 0); err != nil { + if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{}); err != nil { t.Fatalf("direct claim waited on probe gate: %v", err) } } @@ -764,17 +762,14 @@ func TestRefillReactivelyFillsToTarget(t *testing.T) { m.pools[testKey].goldenDir = "/goldens/x" m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 10 && infos[0].Refilling == 0 - }) + waitWarm(t, m, 10) if n := eng.cloneCount(); n != 10 { t.Errorf("clones=%d, want 10", n) } }) } -func TestGoldenBuildPipeline(t *testing.T) { +func TestGoldenBuildExportsAndDestroysItsBuilder(t *testing.T) { synctest.Test(t, func(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) @@ -796,10 +791,7 @@ func TestGoldenBuildPipeline(t *testing.T) { } m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 1 - }) + waitWarm(t, m, 1) }) } @@ -860,7 +852,7 @@ func TestProbeReadyWaitsForVsock(t *testing.T) { eng.vsockLateN = 2 m := newTestManager(t, eng) - sb, err := claimAny(t.Context(), m, testKey, 0) + sb, err := claimAny(t.Context(), m, testKey, ClaimOptions{}) if err != nil { t.Fatalf("Claim: %v", err) } @@ -1001,8 +993,7 @@ func TestRenewRollsBackOnPersistFailure(t *testing.T) { m := newTestManager(t, newFakeEngine()) sb := mustClaim(t, m, testKey) before, beforeLease := sb.Deadline, sb.LeaseSeconds - broken := filepath.Join(t.TempDir(), "gone", "claims.json") - m.store.path = broken + breakStore(t, m) if _, err := m.Renew(t.Context(), sb.ID, Cred{Token: sb.Token}, time.Hour, ""); err == nil { t.Fatal("renew reported success while the claim store was unwritable") @@ -1010,7 +1001,7 @@ func TestRenewRollsBackOnPersistFailure(t *testing.T) { if !sb.Deadline.Equal(before) || sb.LeaseSeconds != beforeLease { t.Errorf("deadline %v lease %ds after a failed persist, want %v / %ds", sb.Deadline, sb.LeaseSeconds, before, beforeLease) } - if err := os.MkdirAll(filepath.Dir(broken), 0o750); err != nil { + if err := os.MkdirAll(filepath.Dir(m.store.path), 0o750); err != nil { t.Fatalf("heal store dir: %v", err) } granted, err := m.Renew(t.Context(), sb.ID, Cred{Token: sb.Token}, time.Hour, "") @@ -1063,15 +1054,13 @@ func TestOperatorPortDialNeverWakesAPausedSandbox(t *testing.T) { m := newTestManager(t, eng) operator := Cred{Operator: true} - hibernated := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), hibernated.ID, Cred{Token: hibernated.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + paused := mustClaim(t, m, testKey) + hibernateClaim(t, m, paused) restores := len(eng.restores) - if _, err := m.DialPort(t.Context(), hibernated.ID, operator, 49983); !errors.Is(err, ErrPaused) { + if _, err := m.DialPort(t.Context(), paused.ID, operator, 49983); !errors.Is(err, ErrPaused) { t.Errorf("hibernated: %v, want ErrPaused", err) } - if len(eng.restores) != restores || hibernated.HibernateSnap == "" { + if len(eng.restores) != restores || paused.HibernateSnap == "" { t.Error("an operator dial restored a hibernated sandbox") } @@ -1081,16 +1070,16 @@ func TestOperatorPortDialNeverWakesAPausedSandbox(t *testing.T) { t.Errorf("mid-transition: %v, want ErrPaused at once", err) } running.Transition.Unlock() - for name, set := range map[string]func(){ - "pending hibernate": func() { running.PendingSnap = "snap" }, - "archived": func() { running.PendingSnap, running.ArchiveCk = "", "ck" }, + for _, tt := range []struct{ name, pending, archive string }{ + {"pending hibernate", "snap", ""}, + {"archived", "", "ck"}, } { m.mu.Lock() - set() + running.PendingSnap, running.ArchiveCk = tt.pending, tt.archive m.store.set(running) m.mu.Unlock() if _, err := m.DialPort(t.Context(), running.ID, operator, 49983); !errors.Is(err, ErrPaused) { - t.Errorf("%s: %v, want ErrPaused", name, err) + t.Errorf("%s: %v, want ErrPaused", tt.name, err) } } if _, err := m.DialPort(t.Context(), "sb_missing", operator, 49983); !errors.Is(err, ErrUnknownSandbox) { @@ -1142,9 +1131,7 @@ func TestTokenPortDialStillWakesAHibernatedSandbox(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, sb) restores := len(eng.restores) _, _ = m.DialPort(t.Context(), sb.ID, Cred{Token: sb.Token}, 49983) if len(eng.restores) != restores+1 || sb.HibernateSnap != "" { @@ -1199,80 +1186,65 @@ func (c *closeWriteConn) CloseWrite() error { return nil } +type fakeVMEvent struct { + sync []engine.VMStatus + change engine.VMChange +} + type fakeEngine struct { - mu sync.Mutex - vms map[string]string + mu sync.Mutex + + vms map[string]string + stopped map[string]bool + creating map[string]bool + disks map[string][]types.VMDisk + pids map[string]int + sockRoot string + tap string + vsockLateN int + clones []string cloneFroms []string colds []string coldStorage []string - removes []string starts []string stops []string + removes []string probeTimeouts []time.Duration - volumeSpecs []engine.VolumeSpec - volumeMounts []types.Volume - volumeOps []string - attachDirty map[string]bool - removeSeenOps map[string][]string - removeSeenDirty map[string][]string - dirtyPaths []string - laneMarks []string syncs []string - - hibernates, restores, snapRemoves []string - snapSaves, exports, snapshots []string - exportContent []byte - exportFiles map[string]string - startedReaders []string - attachedCopies []string - attachedDirFiles []string - attachStall chan struct{} - readerDials []string - startReaderErr error - readerDialErr error - readerPeers chan net.Conn - syncErr error - caInstalls []string - warmups [][]string - warmupSocks []string - warmupAfterSnap bool - warmupErr error - warmupHook func() - trims []string - trimErr error - staleReconciles []string - installCAErr error - guestDial func() (net.Conn, error) - diskAttachErr error - diskAttachErrFor string - diskAttachCancel context.CancelFunc - mountVolumeErr error - unmountVolumeErr error - detachVolumeErr error - detachUnclean bool - liveVolumeSupportErr error - liveVolumeHook func(string) error - inspectHook func() - stopped map[string]bool - disks map[string][]types.VMDisk - creating map[string]bool - staleOutcome engine.StaleCreateOutcome - staleErr error - pids map[string]int - vsockLateN int - sockRoot string - - cloneErr, runColdErr, probeErr, hibernateErr, restoreErr, snapListErr, snapSaveErr, startErr error - removeErrFor string - cloneFailNth int - hibernateErrCompletes bool - tap string - listCount int - inspects int - vmEvents chan fakeVMEvent - - attachRendezvous *sync.WaitGroup + laneMarks []string + caInstalls []string + trims []string + staleReconciles []string + listCount int + inspects int + + hibernates []string + restores []string + snapRemoves []string + snapSaves []string + exports []string + snapshots []string + exportContent []byte + exportFiles map[string]string + snapListCount int + + cloneErr error + runColdErr error + probeErr error + hibernateErr error + restoreErr error + snapListErr error + snapSaveErr error + startErr error + syncErr error + trimErr error + installCAErr error + removeErrFor string + cloneFailNth int + hibernateErrCompletes bool + staleOutcome engine.StaleCreateOutcome + staleErr error cloneStall chan struct{} probeStall chan struct{} @@ -1281,7 +1253,46 @@ type fakeEngine struct { exportStall chan struct{} snapRemoveStall chan struct{} snapListStall chan struct{} - snapListCount int + vmEvents chan fakeVMEvent + guestDial func() (net.Conn, error) + + volumeSpecs []engine.VolumeSpec + volumeMounts []types.Volume + volumeOps []string + attachDirty map[string]bool + removeSeenOps map[string][]string + removeSeenDirty map[string][]string + dirtyPaths []string + diskAttachErr error + diskAttachErrFor string + diskAttachCancel context.CancelFunc + attachRendezvous *sync.WaitGroup + mountVolumeErr error + unmountVolumeErr error + detachVolumeErr error + detachUnclean bool + liveVolumeSupportErr error + liveVolumeHook func(string) error + inspectHook func() + + startedReaders []string + attachedCopies []string + attachedDirFiles []string + attachStall chan struct{} + readerDials []string + startReaderErr error + readerDialErr error + readerPeers chan net.Conn + + warmups [][]string + warmupSocks []string + warmupAfterSnap bool + warmupErr error + warmupHook func() + settles int + settleAfterSnap bool + settleAfterWarmup bool + settleErr error imageIDs map[string]string imageErr error @@ -1291,10 +1302,6 @@ type fakeEngine struct { metadataErr error guestEnvs map[string]string guestEnvErr error - - settles int - settleAfterSnap, settleAfterWarmup bool - settleErr error } func newFakeEngine() *fakeEngine { @@ -1680,7 +1687,7 @@ func (f *fakeEngine) MountLiveVolume(ctx context.Context, sock, name, mount stri return f.MountVolume(ctx, sock, name, mount, rw) } -func (f *fakeEngine) DetachVolume(ctx context.Context, _, sock, name, mount string) (cleanRW bool, err error) { +func (f *fakeEngine) DetachVolume(ctx context.Context, _, sock, name, mount string) (bool, error) { if f.liveVolumeHook != nil { if err := f.liveVolumeHook("detach"); err != nil { return false, err @@ -1946,28 +1953,142 @@ func (f *fakeEngine) snapRemoved(name string) bool { return slices.Contains(f.snapRemoves, name) } +func (m *Manager) epochOf(sb *types.Sandbox) int64 { + m.mu.Lock() + defer m.mu.Unlock() + return sb.ArchiveEpoch +} + +func (m *Manager) tripFence() { + ch := make(chan struct{}) + close(ch) + m.fence.Store(&fenceState{ch: ch, fenced: true}) +} + +func (m *Manager) liftFence() { + m.fence.Store(&fenceState{ch: make(chan struct{})}) +} + func newTestManager(t testing.TB, eng *fakeEngine, pools ...config.PoolSpec) *Manager { t.Helper() return newTestManagerAt(t, eng, t.TempDir(), pools...) } -func claimAny(ctx context.Context, m *Manager, key types.PoolKey, ttl time.Duration) (*types.Sandbox, error) { - sb, err := m.ClaimWarm(ctx, key, ClaimOptions{TTL: ttl}) +func claimAny(ctx context.Context, m *Manager, key types.PoolKey, o ClaimOptions) (*types.Sandbox, error) { + sb, err := m.ClaimWarm(ctx, key, o) if errors.Is(err, ErrNoWarm) { - return m.ClaimProvision(ctx, key, ClaimOptions{TTL: ttl}) + return m.ClaimProvision(ctx, key, o) } return sb, err } func mustClaim(t *testing.T, m *Manager, key types.PoolKey) *types.Sandbox { t.Helper() - sb, err := claimAny(t.Context(), m, key, 0) + sb, err := claimAny(t.Context(), m, key, ClaimOptions{}) if err != nil { t.Fatalf("Claim: %v", err) } return sb } +func hibernateClaim(t *testing.T, m *Manager, sb *types.Sandbox) { + t.Helper() + if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Fatalf("hibernate: %v", err) + } +} + +func mustAdopt(t *testing.T, m *Manager, sb *types.Sandbox, step string) { + t.Helper() + if adopted, err := m.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { + t.Fatalf("%s: %v %v, want adopted", step, adopted, err) + } +} + +func mustNotAdopt(t *testing.T, m *Manager, sb *types.Sandbox, step string) { + t.Helper() + if adopted, err := m.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { + t.Fatalf("%s: %v %v, want refused", step, adopted, err) + } +} + +func assertNotAdopted(t *testing.T, m *Manager, sb *types.Sandbox, step string) { + t.Helper() + if adopted, err := m.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { + t.Errorf("%s: %v %v, want refused", step, adopted, err) + } +} + +func breakStore(t *testing.T, m *Manager) { + t.Helper() + m.store.path = filepath.Join(t.TempDir(), "gone", "claims.json") +} + +func healStore(t *testing.T, m *Manager) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(m.store.path), 0o750); err != nil { + t.Fatalf("heal store: %v", err) + } + waitFor(t, m.store.synced) +} + +func startLease(t *testing.T, m *Manager) context.Context { + t.Helper() + ctx, cancel := context.WithCancel(t.Context()) + t.Cleanup(cancel) + go m.keepLease(ctx) + return ctx +} + +func startWatch(t *testing.T, m *Manager) { + t.Helper() + ctx, cancel := context.WithCancel(t.Context()) + t.Cleanup(cancel) + go m.watchVMMs(ctx) +} + +func fakeOf(m *Manager) *fakeEngine { + return m.eng.(*fakeEngine) +} + +func seedClaims(t *testing.T, dataDir string, claims map[string]*types.Sandbox) { + t.Helper() + if err := newClaimStore(dataDir, false).save(claims); err != nil { + t.Fatalf("setup: %v", err) + } +} + +func journaled(t *testing.T, m *Manager) map[string]*types.Sandbox { + t.Helper() + claims, err := newClaimStore(m.dataDir, false).load() + if err != nil { + t.Fatalf("load journal: %v", err) + } + return claims +} + +func claimTenant(t *testing.T, m *Manager, tenant string) *types.Sandbox { + t.Helper() + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: tenant}) + if err != nil { + t.Fatalf("claim %q: %v", tenant, err) + } + return sb +} + +func tplKey(template string) types.PoolKey { + return types.PoolKey{Template: template, Net: testKey.Net, Size: testKey.Size} +} + +func buildGoldenAt(t *testing.T, m *Manager, key types.PoolKey, imageID string) string { + t.Helper() + final := filepath.Join(m.goldensDir(), key.Hash()) + if err := m.buildGoldenSteps(t.Context(), m.view.Load(), key, "sbx-gb-"+key.Hash(), "snap", final, imageID); err != nil { + t.Fatalf("buildGoldenSteps %s: %v", key.Template, err) + } + return final +} + func newTestManagerAt(t testing.TB, eng *fakeEngine, dataDir string, pools ...config.PoolSpec) *Manager { t.Helper() cfg := &config.Config{DataDir: dataDir, Pools: pools} @@ -2045,14 +2166,223 @@ func waitFor(t *testing.T, cond func() bool) { t.Fatal("condition not met within 3s") } -func seedGolden(t *testing.T, m *Manager, imageID string) string { +func waitWarm(t *testing.T, m *Manager, n int) { + t.Helper() + waitFor(t, func() bool { + infos, _ := m.Info() + return infos[0].Warm == n && infos[0].Refilling == 0 + }) +} + +func refilledTo(t *testing.T, m *Manager, key types.PoolKey, n int) bool { + m.refillOnce(t.Context()) + return warmLen(m, key) == n +} + +func waitPoolSettled(t *testing.T, m *Manager, key types.PoolKey) { t.Helper() - g := filepath.Join(m.goldensDir(), testKey.Hash()) + waitFor(t, func() bool { + m.mu.Lock() + defer m.mu.Unlock() + p := m.pools[key] + return p != nil && !p.building && p.refilling == 0 + }) +} + +func warmLen(m *Manager, key types.PoolKey) int { + m.mu.Lock() + defer m.mu.Unlock() + if p := m.pools[key]; p != nil { + return len(p.warm) + } + return 0 +} + +func seedGolden(t *testing.T, m *Manager, key types.PoolKey, imageID string) string { + t.Helper() + g := filepath.Join(m.goldensDir(), key.Hash()) if err := os.MkdirAll(g, 0o750); err != nil { t.Fatalf("seed golden: %v", err) } - if err := os.WriteFile(g+goldenStampSuffix, []byte(m.goldenStamp(m.view.Load(), testKey, imageID)), 0o644); err != nil { + if err := os.WriteFile(g+goldenStampSuffix, []byte(m.goldenStamp(m.view.Load(), key, imageID)), 0o644); err != nil { t.Fatalf("seed golden stamp: %v", err) } return g } + +func ckExists(t *testing.T, m *Manager, ck string) bool { + t.Helper() + _, _, _, err := m.ckpts.Fetch(t.Context(), ck) //nolint:dogsled + if err != nil && !errors.Is(err, store.ErrNotFound) { + t.Fatalf("fetch %s: %v", ck, err) + } + return err == nil +} + +func mustArchive(t *testing.T, m *Manager, sb *types.Sandbox) { + t.Helper() + hibernateClaim(t, m, sb) + if err := m.archive(t.Context(), sb); err != nil { + t.Fatalf("archive: %v", err) + } +} + +func archivePool(deleteAfter int) config.PoolSpec { + return config.PoolSpec{ + PoolKey: testKey, + Warm: 1, + IdleHibernateSeconds: 1, + ArchiveAfterSeconds: 2, + ArchiveDeleteAfterSeconds: deleteAfter, + } +} + +func backdate(m *Manager, sb *types.Sandbox, by time.Duration) { + m.mu.Lock() + sb.TouchAt(time.Now().Add(-by)) + m.mu.Unlock() +} + +func hibernated(m *Manager) int { + _, g := m.Info() + return g.Hibernated +} + +func sweepIdle(t *testing.T, m *Manager) { + t.Helper() + m.idleOnce(t.Context()) + waitFor(t, func() bool { return !m.idleSweep.Load() }) +} + +func expire(m *Manager, sb *types.Sandbox) { + m.mu.Lock() + sb.Deadline = time.Now().Add(-time.Second) + m.mu.Unlock() +} + +func egressManager(t testing.TB, eng *fakeEngine, pools ...config.PoolSpec) *Manager { + t.Helper() + return egressManagerAt(t, eng, t.TempDir(), pools...) +} + +func withOutbound(t *testing.T, m *Manager, edit func(*outbound.Options)) { + t.Helper() + ca, err := outbound.LoadCA(m.cfg) + if err != nil { + t.Fatalf("load egress ca: %v", err) + } + o := m.outboundOptions(ca) + edit(&o) + m.out = outbound.New(o) +} + +func refillWarmVM(t *testing.T, m *Manager) *types.Sandbox { + t.Helper() + seedGolden(t, m, testKey, "") + m.mu.Lock() + m.adoptGolden(m.pools[testKey], "") + m.mu.Unlock() + m.refillOnce(t.Context()) + var warm *types.Sandbox + waitFor(t, func() bool { + m.mu.Lock() + defer m.mu.Unlock() + if p := m.pools[testKey]; p != nil && len(p.warm) > 0 { + warm = p.warm[0] + } + return warm != nil + }) + return warm +} + +func usageEventsOf(t *testing.T, m *Manager, id string) string { + t.Helper() + var events []string + for _, ev := range usageJournal(t, m) { + if ev.ID == id { + events = append(events, ev.Event) + } + } + return strings.Join(events, ",") +} + +func usageJournal(t *testing.T, m *Manager) []usage.Event { + t.Helper() + raw, err := os.ReadFile(filepath.Join(m.dataDir, usage.JournalFile)) + if err != nil { + t.Fatalf("read usage journal: %v", err) + } + var events []usage.Event + for line := range strings.SplitSeq(strings.TrimSpace(string(raw)), "\n") { + var ev usage.Event + if err := json.Unmarshal([]byte(line), &ev); err != nil { + t.Fatalf("bad journal line %q: %v", line, err) + } + events = append(events, ev) + } + return events +} + +func setTenantCaps(t *testing.T, m *Manager, caps map[string]int) { + t.Helper() + records := make([]tenants.TenantRecord, 0, len(caps)) + for name, limit := range caps { + records = append(records, tenants.TenantRecord{Name: name, TokenSHA256: tenants.TokenSHA256("tok-" + name), MaxClaims: limit}) + } + src, err := file.Open(t.Context(), t.TempDir(), "") + if err != nil { + t.Fatalf("tenant set: %v", err) + } + if _, err = src.Replace(t.Context(), records); err != nil { + t.Fatalf("tenant records: %v", err) + } + m.tenants = src +} + +func putTenants(t *testing.T, m *Manager, specs ...tenants.TenantSpec) { + t.Helper() + if len(specs) == 0 { + return + } + if err := m.SetTenants(t.Context(), specs); err != nil { + t.Fatalf("set tenants: %v", err) + } +} + +func replicateNow(t *testing.T, m *Manager, sb *types.Sandbox) { + t.Helper() + if err := m.replicate(t.Context(), sb); err != nil { + t.Fatalf("replicate %s: %v", sb.ID, err) + } +} + +func newReplicaCluster(t *testing.T) *failoverCluster { + t.Helper() + c := newFailoverCluster(t) + c.replicate = replicateEvery + return c +} + +func volumeHoldersOf(m *Manager, name string) volumeHolders { + m.mu.Lock() + defer m.mu.Unlock() + return m.catalog.holders[name] +} + +func newVolumeManagerAt(t *testing.T, eng *fakeEngine, dataDir string, volumes []config.VolumeSpec) *Manager { + t.Helper() + cfg := &config.Config{DataDir: dataDir, Volumes: volumes} + m, err := NewManager(t.Context(), cfg, eng, testSecrets(t), testBackends(t, cfg)) + if err != nil { + t.Fatalf("setup manager: %v", err) + } + return m +} + +func liveCatalog(t *testing.T) []config.VolumeSpec { + t.Helper() + return []config.VolumeSpec{ + {Name: "data", Path: writeVolumeImage(t, "data.img", "data"), Writable: true}, + {Name: "other", Path: writeVolumeImage(t, "other.img", "other"), Writable: true}, + } +} diff --git a/sandboxd/pool/poolstore_test.go b/sandboxd/pool/poolstore_test.go index 5c3f38ea5..9a56b7683 100644 --- a/sandboxd/pool/poolstore_test.go +++ b/sandboxd/pool/poolstore_test.go @@ -242,7 +242,7 @@ func TestAPutTheCellCannotStoreChangesNothing(t *testing.T) { func TestBootPrefersTheCellSetOverPoolsJSON(t *testing.T) { cached := poolset.Set{Pools: []config.PoolSpec{{PoolKey: apiKey, Warm: 3}}} unfit := poolset.Set{Pools: []config.PoolSpec{{Template: "eg:24.04", Net: types.NetEgress, Size: types.SizeSmall, Warm: 1}}} - for name, tc := range map[string]struct { + for name, tt := range map[string]struct { shared *fakeShared wantAPI bool version int64 @@ -258,18 +258,18 @@ func TestBootPrefersTheCellSetOverPoolsJSON(t *testing.T) { if err := m.poolStore.commit(m.poolStore.seq.Add(1), cached); err != nil { t.Fatalf("seed pools.json: %v", err) } - m.poolShared = tc.shared + m.poolShared = tt.shared if err := m.adoptPools(t.Context()); err != nil { t.Fatalf("adoptPools: %v", err) } - if got := hasAPISet(m); got != tc.wantAPI { - t.Errorf("API set adopted = %v, want %v", got, tc.wantAPI) + if got := hasAPISet(m); got != tt.wantAPI { + t.Errorf("API set adopted = %v, want %v", got, tt.wantAPI) } - if m.poolVersion != tc.version { - t.Errorf("applied version %d, want %d", m.poolVersion, tc.version) + if m.poolVersion != tt.version { + t.Errorf("applied version %d, want %d", m.poolVersion, tt.version) } - if set, err := m.poolStore.load(); err != nil || (set.ConfigSeed == m.configSeedHash) != tc.recached { - t.Errorf("pools.json %+v %v, want recached=%v", set, err, tc.recached) + if set, err := m.poolStore.load(); err != nil || (set.ConfigSeed == m.configSeedHash) != tt.recached { + t.Errorf("pools.json %+v %v, want recached=%v", set, err, tt.recached) } }) } diff --git a/sandboxd/pool/promote_test.go b/sandboxd/pool/promote_test.go index 9bd4f1c2b..5f65fdc9b 100644 --- a/sandboxd/pool/promote_test.go +++ b/sandboxd/pool/promote_test.go @@ -30,7 +30,7 @@ func TestPromoteThenClaimClonesFromTemplate(t *testing.T) { if len(eng.snapSaves) != 1 || !slices.Contains(eng.snapRemoves, eng.snapSaves[0]) { t.Errorf("snapSaves=%v snapRemoves=%v, want one transient snapshot dropped", eng.snapSaves, eng.snapRemoves) } - key := types.PoolKey{Template: "tpl:x", Net: parent.Key.Net, Size: parent.Key.Size} + key := tplKey("tpl:x") if gotKey != key { t.Errorf("returned key %+v, want %+v (the parent's axes)", gotKey, key) } @@ -42,7 +42,7 @@ func TestPromoteThenClaimClonesFromTemplate(t *testing.T) { t.Fatalf("template export missing: %v", err) } - child, err := claimAny(t.Context(), m, key, 0) + child, err := claimAny(t.Context(), m, key, ClaimOptions{}) if err != nil { t.Fatalf("Claim promoted template: %v", err) } @@ -90,7 +90,7 @@ func TestRepromoteContentDigestTracksExportBytes(t *testing.T) { if thirdDigest == firstDigest { t.Errorf("changed export bytes kept digest %q", thirdDigest) } - child, err := claimAny(t.Context(), m, thirdKey, 0) + child, err := claimAny(t.Context(), m, thirdKey, ClaimOptions{}) if err != nil { t.Fatalf("claim latest generation: %v", err) } @@ -103,9 +103,7 @@ func TestPromoteHibernatedUsesWakeImage(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) parent := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), parent.ID, Cred{Token: parent.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, parent) if _, _, err := m.Promote(t.Context(), parent.ID, Cred{Token: parent.Token}, "tpl:hib", ""); err != nil { t.Fatalf("Promote: %v", err) @@ -121,7 +119,7 @@ func TestPromoteHibernatedUsesWakeImage(t *testing.T) { } } -func TestPromoteValidations(t *testing.T) { +func TestPromoteRejectsABadNameTokenOrPooledKeyBeforeASnapshot(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 0}) parent := mustClaim(t, m, testKey) @@ -178,7 +176,7 @@ func TestTemplateDigestGuardReadsTheSharedRecord(t *testing.T) { } } -func TestDeleteTemplate(t *testing.T) { +func TestDeleteTemplateRefusesPooledUnknownAndReplacedTemplates(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 0}) parent := mustClaim(t, m, testKey) @@ -186,12 +184,12 @@ func TestDeleteTemplate(t *testing.T) { if err != nil { t.Fatalf("Promote: %v", err) } - key := types.PoolKey{Template: "tpl:del", Net: testKey.Net, Size: testKey.Size} + key := tplKey("tpl:del") if err := m.DeleteTemplate(t.Context(), testKey, "", ""); !errors.Is(err, ErrPooledTemplate) { t.Errorf("pooled delete: %v, want ErrPooledTemplate", err) } - if err := m.DeleteTemplate(t.Context(), types.PoolKey{Template: "nope", Net: testKey.Net, Size: testKey.Size}, "", ""); !errors.Is(err, ErrUnknownTemplate) { + if err := m.DeleteTemplate(t.Context(), tplKey("nope"), "", ""); !errors.Is(err, ErrUnknownTemplate) { t.Errorf("unknown delete: %v, want ErrUnknownTemplate", err) } if err := m.DeleteTemplate(t.Context(), key, "", "sha256:observed-elsewhere"); !errors.Is(err, ErrTemplateReplaced) || !m.HasGolden(t.Context(), key, "") { @@ -205,7 +203,7 @@ func TestDeleteTemplate(t *testing.T) { } before := len(eng.colds) - if _, err := claimAny(t.Context(), m, key, 0); err != nil { + if _, err := claimAny(t.Context(), m, key, ClaimOptions{}); err != nil { t.Fatalf("Claim after delete: %v", err) } if len(eng.colds) != before+1 { @@ -216,11 +214,11 @@ func TestDeleteTemplate(t *testing.T) { func TestTemplateRecordLockEvictsWithTheRecord(t *testing.T) { m := newTestManager(t, newFakeEngine(), config.PoolSpec{PoolKey: testKey, Warm: 0}) parent := mustClaim(t, m, testKey) - key := types.PoolKey{Template: "tpl:evict", Net: testKey.Net, Size: testKey.Size} + key := tplKey("tpl:evict") id := store.TemplateID(key.Hash()) base := lockCount(m) - if _, err := claimAny(t.Context(), m, key, 0); err != nil { + if _, err := claimAny(t.Context(), m, key, ClaimOptions{}); err != nil { t.Fatalf("Claim of an unpromoted key: %v", err) } if hasRecLock(m, id) { @@ -274,10 +272,7 @@ func TestResolveGoldenSkipsPromotedEgressTemplate(t *testing.T) { func TestTemplateTenantScopedDelete(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - parent, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("claim: %v", err) - } + parent := claimTenant(t, m, "acme") key, _, err := m.Promote(t.Context(), parent.ID, Cred{Token: parent.Token}, "tpl:tenant", "acme") if err != nil { t.Fatalf("Promote: %v", err) @@ -314,14 +309,11 @@ func TestPromoteFailsClosedOnMetaError(t *testing.T) { } eng := newFakeEngine() m := newTestManager(t, eng) - a, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("claim: %v", err) - } + a := claimTenant(t, m, "acme") if _, _, err := m.Promote(t.Context(), a.ID, Cred{Token: a.Token}, "shared:v1", "acme"); err != nil { t.Fatalf("promote: %v", err) } - key := types.PoolKey{Template: "shared:v1", Net: testKey.Net, Size: testKey.Size} + key := tplKey("shared:v1") meta := filepath.Join(m.dataDir, "checkpoints", store.TemplateID(key.Hash()), store.MetaFile) if err := os.Chmod(meta, 0o000); err != nil { t.Fatalf("chmod: %v", err) @@ -335,16 +327,15 @@ func TestPromoteFailsClosedOnMetaError(t *testing.T) { func TestPromoteRefusesCrossTenantOverwrite(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - claim := func(tenant string) *types.Sandbox { return claimTenant(t, m, tenant) } - a := claim("acme") + a := claimTenant(t, m, "acme") if _, _, err := m.Promote(t.Context(), a.ID, Cred{Token: a.Token}, "shared:v1", "acme"); err != nil { t.Fatalf("acme promote: %v", err) } - b := claim("beta") + b := claimTenant(t, m, "beta") if _, _, err := m.Promote(t.Context(), b.ID, Cred{Token: b.Token}, "shared:v1", "beta"); !errors.Is(err, ErrTemplateOwned) { t.Errorf("beta overwrite: %v, want ErrTemplateOwned", err) } - r := claim("") + r := claimTenant(t, m, "") if _, _, err := m.Promote(t.Context(), r.ID, Cred{Token: r.Token}, "shared:v1", ""); err != nil { t.Errorf("root replace: %v, want ok", err) } @@ -353,14 +344,13 @@ func TestPromoteRefusesCrossTenantOverwrite(t *testing.T) { func TestTemplateClaimIsTenantScoped(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - claim := func(tenant string) *types.Sandbox { return claimTenant(t, m, tenant) } - a := claim("acme") + a := claimTenant(t, m, "acme") private, _, err := m.Promote(t.Context(), a.ID, Cred{Token: a.Token}, "acme-private", "acme") if err != nil { t.Fatalf("acme promote: %v", err) } - r := claim("") + r := claimTenant(t, m, "") shared, _, err := m.Promote(t.Context(), r.ID, Cred{Token: r.Token}, "ops-shared", "") if err != nil { t.Fatalf("root promote: %v", err) @@ -369,7 +359,7 @@ func TestTemplateClaimIsTenantScoped(t *testing.T) { if _, err := m.ClaimProvision(t.Context(), private, ClaimOptions{RequirePromoted: true, TTL: time.Hour, Tenant: "beta"}); !errors.Is(err, ErrUnknownTemplate) { t.Errorf("beta claiming acme's template: %v, want ErrUnknownTemplate", err) } - for _, tc := range []struct { + for _, tt := range []struct { name string key types.PoolKey tenant string @@ -378,8 +368,8 @@ func TestTemplateClaimIsTenantScoped(t *testing.T) { {"root claims a tenant's", private, ""}, {"tenant claims a root template", shared, "beta"}, } { - t.Run(tc.name, func(t *testing.T) { - sb, err := m.ClaimProvision(t.Context(), tc.key, ClaimOptions{RequirePromoted: true, TTL: time.Hour, Tenant: tc.tenant}) + t.Run(tt.name, func(t *testing.T) { + sb, err := m.ClaimProvision(t.Context(), tt.key, ClaimOptions{RequirePromoted: true, TTL: time.Hour, Tenant: tt.tenant}) if err != nil { t.Fatalf("claim: %v", err) } @@ -392,10 +382,7 @@ func TestTemplateClaimIsTenantScoped(t *testing.T) { func TestHasPromotedTemplateIsTenantScoped(t *testing.T) { m := newTestManager(t, newFakeEngine()) - a, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("claim: %v", err) - } + a := claimTenant(t, m, "acme") key, _, err := m.Promote(t.Context(), a.ID, Cred{Token: a.Token}, "acme-private", "acme") if err != nil { t.Fatalf("promote: %v", err) @@ -413,10 +400,7 @@ func TestHasPromotedTemplateIsTenantScoped(t *testing.T) { func TestTemplateHashesAreTenantScoped(t *testing.T) { m := tenantManager(t, t.TempDir(), tenants.TenantSpec{Name: "acme", Token: "acme-tok"}) - a, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("claim: %v", err) - } + a := claimTenant(t, m, "acme") key, _, err := m.Promote(t.Context(), a.ID, Cred{Token: a.Token}, "acme-private", "acme") if err != nil { t.Fatalf("promote: %v", err) @@ -475,7 +459,7 @@ func TestTemplatesListWhatTheNodeHoldsAcrossARestart(t *testing.T) { if err != nil { t.Fatalf("Promote tpl:b: %v", err) } - oldID := store.TemplateID(types.PoolKey{Template: "tpl:old", Net: testKey.Net, Size: testKey.Size}.Hash()) + oldID := store.TemplateID(tplKey("tpl:old").Hash()) staging := stageTemplate(t, m, oldID) if err = os.WriteFile(filepath.Join(staging, store.MetaFile), []byte(`{"id":"`+oldID+`","created_at":"2026-01-01T00:00:00Z"}`), 0o600); err != nil { t.Fatalf("write keyless meta: %v", err) @@ -492,8 +476,8 @@ func TestTemplatesListWhatTheNodeHoldsAcrossARestart(t *testing.T) { t.Fatalf("Templates before %+v, after restart %+v, want tpl:a and tpl:b", before, got) } for _, want := range []TemplateInfo{{Key: keyA, ContentDigest: digestA}, {Key: keyB, ContentDigest: digestB, Tenant: "acme"}} { - i := slices.IndexFunc(got, func(t TemplateInfo) bool { return t.Key == want.Key }) - j := slices.IndexFunc(before, func(t TemplateInfo) bool { return t.Key == want.Key }) + i := slices.IndexFunc(got, func(ti TemplateInfo) bool { return ti.Key == want.Key }) + j := slices.IndexFunc(before, func(ti TemplateInfo) bool { return ti.Key == want.Key }) if i < 0 || j < 0 || got[i].ContentDigest != want.ContentDigest || got[i].Tenant != want.Tenant || got[i].CreatedAt.IsZero() || !got[i].CreatedAt.Equal(before[j].CreatedAt) || got[i].CPUCount != spec.CPU || got[i].MemTotalBytes != spec.MemoryBytes { @@ -526,7 +510,7 @@ func TestTemplateLabelsPersistUntilTheTemplateIsRepromotedOrDeleted(t *testing.T } labelsOf := func(m *Manager) types.Metadata { t.Helper() - i := slices.IndexFunc(m.Templates(), func(t TemplateInfo) bool { return t.Key == key }) + i := slices.IndexFunc(m.Templates(), func(ti TemplateInfo) bool { return ti.Key == key }) if i < 0 { t.Fatalf("tpl:l not listed") } @@ -549,7 +533,7 @@ func TestTemplateLabelsPersistUntilTheTemplateIsRepromotedOrDeleted(t *testing.T if err = m.SetTemplateLabels(t.Context(), key, types.Metadata{"x": "y"}, "acme", ""); !errors.Is(err, ErrUnknownTemplate) { t.Errorf("a tenant labeling the operator's template: %v, want ErrUnknownTemplate", err) } - if err = m.SetTemplateLabels(t.Context(), types.PoolKey{Template: "tpl:none", Net: testKey.Net, Size: testKey.Size}, want, "", ""); !errors.Is(err, ErrUnknownTemplate) { + if err = m.SetTemplateLabels(t.Context(), tplKey("tpl:none"), want, "", ""); !errors.Is(err, ErrUnknownTemplate) { t.Errorf("an unknown template: %v, want ErrUnknownTemplate", err) } if _, _, err = m.Promote(t.Context(), parent.ID, Cred{Token: parent.Token}, "tpl:l", ""); err != nil { @@ -586,7 +570,7 @@ func TestRejectedLabelWritesLeakNoRecLock(t *testing.T) { m := newTestManager(t, newFakeEngine()) base := lockCount(m) for _, name := range []string{"tpl:absent", "tpl:gone", "tpl:never"} { - key := types.PoolKey{Template: name, Net: testKey.Net, Size: testKey.Size} + key := tplKey(name) if err := m.SetTemplateLabels(t.Context(), key, types.Metadata{"a": "1"}, "", ""); !errors.Is(err, ErrUnknownTemplate) { t.Errorf("labels on %q: %v, want ErrUnknownTemplate", name, err) } @@ -616,7 +600,7 @@ func TestClaimFromATemplateRunsBesideItsRepromote(t *testing.T) { defer cancel() claimed := make(chan error, 1) go func() { - _, claimErr := claimAny(ctx, m, key, 0) + _, claimErr := claimAny(ctx, m, key, ClaimOptions{}) claimed <- claimErr }() select { @@ -633,13 +617,16 @@ func TestClaimFromATemplateRunsBesideItsRepromote(t *testing.T) { } } -func claimTenant(t *testing.T, m *Manager, tenant string) *types.Sandbox { - t.Helper() - sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: tenant}) - if err != nil { - t.Fatalf("claim %q: %v", tenant, err) - } - return sb +type slowPrepareStore struct { + store.Store + entered chan struct{} + release chan struct{} +} + +func (s *slowPrepareStore) PrepareDigested(ctx context.Context, staging, id, lineage string) (string, store.CommitFunc, error) { + close(s.entered) + <-s.release + return s.Store.PrepareDigested(ctx, staging, id, lineage) } func stageTemplate(t *testing.T, m *Manager, id string) string { @@ -653,15 +640,3 @@ func stageTemplate(t *testing.T, m *Manager, id string) string { } return staging } - -type slowPrepareStore struct { - store.Store - entered chan struct{} - release chan struct{} -} - -func (s *slowPrepareStore) PrepareDigested(ctx context.Context, staging, id, lineage string) (string, store.CommitFunc, error) { - close(s.entered) - <-s.release - return s.Store.PrepareDigested(ctx, staging, id, lineage) -} diff --git a/sandboxd/pool/reader.go b/sandboxd/pool/reader.go index 85dd05e35..355090928 100644 --- a/sandboxd/pool/reader.go +++ b/sandboxd/pool/reader.go @@ -26,6 +26,22 @@ const ( envelopeFile = "snapshot.json" ) +type readerSet struct { + mu sync.Mutex + live map[string]*reader + flights singleflight.Group +} + +type reader struct { + claim string + source string + vm string + sock string + dir string + conns int + idle *time.Timer +} + // DialReader opens a silkd connection to a sleeping claim's reader VM without waking the claim. func (m *Manager) DialReader(ctx context.Context, id, token string) (conn net.Conn, source string, err error) { sb, ok := m.claim(id, token) @@ -260,22 +276,6 @@ func (m *Manager) destroyReader(ctx context.Context, r *reader) { } } -type readerSet struct { - mu sync.Mutex - live map[string]*reader - flights singleflight.Group -} - -type reader struct { - claim string - source string - vm string - sock string - dir string - conns int - idle *time.Timer -} - // sleepSource names a settled sleep's snapshot or record; callers hold m.mu. func sleepSource(sb *types.Sandbox) string { if sb.Failed != "" || sb.PendingSnap != "" || !sb.ReaderSafe { diff --git a/sandboxd/pool/reader_test.go b/sandboxd/pool/reader_test.go index 7f62a4925..553f15606 100644 --- a/sandboxd/pool/reader_test.go +++ b/sandboxd/pool/reader_test.go @@ -467,18 +467,15 @@ func TestReaderRefusesAnArchiveWithoutUsableAttachFiles(t *testing.T) { func TestReaderRefusesWhatAWakeRefuses(t *testing.T) { eng := newFakeEngine() readerExport(eng) - pool := archivePool(0) - pool.Reader = true - cfg := &config.Config{DataDir: t.TempDir(), APIToken: "root", Pools: []config.PoolSpec{pool}} + spec := archivePool(0) + spec.Reader = true + cfg := &config.Config{DataDir: t.TempDir(), APIToken: "root", Pools: []config.PoolSpec{spec}} m, err := NewManager(t.Context(), cfg, eng, testSecrets(t), testBackends(t, cfg)) if err != nil { t.Fatalf("setup manager: %v", err) } putTenants(t, m, tenants.TenantSpec{Name: "acme", Token: "acme-tok"}) - sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("claim: %v", err) - } + sb := claimTenant(t, m, "acme") mustArchive(t, m, sb) seedReaderVMs(m, testKey, "sbx-reader-1") if err = m.DeleteTenant(t.Context(), "acme"); err != nil { @@ -603,12 +600,7 @@ func TestWakeAgentSocketStampsActivityOnlyWhenItServesTheGuest(t *testing.T) { for _, on := range []bool{false, true} { t.Run(fmt.Sprintf("reader=%t", on), func(t *testing.T) { eng := newFakeEngine() - var m *Manager - if on { - m = newReaderManager(t, eng) - } else { - m = newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) - } + m := readerManagerIf(t, eng, on) sb := mustClaim(t, m, testKey) backdate(m, sb, time.Hour) _, _, done, err := m.WakeAgentSocketOrRead(t.Context(), sb.ID, sb.Token) @@ -674,7 +666,7 @@ func TestFailedPreHibernateSyncSendsReadsToTheWakePath(t *testing.T) { hibernateClaim(t, m, sb) seedReaderVMs(m, testKey, "sbx-reader-1") - if got, _ := newClaimStore(m.dataDir, false).load(); got[sb.ID] == nil || got[sb.ID].ReaderSafe != tt.safe { + if got := journaled(t, m); got[sb.ID] == nil || got[sb.ID].ReaderSafe != tt.safe { t.Errorf("persisted reader_safe = %+v, want %t", got[sb.ID], tt.safe) } if got := m.SleepingSource(sb.ID, sb.Token); (got != "") != tt.safe { @@ -711,12 +703,12 @@ func TestWakeClearsReaderSafe(t *testing.T) { } else { done() } - if got, _ := newClaimStore(m.dataDir, false).load(); got[sb.ID] == nil || got[sb.ID].ReaderSafe { + if got := journaled(t, m); got[sb.ID] == nil || got[sb.ID].ReaderSafe { t.Errorf("persisted claim after a wake = %+v, want reader_safe cleared", got[sb.ID]) } } -func TestReaderOffWakesAsBefore(t *testing.T) { +func TestReaderOffRefusesADialAndWakesTheClaim(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) @@ -757,25 +749,22 @@ func TestHibernateSyncsTheGuestOnlyWithTheReaderOn(t *testing.T) { for _, on := range []bool{true, false} { t.Run(fmt.Sprintf("reader=%t", on), func(t *testing.T) { eng := newFakeEngine() - var m *Manager - if on { - m = newReaderManager(t, eng) - } else { - m = newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) - } + m := readerManagerIf(t, eng, on) sb := mustClaim(t, m, testKey) eng.hibernateStall = make(chan struct{}) - hibernated := make(chan struct{}) + captured := make(chan struct{}) go func() { - defer close(hibernated) - hibernateClaim(t, m, sb) + defer close(captured) + if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { + t.Errorf("hibernate: %v", err) + } }() - waitFor(t, func() bool { eng.mu.Lock(); defer eng.mu.Unlock(); return len(eng.hibernates) == 1 }) + waitFor(t, func() bool { return eng.hibernateCount() == 1 }) eng.mu.Lock() syncedBeforeCapture := slices.Clone(eng.syncs) eng.mu.Unlock() close(eng.hibernateStall) - <-hibernated + <-captured want := []string(nil) if on { @@ -803,6 +792,35 @@ func TestReconcileClearsReaderLeftovers(t *testing.T) { } } +type fetchCountingStore struct { + store.Store + mu sync.Mutex + fetches int + fetched []string +} + +func (s *fetchCountingStore) Fetch(ctx context.Context, id string) (string, []byte, string, error) { + s.mu.Lock() + s.fetches++ + s.mu.Unlock() + return s.Store.Fetch(ctx, id) +} + +func (s *fetchCountingStore) FetchFile(ctx context.Context, id, rel, dst string) error { + s.mu.Lock() + s.fetched = append(s.fetched, rel) + s.mu.Unlock() + return s.Store.FetchFile(ctx, id, rel, dst) +} + +func (s *fetchCountingStore) files() []string { + s.mu.Lock() + defer s.mu.Unlock() + out := slices.Clone(s.fetched) + slices.Sort(out) + return out +} + func newReaderManager(t *testing.T, eng *fakeEngine, extra ...config.PoolSpec) *Manager { t.Helper() cfg := &config.Config{DataDir: t.TempDir(), Pools: append([]config.PoolSpec{{PoolKey: testKey, Warm: 1, Reader: true}}, extra...)} @@ -813,16 +831,17 @@ func newReaderManager(t *testing.T, eng *fakeEngine, extra ...config.PoolSpec) * return m } -func seedReaderVMs(m *Manager, key types.PoolKey, names ...string) { - for _, name := range names { - m.pools[key].warm = append(m.pools[key].warm, &types.Sandbox{VMName: name, Key: key, VsockSocket: "/vsock/" + name}) +func readerManagerIf(t *testing.T, eng *fakeEngine, on bool) *Manager { + t.Helper() + if on { + return newReaderManager(t, eng) } + return newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) } -func hibernateClaim(t *testing.T, m *Manager, sb *types.Sandbox) { - t.Helper() - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) +func seedReaderVMs(m *Manager, key types.PoolKey, names ...string) { + for _, name := range names { + m.pools[key].warm = append(m.pools[key].warm, &types.Sandbox{VMName: name, Key: key, VsockSocket: "/vsock/" + name}) } } @@ -840,32 +859,3 @@ func readerLifecycle(eng *fakeEngine) (removes []string, setups int) { defer eng.mu.Unlock() return slices.Clone(eng.removes), len(eng.startedReaders) } - -type fetchCountingStore struct { - store.Store - mu sync.Mutex - fetches int - fetched []string -} - -func (s *fetchCountingStore) Fetch(ctx context.Context, id string) (string, []byte, string, error) { - s.mu.Lock() - s.fetches++ - s.mu.Unlock() - return s.Store.Fetch(ctx, id) -} - -func (s *fetchCountingStore) FetchFile(ctx context.Context, id, rel, dst string) error { - s.mu.Lock() - s.fetched = append(s.fetched, rel) - s.mu.Unlock() - return s.Store.FetchFile(ctx, id, rel, dst) -} - -func (s *fetchCountingStore) files() []string { - s.mu.Lock() - defer s.mu.Unlock() - out := slices.Clone(s.fetched) - slices.Sort(out) - return out -} diff --git a/sandboxd/pool/reconcile_test.go b/sandboxd/pool/reconcile_test.go index b50d78dc3..56335f63a 100644 --- a/sandboxd/pool/reconcile_test.go +++ b/sandboxd/pool/reconcile_test.go @@ -43,11 +43,11 @@ func TestReconcileJournalsAnAdoptedHibernate(t *testing.T) { func TestReconcileStaleCreateSweep(t *testing.T) { tests := []staleCreateCase{ - {"collected frees the record", engine.StaleCreateCollected, nil, false, false, false}, - {"not-found treats the record as gone", engine.StaleCreateNotFound, nil, false, false, false}, - {"busy leaves the in-flight clone", engine.StaleCreateBusy, nil, true, false, true}, - {"not-creating removes normally", engine.StaleCreateNotCreating, nil, false, true, false}, - {"verb failure falls back to remove", "", errors.New("unknown command"), false, true, false}, + {name: "collected frees the record", outcome: engine.StaleCreateCollected}, + {name: "not-found treats the record as gone", outcome: engine.StaleCreateNotFound}, + {name: "busy leaves the in-flight clone", outcome: engine.StaleCreateBusy, wantPresent: true, wantPending: true}, + {name: "not-creating removes normally", outcome: engine.StaleCreateNotCreating, wantRemove: true}, + {name: "verb failure falls back to remove", err: errors.New("unknown command"), wantRemove: true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -84,11 +84,11 @@ func TestReconcileStaleCreateSweep(t *testing.T) { func TestBusyStaleCreateRetryConverges(t *testing.T) { tests := []staleCreateCase{ - {"collected", engine.StaleCreateCollected, nil, false, false, false}, - {"not-found", engine.StaleCreateNotFound, nil, false, false, false}, - {"not-creating", engine.StaleCreateNotCreating, nil, false, true, false}, - {"still busy", engine.StaleCreateBusy, nil, true, false, true}, - {"verb failure", "", errors.New("temporary failure"), true, false, true}, + {name: "collected", outcome: engine.StaleCreateCollected}, + {name: "not-found", outcome: engine.StaleCreateNotFound}, + {name: "not-creating", outcome: engine.StaleCreateNotCreating, wantRemove: true}, + {name: "still busy", outcome: engine.StaleCreateBusy, wantPresent: true, wantPending: true}, + {name: "verb failure", err: errors.New("temporary failure"), wantPresent: true, wantPending: true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -206,7 +206,7 @@ func TestReconcileSweepsInterruptedForkExports(t *testing.T) { func TestReconcileSweepsGoldensNoPoolUses(t *testing.T) { m := newTestManager(t, newFakeEngine(), config.PoolSpec{PoolKey: testKey, Warm: 1}) - pooled := seedGolden(t, m, "") + pooled := seedGolden(t, m, testKey, "") orphan := filepath.Join(m.goldensDir(), otherKey.Hash()) staging := filepath.Join(m.goldensDir(), "x.tmp") for _, d := range []string{orphan, staging} { diff --git a/sandboxd/pool/refill.go b/sandboxd/pool/refill.go index a147e6ef8..f1de8ba14 100644 --- a/sandboxd/pool/refill.go +++ b/sandboxd/pool/refill.go @@ -43,7 +43,7 @@ func (m *Manager) refillOnce(ctx context.Context) { // SetPools leaves a removed pool in place while a build, refill or claim clone is in flight if p.removed { if p.settled() { - m.dropPool(ctx, p) + m.dropPoolLocked(ctx, p) } continue } @@ -294,8 +294,8 @@ func (m *Manager) adoptGolden(p *pool, imageID string) bool { return true } -// dropPool runs under m.mu: the rename hides the golden from adoptGolden, and the slow unlink runs off the lock. -func (m *Manager) dropPool(ctx context.Context, p *pool) { +// dropPoolLocked's rename hides the golden from adoptGolden, and the slow unlink runs off the lock. +func (m *Manager) dropPoolLocked(ctx context.Context, p *pool) { delete(m.pools, p.key) g := filepath.Join(m.goldensDir(), p.hash) _ = os.Remove(g + goldenStampSuffix) @@ -305,7 +305,7 @@ func (m *Manager) dropPool(ctx context.Context, p *pool) { } go func() { if os.RemoveAll(doomed) == nil { - log.WithFunc("pool.dropPool").Infof(ctx, "removed the golden of dropped pool %s", p.key.Template) + log.WithFunc("pool.dropPoolLocked").Infof(ctx, "removed the golden of dropped pool %s", p.key.Template) } }() } diff --git a/sandboxd/pool/refillbackoff_test.go b/sandboxd/pool/refillbackoff_test.go index fe2bd6ca9..ef24cca58 100644 --- a/sandboxd/pool/refillbackoff_test.go +++ b/sandboxd/pool/refillbackoff_test.go @@ -106,10 +106,7 @@ func TestRefillOnceHonorsTheBackoff(t *testing.T) { m.mu.Unlock() eng.cloneErr = nil m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 50 - }) + waitWarm(t, m, 50) }) } @@ -118,7 +115,7 @@ func TestAGoldenThatCannotBeClonedIsRebuiltOnce(t *testing.T) { eng := newFakeEngine() eng.cloneErr = errors.New(`vm.restore -> 500 "Memory manager error","Failed to allocate a memory range"`) m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 2}) - seedGolden(t, m, "") + seedGolden(t, m, testKey, "") waitFor(t, func() bool { m.refillOnce(t.Context()) @@ -140,7 +137,7 @@ func TestAGoldenThatStaysUnclonableIsRebuiltAtMostOncePerRetryDelay(t *testing.T eng := newFakeEngine() eng.cloneErr = errors.New(`vm.restore -> 500 "Memory manager error"`) m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 2}) - seedGolden(t, m, "") + seedGolden(t, m, testKey, "") start := time.Now() for time.Since(start) < 3*buildRetryDelay+buildRetryDelay/2 { @@ -160,7 +157,7 @@ func TestARebuiltGoldenSurvivesOneFailedRefill(t *testing.T) { eng := newFakeEngine() eng.cloneErr = errors.New(`vm.restore -> 500 "Memory manager error"`) m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) - seedGolden(t, m, "") + seedGolden(t, m, testKey, "") waitFor(t, func() bool { m.refillOnce(t.Context()) return eng.coldCount() == 1 @@ -214,7 +211,7 @@ func TestAGoldenWhoseClonesSucceedIsNeverRetiredForFailedProbes(t *testing.T) { eng := newFakeEngine() eng.probeErr = errors.New("silkd not ready") m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 2}) - g := seedGolden(t, m, "") + g := seedGolden(t, m, testKey, "") start := time.Now() for time.Since(start) < 2*buildRetryDelay { @@ -237,7 +234,7 @@ func TestAGoldenWhoseClonesSucceedIsNeverRetiredForFailedProbes(t *testing.T) { func TestRetiringAGoldenLeavesItsDirectoryForAClaimMidClone(t *testing.T) { m := newTestManager(t, newFakeEngine(), config.PoolSpec{PoolKey: testKey, Warm: 1}) - g := seedGolden(t, m, "") + g := seedGolden(t, m, testKey, "") m.mu.Lock() p := m.pools[testKey] p.goldenDir, p.goldenGen, p.cloneFails = g, 1, refillFailStreak @@ -436,7 +433,7 @@ func TestAFailingCloneOfAPromotedTemplateDoesNotCount(t *testing.T) { func adoptedGoldenNoWarm(t *testing.T, eng *fakeEngine) (*Manager, *pool, string) { t.Helper() m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey}) - g := seedGolden(t, m, "") + g := seedGolden(t, m, testKey, "") m.refillOnce(t.Context()) waitPoolSettled(t, m, testKey) m.mu.Lock() diff --git a/sandboxd/pool/refillkick_test.go b/sandboxd/pool/refillkick_test.go index 6748b532c..3643b263a 100644 --- a/sandboxd/pool/refillkick_test.go +++ b/sandboxd/pool/refillkick_test.go @@ -86,10 +86,7 @@ func TestNodeAtCapacityParksRefillInsteadOfRetrying(t *testing.T) { m.mu.Unlock() eng.cloneErr = nil m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 4 - }) + waitWarm(t, m, 4) }) } @@ -110,8 +107,5 @@ func TestRefillKeepsItsShareWhileTeardownsQueue(t *testing.T) { waitFor(t, func() bool { return len(m.maintSem) == cap(m.maintSem) }) m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 1 - }) + waitWarm(t, m, 1) } diff --git a/sandboxd/pool/reload.go b/sandboxd/pool/reload.go index 7a707c079..3b4ff6de0 100644 --- a/sandboxd/pool/reload.go +++ b/sandboxd/pool/reload.go @@ -29,7 +29,7 @@ func (m *Manager) ReloadConfig(ctx context.Context, next *config.Config) (Reload if err != nil { return ReloadResult{}, fmt.Errorf("%w: %w", ErrReloadRefused, err) } - res := ReloadResult{Changed: append([]string{}, changed...), Ignored: ignored} + res := ReloadResult{Changed: changed, Ignored: ignored} if len(changed) == 0 { m.cfg = next return res, nil diff --git a/sandboxd/pool/reload_test.go b/sandboxd/pool/reload_test.go index efa6cb645..c96639838 100644 --- a/sandboxd/pool/reload_test.go +++ b/sandboxd/pool/reload_test.go @@ -25,10 +25,7 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/types" ) -var ( - otherKey = types.PoolKey{Template: "other:1", Net: types.NetNone, Size: types.SizeSmall} - interceptOnKey = types.PoolKey{Template: "mitm:1", Net: types.NetNone, Size: types.SizeSmall} -) +var interceptOnKey = types.PoolKey{Template: "mitm:1", Net: types.NetNone, Size: types.SizeSmall} func TestReloadGivesANewKeyItsSettingsBeforeItsGolden(t *testing.T) { eng := newFakeEngine() @@ -366,27 +363,3 @@ func TestAReloadThatUnguardsTheNodeClosesAWarmVMsDoorsAtItsClaim(t *testing.T) { t.Errorf("claim route %q, want none", route) } } - -func refilledTo(t *testing.T, m *Manager, key types.PoolKey, n int) bool { - m.refillOnce(t.Context()) - return warmLen(m, key) == n -} - -func waitPoolSettled(t *testing.T, m *Manager, key types.PoolKey) { - t.Helper() - waitFor(t, func() bool { - m.mu.Lock() - defer m.mu.Unlock() - p := m.pools[key] - return p != nil && !p.building && p.refilling == 0 - }) -} - -func warmLen(m *Manager, key types.PoolKey) int { - m.mu.Lock() - defer m.mu.Unlock() - if p := m.pools[key]; p != nil { - return len(p.warm) - } - return 0 -} diff --git a/sandboxd/pool/replica_test.go b/sandboxd/pool/replica_test.go index 931576b8e..3f026a13b 100644 --- a/sandboxd/pool/replica_test.go +++ b/sandboxd/pool/replica_test.go @@ -4,7 +4,6 @@ import ( "encoding/json/v2" "errors" "net" - "path/filepath" "testing" "testing/synctest" "time" @@ -60,7 +59,7 @@ func TestReplicaRoundPublishesMovesTheRowAndDropsThePrevious(t *testing.T) { if err = json.Unmarshal(next.Claim, claim); err != nil || claim.ID != sb.ID || claim.Token != "" || claim.VMName != "" || claim.ReplicaCk != "" { t.Errorf("row claim %+v %v, want the claim without token, VM or replica", claim, err) } - if a.eng.(*fakeEngine).removed(vm) { + if fakeOf(a).removed(vm) { t.Error("a replication round removed the running VM") } } @@ -81,8 +80,8 @@ func TestReplicaRefusedPutDropsTheCopyAndItsExport(t *testing.T) { if err := a.replicate(t.Context(), sb); !errors.Is(err, archives.ErrLost) { t.Fatalf("round whose row another node took: %v, want ErrLost", err) } - if _, held := a.byID(sb.ID); held || !a.eng.(*fakeEngine).removed(vm) { - t.Errorf("old owner holds the claim %v, VM removed %v, want its copy dropped", held, a.eng.(*fakeEngine).removed(vm)) + if _, held := a.byID(sb.ID); held || !fakeOf(a).removed(vm) { + t.Errorf("old owner holds the claim %v, VM removed %v, want its copy dropped", held, fakeOf(a).removed(vm)) } if metas, err := a.ckpts.Metas(t.Context()); err != nil || len(metas) != 1 { t.Errorf("store holds %d checkpoints %v, want only the adopter's replica", len(metas), err) @@ -109,13 +108,11 @@ func TestReplicaLostPutReplyKeepsTheCheckpointTheRowNames(t *testing.T) { } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") if _, _, _, err := b.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("the adopter's wake from the checkpoint: %v", err) } - if got := b.eng.(*fakeEngine).cloneCount(); got != 1 { + if got := fakeOf(b).cloneCount(); got != 1 { t.Errorf("the wake cloned %d VMs, want the one restored from the checkpoint", got) } @@ -126,8 +123,8 @@ func TestReplicaLostPutReplyKeepsTheCheckpointTheRowNames(t *testing.T) { if !errors.Is(err, ErrUnknownSandbox) { t.Fatalf("the returning owner's settle: %v, want ErrUnknownSandbox", err) } - if _, held := a.byID(sb.ID); held || !a.eng.(*fakeEngine).removed(vm) { - t.Errorf("old owner holds the claim %v, VM removed %v, want its copy dropped", held, a.eng.(*fakeEngine).removed(vm)) + if _, held := a.byID(sb.ID); held || !fakeOf(a).removed(vm) { + t.Errorf("old owner holds the claim %v, VM removed %v, want its copy dropped", held, fakeOf(a).removed(vm)) } } @@ -196,9 +193,7 @@ func TestReplicaRowOfAHibernatedClaimHasNoRetentionDeadline(t *testing.T) { sb := mustClaim(t, a, testKey) replicateNow(t, a, sb) ck := a.replicaOf(sb) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) time.Sleep(2 * time.Second) c.expire("a") @@ -207,9 +202,7 @@ func TestReplicaRowOfAHibernatedClaimHasNoRetentionDeadline(t *testing.T) { if !ok || !rec.Deadline.IsZero() || !ckExists(t, b, ck) { t.Fatalf("row %+v present %v, replica kept %v after a sweep past the retention window, want the row kept with no deadline", rec, ok, ckExists(t, b, ck)) } - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v, want adopted", adopted, err) - } + mustAdopt(t, b, sb, "adopt") moved, _ := b.byID(sb.ID) if moved.Deadline.IsZero() { t.Error("the adopted claim has no retention deadline, want the pool's window from the adoption") @@ -239,9 +232,7 @@ func TestReplicaRowIsNeverReaderSafe(t *testing.T) { } c.expire("a") - if adopted, adoptErr := b.AdoptArchived(t.Context(), sb.ID, sb.Token); adoptErr != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, adoptErr) - } + mustAdopt(t, b, sb, "adopt") sock, _, done, err := b.WakeAgentSocketOrRead(t.Context(), sb.ID, sb.Token) if err != nil || sock == "" { t.Fatalf("read of the adopted replica = %q, %v, want a wake", sock, err) @@ -253,9 +244,7 @@ func TestArchiveRowKeepsReaderSafe(t *testing.T) { c := newFailoverCluster(t) a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) a.mu.Lock() sb.ReaderSafe = true a.mu.Unlock() @@ -269,9 +258,7 @@ func TestArchiveRowKeepsReaderSafe(t *testing.T) { t.Errorf("archive row claim reader_safe %v (%v), want true", claim.ReaderSafe, err) } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") if _, _, _, err := b.WakeAgentSocketOrRead(t.Context(), sb.ID, sb.Token); !errors.Is(err, ErrSleeping) { t.Errorf("read of the adopted archive = %v, want ErrSleeping", err) } @@ -296,7 +283,6 @@ func TestReplicaRetireAfterAnArchiveLeavesTheArchiveRowAlone(t *testing.T) { func TestReplicaClaimThatGainsVolumesStopsBeingReplicated(t *testing.T) { c := newReplicaCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) - a.liftFence() sb := mustClaim(t, a, testKey) replicateNow(t, a, sb) ck := a.replicaOf(sb) @@ -317,11 +303,10 @@ func TestReplicaClaimThatGainsVolumesStopsBeingReplicated(t *testing.T) { func TestReplicaFollowsThePolicyPoolOfAPromotedTemplate(t *testing.T) { c := newReplicaCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) - a.liftFence() sb := mustClaim(t, a, testKey) a.mu.Lock() a.dropClaimLocked(sb) - sb.Key, sb.PolicySource = types.PoolKey{Template: "promoted:v1", Net: types.NetNone, Size: testKey.Size}, testKey + sb.Key, sb.PolicySource = tplKey("promoted:v1"), testKey a.addClaimLocked(sb) a.mu.Unlock() if !sb.Replicated() { @@ -362,8 +347,8 @@ func TestReplicaInFlightRoundCannotRecreateARowTheAdopterFinished(t *testing.T) if rec, ok, _ := c.reg.Get(t.Context(), sb.ID); ok { t.Errorf("row %+v recreated by the old owner's round", rec) } - if _, held := a.byID(sb.ID); held || !a.eng.(*fakeEngine).removed(vm) { - t.Errorf("old owner holds the claim %v, VM removed %v, want its copy dropped", held, a.eng.(*fakeEngine).removed(vm)) + if _, held := a.byID(sb.ID); held || !fakeOf(a).removed(vm) { + t.Errorf("old owner holds the claim %v, VM removed %v, want its copy dropped", held, fakeOf(a).removed(vm)) } if _, held := b.byID(sb.ID); !held { t.Error("the adopter lost the claim it woke") @@ -397,13 +382,9 @@ func TestReplicaArchiveCannotRecreateARowTheAdopterFinished(t *testing.T) { a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) replicateNow(t, a, sb) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") if _, _, _, err := b.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("the adopter's wake: %v", err) } @@ -473,9 +454,7 @@ func TestReplicaRoundAfterAWakeKeepsTheClaimAndWritesAFreshRow(t *testing.T) { waker, holder := a, sb if tt.adopt { c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") waker = b holder, _ = b.byID(sb.ID) } @@ -509,7 +488,7 @@ func TestReplicaFailedCaptureKeepsTheOldReplicaAndRow(t *testing.T) { first := a.replicaOf(sb) before, _, _ := c.reg.Get(t.Context(), sb.ID) - a.eng.(*fakeEngine).snapSaveErr = errors.New("pause failed") + fakeOf(a).snapSaveErr = errors.New("pause failed") if err := a.replicate(t.Context(), sb); err == nil { t.Fatal("a round whose capture failed reported success") } @@ -529,7 +508,7 @@ func TestReplicaRegistryOutagePausesRoundsWithoutACapture(t *testing.T) { c := newReplicaCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) - eng := a.eng.(*fakeEngine) + eng := fakeOf(a) c.reg.SetDown(true) if err := a.replicate(t.Context(), sb); !errors.Is(err, ErrArchiveStoreDown) { @@ -549,17 +528,14 @@ func TestReplicaPassSnapshotsOnlyDueRunningClaims(t *testing.T) { synctest.Test(t, func(t *testing.T) { c := newReplicaCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) - a.liftFence() running := mustClaim(t, a, testKey) paused := mustClaim(t, a, testKey) withVolume := mustClaim(t, a, testKey) - if err := a.Hibernate(t.Context(), paused.ID, Cred{Token: paused.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, paused) a.mu.Lock() withVolume.Volumes = []types.Volume{{Name: "data"}} a.mu.Unlock() - eng := a.eng.(*fakeEngine) + eng := fakeOf(a) base := len(eng.snapSaves) a.replicateOnce(t.Context()) @@ -591,9 +567,7 @@ func TestAFencedNodeNeitherHibernatesNorWakesAReplicatedClaim(t *testing.T) { a := c.node(t, "a", t.TempDir(), keepAnHour) sb, asleep := mustClaim(t, a, testKey), mustClaim(t, a, testKey) replicateNow(t, a, sb) - if err := a.Hibernate(t.Context(), asleep.ID, Cred{Token: asleep.Token}); err != nil { - t.Fatalf("hibernate before the fence: %v", err) - } + hibernateClaim(t, a, asleep) a.tripFence() if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); !errors.Is(err, ErrLeaseLost) { t.Errorf("hibernate while fenced: %v, want ErrLeaseLost", err) @@ -648,7 +622,7 @@ func TestReplicaPassSkipsWhileFenced(t *testing.T) { time.Sleep(replicateEvery * time.Second) a.replicateOnce(t.Context()) synctest.Wait() - if a.replicaOf(sb) != "" || len(a.eng.(*fakeEngine).snapSaves) != 0 { + if a.replicaOf(sb) != "" || len(fakeOf(a).snapSaves) != 0 { t.Error("a fenced node replicated") } }) @@ -657,14 +631,11 @@ func TestReplicaPassSkipsWhileFenced(t *testing.T) { func TestReplicaPassRetiresAReplicaWhenThePoolTurnsItOff(t *testing.T) { c := newReplicaCluster(t) a := c.node(t, "a", t.TempDir(), keepAnHour) - a.liftFence() sb := mustClaim(t, a, testKey) replicateNow(t, a, sb) ck := a.replicaOf(sb) - view := *a.view.Load() - view.poolReplicate = nil - a.view.Store(&view) + disableReplication(a) a.replicateOnce(t.Context()) waitFor(t, func() bool { return !a.replicating.Load() }) if _, ok, _ := c.reg.Get(t.Context(), sb.ID); ok || a.replicaOf(sb) != "" || ckExists(t, a, ck) { @@ -694,8 +665,8 @@ func TestReplicaReleaseDropsTheRowTheReplicaAndTheVM(t *testing.T) { if err := a.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { t.Fatalf("release: %v", err) } - if _, ok, _ := c.reg.Get(t.Context(), sb.ID); ok || ckExists(t, a, ck) || !a.eng.(*fakeEngine).removed(vm) { - t.Errorf("row present %v, replica kept %v, VM removed %v, want the row and replica gone and the VM removed", ok, ckExists(t, a, ck), a.eng.(*fakeEngine).removed(vm)) + if _, ok, _ := c.reg.Get(t.Context(), sb.ID); ok || ckExists(t, a, ck) || !fakeOf(a).removed(vm) { + t.Errorf("row present %v, replica kept %v, VM removed %v, want the row and replica gone and the VM removed", ok, ckExists(t, a, ck), fakeOf(a).removed(vm)) } assertReleasedForGood(t, c, sb) } @@ -707,9 +678,7 @@ func TestReplicaHibernateKeepsTheRowOnTheLastReplica(t *testing.T) { replicateNow(t, a, sb) before, _, _ := c.reg.Get(t.Context(), sb.ID) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) after, ok, _ := c.reg.Get(t.Context(), sb.ID) if !ok || after.Epoch != before.Epoch || after.Checkpoint != before.Checkpoint || !ckExists(t, a, before.Checkpoint) { t.Errorf("row %+v present %v after the hibernate, want it untouched on %s", after, ok, before.Checkpoint) @@ -764,7 +733,7 @@ func TestReplicaExpiryDropsTheRowThenTheClaim(t *testing.T) { t.Fatal("the first reap destroyed the claim before its row was dropped") } a.reapOnce(t.Context()) - waitFor(t, func() bool { return a.eng.(*fakeEngine).removed(vm) }) + waitFor(t, func() bool { return fakeOf(a).removed(vm) }) if _, held := a.byID(sb.ID); held { t.Error("the second reap kept the claim") } @@ -809,7 +778,14 @@ func TestReplicaRestartAdoptsTheRowsCheckpointOfAnInterruptedRound(t *testing.T) if err := a.FlushClaims(); err != nil { t.Fatalf("flush: %v", err) } - second := interruptReplica(t, a, sb) + interrupted, _, publishErr := a.publishCheckpoint(t.Context(), sb, store.CheckpointID(randHex(8)), "replica", sb.Tenant, true) + if publishErr != nil { + t.Fatalf("publish: %v", publishErr) + } + if _, putErr := a.putReplica(t.Context(), sb, interrupted.ID); putErr != nil { + t.Fatalf("put: %v", putErr) + } + second := interrupted.ID restarted := c.restart(t, a) if err := restarted.Reconcile(t.Context()); err != nil { @@ -862,9 +838,7 @@ func TestAReplicaRowWithNoClaimLeftStaysAdoptable(t *testing.T) { restarted := restartWithoutVM(t, c, a, sb, keepAnHour) restarted.liftFence() restarted.sweepOrphanArchives(t.Context()) - if adopted, err := restarted.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Errorf("adopt inside retention: %t %v", adopted, err) - } + mustAdopt(t, restarted, sb, "adopt inside retention") } func TestAReplicaRowOfAKeepForeverPoolKeepsNoDeadline(t *testing.T) { @@ -892,7 +866,7 @@ func TestReconcileWithTheRegistryDownRecordsOnlyTheEndNoOtherNodeCanHold(t *test if err := a.FlushClaims(); err != nil { t.Fatalf("flush: %v", err) } - eng := a.eng.(*fakeEngine) + eng := fakeOf(a) delete(eng.vms, replicated.VMName) delete(eng.vms, plain.VMName) c.reg.SetDown(true) @@ -920,9 +894,7 @@ func TestReplicaRestartDropsAClaimAdoptedElsewhere(t *testing.T) { t.Fatalf("flush: %v", err) } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") c.renew(t, "a") restarted := c.restart(t, a) @@ -947,9 +919,7 @@ func TestReplicaRestartDropsAClaimTheAdopterWokeAndFinished(t *testing.T) { t.Fatalf("flush: %v", err) } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") if _, _, _, err := b.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("adopter's wake: %v", err) } @@ -1112,13 +1082,9 @@ func TestReplicaAdoptionWakesTheClaimFromTheLastReplica(t *testing.T) { replicateNow(t, a, sb) ck := a.replicaOf(sb) - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || adopted { - t.Fatalf("adopt while a's lease is live: %v %v, want refused", adopted, err) - } + mustNotAdopt(t, b, sb, "adopt while a's lease is live") c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt after the lease expired: %v %v, want adopted", adopted, err) - } + mustAdopt(t, b, sb, "adopt after the lease expired") moved, _ := b.byID(sb.ID) if moved.ArchiveCk != ck { t.Fatalf("adopted claim names %q, want the replica %q", moved.ArchiveCk, ck) @@ -1134,7 +1100,7 @@ func TestReplicaAdoptionWakesTheClaimFromTheLastReplica(t *testing.T) { if got, want := usageEventsOf(t, b, sb.ID), "adopt,unarchive"; got != want { t.Errorf("events on the adopter %s, want %s", got, want) } - if got := b.eng.(*fakeEngine).cloneCount(); got != 1 { + if got := fakeOf(b).cloneCount(); got != 1 { t.Errorf("the wake cloned %d VMs, want the one restored from the replica", got) } if _, ok, _ := c.reg.Get(t.Context(), sb.ID); ok || ckExists(t, b, ck) { @@ -1142,134 +1108,13 @@ func TestReplicaAdoptionWakesTheClaimFromTheLastReplica(t *testing.T) { } } -func TestReplicaArchiveRetriedAfterARegistryFaultKeepsTheClaim(t *testing.T) { - for _, tt := range []struct { - name string - reg func(*failoverCluster) *hookRegistry - }{ - {"the replace failed", func(c *failoverCluster) *hookRegistry { return &hookRegistry{Registry: c.reg, failPut: true} }}, - {"the replace landed and its reply was lost", func(c *failoverCluster) *hookRegistry { - return &hookRegistry{Registry: c.reg, losePutReply: true} - }}, - } { - t.Run(tt.name, func(t *testing.T) { - c := newReplicaCluster(t) - a := c.node(t, "a", t.TempDir(), keepAnHour) - sb := mustClaim(t, a, testKey) - replicateNow(t, a, sb) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } - a.archives = tt.reg(c) - if err := a.archive(t.Context(), sb); err == nil { - t.Fatal("archive under the fault succeeded") - } - a.archives = c.reg - if err := a.archive(t.Context(), sb); err != nil { - t.Fatalf("retried archive: %v", err) - } - if _, held := a.byID(sb.ID); !held || sb.ArchiveCk == "" { - t.Fatalf("claim held=%v archive ck %q, want it archived", held, sb.ArchiveCk) - } - rec, ok, err := c.reg.Get(t.Context(), sb.ID) - if err != nil || !ok || rec.Consumed || rec.Checkpoint != sb.ArchiveCk || rec.Epoch != sb.ArchiveEpoch { - t.Errorf("row %+v ok=%v err=%v, want it live on %s at epoch %d", rec, ok, err, sb.ArchiveCk, sb.ArchiveEpoch) - } - }) - } -} - -func TestReplicaRestartMidArchiveKeepsTheClaim(t *testing.T) { - for _, tt := range []struct { - name string - landed bool - }{ - {"the archive row never landed", false}, - {"the archive row landed", true}, - } { - t.Run(tt.name, func(t *testing.T) { - c := newReplicaCluster(t) - dir := t.TempDir() - a := c.node(t, "a", dir, keepAnHour) - sb := mustClaim(t, a, testKey) - vm := lockedVMName(sb) - replicateNow(t, a, sb) - replica := sb.ReplicaCk - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } - interruptArchive(t, a, sb, tt.landed) - eng := a.eng.(*fakeEngine) - - restarted := c.nodeOn(t, eng, "a", dir, keepAnHour) - if err := restarted.Reconcile(t.Context()); err != nil { - t.Fatalf("reconcile: %v", err) - } - rec, ok, err := c.reg.Get(t.Context(), sb.ID) - if err != nil || !ok || rec.Consumed || rec.Checkpoint != replica { - t.Errorf("row %+v ok=%v err=%v, want it live on the replica %s", rec, ok, err, replica) - } - c.renew(t, "a") - restarted.unfence(t.Context(), time.Now()) - held, kept := restarted.byID(sb.ID) - if !kept || eng.removed(vm) { - t.Fatalf("restart dropped a claim nobody adopted: held=%v VM removed=%v", kept, eng.removed(vm)) - } - if err = restarted.archive(t.Context(), held); err != nil { - t.Fatalf("archive after the restart: %v", err) - } - if rec, ok, err = c.reg.Get(t.Context(), sb.ID); err != nil || !ok || rec.Checkpoint != held.ArchiveCk { - t.Errorf("row %+v ok=%v err=%v, want it on %s", rec, ok, err, held.ArchiveCk) - } - }) - } -} - -func TestReplicaArchiveWhoseJournalWriteFailsKeepsTheClaimAcrossARestart(t *testing.T) { - c := newReplicaCluster(t) - dir := t.TempDir() - a := c.node(t, "a", dir, keepAnHour) - sb := mustClaim(t, a, testKey) - vm := lockedVMName(sb) - replicateNow(t, a, sb) - replica := sb.ReplicaCk - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } - broken := filepath.Join(t.TempDir(), "gone", "claims.json") - a.archives = &hookRegistry{Registry: c.reg, beforePut: func() { a.store.path = broken }} - if err := a.archive(t.Context(), sb); err == nil { - t.Fatal("archive with an unwritable journal succeeded") - } - if rec, ok, err := c.reg.Get(t.Context(), sb.ID); err != nil || !ok || rec.Consumed || rec.Owner != "a" { - t.Errorf("row %+v ok=%v err=%v, want this node's live row", rec, ok, err) - } - eng := a.eng.(*fakeEngine) - - restarted := c.nodeOn(t, eng, "a", dir, keepAnHour) - if err := restarted.Reconcile(t.Context()); err != nil { - t.Fatalf("reconcile: %v", err) - } - rec, ok, err := c.reg.Get(t.Context(), sb.ID) - if err != nil || !ok || rec.Consumed || rec.Checkpoint != replica { - t.Errorf("row %+v ok=%v err=%v, want it live on the replica %s", rec, ok, err, replica) - } - c.renew(t, "a") - restarted.unfence(t.Context(), time.Now()) - if _, kept := restarted.byID(sb.ID); !kept || eng.removed(vm) { - t.Fatalf("restart dropped a claim nobody adopted: held=%v VM removed=%v", kept, eng.removed(vm)) - } -} - func TestReplicaEnvRotationOfAHibernatedClaimReachesTheAdopter(t *testing.T) { c := newReplicaCluster(t) a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) patchHostOnly(t, a, sb, "old") replicateNow(t, a, sb) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) replica := sb.ReplicaCk patchHostOnly(t, a, sb, "new") @@ -1285,9 +1130,7 @@ func TestReplicaEnvRotationOfAHibernatedClaimReachesTheAdopter(t *testing.T) { } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") held, ok := b.byID(sb.ID) if !ok { t.Fatal("adopter has no claim") @@ -1302,11 +1145,11 @@ func TestReplicaEnvWriteOfARunningClaimLeavesTheRowToTheRound(t *testing.T) { a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) replicateNow(t, a, sb) - reg := &replaceFailRegistry{Registry: c.reg, fail: true} + reg := &hookRegistry{Registry: c.reg, failPut: true} a.archives = reg patchHostOnly(t, a, sb, "new") - if reg.calls != 0 { - t.Errorf("%d registry replaces for a running replicated claim, want 0", reg.calls) + if reg.replaces != 0 { + t.Errorf("%d registry replaces for a running replicated claim, want 0", reg.replaces) } } @@ -1315,9 +1158,7 @@ func TestReplicaRenewOfAHibernatedClaimReachesTheAdopter(t *testing.T) { a, b := c.node(t, "a", t.TempDir(), keepAnHour), c.node(t, "b", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) replicateNow(t, a, sb) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, a, sb) if _, err := a.Renew(t.Context(), sb.ID, Cred{Token: sb.Token}, 2*time.Hour, types.ExpireArchive); err != nil { t.Fatalf("renew: %v", err) @@ -1327,9 +1168,7 @@ func TestReplicaRenewOfAHibernatedClaimReachesTheAdopter(t *testing.T) { } c.expire("a") - if adopted, err := b.AdoptArchived(t.Context(), sb.ID, sb.Token); err != nil || !adopted { - t.Fatalf("adopt: %v %v", adopted, err) - } + mustAdopt(t, b, sb, "adopt") held, ok := b.byID(sb.ID) if !ok { t.Fatal("adopter has no claim") @@ -1344,10 +1183,8 @@ func TestReplicaRenewOfAHibernatedClaimWithTheRegistryDownIsLoudAndARetryRepairs a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) replicateNow(t, a, sb) - if err := a.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } - reg := &replaceFailRegistry{Registry: c.reg, fail: true} + hibernateClaim(t, a, sb) + reg := &hookRegistry{Registry: c.reg, failPut: true} a.archives = reg if _, err := a.Renew(t.Context(), sb.ID, Cred{Token: sb.Token}, 2*time.Hour, types.ExpireArchive); !errors.Is(err, archives.ErrUnavailable) { @@ -1356,7 +1193,7 @@ func TestReplicaRenewOfAHibernatedClaimWithTheRegistryDownIsLoudAndARetryRepairs if got := rowClaim(t, c, sb); got.OnExpire == types.ExpireArchive { t.Error("row took the renew while the registry was down") } - reg.fail = false + reg.failPut = false if _, err := a.Renew(t.Context(), sb.ID, Cred{Token: sb.Token}, 2*time.Hour, types.ExpireArchive); err != nil { t.Fatalf("resend: %v", err) } @@ -1370,13 +1207,105 @@ func TestReplicaRenewOfARunningClaimLeavesTheRowToTheRound(t *testing.T) { a := c.node(t, "a", t.TempDir(), keepAnHour) sb := mustClaim(t, a, testKey) replicateNow(t, a, sb) - reg := &replaceFailRegistry{Registry: c.reg, fail: true} + reg := &hookRegistry{Registry: c.reg, failPut: true} a.archives = reg if _, err := a.Renew(t.Context(), sb.ID, Cred{Token: sb.Token}, 2*time.Hour, types.ExpireArchive); err != nil { t.Fatalf("renew: %v", err) } - if reg.calls != 0 { - t.Errorf("%d registry replaces for a running replicated claim, want 0", reg.calls) + if reg.replaces != 0 { + t.Errorf("%d registry replaces for a running replicated claim, want 0", reg.replaces) + } +} + +func TestReplicaArchiveRetriedAfterARegistryFaultKeepsTheClaim(t *testing.T) { + for _, tt := range []struct { + name string + reg func(*failoverCluster) *hookRegistry + }{ + {"the replace failed", func(c *failoverCluster) *hookRegistry { return &hookRegistry{Registry: c.reg, failPut: true} }}, + {"the replace landed and its reply was lost", func(c *failoverCluster) *hookRegistry { + return &hookRegistry{Registry: c.reg, losePutReply: true} + }}, + } { + t.Run(tt.name, func(t *testing.T) { + c := newReplicaCluster(t) + a := c.node(t, "a", t.TempDir(), keepAnHour) + sb := mustClaim(t, a, testKey) + replicateNow(t, a, sb) + hibernateClaim(t, a, sb) + a.archives = tt.reg(c) + if err := a.archive(t.Context(), sb); err == nil { + t.Fatal("archive under the fault succeeded") + } + a.archives = c.reg + if err := a.archive(t.Context(), sb); err != nil { + t.Fatalf("retried archive: %v", err) + } + if _, held := a.byID(sb.ID); !held || sb.ArchiveCk == "" { + t.Fatalf("claim held=%v archive ck %q, want it archived", held, sb.ArchiveCk) + } + rec, ok, err := c.reg.Get(t.Context(), sb.ID) + if err != nil || !ok || rec.Consumed || rec.Checkpoint != sb.ArchiveCk || rec.Epoch != sb.ArchiveEpoch { + t.Errorf("row %+v ok=%v err=%v, want it live on %s at epoch %d", rec, ok, err, sb.ArchiveCk, sb.ArchiveEpoch) + } + }) + } +} + +func TestReplicaRestartMidArchiveKeepsTheClaim(t *testing.T) { + for _, tt := range []struct { + name string + interrupt func(t *testing.T, c *failoverCluster, a *Manager, sb *types.Sandbox) + }{ + {"the archive row never landed", func(t *testing.T, _ *failoverCluster, a *Manager, sb *types.Sandbox) { + interruptArchive(t, a, sb, false) + }}, + {"the archive row landed", func(t *testing.T, _ *failoverCluster, a *Manager, sb *types.Sandbox) { + interruptArchive(t, a, sb, true) + }}, + {"the journal write failed", func(t *testing.T, c *failoverCluster, a *Manager, sb *types.Sandbox) { + a.archives = &hookRegistry{Registry: c.reg, beforePut: func() { breakStore(t, a) }} + if err := a.archive(t.Context(), sb); err == nil { + t.Fatal("archive with an unwritable journal succeeded") + } + if rec, ok, err := c.reg.Get(t.Context(), sb.ID); err != nil || !ok || rec.Consumed || rec.Owner != "a" { + t.Errorf("row %+v ok=%v err=%v, want this node's live row", rec, ok, err) + } + }}, + } { + t.Run(tt.name, func(t *testing.T) { + c := newReplicaCluster(t) + dir := t.TempDir() + a := c.node(t, "a", dir, keepAnHour) + sb := mustClaim(t, a, testKey) + vm := lockedVMName(sb) + replicateNow(t, a, sb) + replica := sb.ReplicaCk + hibernateClaim(t, a, sb) + tt.interrupt(t, c, a, sb) + eng := fakeOf(a) + + restarted := c.nodeOn(t, eng, "a", dir, keepAnHour) + if err := restarted.Reconcile(t.Context()); err != nil { + t.Fatalf("reconcile: %v", err) + } + rec, ok, err := c.reg.Get(t.Context(), sb.ID) + if err != nil || !ok || rec.Consumed || rec.Checkpoint != replica { + t.Errorf("row %+v ok=%v err=%v, want it live on the replica %s", rec, ok, err, replica) + } + c.renew(t, "a") + restarted.unfence(t.Context(), time.Now()) + held, kept := restarted.byID(sb.ID) + if !kept || eng.removed(vm) { + t.Fatalf("restart dropped a claim nobody adopted: held=%v VM removed=%v", kept, eng.removed(vm)) + } + if err = restarted.archive(t.Context(), held); err != nil { + t.Fatalf("archive after the restart: %v", err) + } + if rec, ok, err = c.reg.Get(t.Context(), sb.ID); err != nil || !ok || rec.Checkpoint != held.ArchiveCk { + t.Errorf("row %+v ok=%v err=%v, want it on %s", rec, ok, err, held.ArchiveCk) + } + }) } } @@ -1385,7 +1314,7 @@ func restartWithoutVM(t *testing.T, c *failoverCluster, a *Manager, sb *types.Sa if err := a.FlushClaims(); err != nil { t.Fatalf("flush: %v", err) } - eng := a.eng.(*fakeEngine) + eng := fakeOf(a) delete(eng.vms, sb.VMName) restarted := c.nodeOn(t, eng, a.nodeID, a.dataDir, deleteAfter) if err := restarted.Reconcile(t.Context()); err != nil { @@ -1393,29 +1322,3 @@ func restartWithoutVM(t *testing.T, c *failoverCluster, a *Manager, sb *types.Sa } return restarted } - -func newReplicaCluster(t *testing.T) *failoverCluster { - t.Helper() - c := newFailoverCluster(t) - c.replicate = replicateEvery - return c -} - -func replicateNow(t *testing.T, m *Manager, sb *types.Sandbox) { - t.Helper() - if err := m.replicate(t.Context(), sb); err != nil { - t.Fatalf("replicate %s: %v", sb.ID, err) - } -} - -func interruptReplica(t *testing.T, m *Manager, sb *types.Sandbox) string { - t.Helper() - ck, _, err := m.publishCheckpoint(t.Context(), sb, store.CheckpointID(randHex(8)), "replica", sb.Tenant, true) - if err != nil { - t.Fatalf("publish: %v", err) - } - if _, err = m.putReplica(t.Context(), sb, ck.ID); err != nil { - t.Fatalf("put: %v", err) - } - return ck.ID -} diff --git a/sandboxd/pool/setpools.go b/sandboxd/pool/setpools.go index f22558d64..2b4f7913f 100644 --- a/sandboxd/pool/setpools.go +++ b/sandboxd/pool/setpools.go @@ -84,7 +84,7 @@ func (m *Manager) applyPools(ctx context.Context, desired map[types.PoolKey]conf p.removed = !ok trim = append(trim, p.trimWarm(p.effectiveTarget(now))...) if !ok && p.settled() { - m.dropPool(ctx, p) + m.dropPoolLocked(ctx, p) } } for key, spec := range desired { diff --git a/sandboxd/pool/stale_claim_test.go b/sandboxd/pool/stale_claim_test.go index 885069aa7..6559045df 100644 --- a/sandboxd/pool/stale_claim_test.go +++ b/sandboxd/pool/stale_claim_test.go @@ -31,7 +31,7 @@ func TestStaleClaimOperationsPreserveTheCurrentClaim(t *testing.T) { restore() } } - for _, tc := range []struct { + for _, tt := range []struct { name string newManager func(t *testing.T) *Manager prepare func(t *testing.T, m *Manager, sb *types.Sandbox) @@ -48,7 +48,7 @@ func TestStaleClaimOperationsPreserveTheCurrentClaim(t *testing.T) { if !errors.Is(err, ErrUnknownSandbox) { t.Errorf("stale Hibernate: %v, want ErrUnknownSandbox", err) } - if m.eng.(*fakeEngine).hibernateCount() != 0 { + if fakeOf(m).hibernateCount() != 0 { t.Error("stale Hibernate reached the engine") } }, @@ -63,24 +63,24 @@ func TestStaleClaimOperationsPreserveTheCurrentClaim(t *testing.T) { newManager: func(t *testing.T) *Manager { return newFailoverCluster(t).node(t, "a", t.TempDir(), keepAnHour) }, prepare: func(t *testing.T, m *Manager, sb *types.Sandbox) { if err := hibernate(t, m, sb); err != nil { - t.Fatal(err) + t.Fatalf("hibernate: %v", err) } }, run: rollback(func(t *testing.T, m *Manager, sb *types.Sandbox) error { return m.archive(t.Context(), sb) }), }, } { for _, replace := range []bool{false, true} { - t.Run(fmt.Sprintf("%s/replace=%t", tc.name, replace), func(t *testing.T) { - m := tc.newManager(t) + t.Run(fmt.Sprintf("%s/replace=%t", tt.name, replace), func(t *testing.T) { + m := tt.newManager(t) sb := mustClaim(t, m, testKey) ck, err := m.Checkpoint(t.Context(), sb.ID, Cred{Token: sb.Token}, "", "") if err != nil { - t.Fatal(err) + t.Fatalf("checkpoint: %v", err) } - if tc.prepare != nil { - tc.prepare(t, m, sb) + if tt.prepare != nil { + tt.prepare(t, m, sb) } - tc.run(t, m, sb, func() { + tt.run(t, m, sb, func() { m.mu.Lock() defer m.mu.Unlock() m.dropClaimLocked(sb) @@ -92,11 +92,11 @@ func TestStaleClaimOperationsPreserveTheCurrentClaim(t *testing.T) { } }) if err = m.store.commit(m.store.mark()); err != nil { - t.Fatal(err) + t.Fatalf("commit the journal: %v", err) } loaded, err := newClaimStore(m.dataDir, false).load() if err != nil { - t.Fatal(err) + t.Fatalf("load journal: %v", err) } if replace { if loaded[sb.ID] == nil || loaded[sb.ID].ReplicaCk != ck.ID || loaded[sb.ID].PendingSnap != "" { @@ -106,7 +106,7 @@ func TestStaleClaimOperationsPreserveTheCurrentClaim(t *testing.T) { t.Errorf("stale operation resurrected released claim on disk: %+v", loaded[sb.ID]) } if err = m.markArchiveCk(ck.ID); err != nil { - t.Fatal(err) + t.Fatalf("mark the archive checkpoint: %v", err) } m.retryArchiveDeletes(t.Context()) if exists := ckExists(t, m, ck.ID); exists != replace { diff --git a/sandboxd/pool/stats_test.go b/sandboxd/pool/stats_test.go index ddfb142bc..3674d0ea9 100644 --- a/sandboxd/pool/stats_test.go +++ b/sandboxd/pool/stats_test.go @@ -9,7 +9,7 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/config" ) -func TestStatsUnknownSandbox(t *testing.T) { +func TestStatsOfAnUnknownIDIsNotOK(t *testing.T) { m := newTestManager(t, newFakeEngine()) if _, ok := m.Stats(t.Context(), "sb_missing"); ok { t.Error("Stats on an unknown id, want ok=false") @@ -55,9 +55,7 @@ func TestStatsHibernatedIsUnmeasured(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) before := eng.listCalls() st, ok := m.Stats(t.Context(), sb.ID) diff --git a/sandboxd/pool/storage_test.go b/sandboxd/pool/storage_test.go index 6f0291b58..02bc546f9 100644 --- a/sandboxd/pool/storage_test.go +++ b/sandboxd/pool/storage_test.go @@ -11,22 +11,21 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/types" ) +const unsizedStamp = "\x00\x00" + func TestGoldenBuildSizesItsDisk(t *testing.T) { eng := newFakeEngine() - other := types.PoolKey{Template: "py:3.12", Net: types.NetNone, Size: types.SizeSmall} + other := tplKey("py:3.12") m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Storage: "40G"}, config.PoolSpec{PoolKey: other, Warm: 1}) for _, key := range []types.PoolKey{testKey, other} { - final := filepath.Join(m.goldensDir(), key.Hash()) - if err := m.buildGoldenSteps(t.Context(), m.view.Load(), key, "sbx-gb-"+key.Hash(), "snap", final, ""); err != nil { - t.Fatalf("buildGoldenSteps %s: %v", key.Template, err) - } + buildGoldenAt(t, m, key, "") } if !slices.Equal(eng.coldStorage, []string{"40G", ""}) { t.Errorf("cold boot disks %v, want the sized pool at 40 and the other at cocoon's default", eng.coldStorage) } - if got := m.goldenStamp(m.view.Load(), other, ""); got != strings.Join([]string{"", "", ""}, "\x00") { + if got := m.goldenStamp(m.view.Load(), other, ""); got != unsizedStamp { t.Errorf("unsized stamp %q changed, so goldens built before the field would rebuild", got) } if !strings.Contains(m.goldenStamp(m.view.Load(), testKey, ""), "storage=42949672960") { @@ -48,7 +47,7 @@ func TestSizedPoolRebuildsAnUnsizedGolden(t *testing.T) { if err := os.MkdirAll(g, 0o750); err != nil { t.Fatalf("seed golden: %v", err) } - if err := os.WriteFile(g+goldenStampSuffix, []byte(strings.Join([]string{"", "", ""}, "\x00")), 0o644); err != nil { + if err := os.WriteFile(g+goldenStampSuffix, []byte(unsizedStamp), 0o644); err != nil { t.Fatalf("seed stamp: %v", err) } @@ -64,7 +63,7 @@ func TestSizedPoolRebuildsAnUnsizedGolden(t *testing.T) { func TestNoBalloonRebuildsABalloonGolden(t *testing.T) { spec := config.PoolSpec{PoolKey: testKey, Warm: 1} m := newTestManager(t, newFakeEngine(), spec) - seedGolden(t, m, "") + seedGolden(t, m, testKey, "") m.view.Store(newConfigView(&config.Config{NoBalloon: true, Pools: []config.PoolSpec{spec}}, testSecrets(t))) m.mu.Lock() diff --git a/sandboxd/pool/telemetry_test.go b/sandboxd/pool/telemetry_test.go index aec9f1f24..557a12daa 100644 --- a/sandboxd/pool/telemetry_test.go +++ b/sandboxd/pool/telemetry_test.go @@ -25,9 +25,7 @@ func TestUsageJournalRecordsLifecycle(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("Hibernate: %v", err) - } + hibernateClaim(t, m, sb) if _, _, _, err := m.WakeAgentSocket(t.Context(), sb.ID, sb.Token); err != nil { t.Fatalf("Wake: %v", err) } @@ -93,9 +91,7 @@ func TestClaimLatencyCountsPoolClaimsByTierOnly(t *testing.T) { if _, err = m.ClaimCheckpoint(t.Context(), ckpt.ID, ClaimOptions{TTL: time.Hour}); err != nil { t.Fatalf("claim checkpoint: %v", err) } - cold := testKey - cold.Template = "cold:1" - if _, err = m.ClaimProvision(t.Context(), cold, ClaimOptions{TTL: time.Hour}); err != nil { + if _, err = m.ClaimProvision(t.Context(), tplKey("cold:1"), ClaimOptions{TTL: time.Hour}); err != nil { t.Fatalf("cold claim: %v", err) } @@ -151,14 +147,14 @@ func TestQuotaRefusesClaimsPastCap(t *testing.T) { m.maxClaims = 1 first := mustClaim(t, m, testKey) - if _, err := claimAny(t.Context(), m, testKey, time.Hour); !errors.Is(err, ErrQuota) { + if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{TTL: time.Hour}); !errors.Is(err, ErrQuota) { t.Fatalf("claim past cap: %v, want ErrQuota", err) } if err := m.Release(t.Context(), first.ID, Cred{Token: first.Token}); err != nil { t.Fatalf("Release: %v", err) } - if _, err := claimAny(t.Context(), m, testKey, time.Hour); err != nil { + if _, err := claimAny(t.Context(), m, testKey, ClaimOptions{TTL: time.Hour}); err != nil { t.Errorf("claim after release: %v", err) } } @@ -168,10 +164,7 @@ func TestTenantQuotaBindsPerTenant(t *testing.T) { m := newTestManager(t, eng) setTenantCaps(t, m, map[string]int{"acme": 1, "beta": 2}) - first, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("acme claim: %v", err) - } + first := claimTenant(t, m, "acme") if first.Tenant != "acme" { t.Errorf("tenant %q, want acme", first.Tenant) } @@ -192,18 +185,13 @@ func TestTenantQuotaBindsPerTenant(t *testing.T) { if err := m.Release(t.Context(), first.ID, Cred{Token: first.Token}); err != nil { t.Fatalf("Release: %v", err) } - if _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}); err != nil { - t.Errorf("acme claim after release: %v", err) - } + claimTenant(t, m, "acme") } func TestTenantStampedInJournals(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng) - sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("claim: %v", err) - } + sb := claimTenant(t, m, "acme") raw, err := os.ReadFile(filepath.Join(m.dataDir, "claims.json")) if err != nil { @@ -243,18 +231,13 @@ func TestEveryClaimScopedUsageEventCarriesTheTenant(t *testing.T) { if err := m.PutTenant(ctx, tenants.TenantSpec{Name: "acme", Token: "acme-tok"}); err != nil { t.Fatalf("put tenant: %v", err) } - sb, err := m.ClaimProvision(ctx, testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("claim: %v", err) - } + sb := claimTenant(t, m, "acme") cred := Cred{Token: sb.Token} - if _, err = m.Renew(ctx, sb.ID, cred, time.Hour, ""); err != nil { + if _, err := m.Renew(ctx, sb.ID, cred, time.Hour, ""); err != nil { t.Fatalf("renew: %v", err) } - if err = m.Hibernate(ctx, sb.ID, cred); err != nil { - t.Fatalf("hibernate: %v", err) - } - if _, _, _, err = m.WakeAgentSocket(ctx, sb.ID, sb.Token); err != nil { + hibernateClaim(t, m, sb) + if _, _, _, err := m.WakeAgentSocket(ctx, sb.ID, sb.Token); err != nil { t.Fatalf("wake: %v", err) } children, err := m.Fork(ctx, sb.ID, cred, 1, time.Hour, "", "") @@ -310,9 +293,7 @@ func TestEveryEventThatMovesTheDeadlineStatesIt(t *testing.T) { t.Fatalf("renew: %v", err) } assertDeadline("renew") - if err := m.Hibernate(ctx, sb.ID, cred); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) if _, _, _, err := m.WakeAgentSocket(ctx, sb.ID, sb.Token); err != nil { t.Fatalf("wake: %v", err) } @@ -355,9 +336,7 @@ func TestArchiveEventPrecedesAWakeDuringItsTeardown(t *testing.T) { m := newTestManager(t, eng, archivePool(3600)) ctx := t.Context() sb := mustClaim(t, m, testKey) - if err := m.Hibernate(ctx, sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) stall := make(chan struct{}) eng.mu.Lock() eng.removeStall = stall @@ -382,15 +361,12 @@ func TestReconcileRecordsTheEndOfTheClaimsItDrops(t *testing.T) { eng := newFakeEngine() dir := t.TempDir() m := newTestManagerAt(t, eng, dir) - lost, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if err != nil { - t.Fatalf("claim: %v", err) - } + lost := claimTenant(t, m, "acme") kept := mustClaim(t, m, testKey) delete(eng.vms, lost.VMName) m2 := newTestManagerAt(t, eng, dir) - if err = m2.Reconcile(t.Context()); err != nil { + if err := m2.Reconcile(t.Context()); err != nil { t.Fatalf("Reconcile: %v", err) } if got, want := usageEventsOf(t, m2, lost.ID), "claim,reap"; got != want { @@ -410,9 +386,7 @@ func TestReconcileRecordsTheWakeACrashLost(t *testing.T) { dir := t.TempDir() m := newTestManagerAt(t, eng, dir) sb := mustClaim(t, m, testKey) - if err := m.Hibernate(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, sb) delete(eng.stopped, sb.VMName) m2 := newTestManagerAt(t, eng, dir) @@ -700,31 +674,3 @@ func TestARelayedFrameCannotForgeOrDropItsAuditLine(t *testing.T) { } } } - -func usageJournal(t *testing.T, m *Manager) []usage.Event { - t.Helper() - raw, err := os.ReadFile(filepath.Join(m.dataDir, usage.JournalFile)) - if err != nil { - t.Fatalf("read usage journal: %v", err) - } - var events []usage.Event - for line := range strings.SplitSeq(strings.TrimSpace(string(raw)), "\n") { - var ev usage.Event - if err := json.Unmarshal([]byte(line), &ev); err != nil { - t.Fatalf("bad journal line %q: %v", line, err) - } - events = append(events, ev) - } - return events -} - -func usageEventsOf(t *testing.T, m *Manager, id string) string { - t.Helper() - var events []string - for _, ev := range usageJournal(t, m) { - if ev.ID == id { - events = append(events, ev.Event) - } - } - return strings.Join(events, ",") -} diff --git a/sandboxd/pool/tenants_test.go b/sandboxd/pool/tenants_test.go index 403af0dd3..6339fbe9c 100644 --- a/sandboxd/pool/tenants_test.go +++ b/sandboxd/pool/tenants_test.go @@ -23,7 +23,6 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/outbound" "github.com/cocoonstack/sandbox/sandboxd/outbound/outboundtest" "github.com/cocoonstack/sandbox/sandboxd/tenants" - "github.com/cocoonstack/sandbox/sandboxd/tenants/file" tenantspg "github.com/cocoonstack/sandbox/sandboxd/tenants/pg" "github.com/cocoonstack/sandbox/sandboxd/tenants/tenantstest" "github.com/cocoonstack/sandbox/sandboxd/types" @@ -124,19 +123,9 @@ func TestRemovedTenantsClaimIsNeitherArchivedNorWokenNorForked(t *testing.T) { t.Fatalf("setup manager: %v", err) } putTenants(t, m, tenants.TenantSpec{Name: "acme", Token: "acme-tok"}) - claim := func() *types.Sandbox { - t.Helper() - sb, claimErr := m.ClaimProvision(t.Context(), testKey, ClaimOptions{TTL: time.Hour, Tenant: "acme"}) - if claimErr != nil { - t.Fatalf("claim: %v", claimErr) - } - return sb - } - archived, idle, running := claim(), claim(), claim() + archived, idle, running := claimTenant(t, m, "acme"), claimTenant(t, m, "acme"), claimTenant(t, m, "acme") mustArchive(t, m, archived) - if err = m.Hibernate(t.Context(), idle.ID, Cred{Token: idle.Token}); err != nil { - t.Fatalf("hibernate: %v", err) - } + hibernateClaim(t, m, idle) if err = m.DeleteTenant(t.Context(), "acme"); err != nil { t.Fatalf("DeleteTenant: %v", err) } @@ -272,7 +261,7 @@ func TestARuntimeTenantTakesItsEgressClassWithoutARestart(t *testing.T) { t.Fatalf("PutTenant %s: %v", spec.Name, err) } } - unpooled := types.PoolKey{Template: "promoted-name", Net: types.NetNone, Size: types.SizeSmall} + unpooled := tplKey("promoted-name") claimAs := func(token string) (*types.Sandbox, error) { r, rerr := m.TenantByToken(t.Context(), token) if rerr != nil { @@ -386,7 +375,7 @@ func TestTwoNodesOnOneMetaStoreShareTheTenantSet(t *testing.T) { if name, ok := tenantOf(t, b, "u1-tok"); !ok || name != "u-1" { t.Errorf("b resolves a's tenant: %q %v, want u-1 without a fan-out", name, ok) } - unpooled := types.PoolKey{Template: "promoted-name", Net: types.NetNone, Size: types.SizeSmall} + unpooled := tplKey("promoted-name") sb, err := a.ClaimProvision(t.Context(), unpooled, ClaimOptions{TTL: time.Hour, Tenant: "u-1"}) if err != nil { t.Fatalf("claim on a: %v", err) @@ -411,9 +400,12 @@ func TestTwoNodesOnOneMetaStoreShareTheTenantSet(t *testing.T) { func tenantOf(t *testing.T, m *Manager, token string) (string, bool) { t.Helper() r, err := m.TenantByToken(t.Context(), token) - if err != nil { + if errors.Is(err, ErrUnknownTenant) { return "", false } + if err != nil { + t.Fatalf("tenant by token: %v", err) + } return r.Name, true } @@ -426,39 +418,13 @@ func listTenants(t *testing.T, m *Manager) []TenantInfo { return page.Tenants } -func tenantManager(t *testing.T, dir string, tenants ...tenants.TenantSpec) *Manager { +func tenantManager(t *testing.T, dir string, specs ...tenants.TenantSpec) *Manager { t.Helper() cfg := &config.Config{DataDir: dir, APIToken: "root"} m, err := NewManager(t.Context(), cfg, newFakeEngine(), testSecrets(t), testBackends(t, cfg)) if err != nil { t.Fatalf("setup manager: %v", err) } - putTenants(t, m, tenants...) + putTenants(t, m, specs...) return m } - -func putTenants(t *testing.T, m *Manager, tenants ...tenants.TenantSpec) { - t.Helper() - if len(tenants) == 0 { - return - } - if err := m.SetTenants(t.Context(), tenants); err != nil { - t.Fatalf("set tenants: %v", err) - } -} - -func setTenantCaps(t *testing.T, m *Manager, caps map[string]int) { - t.Helper() - records := make([]tenants.TenantRecord, 0, len(caps)) - for name, limit := range caps { - records = append(records, tenants.TenantRecord{Name: name, TokenSHA256: tenants.TokenSHA256("tok-" + name), MaxClaims: limit}) - } - src, err := file.Open(t.Context(), t.TempDir(), "") - if err != nil { - t.Fatalf("tenant set: %v", err) - } - if _, err = src.Replace(t.Context(), records); err != nil { - t.Fatalf("tenant records: %v", err) - } - m.tenants = src -} diff --git a/sandboxd/pool/trim_test.go b/sandboxd/pool/trim_test.go index 9c8bbffca..c07ddb0ca 100644 --- a/sandboxd/pool/trim_test.go +++ b/sandboxd/pool/trim_test.go @@ -12,7 +12,7 @@ import ( func TestCaptureTrimRunsBeforeTheSnapOnlyForItsPool(t *testing.T) { eng := newFakeEngine() - other := types.PoolKey{Template: "py:3.12", Net: testKey.Net, Size: testKey.Size} + other := tplKey("py:3.12") m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, CaptureTrim: true}, config.PoolSpec{PoolKey: other, Warm: 1}) trimmed := mustClaim(t, m, testKey) diff --git a/sandboxd/pool/upstream_test.go b/sandboxd/pool/upstream_test.go index a42946e66..38c2d401b 100644 --- a/sandboxd/pool/upstream_test.go +++ b/sandboxd/pool/upstream_test.go @@ -12,7 +12,7 @@ import ( func TestUpstreamEnvIsAdmittedHostOnlyAndAllowed(t *testing.T) { m := upstreamManager(t, "127.0.0.1") - for name, tc := range map[string]struct { + for name, tt := range map[string]struct { v types.EnvVar ok bool }{ @@ -22,9 +22,9 @@ func TestUpstreamEnvIsAdmittedHostOnlyAndAllowed(t *testing.T) { "unlisted": {types.EnvVar{Value: "http://10.0.0.1:3128", Guest: hostOnly}, false}, "malformed": {types.EnvVar{Value: "ftp://127.0.0.1:21", Guest: hostOnly}, false}, } { - err := m.checkEnv(types.Env{"EGRESS_UPSTREAM": tc.v}) - if tc.ok != (err == nil) || (err != nil && !errors.Is(err, ErrBadEnv)) { - t.Errorf("%s: %v, want ok=%v", name, err, tc.ok) + err := m.checkEnv(types.Env{"EGRESS_UPSTREAM": tt.v}) + if tt.ok != (err == nil) || (err != nil && !errors.Is(err, ErrBadEnv)) { + t.Errorf("%s: %v, want ok=%v", name, err, tt.ok) } } editEgress(t, m, func(cfg *config.Config) { cfg.EgressUpstream = nil }) diff --git a/sandboxd/pool/volume.go b/sandboxd/pool/volume.go index 828ed76c8..400b48e42 100644 --- a/sandboxd/pool/volume.go +++ b/sandboxd/pool/volume.go @@ -227,7 +227,7 @@ func (m *Manager) applyVolume(ctx context.Context, sb *types.Sandbox, volume res // quiesceVolumes unmounts a claim's writable mounts in reverse; an unclean unmount keeps the marker. func (m *Manager) quiesceVolumes(ctx context.Context, sb *types.Sandbox) volumeTeardown { td := volumeTeardown{holds: heldVolumes(sb)} - if writableMounts(sb.Volumes) == 0 { + if !slices.ContainsFunc(sb.Volumes, needsUnmount) { return td } catalog := m.catalogSnapshot() @@ -334,16 +334,6 @@ func (m *Manager) checkResolvedLocked(volumes []resolvedVolume) error { return nil } -func writableMounts(volumes []types.Volume) int { - mounts := 0 - for _, volume := range volumes { - if needsUnmount(volume) { - mounts++ - } - } - return mounts -} - func needsUnmount(volume types.Volume) bool { return volume.RW() && !volume.AttachOnly } diff --git a/sandboxd/pool/volume_acl_test.go b/sandboxd/pool/volume_acl_test.go index f4136667f..c05568735 100644 --- a/sandboxd/pool/volume_acl_test.go +++ b/sandboxd/pool/volume_acl_test.go @@ -15,7 +15,7 @@ func TestRuntimeCatalogEmptyACLIsPublic(t *testing.T) { m := newVolumeManager(t, eng, nil) spec := types.VolumeRegistration{Path: writeVolumeImage(t, "home.img", "private"), Writable: true, Tenants: []string{}} if err := m.RegisterVolume(t.Context(), "home", spec); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } for _, current := range []*Manager{m, newVolumeManagerAt(t, eng, m.dataDir, nil)} { if got := current.Volumes("acme", nil); len(got) != 1 || got[0].Name != "home" { @@ -29,7 +29,7 @@ func TestRuntimeCatalogEmptyACLIsPublic(t *testing.T) { t.Fatalf("tenant attachment: %v", err) } if err = current.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatal(err) + t.Fatalf("Release: %v", err) } } } @@ -39,7 +39,7 @@ func TestInvalidStaticTenantListNeverBecomesPublic(t *testing.T) { file := filepath.Join(t.TempDir(), "config.json") raw := fmt.Sprintf(`{"pools":[],"volumes":[{"name":"data","path":%q,"tenants":["has space"]}]}`, path) if err := os.WriteFile(file, []byte(raw), 0o600); err != nil { - t.Fatal(err) + t.Fatalf("WriteFile: %v", err) } cfg, err := config.Load(file) if err != nil { diff --git a/sandboxd/pool/volume_attach_test.go b/sandboxd/pool/volume_attach_test.go index bb8ebe0df..fb27d12e3 100644 --- a/sandboxd/pool/volume_attach_test.go +++ b/sandboxd/pool/volume_attach_test.go @@ -112,12 +112,8 @@ func TestClaimProvisionAttachFailureKeepsHoldsUntilRemoval(t *testing.T) { claimErr <- err }() attaches.Wait() + vmName, vmCount := soleVM(eng) eng.mu.Lock() - vmCount := len(eng.vms) - var vmName string - for name := range eng.vms { - vmName = name - } eng.removeErrFor = vmName eng.attachRendezvous = nil eng.mu.Unlock() @@ -213,13 +209,7 @@ func TestFinalizeQuotaFailureKeepsHoldsUntilRemoval(t *testing.T) { claimErr <- err }() waitFor(t, func() bool { return slices.Contains(eng.volumeOpsLog(), "attach:scratch") }) - eng.mu.Lock() - vmCount := len(eng.vms) - var vmName string - for name := range eng.vms { - vmName = name - } - eng.mu.Unlock() + vmName, vmCount := soleVM(eng) filler, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{}) if err != nil { t.Fatalf("volume-less claim: %v", err) @@ -286,12 +276,7 @@ func TestFinalizeTenantQuotaFailureQuiescesAndUncountsTenant(t *testing.T) { claimErr <- err }() waitFor(t, func() bool { return slices.Contains(eng.volumeOpsLog(), "attach:scratch") }) - eng.mu.Lock() - var vmName string - for name := range eng.vms { - vmName = name - } - eng.mu.Unlock() + vmName, _ := soleVM(eng) if _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Tenant: "acme"}); err != nil { t.Fatalf("volume-less tenant claim: %v", err) } @@ -394,26 +379,11 @@ func TestQuiesceVolumesSkipsAttachOnlyEntries(t *testing.T) { } } -func TestWritableMountsCountsOnlyQuiescableEntries(t *testing.T) { - for _, tt := range []struct { - name string - volumes []types.Volume - want int - }{ - {"none", nil, 0}, - {"read-only mount", []types.Volume{{Name: "a", Mount: "/a"}}, 0}, - {"attach-only writer", []types.Volume{{Name: "a", Mode: types.VolumeModeRW, AttachOnly: true}}, 0}, - {"writable mount", []types.Volume{{Name: "a", Mount: "/a", Mode: types.VolumeModeRW}}, 1}, - {"mixed", []types.Volume{ - {Name: "a", Mount: "/a", Mode: types.VolumeModeRW}, - {Name: "b", Mode: types.VolumeModeRW, AttachOnly: true}, - {Name: "c", Mount: "/c"}, - }, 1}, - } { - t.Run(tt.name, func(t *testing.T) { - if got := writableMounts(tt.volumes); got != tt.want { - t.Errorf("writableMounts(%v)=%d, want %d", tt.volumes, got, tt.want) - } - }) +func soleVM(eng *fakeEngine) (name string, count int) { + eng.mu.Lock() + defer eng.mu.Unlock() + for vm := range eng.vms { + name = vm } + return name, len(eng.vms) } diff --git a/sandboxd/pool/volume_catalog_ordering_test.go b/sandboxd/pool/volume_catalog_ordering_test.go index 1eda02a31..66e67711c 100644 --- a/sandboxd/pool/volume_catalog_ordering_test.go +++ b/sandboxd/pool/volume_catalog_ordering_test.go @@ -17,11 +17,11 @@ func TestRuntimeCatalogFailedDeleteRetriesAbsence(t *testing.T) { m := newVolumeManager(t, eng, nil) spec := types.VolumeRegistration{Path: writeVolumeImage(t, "old", "bytes")} if err := m.RegisterVolume(t.Context(), "data", spec); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } restore := blockCatalogRename(t, m) if err := m.UnregisterVolume(t.Context(), "data"); !errors.Is(err, ErrVolumeCatalogUnavailable) { - t.Fatal(err) + t.Fatalf("UnregisterVolume: %v, want ErrVolumeCatalogUnavailable", err) } if got := m.VolumeNames(); len(got) != 0 { t.Fatalf("failed delete remained published: %v", got) @@ -35,7 +35,7 @@ func TestRuntimeCatalogFailedDeleteRetriesAbsence(t *testing.T) { } if !replace { if err := m.UnregisterVolume(t.Context(), "data"); err != nil { - t.Fatal(err) + t.Fatalf("UnregisterVolume: %v", err) } if got := newVolumeManagerAt(t, eng, m.dataDir, nil).VolumeNames(); len(got) != 0 { t.Fatalf("retry did not persist absence: %v", got) @@ -44,7 +44,7 @@ func TestRuntimeCatalogFailedDeleteRetriesAbsence(t *testing.T) { } spec.Path = writeVolumeImage(t, "new", "replacement") if err := m.RegisterVolume(t.Context(), "data", spec); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } restarted := newVolumeManagerAt(t, eng, m.dataDir, nil) resolved, err := restarted.resolveVolumes(t.Context(), "", []types.Volume{{Name: "data"}}) @@ -60,11 +60,11 @@ func TestRuntimeCatalogFailedPutIsNotReplayed(t *testing.T) { spec := types.VolumeRegistration{Path: writeVolumeImage(t, "data", "bytes")} restore := blockCatalogRename(t, m) if err := m.RegisterVolume(t.Context(), "data", spec); !errors.Is(err, ErrVolumeCatalogUnavailable) { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v, want ErrVolumeCatalogUnavailable", err) } restore() if err := m.RegisterVolume(t.Context(), "replacement", spec); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } if got := newVolumeManagerAt(t, newFakeEngine(), m.dataDir, nil).VolumeNames(); !slices.Equal(got, []string{"replacement"}) { t.Fatalf("failed put was replayed: %v", got) @@ -79,15 +79,15 @@ func TestRuntimeCatalogRenameSucceededBeforeDirectoryOpenFailed(t *testing.T) { spec := types.VolumeRegistration{Path: writeVolumeImage(t, "data", "bytes")} if remove { if err := m.RegisterVolume(t.Context(), "data", spec); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } } if err := os.Chmod(m.dataDir, 0o300); err != nil { - t.Fatal(err) + t.Fatalf("Chmod: %v", err) } restore := func() { if err := os.Chmod(m.dataDir, 0o700); err != nil { - t.Fatal(err) + t.Fatalf("Chmod: %v", err) } } t.Cleanup(restore) diff --git a/sandboxd/pool/volume_catalog_test.go b/sandboxd/pool/volume_catalog_test.go index b9f989eea..323a35f09 100644 --- a/sandboxd/pool/volume_catalog_test.go +++ b/sandboxd/pool/volume_catalog_test.go @@ -22,7 +22,7 @@ func TestRuntimeCatalogPersistsNormalizedSemantics(t *testing.T) { spec := types.VolumeRegistration{Path: filepath.Dir(path) + "/./runtime", Writable: true, Tenants: []string{"beta", "acme", "acme"}} before := m.catalogSnapshot() if err := m.RegisterVolume(t.Context(), "runtime", spec); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } if before["runtime"] != nil || spec.Tenants[0] != "beta" { t.Fatal("registration mutated a captured catalog or the caller's spec") @@ -33,7 +33,7 @@ func TestRuntimeCatalogPersistsNormalizedSemantics(t *testing.T) { } raw, err := os.ReadFile(filepath.Join(m.dataDir, runtimeVolumesFile)) if err != nil { - t.Fatal(err) + t.Fatalf("ReadFile: %v", err) } var saved map[string]types.VolumeRegistration if err = json.Unmarshal(raw, &saved); err != nil || len(saved) != 1 || saved["runtime"].Path != path || !slices.Equal(saved["runtime"].Tenants, normalized.Tenants) { @@ -44,7 +44,7 @@ func TestRuntimeCatalogPersistsNormalizedSemantics(t *testing.T) { } sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Volumes: []types.Volume{{Name: "runtime"}}}) if err != nil { - t.Fatal(err) + t.Fatalf("ClaimProvision: %v", err) } m2 := newVolumeManagerAt(t, eng, m.dataDir, nil) if got := m2.Volumes("acme", nil); len(got) != 1 || got[0].Name != "runtime" || !got[0].Available || !got[0].Writable { @@ -54,7 +54,7 @@ func TestRuntimeCatalogPersistsNormalizedSemantics(t *testing.T) { t.Fatalf("ACL changed at restart: %+v", got) } if err = m2.Reconcile(t.Context()); err != nil { - t.Fatal(err) + t.Fatalf("Reconcile: %v", err) } if _, err = m2.ClaimProvision(t.Context(), testKey, ClaimOptions{Volumes: []types.Volume{{Name: "runtime", Mode: types.VolumeModeRW}}}); !errors.Is(err, ErrVolumeBusy) { t.Fatalf("recovered holder missing: %v", err) @@ -68,11 +68,11 @@ func TestRuntimeCatalogPersistsNormalizedSemantics(t *testing.T) { t.Fatalf("unregister of a held volume: %v", err) } if err = m2.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatal(err) + t.Fatalf("Release: %v", err) } for range 2 { if err = m2.UnregisterVolume(t.Context(), "runtime"); err != nil { - t.Fatal(err) + t.Fatalf("UnregisterVolume: %v", err) } } if got, err := os.ReadFile(path); err != nil || string(got) != "backing bytes stay intact" { @@ -87,7 +87,7 @@ func TestRuntimeCatalogChangesAreAudited(t *testing.T) { cfg := &config.Config{DataDir: t.TempDir(), AuditLog: true} m, initErr := NewManager(t.Context(), cfg, newFakeEngine(), testSecrets(t), testBackends(t, cfg)) if initErr != nil { - t.Fatal(initErr) + t.Fatalf("NewManager: %v", initErr) } spec := types.VolumeRegistration{Path: "/images/data"} restore := blockCatalogRename(t, m) @@ -97,7 +97,7 @@ func TestRuntimeCatalogChangesAreAudited(t *testing.T) { restore() for range 2 { if err := m.RegisterVolume(t.Context(), "data", spec); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } } restore = blockCatalogRename(t, m) @@ -107,12 +107,12 @@ func TestRuntimeCatalogChangesAreAudited(t *testing.T) { restore() for range 2 { if err := m.UnregisterVolume(t.Context(), "data"); err != nil { - t.Fatal(err) + t.Fatalf("UnregisterVolume: %v", err) } } raw, err := os.ReadFile(filepath.Join(m.dataDir, "audit.jsonl")) if err != nil { - t.Fatal(err) + t.Fatalf("ReadFile: %v", err) } lines := strings.Split(strings.TrimSpace(string(raw)), "\n") want := []auditFrame{{Op: "volumes", Added: []string{"data"}}, {Op: "volumes", Removed: []string{"data"}}, {Op: "volumes", Removed: []string{"data"}}} @@ -130,15 +130,15 @@ func TestRuntimeCatalogChangesAreAudited(t *testing.T) { func TestRuntimeCatalogRegistersUncheckedBacking(t *testing.T) { link := filepath.Join(t.TempDir(), "backing-link") if err := os.Symlink(writeVolumeImage(t, "backing", "bytes"), link); err != nil { - t.Fatal(err) + t.Fatalf("Symlink: %v", err) } - for _, tc := range []struct { + for _, tt := range []struct { name, path string wantErr error }{{"symlink", link, nil}, {"missing", filepath.Join(t.TempDir(), "missing"), ErrVolumeUnavailable}} { - t.Run(tc.name, func(t *testing.T) { + t.Run(tt.name, func(t *testing.T) { m := newVolumeManager(t, newFakeEngine(), nil) - registration := types.VolumeRegistration{Path: tc.path, Writable: true} + registration := types.VolumeRegistration{Path: tt.path, Writable: true} if err := m.RegisterVolume(t.Context(), "runtime", registration); err != nil { t.Fatalf("register: %v", err) } @@ -146,14 +146,14 @@ func TestRuntimeCatalogRegistersUncheckedBacking(t *testing.T) { if err := m2.RegisterVolume(t.Context(), "runtime", registration); err != nil { t.Fatalf("idempotent retry after restart: %v", err) } - if infos := m2.Volumes("", nil); len(infos) != 1 || infos[0].Available != (tc.wantErr == nil) { + if infos := m2.Volumes("", nil); len(infos) != 1 || infos[0].Available != (tt.wantErr == nil) { t.Fatalf("availability: %+v", infos) } - if _, err := m2.resolveVolumes(t.Context(), "", []types.Volume{{Name: "runtime"}}); !errors.Is(err, tc.wantErr) { - t.Fatalf("resolve = %v, want %v", err, tc.wantErr) + if _, err := m2.resolveVolumes(t.Context(), "", []types.Volume{{Name: "runtime"}}); !errors.Is(err, tt.wantErr) { + t.Fatalf("resolve = %v, want %v", err, tt.wantErr) } if err := m2.UnregisterVolume(t.Context(), "runtime"); err != nil { - t.Fatal(err) + t.Fatalf("UnregisterVolume: %v", err) } }) } @@ -165,14 +165,14 @@ func TestRuntimeCatalogRejectsCollisionsAndInvalidRegistrations(t *testing.T) { other := writeVolumeImage(t, "other-runtime", "bytes") m := newVolumeManager(t, eng, append(slices.Clone(static), config.VolumeSpec{Name: "unexplained", Path: other})) if _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Volumes: []types.Volume{{Name: "unexplained"}}}); err != nil { - t.Fatal(err) + t.Fatalf("ClaimProvision: %v", err) } m = newVolumeManagerAt(t, eng, m.dataDir, static) if err := m.Reconcile(t.Context()); err != nil { - t.Fatal(err) + t.Fatalf("Reconcile: %v", err) } alias := filepath.Dir(static[0].Path) + "/./" + filepath.Base(static[0].Path) - for _, tc := range []struct { + for _, tt := range []struct { name string spec types.VolumeRegistration want error @@ -190,20 +190,20 @@ func TestRuntimeCatalogRejectsCollisionsAndInvalidRegistrations(t *testing.T) { {"runtime", types.VolumeRegistration{Path: other, Tenants: []string{"_leading"}}, ErrBadVolume}, {"runtime", types.VolumeRegistration{Path: other, Tenants: []string{strings.Repeat("a", 64)}}, ErrBadVolume}, } { - err := m.RegisterVolume(t.Context(), tc.name, tc.spec) - if !errors.Is(err, tc.want) { - t.Errorf("register %s %+v = %v, want %v", tc.name, tc.spec, err, tc.want) + err := m.RegisterVolume(t.Context(), tt.name, tt.spec) + if !errors.Is(err, tt.want) { + t.Errorf("register %s %+v = %v, want %v", tt.name, tt.spec, err, tt.want) } } if _, err := os.Stat(filepath.Join(m.dataDir, runtimeVolumesFile)); !errors.Is(err, os.ErrNotExist) || m.catalog.entries["runtime"] != nil { t.Fatalf("rejected registration persisted: stat=%v entry=%+v", err, m.catalog.entries["runtime"]) } - for _, tc := range []struct { + for _, tt := range []struct { name string want error }{{"data", ErrVolumeRegistrationConflict}, {"unexplained", ErrVolumeRegistrationConflict}, {"bad/name", ErrBadVolume}} { - if err := m.UnregisterVolume(t.Context(), tc.name); !errors.Is(err, tc.want) { - t.Errorf("unregister %s = %v, want %v", tc.name, err, tc.want) + if err := m.UnregisterVolume(t.Context(), tt.name); !errors.Is(err, tt.want) { + t.Errorf("unregister %s = %v, want %v", tt.name, err, tt.want) } } if err := m.RegisterVolume(t.Context(), "runtime", types.VolumeRegistration{Path: "/images/data", Tenants: []string{"future-tenant", "team/project:v1"}}); err != nil { @@ -217,7 +217,7 @@ func TestRuntimeCatalogVolumeRootsBoundRegistration(t *testing.T) { cfg := &config.Config{DataDir: dataDir, VolumeRoots: roots} return NewManager(t.Context(), cfg, eng, testSecrets(t), testBackends(t, cfg)) } - for _, tc := range []struct { + for _, tt := range []struct { name, path string roots []string want error @@ -230,13 +230,13 @@ func TestRuntimeCatalogVolumeRootsBoundRegistration(t *testing.T) { {"filesystem root", "/etc/a.img", []string{"/"}, nil}, {"no roots", "/etc/a.img", nil, nil}, } { - t.Run(tc.name, func(t *testing.T) { - m, err := newRootedManager(t, newFakeEngine(), t.TempDir(), tc.roots...) + t.Run(tt.name, func(t *testing.T) { + m, err := newRootedManager(t, newFakeEngine(), t.TempDir(), tt.roots...) if err != nil { - t.Fatal(err) + t.Fatalf("newRootedManager: %v", err) } - if err := m.RegisterVolume(t.Context(), "runtime", types.VolumeRegistration{Path: tc.path}); !errors.Is(err, tc.want) { - t.Fatalf("register %s under %v = %v, want %v", tc.path, tc.roots, err, tc.want) + if err := m.RegisterVolume(t.Context(), "runtime", types.VolumeRegistration{Path: tt.path}); !errors.Is(err, tt.want) { + t.Fatalf("register %s under %v = %v, want %v", tt.path, tt.roots, err, tt.want) } }) } @@ -245,7 +245,7 @@ func TestRuntimeCatalogVolumeRootsBoundRegistration(t *testing.T) { images := map[string]string{"outside": writeVolumeImage(t, "outside", "bytes"), "inside": writeVolumeImage(t, "inside", "bytes")} for name, path := range images { if err := m.RegisterVolume(t.Context(), name, types.VolumeRegistration{Path: path}); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } } outside, inside := filepath.Dir(images["outside"]), filepath.Dir(images["inside"]) @@ -255,7 +255,7 @@ func TestRuntimeCatalogVolumeRootsBoundRegistration(t *testing.T) { } before, err := os.ReadFile(filepath.Join(m.dataDir, runtimeVolumesFile)) if err != nil { - t.Fatal(err) + t.Fatalf("ReadFile: %v", err) } if _, err = newRootedManager(t, eng, m.dataDir, inside); err == nil { t.Fatal("restart accepted a persisted registration outside volume_roots") @@ -265,7 +265,7 @@ func TestRuntimeCatalogVolumeRootsBoundRegistration(t *testing.T) { t.Fatalf("refused startup changed saved registrations: %s, %v", after, err) } if err := wide.UnregisterVolume(t.Context(), "outside"); err != nil { - t.Fatal(err) + t.Fatalf("UnregisterVolume: %v", err) } if narrowed, err := newRootedManager(t, eng, m.dataDir, inside); err != nil || !slices.Equal(narrowed.VolumeNames(), []string{"inside"}) { t.Fatalf("restart after unregistering the out-of-root entry: %v", err) @@ -277,7 +277,7 @@ func TestRuntimeCatalogFailedWriteLeavesUnrelatedAdmissionsAvailable(t *testing. register := func(m *Manager, spec types.VolumeRegistration) error { return m.RegisterVolume(t.Context(), "affected", spec) } - for _, tc := range []struct { + for _, tt := range []struct { name string before []string write func(*Manager, types.VolumeRegistration) error @@ -287,34 +287,34 @@ func TestRuntimeCatalogFailedWriteLeavesUnrelatedAdmissionsAvailable(t *testing. {"register", []string{"healthy"}, register, []types.Volume{{Name: "healthy"}}, nil, ErrVolumeUnavailable}, {"unregister", []string{"healthy", "affected"}, unregister, []types.Volume{{Name: "healthy"}, {Name: "affected"}}, ErrVolumeUnavailable, ErrVolumeUnavailable}, } { - t.Run(tc.name, func(t *testing.T) { + t.Run(tt.name, func(t *testing.T) { eng := newFakeEngine() m, sb := newTenantVolumeManager(t, eng, liveCatalog(t)) specs := map[string]types.VolumeRegistration{ "healthy": {Path: writeVolumeImage(t, "healthy", "bytes")}, "affected": {Path: writeVolumeImage(t, "affected", "bytes")}, } - for _, name := range tc.before { + for _, name := range tt.before { if err := m.RegisterVolume(t.Context(), name, specs[name]); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } } - resolved, err := m.resolveVolumes(t.Context(), "", tc.resolved) + resolved, err := m.resolveVolumes(t.Context(), "", tt.resolved) if err != nil { - t.Fatal(err) + t.Fatalf("resolveVolumes: %v", err) } blockCatalogRename(t, m) - if err = tc.write(m, specs["affected"]); !errors.Is(err, ErrVolumeCatalogUnavailable) { + if err = tt.write(m, specs["affected"]); !errors.Is(err, ErrVolumeCatalogUnavailable) { t.Fatalf("obstructed catalog write: %v", err) } holds, err := m.admitVolumes("", resolved) m.unreserveVolumes(holds) - if !errors.Is(err, tc.wantResolved) { - t.Fatalf("admission resolved before the write = %v, want %v", err, tc.wantResolved) + if !errors.Is(err, tt.wantResolved) { + t.Fatalf("admission resolved before the write = %v, want %v", err, tt.wantResolved) } for _, volumes := range [][]types.Volume{{{Name: "affected"}}, {{Name: "healthy"}, {Name: "affected"}}} { - if _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Volumes: volumes}); !errors.Is(err, tc.want) { - t.Errorf("root claim %v = %v, want %v", volumes, err, tc.want) + if _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Volumes: volumes}); !errors.Is(err, tt.want) { + t.Errorf("root claim %v = %v, want %v", volumes, err, tt.want) } } if len(eng.volumeSpecs) != 0 { @@ -341,14 +341,14 @@ func TestResolvedCatalogBindingRequiresSameEntry(t *testing.T) { m := newVolumeManager(t, newFakeEngine(), nil) spec := types.VolumeRegistration{Path: writeVolumeImage(t, "runtime", "bytes")} if err := m.RegisterVolume(t.Context(), "runtime", spec); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } resolved, err := m.resolveVolumes(t.Context(), "", []types.Volume{{Name: "runtime"}}) if err != nil { - t.Fatal(err) + t.Fatalf("resolveVolumes: %v", err) } if err = m.RegisterVolume(t.Context(), "other", types.VolumeRegistration{Path: writeVolumeImage(t, "other", "bytes")}); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } holds, err := m.admitVolumes("", resolved) if err != nil { @@ -356,10 +356,10 @@ func TestResolvedCatalogBindingRequiresSameEntry(t *testing.T) { } m.unreserveVolumes(holds) if err = m.UnregisterVolume(t.Context(), "runtime"); err != nil { - t.Fatal(err) + t.Fatalf("UnregisterVolume: %v", err) } if err = m.RegisterVolume(t.Context(), "runtime", spec); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } if _, err = m.admitVolumes("", resolved); !errors.Is(err, ErrVolumeUnavailable) { t.Fatalf("same-file reuse admitted stale resolution: %v", err) @@ -371,7 +371,7 @@ func TestResolvedCatalogBindingRequiresSameEntry(t *testing.T) { func TestRuntimeCatalogRejectsInvalidMetadata(t *testing.T) { static := liveCatalog(t) - for _, tc := range []struct{ name, raw string }{ + for _, tt := range []struct{ name, raw string }{ {"malformed", "{"}, {"name", `{"bad/name":{"path":"/data"}}`}, {"relative", `{"runtime":{"path":"relative"}}`}, @@ -381,15 +381,15 @@ func TestRuntimeCatalogRejectsInvalidMetadata(t *testing.T) { {"static-name", `{"data":{"path":"/other"}}`}, {"static-path", fmt.Sprintf(`{"runtime":{"path":%q}}`, static[0].Path)}, } { - t.Run(tc.name, func(t *testing.T) { + t.Run(tt.name, func(t *testing.T) { eng := newFakeEngine() m := newVolumeManager(t, eng, static) - if err := os.WriteFile(filepath.Join(m.dataDir, runtimeVolumesFile), []byte(tc.raw), 0o600); err != nil { - t.Fatal(err) + if err := os.WriteFile(filepath.Join(m.dataDir, runtimeVolumesFile), []byte(tt.raw), 0o600); err != nil { + t.Fatalf("WriteFile: %v", err) } cfg := &config.Config{DataDir: m.dataDir, Volumes: static} if _, err := NewManager(t.Context(), cfg, eng, testSecrets(t), testBackends(t, cfg)); err == nil { - t.Fatalf("accepted %s metadata", tc.name) + t.Fatalf("accepted %s metadata", tt.name) } }) } @@ -400,19 +400,19 @@ func blockCatalogRename(t *testing.T, m *Manager) func() { record := filepath.Join(m.dataDir, runtimeVolumesFile) raw, err := os.ReadFile(record) if err != nil && !errors.Is(err, os.ErrNotExist) { - t.Fatal(err) + t.Fatalf("ReadFile: %v", err) } existed := err == nil if err := errors.Join(os.RemoveAll(record), os.Mkdir(record, 0o700)); err != nil { - t.Fatal(err) + t.Fatalf("block the catalog rename: %v", err) } restore := sync.OnceFunc(func() { if err := os.Remove(record); err != nil { - t.Fatal(err) + t.Fatalf("Remove: %v", err) } if existed { if err := os.WriteFile(record, raw, 0o600); err != nil { - t.Fatal(err) + t.Fatalf("WriteFile: %v", err) } } }) diff --git a/sandboxd/pool/volume_cleanup_test.go b/sandboxd/pool/volume_cleanup_test.go index 3cba011d9..5fd1b8093 100644 --- a/sandboxd/pool/volume_cleanup_test.go +++ b/sandboxd/pool/volume_cleanup_test.go @@ -13,11 +13,11 @@ func TestDispatchedCleanupRetainsCatalogReference(t *testing.T) { base := newFakeEngine() m := newVolumeManager(t, base, nil) if err := m.RegisterVolume(t.Context(), "data", types.VolumeRegistration{Path: writeVolumeImage(t, "runtime", "bytes")}); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Volumes: []types.Volume{{Name: "data"}}}) if err != nil { - t.Fatal(err) + t.Fatalf("ClaimProvision: %v", err) } base.removeErrFor = sb.VMName if err = m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err == nil { @@ -32,7 +32,7 @@ func TestDispatchedCleanupRetainsCatalogReference(t *testing.T) { base.removeErrFor = "" m.retryRemovals(t.Context()).Wait() if err := m.UnregisterVolume(t.Context(), "data"); err != nil { - t.Fatal(err) + t.Fatalf("UnregisterVolume: %v", err) } } @@ -40,11 +40,11 @@ func TestOwnershipTransferReleasesRunningVolumeHoldAfterRemoval(t *testing.T) { base := newFakeEngine() m := newVolumeManager(t, base, nil) if err := m.RegisterVolume(t.Context(), "data", types.VolumeRegistration{Path: writeVolumeImage(t, "runtime", "bytes"), Writable: true}); err != nil { - t.Fatal(err) + t.Fatalf("RegisterVolume: %v", err) } sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Volumes: []types.Volume{{Name: "data"}}}) if err != nil { - t.Fatal(err) + t.Fatalf("ClaimProvision: %v", err) } writer := ClaimOptions{Volumes: []types.Volume{{Name: "data", Mode: types.VolumeModeRW}}} base.removeErrFor = sb.VMName diff --git a/sandboxd/pool/volume_convergence_test.go b/sandboxd/pool/volume_convergence_test.go index 6ce2456b7..2bde19c34 100644 --- a/sandboxd/pool/volume_convergence_test.go +++ b/sandboxd/pool/volume_convergence_test.go @@ -20,12 +20,12 @@ func TestSettledVolumesDoNotCallDevicesOrResolveCatalog(t *testing.T) { m := newVolumeManager(t, eng, catalog) sb := mustClaim(t, m, testKey) if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, []types.Volume{{Name: "data"}}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } delete(m.catalog.entries, "data") before := len(eng.volumeSpecs) if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, []types.Volume{{Name: "data"}, {Name: "other"}}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if len(eng.volumeSpecs) != before+1 || eng.volumeSpecs[before].Name != "other" { t.Fatalf("retained volume reapplied: %v", eng.volumeSpecs) @@ -33,7 +33,7 @@ func TestSettledVolumesDoNotCallDevicesOrResolveCatalog(t *testing.T) { before = len(eng.volumeSpecs) journaled := m.store.mark() if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, []types.Volume{{Name: "data"}, {Name: "other"}}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if len(eng.volumeSpecs) != before || m.store.mark() != journaled { t.Fatal("identical target reached devices or the claims journal") @@ -44,10 +44,10 @@ func TestRemovedTenantRetainsAndRepairsHeldVolumes(t *testing.T) { eng := newFakeEngine() m, sb := newTenantVolumeManager(t, eng, liveCatalog(t)) if _, err := m.SetVolumes(t.Context(), sb.ID, "acme", []types.Volume{{Name: "data"}, {Name: "other"}}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if err := m.DeleteTenant(t.Context(), "acme"); err != nil { - t.Fatal(err) + t.Fatalf("DeleteTenant: %v", err) } if _, err := m.PatchVolumes(t.Context(), sb.ID, "", types.VolumePatch{Remove: []string{"other"}}); err != nil { t.Fatalf("remove other while retaining data: %v", err) @@ -74,7 +74,7 @@ func TestReaddedDetachIsPendingAtTheDurableIntentBoundary(t *testing.T) { sb := mustClaim(t, m, testKey) target := []types.Volume{{Name: "data"}} if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, target); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } eng.detachVolumeErr = errors.New("guest unmounted; eject outcome unknown") if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, []types.Volume{}); err == nil { @@ -82,16 +82,16 @@ func TestReaddedDetachIsPendingAtTheDurableIntentBoundary(t *testing.T) { } ctx, run, err := m.beginVolumeRun(t.Context(), sb) if err != nil { - t.Fatal(err) + t.Fatalf("beginVolumeRun: %v", err) } target = slices.Clone(sb.Volumes) if checkErr := m.prepareVolumeTarget(ctx, sb, run, target, nil, nil); checkErr != nil { - t.Fatal(checkErr) + t.Fatalf("prepareVolumeTarget: %v", checkErr) } m.finishVolumeRun(sb, run) m2 := newVolumeManagerAt(t, eng, m.dataDir, catalog) if checkErr := m2.Reconcile(t.Context()); checkErr != nil { - t.Fatal(checkErr) + t.Fatalf("Reconcile: %v", checkErr) } view, err := m2.GetVolumes(sb.ID, sb.Tenant) if err != nil || len(view.Volumes) != 0 || len(view.VolumesWant) != 1 || view.States[0].State != types.VolumePending || view.Converging { @@ -103,7 +103,7 @@ func TestReaddedDetachIsPendingAtTheDurableIntentBoundary(t *testing.T) { eng.detachVolumeErr = nil before := len(eng.volumeSpecs) if _, err := m2.SetVolumes(t.Context(), sb.ID, sb.Tenant, target); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if len(eng.volumeSpecs) != before+1 { t.Fatal("readded volume was not repaired") @@ -116,13 +116,13 @@ func TestReleaseBeforeReservationDoesNotLeakAdmission(t *testing.T) { sb := mustClaim(t, m, testKey) ctx, run, err := m.beginVolumeRun(t.Context(), sb) if err != nil { - t.Fatal(err) + t.Fatalf("beginVolumeRun: %v", err) } if err := m.checkVolumeRun(ctx, sb, run); err != nil { - t.Fatal(err) + t.Fatalf("checkVolumeRun: %v", err) } if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatal(err) + t.Fatalf("Release: %v", err) } target := []types.Volume{{Name: "data", Mount: "/volumes/data"}} if err := m.prepareVolumeTarget(ctx, sb, run, target, target, nil); err == nil { @@ -151,7 +151,7 @@ func TestVolumeDiagnosticsTrackActualWorkerAndRetry(t *testing.T) { } close(eng.resume) if checkErr := <-done; !errors.Is(checkErr, context.Canceled) { - t.Fatal(checkErr) + t.Fatalf("canceled worker: %v, want context.Canceled", checkErr) } view, err = m.GetVolumes(sb.ID, sb.Tenant) if err != nil || view.Converging || view.States[0].Error == "" { @@ -178,7 +178,7 @@ func TestReleaseDropsVolumeDiagnostics(t *testing.T) { m := newVolumeManager(t, eng, liveCatalog(t)) sb := mustClaim(t, m, testKey) if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, []types.Volume{{Name: "data"}}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } eng.detachVolumeErr = unmountConflict() if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, []types.Volume{}); err == nil { @@ -186,7 +186,7 @@ func TestReleaseDropsVolumeDiagnostics(t *testing.T) { } eng.detachVolumeErr = nil if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatal(err) + t.Fatalf("Release: %v", err) } m.mu.Lock() defer m.mu.Unlock() @@ -211,7 +211,7 @@ func TestLiveVolumeRunStampsActivityBeforeDeviceCalls(t *testing.T) { idle := time.Since(sb.LastSeen()) close(eng.resume) if err := <-done; err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if idle > time.Minute { t.Fatalf("a running volume change left the sandbox idle for %v", idle) @@ -260,7 +260,7 @@ func TestRejectedVolumeTargetPreservesConvergenceDiagnostics(t *testing.T) { m := newVolumeManager(t, eng, liveCatalog(t)) sb := mustClaim(t, m, testKey) if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, []types.Volume{{Name: "data"}}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if priorConflict { eng.detachVolumeErr = unmountConflict() @@ -270,7 +270,7 @@ func TestRejectedVolumeTargetPreservesConvergenceDiagnostics(t *testing.T) { } before, err := m.GetVolumes(sb.ID, sb.Tenant) if err != nil { - t.Fatal(err) + t.Fatalf("GetVolumes: %v", err) } if _, checkErr := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, []types.Volume{{Name: "data"}, {Name: "data"}}); !errors.Is(checkErr, ErrBadVolume) { t.Fatalf("invalid target: %v", checkErr) @@ -290,7 +290,7 @@ func TestUnsupportedRuntimeDoesNotChangeVolumeIntent(t *testing.T) { eng.liveVolumeSupportErr = &engine.LiveVolumeUnsupported{} before, err := os.ReadFile(m.store.path) if err != nil { - t.Fatal(err) + t.Fatalf("ReadFile: %v", err) } for _, patch := range []bool{false, true} { if patch { @@ -304,7 +304,7 @@ func TestUnsupportedRuntimeDoesNotChangeVolumeIntent(t *testing.T) { } after, err := os.ReadFile(m.store.path) if err != nil { - t.Fatal(err) + t.Fatalf("ReadFile: %v", err) } if string(before) != string(after) || len(eng.volumeSpecs) != 0 || volumeHoldersOf(m, "data") != (volumeHolders{}) { t.Fatal("unsupported runtime changed target, hold, or device") @@ -326,7 +326,7 @@ func TestVolumeHoldsRejectCaptureBeforeTrim(t *testing.T) { case "admitted": _, run, err := m.beginVolumeRun(t.Context(), sb) if err != nil { - t.Fatal(err) + t.Fatalf("beginVolumeRun: %v", err) } m.mu.Lock() m.adoptVolumes(held) @@ -369,7 +369,7 @@ func TestUncertainTargetCommitReleasesOnlyConfirmedPrecleanedHolds(t *testing.T) eng.liveVolumeHook = func(verb string) error { if verb == "detach" { if err := os.Mkdir(m.store.path+".tmp", 0o700); err != nil { - t.Fatal(err) + t.Fatalf("Mkdir: %v", err) } } return nil @@ -396,18 +396,18 @@ func TestUncertainTargetCommitReleasesOnlyConfirmedPrecleanedHolds(t *testing.T) }) } if checkErr := os.Remove(m.store.path + ".tmp"); checkErr != nil { - t.Fatal(checkErr) + t.Fatalf("Remove: %v", checkErr) } eng.liveVolumeHook = nil if durable == "new" { if checkErr := m.store.commit(m.store.mark()); checkErr != nil { - t.Fatal(checkErr) + t.Fatalf("commit: %v", checkErr) } waitFor(t, func() bool { return !m.store.recommitting.Load() }) } m2 := newVolumeManagerAt(t, eng, m.dataDir, catalog) if checkErr := m2.Reconcile(t.Context()); checkErr != nil { - t.Fatal(checkErr) + t.Fatalf("Reconcile: %v", checkErr) } view, err = m2.SetVolumes(t.Context(), sb.ID, sb.Tenant, target) if err != nil || len(view.VolumesWant) != 1 || view.VolumesWant[0].Name != "other" { diff --git a/sandboxd/pool/volume_mutation_test.go b/sandboxd/pool/volume_mutation_test.go index 6a9162e64..694974b75 100644 --- a/sandboxd/pool/volume_mutation_test.go +++ b/sandboxd/pool/volume_mutation_test.go @@ -30,7 +30,7 @@ func TestLiveVolumeFourModesAndMixedRoundtrip(t *testing.T) { sb := mustClaim(t, m, testKey) view, err := m.PatchVolumes(t.Context(), sb.ID, "", types.VolumePatch{Add: request}) if err != nil { - t.Fatal(err) + t.Fatalf("PatchVolumes: %v", err) } if len(view.Volumes) != 4 || len(eng.volumeMounts) != 2 { t.Fatalf("mixed modes: view=%+v mounts=%+v", view, eng.volumeMounts) @@ -42,7 +42,7 @@ func TestLiveVolumeFourModesAndMixedRoundtrip(t *testing.T) { } before := slices.Clone(eng.volumeOpsLog()) if _, err = m.SetVolumes(t.Context(), sb.ID, "", view.VolumesWant); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if !slices.Equal(before, eng.volumeOpsLog()) { t.Fatal("GET-to-PUT roundtrip changed devices") @@ -62,14 +62,14 @@ func TestLiveVolumeFourModesAndMixedRoundtrip(t *testing.T) { assertLiveJournal(t, m, sb) raw, err := os.ReadFile(m.store.path) if err != nil { - t.Fatal(err) + t.Fatalf("ReadFile: %v", err) } if err = os.WriteFile(m.store.path, bytes.ReplaceAll(raw, []byte(`,"attach_only":true`), nil), 0o600); err != nil { - t.Fatal(err) + t.Fatalf("WriteFile: %v", err) } restarted := newVolumeManagerAt(t, eng, m.dataDir, catalog) if err = restarted.Reconcile(t.Context()); err != nil { - t.Fatal(err) + t.Fatalf("Reconcile: %v", err) } recovered, err := restarted.GetVolumes(sb.ID, "") if err != nil || !slices.Equal(recovered.Volumes, view.Volumes) { @@ -79,7 +79,7 @@ func TestLiveVolumeFourModesAndMixedRoundtrip(t *testing.T) { t.Fatalf("restart lost writer exclusion: %v", err) } if _, err = restarted.SetVolumes(t.Context(), sb.ID, "", []types.Volume{}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } for _, spec := range catalog { if volumeDirty(spec.Path) || volumeHoldersOf(restarted, spec.Name) != (volumeHolders{}) { @@ -161,7 +161,7 @@ func TestLiveAttachFailureSurvivesRestartAndCanBeDetached(t *testing.T) { } m2 := newVolumeManagerAt(t, eng, m.dataDir, catalog) if reconcileErr := m2.Reconcile(t.Context()); reconcileErr != nil { - t.Fatal(reconcileErr) + t.Fatalf("Reconcile: %v", reconcileErr) } if volumeHoldersOf(m2, "data").readers != 1 { t.Fatal("restart lost pending hold") @@ -175,7 +175,7 @@ func TestLiveAttachFailureSurvivesRestartAndCanBeDetached(t *testing.T) { } eng.detachVolumeErr = nil if _, err := m2.PatchVolumes(t.Context(), sb.ID, sb.Tenant, types.VolumePatch{Remove: []string{"data"}}); err != nil { - t.Fatal(err) + t.Fatalf("PatchVolumes: %v", err) } if volumeHoldersOf(m2, "data") != (volumeHolders{}) { t.Fatal("successful cleanup leaked hold") @@ -210,7 +210,7 @@ func TestLiveVolumePersistenceFailureRetainsRecoverableState(t *testing.T) { } view, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, request) if err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if slices.ContainsFunc(view.States, func(s types.VolumeState) bool { return s.Error != "" }) { t.Fatalf("converged view kept a stale error: %+v", view.States) @@ -226,15 +226,7 @@ func TestLiveDetachResultWriteFailureSurvivesTwoRestarts(t *testing.T) { m := newVolumeManager(t, eng, catalog) a := mustClaim(t, m, testKey) if _, err := m.SetVolumes(t.Context(), a.ID, a.Tenant, []types.Volume{{Name: "data"}}); err != nil { - t.Fatal(err) - } - loadClaims := func() map[string]*types.Sandbox { - t.Helper() - claims, err := newClaimStore(m.dataDir, false).load() - if err != nil { - t.Fatal(err) - } - return claims + t.Fatalf("SetVolumes: %v", err) } journalPath := m.store.path m.store.recommitting.Store(true) @@ -262,7 +254,7 @@ func TestLiveDetachResultWriteFailureSurvivesTwoRestarts(t *testing.T) { if got := volumeHoldersOf(m, "data"); got != (volumeHolders{}) { t.Fatalf("successful detach retained live holders: %+v", got) } - stale := loadClaims()[a.ID] + stale := journaled(t, m)[a.ID] wantReader := []types.Volume{{Name: "data", Mount: "/volumes/data"}} if stale == nil || !slices.Equal(stale.Volumes, wantReader) || stale.VolumesWant == nil || len(stale.VolumesWant) != 0 { t.Fatalf("journal lost stale applied reader or empty target: %+v", stale) @@ -270,7 +262,7 @@ func TestLiveDetachResultWriteFailureSurvivesTwoRestarts(t *testing.T) { m2 := newVolumeManagerAt(t, eng, m.dataDir, catalog) if reconcileErr := m2.Reconcile(t.Context()); reconcileErr != nil { - t.Fatal(reconcileErr) + t.Fatalf("Reconcile: %v", reconcileErr) } writer := ClaimOptions{Volumes: []types.Volume{{Name: "data", Mode: types.VolumeModeRW}}} if _, claimErr := m2.ClaimProvision(t.Context(), testKey, writer); !errors.Is(claimErr, ErrVolumeBusy) { @@ -288,7 +280,7 @@ func TestLiveDetachResultWriteFailureSurvivesTwoRestarts(t *testing.T) { if err != nil { t.Fatalf("writer after recovered detach: %v", err) } - claims := loadClaims() + claims := journaled(t, m) wantWriter := []types.Volume{{Name: "data", Mount: "/volumes/data", Mode: types.VolumeModeRW}} if savedA, savedB := claims[a.ID], claims[b.ID]; savedA == nil || savedB == nil || len(savedA.Volumes) != 0 || len(savedA.VolumesWant) != 0 || !slices.Equal(savedB.Volumes, wantWriter) { t.Fatalf("successful writer kept obsolete reader journal: A=%+v B=%+v", savedA, savedB) @@ -296,7 +288,7 @@ func TestLiveDetachResultWriteFailureSurvivesTwoRestarts(t *testing.T) { m3 := newVolumeManagerAt(t, eng, m.dataDir, catalog) if reconcileErr := m3.Reconcile(t.Context()); reconcileErr != nil { - t.Fatal(reconcileErr) + t.Fatalf("Reconcile: %v", reconcileErr) } if _, claimErr := m3.ClaimProvision(t.Context(), testKey, writer); !errors.Is(claimErr, ErrVolumeBusy) { t.Fatalf("second restart admitted competing writer: %v", claimErr) @@ -329,7 +321,7 @@ func TestLiveBatchPartialFailureRetriesWithoutDoubleHolds(t *testing.T) { } eng.diskAttachErrFor = "" if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, request); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } for _, name := range []string{"data", "other"} { if volumeHoldersOf(m, name).readers != 1 { @@ -417,7 +409,7 @@ func TestLiveVolumeDoesNotBlockLifecycle(t *testing.T) { t.Fatal("teardown leaked holds") } } else if attachErr != nil { - t.Fatal(attachErr) + t.Fatalf("attach: %v", attachErr) } }) } @@ -487,12 +479,12 @@ func TestLiveVolumeAdmissionAndState(t *testing.T) { sb.OnExpire, want = types.ExpireArchive, ErrVolumeCapture case "dirty": if err := markVolumeDirty(catalog[0].Path); err != nil { - t.Fatal(err) + t.Fatalf("markVolumeDirty: %v", err) } want = ErrVolumeNeedsRecovery case "writer": if _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Volumes: []types.Volume{{Name: "data", Mode: "rw"}}}); err != nil { - t.Fatal(err) + t.Fatalf("ClaimProvision: %v", err) } want = ErrVolumeBusy } @@ -516,10 +508,10 @@ func TestLiveVolumesRespectOwningTenantAndRemoval(t *testing.T) { } } if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, []types.Volume{{Name: "data"}}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if err := m.DeleteTenant(t.Context(), "acme"); err != nil { - t.Fatal(err) + t.Fatalf("DeleteTenant: %v", err) } if _, err := m.SetVolumes(t.Context(), sb.ID, "", []types.Volume{{Name: "data"}}); err != nil { t.Fatalf("held volume after tenant removal: %v", err) @@ -543,21 +535,21 @@ func TestDesiredVolumesJournalMigration(t *testing.T) { store := newClaimStore(t.TempDir(), false) raw := `{"sb_1":{"id":"sb_1","volumes":[{"name":"data","mount":"/data"}]` + tt.field + `}}` if err := os.WriteFile(store.path, []byte(raw), 0o600); err != nil { - t.Fatal(err) + t.Fatalf("WriteFile: %v", err) } claims, err := store.load() if err != nil { - t.Fatal(err) + t.Fatalf("load: %v", err) } if got := claims["sb_1"].VolumesWant; !slices.Equal(got, tt.want) { t.Fatalf("loaded target %v, want %v", got, tt.want) } if err = store.save(claims); err != nil { - t.Fatal(err) + t.Fatalf("save: %v", err) } rawBytes, err := os.ReadFile(store.path) if err != nil { - t.Fatal(err) + t.Fatalf("ReadFile: %v", err) } if strings.Contains(string(rawBytes), `"volumes_want":`) != (tt.field != "") { t.Fatalf("persisted target %s, want field only when it differs", rawBytes) @@ -573,7 +565,7 @@ func TestFailedDetachReaddRepairsAppliedMountAfterRestart(t *testing.T) { sb := mustClaim(t, m, testKey) target := []types.Volume{{Name: "data"}} if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, target); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } eng.detachVolumeErr = errors.New("eject did not complete") view, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, nil) @@ -582,7 +574,7 @@ func TestFailedDetachReaddRepairsAppliedMountAfterRestart(t *testing.T) { } m2 := newVolumeManagerAt(t, eng, m.dataDir, catalog) if reconcileErr := m2.Reconcile(t.Context()); reconcileErr != nil { - t.Fatal(reconcileErr) + t.Fatalf("Reconcile: %v", reconcileErr) } before := len(eng.volumeMounts) eng.detachVolumeErr = nil @@ -625,7 +617,7 @@ func TestSupersedingTargetPrecleansUncertainAddBeforeJournaling(t *testing.T) { assertLiveJournal(t, m, sb) close(resume) if err := <-done; err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if volumeHoldersOf(m, "data") != (volumeHolders{}) || volumeHoldersOf(m, "other").readers != 1 { t.Fatal("superseding target leaked or lost holds") @@ -735,10 +727,10 @@ func TestConcurrentVolumePatchesUseLatestDesiredSet(t *testing.T) { } close(eng.resume) if err := <-first; err != nil { - t.Fatal(err) + t.Fatalf("first patch: %v", err) } if err := <-second; err != nil { - t.Fatal(err) + t.Fatalf("second patch: %v", err) } view, err := m.GetVolumes(sb.ID, sb.Tenant) if err != nil || len(view.Volumes) != 2 || len(view.VolumesWant) != 2 { @@ -764,7 +756,7 @@ func TestSyncIntentFailureRetainsConservativeDesiredHold(t *testing.T) { } restore() if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, nil); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if volumeHoldersOf(m, "data") != (volumeHolders{}) { t.Fatal("cleanup leaked conservative hold") @@ -818,7 +810,7 @@ func TestFailedReleaseRestoresVolumeHoldOwnership(t *testing.T) { } restore() if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, []types.Volume{{Name: "data"}}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if volumeHoldersOf(m, "data").readers != 1 { t.Fatal("retry duplicated admission hold") @@ -858,7 +850,7 @@ func TestRetriedTargetPersistsIntentBeforeDeviceCalls(t *testing.T) { } restore() if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, request); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } assertLiveJournal(t, m, sb) } @@ -885,7 +877,7 @@ func TestLiveAttachBoundsWholeDeviceOperation(t *testing.T) { m.eng = eng start := time.Now() if _, err := m.SetVolumes(t.Context(), sb.ID, sb.Tenant, []types.Volume{{Name: "data"}}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if deadline := <-eng.deadlines; !deadline.Equal(start.Add(engine.VolumeMutationTimeout)) { t.Fatalf("live attach deadline=%v, want %v", deadline, start.Add(engine.VolumeMutationTimeout)) @@ -933,10 +925,10 @@ func TestRemovedTenantCanRemoveReadOnlyVolumeBesideWritableVolume(t *testing.T) eng := newFakeEngine() m, sb := newTenantVolumeManager(t, eng, liveCatalog(t), types.Volume{Name: "other", Mode: "rw"}) if _, err := m.PatchVolumes(t.Context(), sb.ID, sb.Tenant, types.VolumePatch{Add: []types.Volume{{Name: "data"}}}); err != nil { - t.Fatal(err) + t.Fatalf("PatchVolumes: %v", err) } if err := m.DeleteTenant(t.Context(), "acme"); err != nil { - t.Fatal(err) + t.Fatalf("DeleteTenant: %v", err) } view, err := m.PatchVolumes(t.Context(), sb.ID, sb.Tenant, types.VolumePatch{Remove: []string{"data"}}) if err != nil || len(view.Volumes) != 1 || view.Volumes[0].Name != "other" || !view.Volumes[0].RW() { @@ -990,25 +982,17 @@ func (e *stalledDetachEngine) DetachVolume(ctx context.Context, _, _, _, _ strin return false, errors.New("signal: killed") } -func liveCatalog(t *testing.T) []config.VolumeSpec { - t.Helper() - return []config.VolumeSpec{ - {Name: "data", Path: writeVolumeImage(t, "data.img", "data"), Writable: true}, - {Name: "other", Path: writeVolumeImage(t, "other.img", "other"), Writable: true}, - } -} - func newTenantVolumeManager(t *testing.T, eng *fakeEngine, catalog []config.VolumeSpec, volumes ...types.Volume) (*Manager, *types.Sandbox) { t.Helper() cfg := &config.Config{DataDir: t.TempDir(), APIToken: "root", Volumes: catalog} m, err := NewManager(t.Context(), cfg, eng, testSecrets(t), testBackends(t, cfg)) if err != nil { - t.Fatal(err) + t.Fatalf("NewManager: %v", err) } putTenants(t, m, tenants.TenantSpec{Name: "acme", Token: "acme-token"}) sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Tenant: "acme", Volumes: volumes}) if err != nil { - t.Fatal(err) + t.Fatalf("ClaimProvision: %v", err) } return m, sb } @@ -1017,7 +1001,7 @@ func assertLiveJournal(t *testing.T, m *Manager, sb *types.Sandbox) { t.Helper() claims, err := newClaimStore(m.dataDir, false).load() if err != nil { - t.Fatal(err) + t.Fatalf("load journal: %v", err) } got := claims[sb.ID] if got == nil || !slices.Equal(got.Volumes, sb.Volumes) { @@ -1027,7 +1011,7 @@ func assertLiveJournal(t *testing.T, m *Manager, sb *types.Sandbox) { t.Fatalf("journal target: %+v, want %+v", got.VolumesWant, sb.VolumesWant) } if _, err := os.Stat(m.store.path); err != nil { - t.Fatal(err) + t.Fatalf("Stat: %v", err) } } @@ -1035,7 +1019,7 @@ func beginAdmittedVolumeRun(t *testing.T, m *Manager, sb *types.Sandbox) *volume t.Helper() _, run, err := m.beginVolumeRun(t.Context(), sb) if err != nil { - t.Fatal(err) + t.Fatalf("beginVolumeRun: %v", err) } m.mu.Lock() defer m.mu.Unlock() @@ -1052,7 +1036,7 @@ func breakClaimStore(t *testing.T, m *Manager) (restore func()) { err := os.Mkdir(tmp, 0o700) m.store.writeMu.Unlock() if err != nil { - t.Fatal(err) + t.Fatalf("break journal: %v", err) } return func() { t.Helper() @@ -1060,7 +1044,7 @@ func breakClaimStore(t *testing.T, m *Manager) (restore func()) { err := os.Remove(tmp) m.store.writeMu.Unlock() if err != nil { - t.Fatal(err) + t.Fatalf("restore journal: %v", err) } waitFor(t, func() bool { return !m.store.recommitting.Load() }) } diff --git a/sandboxd/pool/volume_replica_test.go b/sandboxd/pool/volume_replica_test.go index 065ae4d1b..9206f4260 100644 --- a/sandboxd/pool/volume_replica_test.go +++ b/sandboxd/pool/volume_replica_test.go @@ -18,7 +18,7 @@ func TestReplicaRecoveryDoesNotRepublishReadOnlyHold(t *testing.T) { t.Fatalf("read-only live attachment: %+v, %v", view, err) } if _, err := c.reg.Consume(t.Context(), sb.ID, sb.ArchiveEpoch); err != nil { - t.Fatal(err) + t.Fatalf("consume: %v", err) } rec, ok, err := c.reg.Get(t.Context(), sb.ID) if err != nil || !ok || !rec.Consumed { @@ -66,7 +66,7 @@ func TestReplicaCaptureRefusesPendingLiveWriter(t *testing.T) { m, c := newReplicaVolumeManager(t) disableReplication(m) sb := mustClaim(t, m, testKey) - eng := m.eng.(*fakeEngine) + eng := fakeOf(m) entered, resume := make(chan struct{}), make(chan struct{}) release := sync.OnceFunc(func() { close(resume) }) eng.liveVolumeHook = func(verb string) error { diff --git a/sandboxd/pool/volume_rw_test.go b/sandboxd/pool/volume_rw_test.go index 8d2713038..dbc7b6d8d 100644 --- a/sandboxd/pool/volume_rw_test.go +++ b/sandboxd/pool/volume_rw_test.go @@ -17,34 +17,34 @@ import ( ) func TestLiveWritableDetachRequiresCleanEvidence(t *testing.T) { - for _, tc := range []struct { + for _, tt := range []struct { name string unclean bool }{ {"clean unmount", false}, {"unclean unmount", true}, } { - t.Run(tc.name, func(t *testing.T) { + t.Run(tt.name, func(t *testing.T) { base := newFakeEngine() catalog := liveCatalog(t) m := newVolumeManager(t, base, catalog) - base.detachUnclean = tc.unclean + base.detachUnclean = tt.unclean sb := mustClaim(t, m, testKey) if _, err := m.SetVolumes(t.Context(), sb.ID, "", []types.Volume{{Name: "data", Mode: types.VolumeModeRW}}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } path := catalog[0].Path if !volumeDirty(path) || !base.attachDirty["data"] { t.Fatal("writer was attached before dirty marker") } if _, err := m.SetVolumes(t.Context(), sb.ID, "", []types.Volume{}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } - if volumeDirty(path) != tc.unclean { - t.Fatalf("marker disagrees with clean evidence: unclean=%t dirty=%t", tc.unclean, volumeDirty(path)) + if volumeDirty(path) != tt.unclean { + t.Fatalf("marker disagrees with clean evidence: unclean=%t dirty=%t", tt.unclean, volumeDirty(path)) } _, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Volumes: []types.Volume{{Name: "data"}}}) - if !tc.unclean && err != nil || tc.unclean && !errors.Is(err, ErrVolumeNeedsRecovery) { + if !tt.unclean && err != nil || tt.unclean && !errors.Is(err, ErrVolumeNeedsRecovery) { t.Fatalf("reader admission after detach: %v", err) } }) @@ -57,7 +57,7 @@ func TestLiveWritableFailedHostDetachKeepsMarkerAndHold(t *testing.T) { m := newVolumeManager(t, eng, catalog) sb := mustClaim(t, m, testKey) if _, err := m.SetVolumes(t.Context(), sb.ID, "", []types.Volume{{Name: "data", Mode: types.VolumeModeRW}}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } eng.detachVolumeErr = errors.New("host removal uncertain") if _, err := m.SetVolumes(t.Context(), sb.ID, "", []types.Volume{}); err == nil { @@ -72,7 +72,7 @@ func TestLiveWritableFailedHostDetachKeepsMarkerAndHold(t *testing.T) { eng.detachVolumeErr = nil eng.detachUnclean = true if _, err := m.SetVolumes(t.Context(), sb.ID, "", []types.Volume{}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if !volumeDirty(catalog[0].Path) { t.Fatal("absent-device retry invented clean evidence") @@ -93,7 +93,7 @@ func TestLiveWritableUncertainAdditionClearsOnlyAfterDetach(t *testing.T) { } eng.mountVolumeErr = nil if _, err := m.SetVolumes(t.Context(), sb.ID, "", []types.Volume{}); err != nil { - t.Fatal(err) + t.Fatalf("SetVolumes: %v", err) } if volumeDirty(catalog[0].Path) || volumeHoldersOf(m, "data") != (volumeHolders{}) { t.Fatal("uncertain addition cleanup retained marker or hold") @@ -153,11 +153,11 @@ func TestReleaseKeepsMarkerAfterUncleanUnmount(t *testing.T) { m := newVolumeManager(t, eng, []config.VolumeSpec{{Name: "scratch", Path: path, Writable: true}}) sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Volumes: []types.Volume{{Name: "scratch", Mode: types.VolumeModeRW}}}) if err != nil { - t.Fatal(err) + t.Fatalf("ClaimProvision: %v", err) } eng.detachUnclean = true if err := m.Release(t.Context(), sb.ID, Cred{Token: sb.Token}); err != nil { - t.Fatal(err) + t.Fatalf("Release: %v", err) } if !volumeDirty(path) { t.Error("release cleared the marker after an unclean unmount") @@ -713,19 +713,3 @@ func claimTwoWritableVolumes(t *testing.T) (*Manager, *fakeEngine, *types.Sandbo } return m, base, sb, paths } - -func newVolumeManagerAt(t *testing.T, eng *fakeEngine, dataDir string, volumes []config.VolumeSpec) *Manager { - t.Helper() - cfg := &config.Config{DataDir: dataDir, Volumes: volumes} - m, err := NewManager(t.Context(), cfg, eng, testSecrets(t), testBackends(t, cfg)) - if err != nil { - t.Fatalf("setup manager: %v", err) - } - return m -} - -func volumeHoldersOf(m *Manager, name string) volumeHolders { - m.mu.Lock() - defer m.mu.Unlock() - return m.catalog.holders[name] -} diff --git a/sandboxd/pool/warmup_test.go b/sandboxd/pool/warmup_test.go index eab3d8433..bcc7decc9 100644 --- a/sandboxd/pool/warmup_test.go +++ b/sandboxd/pool/warmup_test.go @@ -16,10 +16,7 @@ func TestGoldenBuildRunsWarmupBeforeSnapshot(t *testing.T) { eng := newFakeEngine() argv := []string{"node", "-e", "0"} m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Warmup: argv}) - final := filepath.Join(m.goldensDir(), testKey.Hash()) - if err := m.buildGoldenSteps(t.Context(), m.view.Load(), testKey, "sbx-gb", "snap", final, ""); err != nil { - t.Fatalf("buildGoldenSteps: %v", err) - } + final := buildGoldenAt(t, m, testKey, "") if len(eng.warmups) != 1 || !slices.Equal(eng.warmups[0], argv) { t.Fatalf("warmups = %v, want [%v]", eng.warmups, argv) } @@ -47,10 +44,7 @@ func TestGoldenBuildSettlesTheGuestBetweenWarmupAndSnapshot(t *testing.T) { eng := newFakeEngine() eng.settleErr = tt.err m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1, Warmup: []string{"true"}}) - final := filepath.Join(m.goldensDir(), testKey.Hash()) - if err := m.buildGoldenSteps(t.Context(), m.view.Load(), testKey, "sbx-gb", "snap", final, ""); err != nil { - t.Fatalf("buildGoldenSteps: %v", err) - } + buildGoldenAt(t, m, testKey, "") if eng.settles != 1 || eng.settleAfterSnap || !eng.settleAfterWarmup || len(eng.snapSaves) != 1 { t.Errorf("settles %d, after snapshot %t, after warmup %t, snapshots %d", eng.settles, eng.settleAfterSnap, eng.settleAfterWarmup, len(eng.snapSaves)) } @@ -61,10 +55,7 @@ func TestGoldenBuildSettlesTheGuestBetweenWarmupAndSnapshot(t *testing.T) { func TestGoldenBuildSkipsWarmupWhenUnset(t *testing.T) { eng := newFakeEngine() m := newTestManager(t, eng, config.PoolSpec{PoolKey: testKey, Warm: 1}) - final := filepath.Join(m.goldensDir(), testKey.Hash()) - if err := m.buildGoldenSteps(t.Context(), m.view.Load(), testKey, "sbx-gb", "snap", final, ""); err != nil { - t.Fatalf("buildGoldenSteps: %v", err) - } + final := buildGoldenAt(t, m, testKey, "") if len(eng.warmups) != 0 { t.Errorf("Warmup called %d times for a pool without one", len(eng.warmups)) } @@ -118,10 +109,7 @@ func TestRefillWarmsEveryClone(t *testing.T) { m.pools[testKey].goldenDir = "/goldens/x" m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 2 && infos[0].Refilling == 0 - }) + waitWarm(t, m, 2) eng.mu.Lock() warmups, socks := slices.Clone(eng.warmups), slices.Clone(eng.warmupSocks) eng.mu.Unlock() @@ -150,9 +138,6 @@ func TestRefillCloneWarmupFailureCleansUp(t *testing.T) { eng.warmupErr = nil eng.mu.Unlock() m.refillOnce(t.Context()) - waitFor(t, func() bool { - infos, _ := m.Info() - return infos[0].Warm == 1 && infos[0].Refilling == 0 - }) + waitWarm(t, m, 1) }) } diff --git a/sandboxd/server/exec_test.go b/sandboxd/server/exec_test.go index 8fd238757..8965ca275 100644 --- a/sandboxd/server/exec_test.go +++ b/sandboxd/server/exec_test.go @@ -23,18 +23,18 @@ func TestExecReturnsBufferedOutput(t *testing.T) { ts, _ := newRelayServer(t, func(conn net.Conn) { defer conn.Close() r := bufio.NewReader(conn) - line, err := r.ReadBytes('\n') + frame, err := r.ReadBytes('\n') if err != nil { return } - gotExec, _ = wire.DecodeRequest(bytes.TrimSpace(line)) - line, err = r.ReadBytes('\n') + gotExec, _ = wire.DecodeRequest(bytes.TrimSpace(frame)) + frame, err = r.ReadBytes('\n') if err != nil { return } - gotStdinClose, _ = wire.DecodeRequest(bytes.TrimSpace(line)) + gotStdinClose, _ = wire.DecodeRequest(bytes.TrimSpace(frame)) _, _ = io.WriteString(conn, `{"type":"started","pid":7}`+"\n") - _, _ = io.WriteString(conn, `{"type":"stdout","data":"`+base64.StdEncoding.EncodeToString([]byte("v22\n"))+`"}`+"\n") + _, _ = io.WriteString(conn, stdoutFrame("v22\n")) _, _ = io.WriteString(conn, `{"type":"stderr","data":"`+base64.StdEncoding.EncodeToString([]byte("warn\n"))+`"}`+"\n") _, _ = io.WriteString(conn, `{"type":"exit","code":3}`+"\n") }) @@ -65,7 +65,7 @@ func TestExecKeepsInvalidUTF8OutputAsReplacementCharacters(t *testing.T) { _, _ = r.ReadBytes('\n') _, _ = r.ReadBytes('\n') _, _ = io.WriteString(conn, `{"type":"started","pid":7}`+"\n") - _, _ = io.WriteString(conn, `{"type":"stdout","data":"`+base64.StdEncoding.EncodeToString([]byte("ok\xff\n"))+`"}`+"\n") + _, _ = io.WriteString(conn, stdoutFrame("ok\xff\n")) _, _ = io.WriteString(conn, `{"type":"exit","code":0}`+"\n") }) status, body := postExec(t, ts, `{"argv":["cat","blob"],"timeout_seconds":5}`) @@ -93,7 +93,7 @@ func TestExecAuditsTheFrameWithoutItsDelimiter(t *testing.T) { _, _ = io.WriteString(conn, `{"type":"started","pid":7}`+"\n"+`{"type":"exit","code":0}`+"\n") }) audited := make(chan []byte, 1) - srv.mgr.(*fakeManager).audited = func(_ string, line []byte) { audited <- bytes.Clone(line) } + srv.mgr.(*fakeManager).audited = func(_ string, frame []byte) { audited <- bytes.Clone(frame) } if status, body := postExec(t, ts, `{"argv":["true"]}`); status != http.StatusOK { t.Fatalf("status %d, want 200: %s", status, body) } @@ -142,28 +142,15 @@ func TestExecMapsSilkdErrors(t *testing.T) { } func TestExecTimeoutKillsTheCommand(t *testing.T) { - var conns atomic.Int32 killed := make(chan uint32, 1) closed := make(chan struct{}) - ts, _ := newRelayServer(t, func(conn net.Conn) { - r := bufio.NewReader(conn) - if conns.Add(1) == 1 { - defer close(closed) - _, _ = r.ReadBytes('\n') - _, _ = io.WriteString(conn, `{"type":"started","pid":7}`+"\n") - _, _ = r.ReadBytes('\n') - _, _ = r.ReadBytes('\n') - return - } - defer conn.Close() - line, _ := r.ReadBytes('\n') - if req, _ := wire.DecodeRequest(bytes.TrimSpace(line)); req != nil { - if kill, ok := req.(*wire.Kill); ok { - killed <- kill.PID - } - } - _, _ = io.WriteString(conn, `{"type":"done"}`+"\n") - }) + ts, _ := newRelayServer(t, killWatchGuest(killed, func(conn net.Conn, r *bufio.Reader) { + defer close(closed) + _, _ = r.ReadBytes('\n') + _, _ = io.WriteString(conn, `{"type":"started","pid":7}`+"\n") + _, _ = r.ReadBytes('\n') + _, _ = r.ReadBytes('\n') + })) if status, _ := postExec(t, ts, `{"argv":["sleep","60"],"timeout_seconds":1}`); status != http.StatusGatewayTimeout { t.Fatalf("status %d, want 504", status) } @@ -172,14 +159,7 @@ func TestExecTimeoutKillsTheCommand(t *testing.T) { case <-time.After(3 * time.Second): t.Fatal("guest conn still open after the timeout") } - select { - case pid := <-killed: - if pid != 7 { - t.Errorf("killed pid %d, want 7", pid) - } - case <-time.After(3 * time.Second): - t.Fatal("no kill reached the guest after the timeout") - } + expectKill(t, killed, "timeout") } func TestExecRejectsUnknownFields(t *testing.T) { @@ -215,18 +195,13 @@ func TestExecGuestClosingAfterTheStartWakesOnce(t *testing.T) { return "/v/sock", nil }, } - dialer := &fakeDialer{dial: func(context.Context, string) (net.Conn, error) { - relayEnd, guestEnd := net.Pipe() - go func() { - defer guestEnd.Close() - r := bufio.NewReader(guestEnd) - _, _ = r.ReadBytes('\n') - _, _ = r.ReadBytes('\n') - _, _ = io.WriteString(guestEnd, `{"type":"started","pid":7}`+"\n") - }() - return relayEnd, nil - }} - ts := serveTestServer(t, New("", "node:7777", mgr, dialer, nil, nil, nil, nil)) + ts := newTestServer(t, "", mgr, pipeDialer(func(guestEnd net.Conn) { + defer guestEnd.Close() + r := bufio.NewReader(guestEnd) + _, _ = r.ReadBytes('\n') + _, _ = r.ReadBytes('\n') + _, _ = io.WriteString(guestEnd, `{"type":"started","pid":7}`+"\n") + })) if status, body := postExec(t, ts, `{"argv":["sleep","60"]}`); status != http.StatusBadGateway { t.Fatalf("status %d, want 502: %s", status, body) @@ -246,19 +221,13 @@ func TestExecTimeoutStartsAfterWake(t *testing.T) { return "/v/sock", nil }, } - dialer := &fakeDialer{dial: func(context.Context, string) (net.Conn, error) { - relayEnd, guestEnd := net.Pipe() - go func() { - defer guestEnd.Close() - r := bufio.NewReader(guestEnd) - _, _ = r.ReadBytes('\n') - _, _ = r.ReadBytes('\n') - _, _ = io.WriteString(guestEnd, `{"type":"exit","code":0}`+"\n") - }() - return relayEnd, nil - }} - srv := New("", "node:7777", mgr, dialer, nil, nil, nil, nil) - ts := serveTestServer(t, srv) + ts := newTestServer(t, "", mgr, pipeDialer(func(guestEnd net.Conn) { + defer guestEnd.Close() + r := bufio.NewReader(guestEnd) + _, _ = r.ReadBytes('\n') + _, _ = r.ReadBytes('\n') + _, _ = io.WriteString(guestEnd, `{"type":"exit","code":0}`+"\n") + })) if status, body := postExec(t, ts, `{"argv":["true"],"timeout_seconds":1}`); status != http.StatusOK { t.Fatalf("status %d, want 200: %s", status, body) @@ -269,26 +238,13 @@ func TestExecTimeoutStartsAfterWake(t *testing.T) { } func TestExecOverTheNodeBudgetIs503AndKillsTheCommand(t *testing.T) { - var conns atomic.Int32 killed := make(chan uint32, 1) - ts, srv := newRelayServer(t, func(conn net.Conn) { - r := bufio.NewReader(conn) - if conns.Add(1) == 1 { - _, _ = r.ReadBytes('\n') - _, _ = io.WriteString(conn, `{"type":"started","pid":7}`+"\n") - _, _ = io.WriteString(conn, `{"type":"stdout","data":"`+base64.StdEncoding.EncodeToString([]byte("x"))+`"}`+"\n") - _, _ = io.Copy(io.Discard, r) - return - } - defer conn.Close() - line, _ := r.ReadBytes('\n') - if req, _ := wire.DecodeRequest(bytes.TrimSpace(line)); req != nil { - if kill, ok := req.(*wire.Kill); ok { - killed <- kill.PID - } - } - _, _ = io.WriteString(conn, `{"type":"done"}`+"\n") - }) + ts, srv := newRelayServer(t, killWatchGuest(killed, func(conn net.Conn, r *bufio.Reader) { + _, _ = r.ReadBytes('\n') + _, _ = io.WriteString(conn, `{"type":"started","pid":7}`+"\n") + _, _ = io.WriteString(conn, stdoutFrame("x")) + _, _ = io.Copy(io.Discard, r) + })) if !srv.execBudget.TryAcquire(execBufferBudget) { t.Fatal("budget not empty at start") } @@ -297,53 +253,26 @@ func TestExecOverTheNodeBudgetIs503AndKillsTheCommand(t *testing.T) { if resp.StatusCode != http.StatusServiceUnavailable || resp.Header.Get("Retry-After") != "1" { t.Fatalf("status %d Retry-After %q, want 503 and 1", resp.StatusCode, resp.Header.Get("Retry-After")) } - select { - case pid := <-killed: - if pid != 7 { - t.Errorf("killed pid %d, want 7", pid) - } - case <-time.After(3 * time.Second): - t.Fatal("no kill reached the guest after the budget refusal") - } + expectKill(t, killed, "budget refusal") } func TestExecOverTheOutputCapIs413AndKillsTheCommand(t *testing.T) { - var conns atomic.Int32 killed := make(chan uint32, 1) - chunk := `{"type":"stdout","data":"` + base64.StdEncoding.EncodeToString(bytes.Repeat([]byte("x"), 1<<20)) + `"}` + "\n" - ts, _ := newRelayServer(t, func(conn net.Conn) { - r := bufio.NewReader(conn) - if conns.Add(1) == 1 { - _, _ = r.ReadBytes('\n') - _, _ = io.WriteString(conn, `{"type":"started","pid":7}`+"\n") - for range execOutputCap>>20 + 1 { - if _, err := io.WriteString(conn, chunk); err != nil { - return - } - } - _, _ = io.Copy(io.Discard, r) - return - } - defer conn.Close() - line, _ := r.ReadBytes('\n') - if req, _ := wire.DecodeRequest(bytes.TrimSpace(line)); req != nil { - if kill, ok := req.(*wire.Kill); ok { - killed <- kill.PID + chunk := stdoutFrame(strings.Repeat("x", 1<<20)) + ts, _ := newRelayServer(t, killWatchGuest(killed, func(conn net.Conn, r *bufio.Reader) { + _, _ = r.ReadBytes('\n') + _, _ = io.WriteString(conn, `{"type":"started","pid":7}`+"\n") + for range execOutputCap>>20 + 1 { + if _, err := io.WriteString(conn, chunk); err != nil { + return } } - _, _ = io.WriteString(conn, `{"type":"done"}`+"\n") - }) + _, _ = io.Copy(io.Discard, r) + })) if status, _ := postExec(t, ts, `{"argv":["yes"]}`); status != http.StatusRequestEntityTooLarge { t.Fatalf("status %d, want 413", status) } - select { - case pid := <-killed: - if pid != 7 { - t.Errorf("killed pid %d, want 7", pid) - } - case <-time.After(3 * time.Second): - t.Fatal("no kill reached the guest after the output cap") - } + expectKill(t, killed, "output cap") } func TestExecReturnsItsBudgetAfterAnswering(t *testing.T) { @@ -352,7 +281,7 @@ func TestExecReturnsItsBudgetAfterAnswering(t *testing.T) { r := bufio.NewReader(conn) _, _ = r.ReadBytes('\n') _, _ = io.WriteString(conn, `{"type":"started","pid":7}`+"\n") - _, _ = io.WriteString(conn, `{"type":"stdout","data":"`+base64.StdEncoding.EncodeToString([]byte("v22\n"))+`"}`+"\n") + _, _ = io.WriteString(conn, stdoutFrame("v22\n")) _, _ = io.WriteString(conn, `{"type":"exit","code":0}`+"\n") }) if status, body := postExec(t, ts, `{"argv":["node","-v"]}`); status != http.StatusOK { @@ -375,11 +304,40 @@ func (m *execManager) WakeAgentSocket(ctx context.Context, id, token string) (st func postExec(t *testing.T, ts *httptest.Server, body string) (int, []byte) { t.Helper() - resp := postJSON(t, ts.URL+"/v1/sandboxes/sb_1/exec", "tok", body) - defer resp.Body.Close() - out, err := io.ReadAll(resp.Body) - if err != nil { - t.Fatalf("read response: %v", err) + return bodyOf(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/exec", "tok", body) +} + +func killWatchGuest(killed chan<- uint32, first func(net.Conn, *bufio.Reader)) guestFunc { + var conns atomic.Int32 + return func(conn net.Conn) { + r := bufio.NewReader(conn) + if conns.Add(1) == 1 { + first(conn, r) + return + } + defer conn.Close() + frame, _ := r.ReadBytes('\n') + if req, _ := wire.DecodeRequest(bytes.TrimSpace(frame)); req != nil { + if kill, ok := req.(*wire.Kill); ok { + killed <- kill.PID + } + } + _, _ = io.WriteString(conn, `{"type":"done"}`+"\n") + } +} + +func expectKill(t *testing.T, killed <-chan uint32, after string) { + t.Helper() + select { + case pid := <-killed: + if pid != 7 { + t.Errorf("killed pid %d, want 7", pid) + } + case <-time.After(3 * time.Second): + t.Fatalf("no kill reached the guest after the %s", after) } - return resp.StatusCode, out +} + +func stdoutFrame(data string) string { + return `{"type":"stdout","data":"` + base64.StdEncoding.EncodeToString([]byte(data)) + `"}` + "\n" } diff --git a/sandboxd/server/port_test.go b/sandboxd/server/port_test.go index 0c2cd54ac..440dc2384 100644 --- a/sandboxd/server/port_test.go +++ b/sandboxd/server/port_test.go @@ -65,10 +65,8 @@ func TestPortRelayRejectsNonUpgradeWithoutDialing(t *testing.T) { return nil, pool.ErrUnknownSandbox }} ts := newTestServer(t, "", mgr, nil) - resp := getPort(t, ts, "/v1/sandboxes/sb_1/ports/49983", "tok", false) - defer resp.Body.Close() - if resp.StatusCode != http.StatusUpgradeRequired { - t.Errorf("status %d, want 426", resp.StatusCode) + if code := portStatus(t, ts, "/v1/sandboxes/sb_1/ports/49983", "tok", false); code != http.StatusUpgradeRequired { + t.Errorf("status %d, want 426", code) } if dialed { t.Error("a non-upgrade GET dialed the guest; it must not wake a hibernated sandbox") @@ -77,10 +75,8 @@ func TestPortRelayRejectsNonUpgradeWithoutDialing(t *testing.T) { func TestPortRelayRequiresToken(t *testing.T) { ts := newTestServer(t, "", &fakeManager{}, nil) - resp := getPort(t, ts, "/v1/sandboxes/sb_1/ports/49983", "", true) - defer resp.Body.Close() - if resp.StatusCode != http.StatusUnauthorized { - t.Errorf("status %d, want 401", resp.StatusCode) + if code := portStatus(t, ts, "/v1/sandboxes/sb_1/ports/49983", "", true); code != http.StatusUnauthorized { + t.Errorf("status %d, want 401", code) } } @@ -88,10 +84,8 @@ func TestPortRelayRejectsBadPort(t *testing.T) { ts := newTestServer(t, "", &fakeManager{}, nil) for _, port := range []string{"0", "65536", "http", "-1", "+22"} { t.Run(port, func(t *testing.T) { - resp := getPort(t, ts, "/v1/sandboxes/sb_1/ports/"+port, "tok", true) - defer resp.Body.Close() - if resp.StatusCode != http.StatusBadRequest { - t.Errorf("status %d, want 400", resp.StatusCode) + if code := portStatus(t, ts, "/v1/sandboxes/sb_1/ports/"+port, "tok", true); code != http.StatusBadRequest { + t.Errorf("status %d, want 400", code) } }) } @@ -111,10 +105,8 @@ func TestPortRelayMapsDialErrors(t *testing.T) { t.Run(tt.name, func(t *testing.T) { mgr := &fakeManager{dialPort: func(string, string, uint16) (net.Conn, error) { return nil, tt.err }} ts := newTestServer(t, "", mgr, nil) - resp := getPort(t, ts, "/v1/sandboxes/sb_1/ports/49983", "tok", true) - defer resp.Body.Close() - if resp.StatusCode != tt.want { - t.Errorf("status %d, want %d", resp.StatusCode, tt.want) + if code := portStatus(t, ts, "/v1/sandboxes/sb_1/ports/49983", "tok", true); code != tt.want { + t.Errorf("status %d, want %d", code, tt.want) } }) } @@ -128,10 +120,8 @@ func TestPortRelayTakesTheRootTokenAsTheOperator(t *testing.T) { }} ts := newTestServer(t, "root-secret", mgr, nil) for _, token := range []string{"root-secret", "sandbox-token"} { - resp := getPort(t, ts, "/v1/sandboxes/sb_1/ports/49983", token, true) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusConflict { - t.Errorf("%s: status %d, want 409 for a paused sandbox", token, resp.StatusCode) + if code := portStatus(t, ts, "/v1/sandboxes/sb_1/ports/49983", token, true); code != http.StatusConflict { + t.Errorf("%s: status %d, want 409 for a paused sandbox", token, code) } } if len(got) != 2 || got[0] != "" || got[1] != "sandbox-token" { @@ -219,17 +209,15 @@ func TestSpliceTearsDownWhenTheGuestFails(t *testing.T) { }) } -func TestPortRelayRefusedAfterCloseRelays(t *testing.T) { - ts, srv := newPortServer(t, func(c net.Conn) { _, _ = io.Copy(io.Discard, c) }) - srv.CloseRelays() +type failedGuestConn struct { + net.Conn +} - conn := dialPortConn(t, ts, portRequest) - if _, err := io.Copy(io.Discard, conn); err != nil { - t.Errorf("late relay not refused promptly: %v", err) - } +func (c *failedGuestConn) Read([]byte) (int, error) { + return 0, errors.New("guest connection reset") } -func newPortServer(t *testing.T, guest func(net.Conn)) (*httptest.Server, *Server) { +func newPortServer(t *testing.T, guest guestFunc) (*httptest.Server, *Server) { t.Helper() mgr := &fakeManager{dialPort: func(string, string, uint16) (net.Conn, error) { relayEnd, guestEnd := net.Pipe() @@ -254,7 +242,7 @@ func dialPortConn(t *testing.T, ts *httptest.Server, req string) net.Conn { return conn } -func getPort(t *testing.T, ts *httptest.Server, path, token string, upgrade bool) *http.Response { +func portStatus(t *testing.T, ts *httptest.Server, path, token string, upgrade bool) int { t.Helper() req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, ts.URL+path, nil) if err != nil { @@ -271,13 +259,6 @@ func getPort(t *testing.T, ts *httptest.Server, path, token string, upgrade bool if err != nil { t.Fatalf("do: %v", err) } - return resp -} - -type failedGuestConn struct { - net.Conn -} - -func (c *failedGuestConn) Read([]byte) (int, error) { - return 0, errors.New("guest connection reset") + _ = resp.Body.Close() + return resp.StatusCode } diff --git a/sandboxd/server/reader_test.go b/sandboxd/server/reader_test.go index 8dbd0118c..84e5bbd3e 100644 --- a/sandboxd/server/reader_test.go +++ b/sandboxd/server/reader_test.go @@ -104,7 +104,7 @@ func TestSleepingSecondInfoBeforeACallWakes(t *testing.T) { } } -func TestSleepingFirstWriteWakesAsBefore(t *testing.T) { +func TestSleepingFirstWriteWakesTheSandboxWithoutAReader(t *testing.T) { for name, first := range map[string]string{"exec": execFrame, "write": line(wire.FsWrite{Path: "/x"}), "port": line(wire.PortForward{Port: 80})} { t.Run(name, func(t *testing.T) { env := newSleepingEnv(t) @@ -352,16 +352,6 @@ func TestSleepingCloseRelaysEndsReaderConnections(t *testing.T) { } } -func captureLogs(t *testing.T) *logSink { - t.Helper() - sink := &logSink{} - logger := log.GetGlobalLogger() - prev := *logger - *logger = zerolog.New(sink) - t.Cleanup(func() { *logger = prev }) - return sink -} - type logSink struct { mu sync.Mutex buf strings.Builder @@ -395,74 +385,11 @@ type sleepingEnv struct { readerClosed chan struct{} } -func newSleepingEnv(t *testing.T) *sleepingEnv { - t.Helper() - env := &sleepingEnv{frames: &frameLog{}, guestDials: &atomic.Int32{}, readerClosed: make(chan struct{}, 4)} - env.reader = &fakeReader{src: "snap-1", serve: tagServer("reader", env.frames, env.readerClosed)} - env.manager = &fakeManager{reader: env.reader} - dialer := &fakeDialer{dial: func(context.Context, string) (net.Conn, error) { - env.guestDials.Add(1) - relayEnd, guestEnd := net.Pipe() - go tagServer("guest", env.frames, nil)(guestEnd) - return relayEnd, nil - }} - env.srv = New("", "node:7777", env.manager, dialer, nil, nil, nil, nil) - env.ts = httptest.NewServer(env.srv.Handler()) - t.Cleanup(func() { - env.srv.CloseRelays() - env.ts.Close() - }) - return env -} - func (e *sleepingEnv) upgrade(t *testing.T) (net.Conn, *bufio.Reader) { t.Helper() return upgradeConn(t, e.ts) } -func send(t *testing.T, conn net.Conn, frame string) { - t.Helper() - if _, err := io.WriteString(conn, frame); err != nil { - t.Fatalf("write frame: %v", err) - } -} - -func expectLine(t *testing.T, r *bufio.Reader, want string) { - t.Helper() - got, err := r.ReadString('\n') - if err != nil { - t.Fatalf("read reply (want %q): %v", want, err) - } - if strings.TrimSpace(got) != want { - t.Fatalf("reply %q, want %q", got, want) - } -} - -func tagServer(tag string, frames *frameLog, closed chan<- struct{}) func(net.Conn) { - return func(c net.Conn) { - defer func() { - _ = c.Close() - if closed != nil { - select { - case closed <- struct{}{}: - default: - } - } - }() - r := bufio.NewReaderSize(c, 1<<20) - for { - got, err := r.ReadString('\n') - if err != nil { - return - } - frames.add(tag + ":" + got) - if _, err = io.WriteString(c, tag+"\n"); err != nil { - return - } - } - } -} - type frameLog struct { mu sync.Mutex entries []string @@ -486,7 +413,7 @@ type fakeReader struct { wakes int dials int dialErr error - serve func(net.Conn) + serve guestFunc } func (f *fakeReader) source() string { @@ -534,3 +461,76 @@ func (f *fakeReader) dial() (net.Conn, string, error) { go f.serve(readerEnd) return relayEnd, f.src, nil } + +func captureLogs(t *testing.T) *logSink { + t.Helper() + sink := &logSink{} + logger := log.GetGlobalLogger() + prev := *logger + *logger = zerolog.New(sink) + t.Cleanup(func() { *logger = prev }) + return sink +} + +func newSleepingEnv(t *testing.T) *sleepingEnv { + t.Helper() + env := &sleepingEnv{frames: &frameLog{}, guestDials: &atomic.Int32{}, readerClosed: make(chan struct{}, 4)} + env.reader = &fakeReader{src: "snap-1", serve: tagServer("reader", env.frames, env.readerClosed)} + env.manager = &fakeManager{reader: env.reader} + dialer := &fakeDialer{dial: func(context.Context, string) (net.Conn, error) { + env.guestDials.Add(1) + relayEnd, guestEnd := net.Pipe() + go tagServer("guest", env.frames, nil)(guestEnd) + return relayEnd, nil + }} + env.srv = New("", "node:7777", env.manager, dialer, nil, nil, nil, nil) + env.ts = httptest.NewServer(env.srv.Handler()) + t.Cleanup(func() { + env.srv.CloseRelays() + env.ts.Close() + }) + return env +} + +func send(t *testing.T, conn net.Conn, frame string) { + t.Helper() + if _, err := io.WriteString(conn, frame); err != nil { + t.Fatalf("write frame: %v", err) + } +} + +func expectLine(t *testing.T, r *bufio.Reader, want string) { + t.Helper() + got, err := r.ReadString('\n') + if err != nil { + t.Fatalf("read reply (want %q): %v", want, err) + } + if strings.TrimSpace(got) != want { + t.Fatalf("reply %q, want %q", got, want) + } +} + +func tagServer(tag string, frames *frameLog, closed chan<- struct{}) guestFunc { + return func(c net.Conn) { + defer func() { + _ = c.Close() + if closed != nil { + select { + case closed <- struct{}{}: + default: + } + } + }() + r := bufio.NewReaderSize(c, 1<<20) + for { + got, err := r.ReadString('\n') + if err != nil { + return + } + frames.add(tag + ":" + got) + if _, err = io.WriteString(c, tag+"\n"); err != nil { + return + } + } + } +} diff --git a/sandboxd/server/relay_test.go b/sandboxd/server/relay_test.go index 69a3e09f2..3ff0ca4b9 100644 --- a/sandboxd/server/relay_test.go +++ b/sandboxd/server/relay_test.go @@ -39,14 +39,21 @@ var ( ) func TestCloseRelaysRefusesLateRelays(t *testing.T) { - ts, srv := newRelayServer(t, func(c net.Conn) { - _, _ = io.Copy(io.Discard, c) - }) - srv.CloseRelays() - - conn := dialPortConn(t, ts, agentRequest) - if _, err := io.Copy(io.Discard, conn); err != nil { - t.Errorf("late relay not refused promptly: %v", err) + for _, tt := range []struct { + name string + serve func(*testing.T, guestFunc) (*httptest.Server, *Server) + request string + }{ + {"agent", newRelayServer, agentRequest}, + {"port", newPortServer, portRequest}, + } { + t.Run(tt.name, func(t *testing.T) { + ts, srv := tt.serve(t, func(c net.Conn) { _, _ = io.Copy(io.Discard, c) }) + srv.CloseRelays() + if _, err := io.Copy(io.Discard, dialPortConn(t, ts, tt.request)); err != nil { + t.Errorf("late relay not refused promptly: %v", err) + } + }) } } @@ -91,11 +98,11 @@ func TestRelayDeliversPipelinedBytes(t *testing.T) { ts, _ := newRelayServer(t, func(c net.Conn) { defer c.Close() r := bufio.NewReader(c) - line, err := r.ReadString('\n') + frame, err := r.ReadString('\n') if err != nil { return } - _, _ = io.WriteString(c, line) + _, _ = io.WriteString(c, frame) }) r := bufio.NewReader(dialPortConn(t, ts, agentRequest+execFrame)) @@ -127,13 +134,8 @@ func TestRelayClientDisconnectClosesGuest(t *testing.T) { func TestRelayEndsWhenItsClaimIsFenced(t *testing.T) { relayWithFence := func(fenced <-chan struct{}) (net.Conn, *bufio.Reader) { - dialer := &fakeDialer{dial: func(context.Context, string) (net.Conn, error) { - relayEnd, guestEnd := net.Pipe() - go func() { _, _ = io.Copy(io.Discard, guestEnd) }() - return relayEnd, nil - }} - srv := New("", "node:7777", &fakeManager{fenced: fenced}, dialer, nil, nil, nil, nil) - return upgradeConn(t, serveTestServer(t, srv)) + guest := pipeDialer(func(c net.Conn) { _, _ = io.Copy(io.Discard, c) }) + return upgradeConn(t, newTestServer(t, "", &fakeManager{fenced: fenced}, guest)) } fenced := make(chan struct{}) conn, r := relayWithFence(fenced) @@ -209,12 +211,12 @@ func TestAuditTeeRecordsRequestLines(t *testing.T) { } { t.Run(tt.name+"/"+rd.name, func(t *testing.T) { var got []string - tee := &auditTee{r: rd.wrap(strings.NewReader(tt.in)), record: func(line []byte) { - if len(line) > pool.AuditLineCap { + tee := &auditTee{r: rd.wrap(strings.NewReader(tt.in)), record: func(frame []byte) { + if len(frame) > pool.AuditLineCap { got = append(got, "oversized") return } - got = append(got, string(line)) + got = append(got, string(frame)) }} var out bytes.Buffer if _, err := io.Copy(&out, tee); err != nil || out.String() != tt.in { @@ -228,33 +230,22 @@ func TestAuditTeeRecordsRequestLines(t *testing.T) { } } -func TestCloseRelaysDrains(t *testing.T) { +func TestCloseRelaysClosesAnOpenClientConn(t *testing.T) { ts, srv := newRelayServer(t, func(c net.Conn) { _, _ = io.Copy(io.Discard, c) }) - _, r := upgradeConn(t, ts) - closed := make(chan struct{}) - go func() { - defer close(closed) - _, _ = io.Copy(io.Discard, r) - }() + conn, r := upgradeConn(t, ts) srv.CloseRelays() - select { - case <-closed: - case <-time.After(3 * time.Second): - t.Fatal("client conn still open after CloseRelays") + _ = conn.SetReadDeadline(time.Now().Add(time.Second)) + if _, err := io.Copy(io.Discard, r); err != nil { + t.Errorf("client after CloseRelays: %v, want EOF", err) } } -func newRelayServer(t *testing.T, guest func(net.Conn)) (*httptest.Server, *Server) { +func newRelayServer(t *testing.T, guest guestFunc) (*httptest.Server, *Server) { t.Helper() - dialer := &fakeDialer{dial: func(context.Context, string) (net.Conn, error) { - relayEnd, guestEnd := net.Pipe() - go guest(guestEnd) - return relayEnd, nil - }} - srv := New("", "node:7777", &fakeManager{}, dialer, nil, nil, nil, nil) + srv := New("", "node:7777", &fakeManager{}, pipeDialer(guest), nil, nil, nil, nil) return serveTestServer(t, srv), srv } diff --git a/sandboxd/server/server_test.go b/sandboxd/server/server_test.go index 1054ada9a..a6c47b953 100644 --- a/sandboxd/server/server_test.go +++ b/sandboxd/server/server_test.go @@ -1,7 +1,6 @@ package server import ( - "bytes" "context" "encoding/json/v2" "errors" @@ -35,7 +34,7 @@ func TestOwnerAddrOmitsAnUnspecifiedHost(t *testing.T) { } } -func TestClaimHappyPath(t *testing.T) { +func TestClaimFillsTheDefaultKeyAndTTL(t *testing.T) { var gotKey types.PoolKey var gotTTL time.Duration mgr := &fakeManager{ @@ -49,19 +48,12 @@ func TestClaimHappyPath(t *testing.T) { } ts := newTestServer(t, "", mgr, nil) - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", - strings.NewReader(`{"template":"rt:24.04","ttl_seconds":60}`)) - if err != nil { - t.Fatalf("claim: %v", err) - } + resp := postJSON(t, ts.URL+"/v1/claim", "", `{"template":"rt:24.04","ttl_seconds":60}`) defer resp.Body.Close() if resp.StatusCode != http.StatusOK { t.Fatalf("status %d, want 200", resp.StatusCode) } - var cr ClaimResponse - if err := json.UnmarshalRead(resp.Body, &cr); err != nil { - t.Fatalf("decode: %v", err) - } + cr := decodeJSON[ClaimResponse](t, resp.Body) if cr.ID != "sb_1" || cr.Token != "tok" || !cr.Deadline.Equal(time.Unix(42, 0)) { t.Errorf("got %+v", cr) } @@ -85,10 +77,7 @@ func TestClaimCarriesAnEgressOptOut(t *testing.T) { } { mgr := &fakeManager{netRoute: types.NetRouteRelay} ts := newTestServer(t, "", mgr, nil) - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(body)) - if err != nil { - t.Fatalf("claim: %v", err) - } + resp := postJSON(t, ts.URL+"/v1/claim", "", body) var cr ClaimResponse decodeErr := json.UnmarshalRead(resp.Body, &cr) _ = resp.Body.Close() @@ -121,13 +110,8 @@ func TestClaimErrorMapping(t *testing.T) { }, } ts := newTestServer(t, "", mgr, nil) - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(tt.body)) - if err != nil { - t.Fatalf("claim: %v", err) - } - defer resp.Body.Close() - if resp.StatusCode != tt.want { - t.Errorf("status %d, want %d", resp.StatusCode, tt.want) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/claim", "", tt.body); code != tt.want { + t.Errorf("status %d, want %d", code, tt.want) } }) } @@ -136,10 +120,8 @@ func TestClaimErrorMapping(t *testing.T) { func TestHibernateVolumeCaptureMapsConflict(t *testing.T) { mgr := &fakeManager{hibernate: func(string, string) error { return pool.ErrVolumeCapture }} ts := newTestServer(t, "", mgr, nil) - resp := doReq(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/hibernate", "tok", "") - defer resp.Body.Close() - if resp.StatusCode != http.StatusConflict { - t.Errorf("status %d, want 409", resp.StatusCode) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/hibernate", "tok", ""); code != http.StatusConflict { + t.Errorf("status %d, want 409", code) } } @@ -160,10 +142,7 @@ func TestRenewGrantsAndReportsTheDeadline(t *testing.T) { if resp.StatusCode != http.StatusOK { t.Fatalf("status %d, want 200", resp.StatusCode) } - var got RenewResponse - if err := json.UnmarshalRead(resp.Body, &got); err != nil { - t.Fatalf("decode: %v", err) - } + got := decodeJSON[RenewResponse](t, resp.Body) if !got.Deadline.Equal(want) { t.Errorf("deadline %v, want %v", got.Deadline, want) } @@ -181,10 +160,8 @@ func TestRenewWithoutABodyTakesTheNodeDefault(t *testing.T) { }} ts := newTestServer(t, "", mgr, nil) - resp := postJSON(t, ts.URL+"/v1/sandboxes/sb_1/renew", "tok", "") - defer resp.Body.Close() - if resp.StatusCode != http.StatusOK { - t.Fatalf("status %d, want 200: an absent body is the documented way to ask for the default", resp.StatusCode) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/renew", "tok", ""); code != http.StatusOK { + t.Fatalf("status %d, want 200: an absent body is the documented way to ask for the default", code) } if !called || gotTTL != 0 { t.Errorf("manager saw ttl %v (called=%v), want the zero that means the node default", gotTTL, called) @@ -199,10 +176,8 @@ func TestRenewMatchesBodyKeysCaseInsensitively(t *testing.T) { }} ts := newTestServer(t, "", mgr, nil) - resp := postJSON(t, ts.URL+"/v1/sandboxes/sb_1/renew", "tok", `{"TTL_Seconds":3600}`) - defer resp.Body.Close() - if resp.StatusCode != http.StatusOK || gotTTL != time.Hour { - t.Errorf("status %d ttl %v, want 200 and 1h", resp.StatusCode, gotTTL) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/renew", "tok", `{"TTL_Seconds":3600}`); code != http.StatusOK || gotTTL != time.Hour { + t.Errorf("status %d ttl %v, want 200 and 1h", code, gotTTL) } } @@ -214,10 +189,8 @@ func TestClaimMatchesBodyKeysCaseInsensitively(t *testing.T) { }} ts := newTestServer(t, "", mgr, nil) - resp := postJSON(t, ts.URL+"/v1/claim", "", `{"Template":"rt:24.04"}`) - defer resp.Body.Close() - if resp.StatusCode != http.StatusOK || gotKey.Template != "rt:24.04" { - t.Errorf("status %d template %q, want 200 and rt:24.04", resp.StatusCode, gotKey.Template) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/claim", "", `{"Template":"rt:24.04"}`); code != http.StatusOK || gotKey.Template != "rt:24.04" { + t.Errorf("status %d template %q, want 200 and rt:24.04", code, gotKey.Template) } } @@ -227,10 +200,8 @@ func TestRenewRejectsWrongToken(t *testing.T) { }} ts := newTestServer(t, "", mgr, nil) - resp := postJSON(t, ts.URL+"/v1/sandboxes/sb_1/renew", "wrong", `{"ttl_seconds":60}`) - defer resp.Body.Close() - if resp.StatusCode != http.StatusNotFound { - t.Errorf("status %d, want 404", resp.StatusCode) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/renew", "wrong", `{"ttl_seconds":60}`); code != http.StatusNotFound { + t.Errorf("status %d, want 404", code) } } @@ -239,10 +210,7 @@ func TestInfoReportsTheNodeAdvertiseAddr(t *testing.T) { resp := doReq(t, http.MethodGet, ts.URL+"/v1/info", "sekret", "") defer resp.Body.Close() - var info InfoResponse - if err := json.UnmarshalRead(resp.Body, &info); err != nil { - t.Fatalf("decode: %v", err) - } + info := decodeJSON[InfoResponse](t, resp.Body) if info.AdvertiseAddr != "node:7777" { t.Errorf("advertise_addr = %q, want node:7777", info.AdvertiseAddr) } @@ -274,9 +242,7 @@ func TestSetTemplateLabelsReplacesTheMapForRootOrTheTenant(t *testing.T) { ts := newTenantTestServer(t, "root", []tenants.TenantSpec{{Name: "acme", Token: "acme-tok"}}, mgr, nil) put := func(t *testing.T, query, token, body string) int { t.Helper() - resp := doReq(t, http.MethodPut, ts.URL+"/v1/templates/labels?"+query, token, body) - _ = resp.Body.Close() - return resp.StatusCode + return statusOf(t, http.MethodPut, ts.URL+"/v1/templates/labels?"+query, token, body) } if code := put(t, "template=app:v1&size=medium", "root", `{"labels":{"v1":"sha256:aa"}}`); code != http.StatusNoContent { @@ -358,9 +324,7 @@ func TestEnvPatchVerb(t *testing.T) { ts := newTenantTestServer(t, "root", []tenants.TenantSpec{{Name: "acme", Token: "acme-tok"}}, mgr, nil) call := func(t *testing.T, token, body string) int { t.Helper() - resp := doReq(t, http.MethodPatch, ts.URL+"/v1/sandboxes/sb_1/env", token, body) - resp.Body.Close() - return resp.StatusCode + return statusOf(t, http.MethodPatch, ts.URL+"/v1/sandboxes/sb_1/env", token, body) } if code := call(t, "root", `{"env":{"GW":{"value":"Bearer a","guest":false},"OLD":null}}`); code != http.StatusNoContent { @@ -405,10 +369,8 @@ func TestClaimRefusesAnAmbiguousHostOnlyFlag(t *testing.T) { for _, path := range []string{"/v1/claim", "/v1/checkpoints/ck_00000000000000aa/claim"} { mgr := &fakeManager{claimCheckpoint: func(string) (*types.Sandbox, error) { return &types.Sandbox{ID: "sb_1"}, nil }} ts := newTestServer(t, "sekret", mgr, nil) - resp := postJSON(t, ts.URL+path, "sekret", `{"template":"rt:24.04","env":{"KEY":`+entry+`}}`) - resp.Body.Close() - if resp.StatusCode != http.StatusBadRequest || mgr.warmCalls+mgr.provisionCalls != 0 || mgr.gotEnv != nil { - t.Errorf("%s %s: %d, manager called %d times, want 400 and none", path, entry, resp.StatusCode, mgr.warmCalls+mgr.provisionCalls) + if code := statusOf(t, http.MethodPost, ts.URL+path, "sekret", `{"template":"rt:24.04","env":{"KEY":`+entry+`}}`); code != http.StatusBadRequest || mgr.warmCalls+mgr.provisionCalls != 0 || mgr.gotEnv != nil { + t.Errorf("%s %s: %d, manager called %d times, want 400 and none", path, entry, code, mgr.warmCalls+mgr.provisionCalls) } } } @@ -425,41 +387,29 @@ func TestAPITokenGuard(t *testing.T) { want int }{ {"claim no token", "/v1/claim", http.MethodPost, "", http.StatusUnauthorized}, - {"claim wrong token", "/v1/claim", http.MethodPost, "Bearer nope", http.StatusUnauthorized}, - {"claim right token", "/v1/claim", http.MethodPost, "Bearer sekret", http.StatusOK}, + {"claim wrong token", "/v1/claim", http.MethodPost, "nope", http.StatusUnauthorized}, + {"claim right token", "/v1/claim", http.MethodPost, "sekret", http.StatusOK}, {"volumes no token", "/v1/volumes", http.MethodGet, "", http.StatusUnauthorized}, - {"volumes right token", "/v1/volumes", http.MethodGet, "Bearer sekret", http.StatusOK}, + {"volumes right token", "/v1/volumes", http.MethodGet, "sekret", http.StatusOK}, {"sandboxes no token", "/v1/sandboxes", http.MethodGet, "", http.StatusUnauthorized}, - {"sandboxes right token", "/v1/sandboxes", http.MethodGet, "Bearer sekret", http.StatusOK}, + {"sandboxes right token", "/v1/sandboxes", http.MethodGet, "sekret", http.StatusOK}, {"info no token", "/v1/info", http.MethodGet, "", http.StatusUnauthorized}, - {"info right token", "/v1/info", http.MethodGet, "Bearer sekret", http.StatusOK}, + {"info right token", "/v1/info", http.MethodGet, "sekret", http.StatusOK}, {"put pools no token", "/v1/pools", http.MethodPut, "", http.StatusUnauthorized}, - {"put pools right token", "/v1/pools", http.MethodPut, "Bearer sekret", http.StatusOK}, + {"put pools right token", "/v1/pools", http.MethodPut, "sekret", http.StatusOK}, {"healthz open", "/healthz", http.MethodGet, "", http.StatusOK}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - var body io.Reader + var body string switch tt.method { case http.MethodPost: - body = strings.NewReader(`{"template":"rt:24.04"}`) + body = `{"template":"rt:24.04"}` case http.MethodPut: - body = strings.NewReader(`{"pools":[{"template":"rt:24.04","net":"none","size":"small","warm":1}]}`) - } - req, err := http.NewRequestWithContext(t.Context(), tt.method, ts.URL+tt.path, body) - if err != nil { - t.Fatalf("request: %v", err) - } - if tt.auth != "" { - req.Header.Set("Authorization", tt.auth) - } - resp, err := http.DefaultClient.Do(req) - if err != nil { - t.Fatalf("do: %v", err) + body = `{"pools":[{"template":"rt:24.04","net":"none","size":"small","warm":1}]}` } - defer resp.Body.Close() - if resp.StatusCode != tt.want { - t.Errorf("status %d, want %d", resp.StatusCode, tt.want) + if got := statusOf(t, tt.method, ts.URL+tt.path, tt.auth, body); got != tt.want { + t.Errorf("status %d, want %d", got, tt.want) } }) } @@ -482,10 +432,7 @@ func TestVolumeCatalogReturnsScopedFleetProjection(t *testing.T) { ts := newMeshTestServer(t, "root", mgr, placer, nil) resp := doReq(t, http.MethodGet, ts.URL+"/v1/volumes", "acme-tok", "") defer resp.Body.Close() - var got VolumeListResponse - if err := json.UnmarshalRead(resp.Body, &got); err != nil { - t.Fatalf("decode: %v", err) - } + got := decodeJSON[VolumeListResponse](t, resp.Body) want := []types.VolumeInfo{{ Name: "imagenet", DefaultMount: "/volumes/imagenet", SizeBytes: 42, Available: true, Nodes: 3, }} @@ -506,10 +453,7 @@ func TestSandboxIndexReturnsScopedAppliedVolumes(t *testing.T) { ts := newTenantTestServer(t, "root", []tenants.TenantSpec{{Name: "acme", Token: "acme-tok"}}, mgr, nil) resp := doReq(t, http.MethodGet, ts.URL+"/v1/sandboxes", "acme-tok", "") defer resp.Body.Close() - var got SandboxListResponse - if err := json.UnmarshalRead(resp.Body, &got); err != nil { - t.Fatalf("decode: %v", err) - } + got := decodeJSON[SandboxListResponse](t, resp.Body) want := []types.Volume{{Name: "imagenet", Mount: "/datasets/imagenet"}} if len(got.Sandboxes) != 1 || got.Sandboxes[0].ID != "sb_1" || !slices.Equal(got.Sandboxes[0].Volumes, want) { t.Errorf("sandboxes = %+v, want sb_1 with %+v", got.Sandboxes, want) @@ -524,10 +468,7 @@ func TestSandboxesReportClaimedAt(t *testing.T) { ts := newTenantTestServer(t, "root", nil, mgr, nil) resp := doReq(t, http.MethodGet, ts.URL+"/v1/sandboxes", "root", "") defer resp.Body.Close() - var got SandboxListResponse - if err := json.UnmarshalRead(resp.Body, &got); err != nil { - t.Fatalf("decode: %v", err) - } + got := decodeJSON[SandboxListResponse](t, resp.Body) if len(got.Sandboxes) != 1 || !got.Sandboxes[0].ClaimedAt.Equal(claimed) { t.Errorf("sandboxes = %+v, want claimed_at %v", got.Sandboxes, claimed) } @@ -545,22 +486,9 @@ func TestSandboxByIDCarriesTheTokenOnlyForTheRootCredential(t *testing.T) { } { t.Run(tt.name, func(t *testing.T) { ts := newTestServer(t, tt.apiToken, mgr, nil) - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, ts.URL+"/v1/sandboxes/sb_1", nil) - if err != nil { - t.Fatalf("request: %v", err) - } - if tt.bearer != "" { - req.Header.Set("Authorization", "Bearer "+tt.bearer) - } - resp, err := http.DefaultClient.Do(req) - if err != nil { - t.Fatalf("sandbox: %v", err) - } + resp := doReq(t, http.MethodGet, ts.URL+"/v1/sandboxes/sb_1", tt.bearer, "") defer resp.Body.Close() - var got pool.SandboxSummary - if err := json.UnmarshalRead(resp.Body, &got); err != nil { - t.Fatalf("decode: %v", err) - } + got := decodeJSON[pool.SandboxSummary](t, resp.Body) if resp.StatusCode != http.StatusOK || got.Token != tt.want { t.Errorf("status %d token %q, want 200 and %q", resp.StatusCode, got.Token, tt.want) } @@ -582,40 +510,27 @@ func TestSandboxesNarrowToAClaimRef(t *testing.T) { return nil }} ts := newTenantTestServer(t, "root", nil, mgr, nil) - resp := doReq(t, http.MethodGet, ts.URL+tt.path, "root", "") - resp.Body.Close() - if resp.StatusCode != http.StatusOK || got != tt.want { - t.Errorf("%s: status %d, manager saw claim_ref %q, want %q", tt.path, resp.StatusCode, got, tt.want) + if code := statusOf(t, http.MethodGet, ts.URL+tt.path, "root", ""); code != http.StatusOK || got != tt.want { + t.Errorf("%s: status %d, manager saw claim_ref %q, want %q", tt.path, code, got, tt.want) } } } func TestTenantAuthMatrix(t *testing.T) { mgr := &fakeManager{tenantClaims: map[string]int{"acme": 2}} - tenants := []tenants.TenantSpec{{Name: "acme", Token: "acme-tok"}, {Name: "beta", Token: "beta-tok"}} - ts := newTenantTestServer(t, "sekret", tenants, mgr, nil) + specs := []tenants.TenantSpec{{Name: "acme", Token: "acme-tok"}, {Name: "beta", Token: "beta-tok"}} + ts := newTenantTestServer(t, "sekret", specs, mgr, nil) do := func(t *testing.T, method, path, auth string) *http.Response { t.Helper() - var body io.Reader + var body string switch { case path == "/v1/claim": - body = strings.NewReader(`{"template":"rt:24.04"}`) + body = `{"template":"rt:24.04"}` case method == http.MethodPut: - body = strings.NewReader(`{"pools":[]}`) - } - req, err := http.NewRequestWithContext(t.Context(), method, ts.URL+path, body) - if err != nil { - t.Fatalf("request: %v", err) - } - if auth != "" { - req.Header.Set("Authorization", "Bearer "+auth) - } - resp, err := http.DefaultClient.Do(req) - if err != nil { - t.Fatalf("do: %v", err) + body = `{"pools":[]}` } - return resp + return doReq(t, method, ts.URL+path, auth, body) } for _, tt := range []struct { @@ -661,13 +576,7 @@ func TestMetricsTenantGauge(t *testing.T) { mgr := &fakeManager{tenantClaims: map[string]int{"acme": 2, "beta": 0}} ts := newTestServer(t, "", mgr, nil) - resp, err := http.Get(ts.URL + "/metrics") - if err != nil { - t.Fatalf("metrics: %v", err) - } - defer resp.Body.Close() - raw, _ := io.ReadAll(resp.Body) - body := string(raw) + body := scrapeMetrics(t, ts.URL) if !strings.Contains(body, `sandboxd_tenant_claims{tenant="acme"} 2`) || !strings.Contains(body, `sandboxd_tenant_claims{tenant="beta"} 0`) { t.Errorf("tenant gauge missing:\n%s", body) @@ -680,13 +589,7 @@ func TestMetricsClaimHistogramIsCumulative(t *testing.T) { latency.Nanos = 1_500_000 ts := newTestServer(t, "", &fakeManager{counters: pool.Counters{ClaimCloneLatency: latency}}, nil) - resp, err := http.Get(ts.URL + "/metrics") - if err != nil { - t.Fatalf("metrics: %v", err) - } - defer resp.Body.Close() - raw, _ := io.ReadAll(resp.Body) - body := string(raw) + body := scrapeMetrics(t, ts.URL) for _, want := range []string{ "# TYPE sandboxd_claim_duration_seconds histogram\n", `sandboxd_claim_duration_seconds_bucket{tier="clone",le="0.0005"} 2` + "\n", @@ -733,18 +636,10 @@ func TestClaimRedirectsAreCounted(t *testing.T) { } { t.Run(tt.name, func(t *testing.T) { ts := newMeshTestServer(t, "", tt.mgr, tt.placer, nil) - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(tt.body)) - if err != nil { - t.Fatalf("claim: %v", err) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/claim", "", tt.body); code != http.StatusOK { + t.Fatalf("claim status %d, want 200", code) } - _ = resp.Body.Close() - metrics, err := http.Get(ts.URL + "/metrics") - if err != nil { - t.Fatalf("metrics: %v", err) - } - defer metrics.Body.Close() - raw, _ := io.ReadAll(metrics.Body) - if want := "sandboxd_claim_redirects_total " + tt.want + "\n"; !strings.Contains(string(raw), want) { + if want, raw := "sandboxd_claim_redirects_total "+tt.want+"\n", scrapeMetrics(t, ts.URL); !strings.Contains(raw, want) { t.Errorf("missing %q in:\n%s", want, raw) } }) @@ -753,10 +648,8 @@ func TestClaimRedirectsAreCounted(t *testing.T) { func TestPutPoolsRejectsUnknownFields(t *testing.T) { ts := newTestServer(t, "sekret", &fakeManager{}, nil) - resp := doReq(t, http.MethodPut, ts.URL+"/v1/pools", "sekret", `{"pools":[{"template":"rt:24.04","net":"none","size":"small","egres":{"allow":[]}}]}`) - defer resp.Body.Close() - if resp.StatusCode != http.StatusBadRequest { - t.Fatalf("status %d, want 400", resp.StatusCode) + if code := statusOf(t, http.MethodPut, ts.URL+"/v1/pools", "sekret", `{"pools":[{"template":"rt:24.04","net":"none","size":"small","egres":{"allow":[]}}]}`); code != http.StatusBadRequest { + t.Fatalf("status %d, want 400", code) } } @@ -773,9 +666,7 @@ func TestInstanceMetadataVerb(t *testing.T) { ts := newTenantTestServer(t, "sekret", []tenants.TenantSpec{{Name: "acme", Token: "acme-tok"}}, mgr, nil) put := func(token, id, body string) int { t.Helper() - resp := doReq(t, http.MethodPut, ts.URL+"/v1/sandboxes/"+id+"/instance-metadata", token, body) - resp.Body.Close() - return resp.StatusCode + return statusOf(t, http.MethodPut, ts.URL+"/v1/sandboxes/"+id+"/instance-metadata", token, body) } valid := `{"region": "local", "tags": ["a", "b"]}` if code := put("sekret", "sb_1", valid); code != http.StatusNoContent { @@ -818,10 +709,7 @@ func TestDrainEndpoints(t *testing.T) { {http.MethodDelete, false}, } { resp := doReq(t, tt.method, ts.URL+"/v1/drain", "sekret", "") - var info InfoResponse - if err := json.UnmarshalRead(resp.Body, &info); err != nil { - t.Fatalf("decode: %v", err) - } + info := decodeJSON[InfoResponse](t, resp.Body) resp.Body.Close() if resp.StatusCode != http.StatusOK || info.Draining != tt.draining || mgr.draining != tt.draining { t.Fatalf("%s: status=%d draining=%v mgr=%v, want 200/%v", tt.method, resp.StatusCode, info.Draining, mgr.draining, tt.draining) @@ -853,16 +741,13 @@ func TestPutPoolsUpdatesTargets(t *testing.T) { if len(got) != 1 || got[0].Template != "rt:24.04" || got[0].Warm != 2 { t.Fatalf("SetPools got %+v", got) } - var out InfoResponse - if err := json.UnmarshalRead(resp.Body, &out); err != nil { - t.Fatalf("decode: %v", err) - } + out := decodeJSON[InfoResponse](t, resp.Body) if len(out.Pools) != 1 || out.Pools[0].Target != 2 || !out.Pools[0].Golden { t.Fatalf("info response %+v, want updated pool", out) } } -func TestSandboxVerbFlows(t *testing.T) { +func TestReleaseAndHibernateMapEachOutcomeToItsStatus(t *testing.T) { verbs := []struct { name string hook func(f *fakeManager, h sandboxVerbFunc) @@ -876,10 +761,10 @@ func TestSandboxVerbFlows(t *testing.T) { err error want int }{ - {"ok", "Bearer tok", nil, http.StatusNoContent}, - {"unknown or bad token", "Bearer bad", pool.ErrUnknownSandbox, http.StatusNotFound}, + {"ok", "tok", nil, http.StatusNoContent}, + {"unknown or bad token", "bad", pool.ErrUnknownSandbox, http.StatusNotFound}, {"missing bearer", "", nil, http.StatusUnauthorized}, - {"engine failure", "Bearer tok", errors.New("engine failed"), http.StatusInternalServerError}, + {"engine failure", "tok", errors.New("engine failed"), http.StatusInternalServerError}, } for _, v := range verbs { for _, tt := range tests { @@ -891,24 +776,11 @@ func TestSandboxVerbFlows(t *testing.T) { return tt.err }) ts := newTestServer(t, "", mgr, nil) - req, err := http.NewRequestWithContext(t.Context(), http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/"+v.name, nil) - if err != nil { - t.Fatalf("request: %v", err) - } - if tt.auth != "" { - req.Header.Set("Authorization", tt.auth) + if got := statusOf(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/"+v.name, tt.auth, ""); got != tt.want { + t.Errorf("status %d, want %d", got, tt.want) } - resp, err := http.DefaultClient.Do(req) - if err != nil { - t.Fatalf("do: %v", err) - } - defer resp.Body.Close() - if resp.StatusCode != tt.want { - t.Errorf("status %d, want %d", resp.StatusCode, tt.want) - } - wantToken := strings.TrimPrefix(tt.auth, "Bearer ") - if tt.auth != "" && (gotID != "sb_1" || gotToken != wantToken) { - t.Errorf("%s called with (%q, %q), want (sb_1, %q)", v.name, gotID, gotToken, wantToken) + if tt.auth != "" && (gotID != "sb_1" || gotToken != tt.auth) { + t.Errorf("%s called with (%q, %q), want (sb_1, %q)", v.name, gotID, gotToken, tt.auth) } }) } @@ -917,7 +789,7 @@ func TestSandboxVerbFlows(t *testing.T) { func TestReleaseOperatorToken(t *testing.T) { const rootTok, sbTok = "sekret", "sb-secret" - tenants := []tenants.TenantSpec{{Name: "acme", Token: "acme-tok"}} + specs := []tenants.TenantSpec{{Name: "acme", Token: "acme-tok"}} tests := []struct { name string auth string @@ -927,10 +799,10 @@ func TestReleaseOperatorToken(t *testing.T) { releaseErr error want int }{ - {"root token releases by id", "Bearer " + rootTok, true, false, "", nil, http.StatusNoContent}, - {"sandbox token releases self", "Bearer " + sbTok, false, true, sbTok, nil, http.StatusNoContent}, - {"tenant token gets no operator release", "Bearer acme-tok", false, true, "acme-tok", pool.ErrUnknownSandbox, http.StatusNotFound}, - {"wrong token 404s", "Bearer nope", false, true, "nope", pool.ErrUnknownSandbox, http.StatusNotFound}, + {"root token releases by id", rootTok, true, false, "", nil, http.StatusNoContent}, + {"sandbox token releases self", sbTok, false, true, sbTok, nil, http.StatusNoContent}, + {"tenant token gets no operator release", "acme-tok", false, true, "acme-tok", pool.ErrUnknownSandbox, http.StatusNotFound}, + {"wrong token 404s", "nope", false, true, "nope", pool.ErrUnknownSandbox, http.StatusNotFound}, {"missing bearer", "", false, false, "", nil, http.StatusUnauthorized}, } for _, tt := range tests { @@ -947,21 +819,9 @@ func TestReleaseOperatorToken(t *testing.T) { return tt.releaseErr }, } - ts := newTenantTestServer(t, rootTok, tenants, mgr, nil) - req, err := http.NewRequestWithContext(t.Context(), http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/release", nil) - if err != nil { - t.Fatalf("request: %v", err) - } - if tt.auth != "" { - req.Header.Set("Authorization", tt.auth) - } - resp, err := http.DefaultClient.Do(req) - if err != nil { - t.Fatalf("do: %v", err) - } - defer resp.Body.Close() - if resp.StatusCode != tt.want { - t.Errorf("status %d, want %d", resp.StatusCode, tt.want) + ts := newTenantTestServer(t, rootTok, specs, mgr, nil) + if got := statusOf(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/release", tt.auth, ""); got != tt.want { + t.Errorf("status %d, want %d", got, tt.want) } if opCalled != tt.wantOp { t.Errorf("operator release called=%v, want %v", opCalled, tt.wantOp) @@ -1027,7 +887,7 @@ func TestAgentErrorPaths(t *testing.T) { } } -func TestForkFlow(t *testing.T) { +func TestForkAnswersOwnedChildrenAndMapsBadRequests(t *testing.T) { mgr := &fakeManager{fork: func(_, token string, count int, ttl time.Duration) ([]*types.Sandbox, error) { switch { case token != "tok": @@ -1048,29 +908,16 @@ func TestForkFlow(t *testing.T) { ts := newTestServer(t, "sekret", mgr, nil) post := func(t *testing.T, auth, body string) *http.Response { - req, err := http.NewRequestWithContext(t.Context(), http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/fork", strings.NewReader(body)) - if err != nil { - t.Fatalf("request: %v", err) - } - if auth != "" { - req.Header.Set("Authorization", auth) - } - resp, err := http.DefaultClient.Do(req) - if err != nil { - t.Fatalf("do: %v", err) - } - return resp + t.Helper() + return postJSON(t, ts.URL+"/v1/sandboxes/sb_1/fork", auth, body) } - resp := post(t, "Bearer sekret", `{"token":"tok","count":2,"ttl_seconds":60}`) + resp := post(t, "sekret", `{"token":"tok","count":2,"ttl_seconds":60}`) defer resp.Body.Close() if resp.StatusCode != http.StatusOK { t.Fatalf("status %d, want 200", resp.StatusCode) } - var fr ForkResponse - if err := json.UnmarshalRead(resp.Body, &fr); err != nil { - t.Fatalf("decode: %v", err) - } + fr := decodeJSON[ForkResponse](t, resp.Body) if len(fr.Children) != 2 || fr.Children[0].ID != "sb_c0" || fr.Children[0].OwnerAddr != "node:7777" { t.Errorf("children %+v, want two with this node as owner", fr.Children) } @@ -1079,11 +926,11 @@ func TestForkFlow(t *testing.T) { name, auth, body string want int }{ - {"bad count", "Bearer sekret", `{"token":"tok","count":17,"ttl_seconds":60}`, http.StatusBadRequest}, - {"bad body", "Bearer sekret", `{oops`, http.StatusBadRequest}, - {"unknown or bad sandbox token", "Bearer sekret", `{"token":"bad","count":1}`, http.StatusNotFound}, + {"bad count", "sekret", `{"token":"tok","count":17,"ttl_seconds":60}`, http.StatusBadRequest}, + {"bad body", "sekret", `{oops`, http.StatusBadRequest}, + {"unknown or bad sandbox token", "sekret", `{"token":"bad","count":1}`, http.StatusNotFound}, {"missing api token", "", `{"token":"tok","count":1}`, http.StatusUnauthorized}, - {"sandbox token is no api token", "Bearer tok", `{"token":"tok","count":1}`, http.StatusUnauthorized}, + {"sandbox token is no api token", "tok", `{"token":"tok","count":1}`, http.StatusUnauthorized}, } { t.Run(tt.name, func(t *testing.T) { resp := post(t, tt.auth, tt.body) @@ -1095,7 +942,7 @@ func TestForkFlow(t *testing.T) { } } -func TestPromoteAndDeleteTemplateFlow(t *testing.T) { +func TestPromoteAndDeleteTemplateMapEachOutcomeToItsStatus(t *testing.T) { var gotKey types.PoolKey mgr := &fakeManager{ promoteContentDigest: "sha256:promoted-digest", @@ -1121,20 +968,12 @@ func TestPromoteAndDeleteTemplateFlow(t *testing.T) { ts := newTestServer(t, "sekret", mgr, nil) promote := func(t *testing.T, auth, body string) (int, PromoteResponse) { - req, _ := http.NewRequestWithContext(t.Context(), http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/promote", strings.NewReader(body)) - if auth != "" { - req.Header.Set("Authorization", auth) - } - resp, err := http.DefaultClient.Do(req) - if err != nil { - t.Fatalf("do: %v", err) - } + t.Helper() + resp := postJSON(t, ts.URL+"/v1/sandboxes/sb_1/promote", auth, body) defer resp.Body.Close() var result PromoteResponse if resp.StatusCode == http.StatusOK { - if err := json.UnmarshalRead(resp.Body, &result); err != nil { - t.Fatalf("decode promote response: %v", err) - } + result = decodeJSON[PromoteResponse](t, resp.Body) } return resp.StatusCode, result } @@ -1142,10 +981,10 @@ func TestPromoteAndDeleteTemplateFlow(t *testing.T) { name, auth, body string want int }{ - {"ok", "Bearer sekret", `{"token":"tok","template":"tpl:x"}`, http.StatusOK}, - {"bad name", "Bearer sekret", `{"token":"tok","template":"_bad"}`, http.StatusBadRequest}, - {"pooled", "Bearer sekret", `{"token":"tok","template":"pooled"}`, http.StatusConflict}, - {"bad sandbox token", "Bearer sekret", `{"token":"bad","template":"tpl:x"}`, http.StatusNotFound}, + {"ok", "sekret", `{"token":"tok","template":"tpl:x"}`, http.StatusOK}, + {"bad name", "sekret", `{"token":"tok","template":"_bad"}`, http.StatusBadRequest}, + {"pooled", "sekret", `{"token":"tok","template":"pooled"}`, http.StatusConflict}, + {"bad sandbox token", "sekret", `{"token":"bad","template":"tpl:x"}`, http.StatusNotFound}, {"missing api token", "", `{"token":"tok","template":"tpl:x"}`, http.StatusUnauthorized}, } { t.Run("promote/"+tt.name, func(t *testing.T) { @@ -1160,25 +999,16 @@ func TestPromoteAndDeleteTemplateFlow(t *testing.T) { } del := func(auth, query string) int { - req, _ := http.NewRequestWithContext(t.Context(), http.MethodDelete, ts.URL+"/v1/templates?"+query, nil) - if auth != "" { - req.Header.Set("Authorization", auth) - } - resp, err := http.DefaultClient.Do(req) - if err != nil { - t.Fatalf("do: %v", err) - } - defer resp.Body.Close() - return resp.StatusCode + return statusOf(t, http.MethodDelete, ts.URL+"/v1/templates?"+query, auth, "") } - if got := del("Bearer sekret", "template=tpl:x&net=none&size=small"); got != http.StatusNoContent { + if got := del("sekret", "template=tpl:x&net=none&size=small"); got != http.StatusNoContent { t.Errorf("delete status %d, want 204", got) } want := types.PoolKey{Template: "tpl:x", Net: types.NetNone, Size: types.SizeSmall} if gotKey != want { t.Errorf("delete key %+v, want %+v (claim defaults applied)", gotKey, want) } - if got := del("Bearer sekret", "template=nope"); got != http.StatusNotFound { + if got := del("sekret", "template=nope"); got != http.StatusNotFound { t.Errorf("unknown delete status %d, want 404", got) } @@ -1187,7 +1017,7 @@ func TestPromoteAndDeleteTemplateFlow(t *testing.T) { } } -func TestCheckpointFlow(t *testing.T) { +func TestCheckpointCanBeTakenClaimedAndDeleted(t *testing.T) { mgr := &fakeManager{ checkpoint: func(id, token, name string) (types.Checkpoint, error) { if id != "sb_1" || token != "tok" { @@ -1245,9 +1075,7 @@ func TestDeleteCheckpointNoForwardQueryParam(t *testing.T) { ts := newTestServer(t, "api", mgr, &fakeDialer{}) del := func(path string) int { - resp := doReq(t, http.MethodDelete, ts.URL+path, "api", "") - defer resp.Body.Close() - return resp.StatusCode + return statusOf(t, http.MethodDelete, ts.URL+path, "api", "") } if got := del("/v1/checkpoints/ck_0011223344556677"); got != http.StatusNoContent { @@ -1298,16 +1126,10 @@ func TestClaimRedirectsOnWarmMiss(t *testing.T) { }} ts := newMeshTestServer(t, "", mgr, &fakePlacer{addrs: []string{"node-b:7777", "node-c:7777"}}, nil) - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(`{"template":"rt:24.04"}`)) - if err != nil { - t.Fatalf("claim: %v", err) - } + resp := postJSON(t, ts.URL+"/v1/claim", "", `{"template":"rt:24.04"}`) defer resp.Body.Close() - var cr ClaimResponse - if err := json.UnmarshalRead(resp.Body, &cr); err != nil { - t.Fatalf("decode: %v", err) - } - if len(cr.Redirect) != 2 || cr.Redirect[0] != "node-b:7777" { + cr := decodeJSON[ClaimResponse](t, resp.Body) + if !slices.Equal(cr.Redirect, []string{"node-b:7777", "node-c:7777"}) { t.Errorf("redirect %v, want [node-b:7777 node-c:7777]", cr.Redirect) } if cr.ID != "" { @@ -1338,15 +1160,9 @@ func TestClaimRedirectsToTemplateOwner(t *testing.T) { } ts := newMeshTestServer(t, "", mgr, &fakePlacer{owners: []string{"node-b:7777"}}, nil) - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(`{"template":"tpl"}`)) - if err != nil { - t.Fatalf("claim: %v", err) - } + resp := postJSON(t, ts.URL+"/v1/claim", "", `{"template":"tpl"}`) defer resp.Body.Close() - var cr ClaimResponse - if err := json.UnmarshalRead(resp.Body, &cr); err != nil { - t.Fatalf("decode: %v", err) - } + cr := decodeJSON[ClaimResponse](t, resp.Body) gotRedirect := len(cr.Redirect) > 0 if gotRedirect != tt.wantRedirect { t.Errorf("redirect %v, want redirect=%v", cr.Redirect, tt.wantRedirect) @@ -1362,17 +1178,13 @@ func TestATemplateDigestGatesDeleteAndLabelWrites(t *testing.T) { mgr := &fakeManager{deleteGolden: func(types.PoolKey) error { return pool.ErrTemplateReplaced }} ts := newTestServer(t, "sekret", mgr, nil) - resp := doReq(t, http.MethodDelete, ts.URL+"/v1/templates?template=ns%2Fapp&digest=sha256%3Aold", "sekret", "") - defer resp.Body.Close() - if resp.StatusCode != http.StatusPreconditionFailed || mgr.gotDigest != "sha256:old" { - t.Errorf("status=%d digest=%q, want 412 for a replaced generation and the digest passed through", resp.StatusCode, mgr.gotDigest) + if code := statusOf(t, http.MethodDelete, ts.URL+"/v1/templates?template=ns%2Fapp&digest=sha256%3Aold", "sekret", ""); code != http.StatusPreconditionFailed || mgr.gotDigest != "sha256:old" { + t.Errorf("status=%d digest=%q, want 412 for a replaced generation and the digest passed through", code, mgr.gotDigest) } mgr.labelErr, mgr.gotDigest = pool.ErrTemplateReplaced, "" - resp = doReq(t, http.MethodPut, ts.URL+"/v1/templates/labels?template=ns%2Fapp&digest=sha256%3Aold", "sekret", `{"labels":{"v1":"sha256:new"}}`) - defer resp.Body.Close() - if resp.StatusCode != http.StatusPreconditionFailed || mgr.gotDigest != "sha256:old" { - t.Errorf("labels: status=%d digest=%q, want 412 for a replaced generation and the digest passed through", resp.StatusCode, mgr.gotDigest) + if code := statusOf(t, http.MethodPut, ts.URL+"/v1/templates/labels?template=ns%2Fapp&digest=sha256%3Aold", "sekret", `{"labels":{"v1":"sha256:new"}}`); code != http.StatusPreconditionFailed || mgr.gotDigest != "sha256:old" { + t.Errorf("labels: status=%d digest=%q, want 412 for a replaced generation and the digest passed through", code, mgr.gotDigest) } } @@ -1380,51 +1192,23 @@ func TestDeleteTemplateRedirectsToOwner(t *testing.T) { mgr := &fakeManager{} ts := newMeshTestServer(t, "", mgr, &fakePlacer{owners: []string{"node-b:7777"}}, nil) - req, err := http.NewRequestWithContext(t.Context(), http.MethodDelete, ts.URL+"/v1/templates?template=tpl", nil) - if err != nil { - t.Fatalf("request: %v", err) - } - resp, err := http.DefaultClient.Do(req) - if err != nil { - t.Fatalf("delete: %v", err) - } + resp := doReq(t, http.MethodDelete, ts.URL+"/v1/templates?template=tpl", "", "") defer resp.Body.Close() if resp.StatusCode != http.StatusOK { t.Fatalf("status %d, want 200 redirect", resp.StatusCode) } - var cr ClaimResponse - if decodeErr := json.UnmarshalRead(resp.Body, &cr); decodeErr != nil { - t.Fatalf("decode: %v", decodeErr) - } + cr := decodeJSON[ClaimResponse](t, resp.Body) if len(cr.Redirect) != 1 || cr.Redirect[0] != "node-b:7777" { t.Errorf("redirect %v, want [node-b:7777]", cr.Redirect) } - req2, err := http.NewRequestWithContext(t.Context(), http.MethodDelete, ts.URL+"/v1/templates?template=tpl&no_redirect=1", nil) - if err != nil { - t.Fatalf("request: %v", err) - } - resp2, err := http.DefaultClient.Do(req2) - if err != nil { - t.Fatalf("delete: %v", err) - } - defer resp2.Body.Close() - if resp2.StatusCode != http.StatusNotFound { - t.Errorf("status %d, want 404 with no_redirect despite known owners", resp2.StatusCode) + if got := statusOf(t, http.MethodDelete, ts.URL+"/v1/templates?template=tpl&no_redirect=1", "", ""); got != http.StatusNotFound { + t.Errorf("status %d, want 404 with no_redirect despite known owners", got) } ts2 := newMeshTestServer(t, "", mgr, &fakePlacer{}, nil) - req3, err := http.NewRequestWithContext(t.Context(), http.MethodDelete, ts2.URL+"/v1/templates?template=tpl", nil) - if err != nil { - t.Fatalf("request: %v", err) - } - resp3, err := http.DefaultClient.Do(req3) - if err != nil { - t.Fatalf("delete: %v", err) - } - defer resp3.Body.Close() - if resp3.StatusCode != http.StatusNotFound { - t.Errorf("status %d, want 404 when no owner known", resp3.StatusCode) + if got := statusOf(t, http.MethodDelete, ts2.URL+"/v1/templates?template=tpl", "", ""); got != http.StatusNotFound { + t.Errorf("status %d, want 404 when no owner known", got) } } @@ -1434,13 +1218,9 @@ func TestClaimProvisionsWhenNoCandidate(t *testing.T) { }} ts := newMeshTestServer(t, "", mgr, &fakePlacer{addrs: nil}, nil) - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(`{"template":"rt:24.04","claim_ref":"ns1/w1"}`)) - if err != nil { - t.Fatalf("claim: %v", err) - } + resp := postJSON(t, ts.URL+"/v1/claim", "", `{"template":"rt:24.04","claim_ref":"ns1/w1"}`) defer resp.Body.Close() - var cr ClaimResponse - _ = json.UnmarshalRead(resp.Body, &cr) + cr := decodeJSON[ClaimResponse](t, resp.Body) if cr.ID != "sb_local" || len(cr.Redirect) != 0 { t.Errorf("got %+v, want local sandbox", cr) } @@ -1486,15 +1266,9 @@ func TestVolumeClaimUsesWarmBeforeRedirectOrProvision(t *testing.T) { } ts := newMeshTestServer(t, "", mgr, placer, nil) - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(`{"template":"rt:24.04","volumes":[{"name":"imagenet"}]}`)) - if err != nil { - t.Fatalf("claim: %v", err) - } + resp := postJSON(t, ts.URL+"/v1/claim", "", `{"template":"rt:24.04","volumes":[{"name":"imagenet"}]}`) defer resp.Body.Close() - var got ClaimResponse - if err := json.UnmarshalRead(resp.Body, &got); err != nil { - t.Fatalf("decode: %v", err) - } + got := decodeJSON[ClaimResponse](t, resp.Body) if resp.StatusCode != http.StatusOK || got.ID != tt.wantID || !slices.Equal(got.Redirect, tt.wantRedirect) { t.Errorf("status=%d response=%+v", resp.StatusCode, got) } @@ -1603,10 +1377,7 @@ func TestVolumeClaimUsesVolumeAndTemplateIntersections(t *testing.T) { if err != nil { t.Fatalf("encode request: %v", err) } - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", bytes.NewReader(body)) - if err != nil { - t.Fatalf("claim: %v", err) - } + resp := postJSON(t, ts.URL+"/v1/claim", "", string(body)) defer resp.Body.Close() if resp.StatusCode != tt.wantStatus { t.Fatalf("status=%d, want %d", resp.StatusCode, tt.wantStatus) @@ -1683,10 +1454,7 @@ func TestVolumeClaimKeepsPoolContentAgainstPeerTemplates(t *testing.T) { if err != nil { t.Fatalf("encode request: %v", err) } - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", bytes.NewReader(body)) - if err != nil { - t.Fatalf("claim: %v", err) - } + resp := postJSON(t, ts.URL+"/v1/claim", "", string(body)) defer resp.Body.Close() if resp.StatusCode != tt.wantStatus { @@ -1712,7 +1480,7 @@ func TestVolumeClaimKeepsPoolContentAgainstPeerTemplates(t *testing.T) { func TestForeignTemplateGossipNeverEscalates(t *testing.T) { hash := ClaimRequest{Template: "tpl"}.Key().Hash() - tenants := []tenants.TenantSpec{{Name: "acme", Token: "acme-tok"}, {Name: "beta", Token: "beta-tok"}} + specs := []tenants.TenantSpec{{Name: "acme", Token: "acme-tok"}, {Name: "beta", Token: "beta-tok"}} for _, tt := range []struct { name, token string volumes []types.Volume @@ -1732,7 +1500,7 @@ func TestForeignTemplateGossipNeverEscalates(t *testing.T) { types.TemplateGossipHash(hash, "acme"): {"peer:7777"}, types.TemplateGossipHash(hash, types.TemplateRootScope): {"peer:7777"}, }} - mgr.tenants = tenants + mgr.tenants = specs ts := newMeshTestServer(t, "root-tok", mgr, placer, nil) body, _ := json.Marshal(ClaimRequest{Template: "tpl", Volumes: tt.volumes}) @@ -1741,10 +1509,7 @@ func TestForeignTemplateGossipNeverEscalates(t *testing.T) { if resp.StatusCode != http.StatusOK { t.Fatalf("status=%d, want 200", resp.StatusCode) } - var cr ClaimResponse - if err := json.UnmarshalRead(resp.Body, &cr); err != nil { - t.Fatalf("decode: %v", err) - } + cr := decodeJSON[ClaimResponse](t, resp.Body) if got := len(cr.Redirect) > 0; got != tt.wantRedirect { t.Errorf("redirect=%v (%v), want %v", got, cr.Redirect, tt.wantRedirect) } @@ -1783,8 +1548,7 @@ func TestARequiredPromotedClaimIsNeverSentToAWarmPeer(t *testing.T) { body, _ := json.Marshal(ClaimRequest{Template: "ns/app", RequirePromoted: tt.require}) resp := postJSON(t, ts.URL+"/v1/claim", "sekret", string(body)) defer resp.Body.Close() - var cr ClaimResponse - _ = json.UnmarshalRead(resp.Body, &cr) + cr := decodeJSON[ClaimResponse](t, resp.Body) if resp.StatusCode != tt.wantStatus || (len(cr.Redirect) > 0) != tt.wantWarm { t.Errorf("status=%d redirect=%v, want %d and warm redirect %v", resp.StatusCode, cr.Redirect, tt.wantStatus, tt.wantWarm) } @@ -1823,10 +1587,8 @@ func TestAPlainClaimRequiringAPromotedTemplateNeverColdBoots(t *testing.T) { ts := newTestServer(t, "sekret", mgr, nil) body, _ := json.Marshal(ClaimRequest{Template: "ns/app", RequirePromoted: tt.require}) - resp := postJSON(t, ts.URL+"/v1/claim", "sekret", string(body)) - defer resp.Body.Close() - if resp.StatusCode != tt.wantStatus || mgr.provisionCalls != 1 || mgr.gotRequirePromoted != tt.wantPromoted { - t.Errorf("status=%d provisions=%d promoted=%v, want %d 1 %v", resp.StatusCode, mgr.provisionCalls, mgr.gotRequirePromoted, tt.wantStatus, tt.wantPromoted) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/claim", "sekret", string(body)); code != tt.wantStatus || mgr.provisionCalls != 1 || mgr.gotRequirePromoted != tt.wantPromoted { + t.Errorf("status=%d provisions=%d promoted=%v, want %d 1 %v", code, mgr.provisionCalls, mgr.gotRequirePromoted, tt.wantStatus, tt.wantPromoted) } if tt.require && mgr.warmCalls != 0 { t.Errorf("a claim requiring the promoted template asked the warm pool %d times", mgr.warmCalls) @@ -1844,14 +1606,8 @@ func TestVolumeClaimValidatesKeyBeforePlacement(t *testing.T) { placer := &fakePlacer{volumeOwners: []string{"volume:7777"}} ts := newMeshTestServer(t, "", mgr, placer, nil) - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader( - `{"template":"rt:24.04","net":"egress","volumes":[{"name":"imagenet"}]}`)) - if err != nil { - t.Fatalf("claim: %v", err) - } - defer resp.Body.Close() - if resp.StatusCode != http.StatusConflict { - t.Errorf("status=%d, want 409", resp.StatusCode) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/claim", "", `{"template":"rt:24.04","net":"egress","volumes":[{"name":"imagenet"}]}`); code != http.StatusConflict { + t.Errorf("status=%d, want 409", code) } if mgr.provisionCalls != 0 || placer.volumeOwnerCalls != 0 || placer.templateVolumeCalls != 0 { t.Errorf("provision=%d volume owners=%d template-volume owners=%d", @@ -1868,13 +1624,8 @@ func TestVolumeClaimQuotaDoesNotRedirect(t *testing.T) { } ts := newMeshTestServer(t, "", mgr, placer, nil) - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(`{"template":"rt:24.04","volumes":[{"name":"imagenet"}]}`)) - if err != nil { - t.Fatalf("claim: %v", err) - } - defer resp.Body.Close() - if resp.StatusCode != http.StatusTooManyRequests { - t.Errorf("status=%d, want 429", resp.StatusCode) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/claim", "", `{"template":"rt:24.04","volumes":[{"name":"imagenet"}]}`); code != http.StatusTooManyRequests { + t.Errorf("status=%d, want 429", code) } if mgr.warmCalls != 1 || mgr.provisionCalls != 0 || placer.candidateCalls != 0 || placer.volumeCandidateCalls != 0 { @@ -1899,13 +1650,8 @@ func TestVolumeClaimRejectsShapeBeforePlacement(t *testing.T) { placer := &fakePlacer{addrs: []string{"warm-peer:7777"}, owners: []string{"owner:7777"}} ts := newMeshTestServer(t, "", mgr, placer, nil) - resp, err := http.Post(ts.URL+"/v1/claim", "application/json", strings.NewReader(body)) - if err != nil { - t.Fatalf("claim: %v", err) - } - resp.Body.Close() - if resp.StatusCode != http.StatusBadRequest { - t.Errorf("body=%s status=%d, want 400", body, resp.StatusCode) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/claim", "", body); code != http.StatusBadRequest { + t.Errorf("body=%s status=%d, want 400", body, code) } if mgr.provisionCalls != 0 || placer.candidateCalls != 0 { t.Errorf("body=%s provision=%d candidates=%d", body, mgr.provisionCalls, placer.candidateCalls) @@ -1913,7 +1659,7 @@ func TestVolumeClaimRejectsShapeBeforePlacement(t *testing.T) { } } -func TestOwnerEndpoint(t *testing.T) { +func TestOwnerEndpointNamesThisNodeForAGoodTokenAnd404sABadOne(t *testing.T) { mgr := &fakeManager{socket: func(_, token string) (string, error) { if token == "good" { return "/v/sock", nil @@ -1927,10 +1673,7 @@ func TestOwnerEndpoint(t *testing.T) { if resp.StatusCode != http.StatusOK { t.Fatalf("status %d, want 200", resp.StatusCode) } - var body struct { - OwnerAddr string `json:"owner_addr"` - } - _ = json.UnmarshalRead(resp.Body, &body) + body := decodeJSON[ownerReply](t, resp.Body) if body.OwnerAddr != "node-a:7777" { t.Errorf("owner %q, want node-a:7777", body.OwnerAddr) } @@ -1953,10 +1696,8 @@ func TestOwnerEndpointAnswers404ForAClaimItsFencedNodeCannotServe(t *testing.T) } ts := newMeshTestServer(t, "", mgr, nil, nil) - resp := doReq(t, http.MethodGet, ts.URL+"/v1/sandboxes/sb_1/owner", "good", "") - defer resp.Body.Close() - if resp.StatusCode != http.StatusNotFound || adopted { - t.Errorf("fenced owner: status %d adopt tried %v, want 404 without an adoption", resp.StatusCode, adopted) + if code := statusOf(t, http.MethodGet, ts.URL+"/v1/sandboxes/sb_1/owner", "good", ""); code != http.StatusNotFound || adopted { + t.Errorf("fenced owner: status %d adopt tried %v, want 404 without an adoption", code, adopted) } } @@ -1972,10 +1713,7 @@ func TestOwnerEndpointAdoptsAnArchivedClaimItDoesNotHold(t *testing.T) { resp := doReq(t, http.MethodGet, ts.URL+"/v1/sandboxes/sb_1/owner", "good", "") defer resp.Body.Close() - var body struct { - OwnerAddr string `json:"owner_addr"` - } - _ = json.UnmarshalRead(resp.Body, &body) + body := decodeJSON[ownerReply](t, resp.Body) if resp.StatusCode != http.StatusOK || body.OwnerAddr != "node-a:7777" { t.Errorf("adopted: status %d owner %q, want 200 node-a:7777", resp.StatusCode, body.OwnerAddr) } @@ -2001,19 +1739,12 @@ func TestPreviewHandlerZeroDeadlineMintsLiveToken(t *testing.T) { ps := NewPreviewServer("secret", "node:7777", "node:7777", &fakePreviewMgr{}) ts := serveTestServer(t, New("", "node:7777", mgr, &fakeDialer{}, nil, nil, nil, ps)) - resp, err := http.Post(ts.URL+"/v1/sandboxes/sb_1/preview", "application/json", - strings.NewReader(`{"token":"tok","port":8080,"ttl_seconds":300}`)) - if err != nil { - t.Fatalf("preview: %v", err) - } + resp := postJSON(t, ts.URL+"/v1/sandboxes/sb_1/preview", "", `{"token":"tok","port":8080,"ttl_seconds":300}`) defer resp.Body.Close() if resp.StatusCode != http.StatusOK { t.Fatalf("status %d, want 200", resp.StatusCode) } - var out PreviewResponse - if err := json.UnmarshalRead(resp.Body, &out); err != nil { - t.Fatalf("decode: %v", err) - } + out := decodeJSON[PreviewResponse](t, resp.Body) _, token, _ := strings.Cut(out.URL, "/p/") token = strings.TrimSuffix(token, "/") claims, ok := ps.verify(token) @@ -2032,14 +1763,8 @@ func TestPreviewHandlerRejectsPortZero(t *testing.T) { ps := NewPreviewServer("secret", "node:7777", "node:7777", &fakePreviewMgr{}) ts := serveTestServer(t, New("", "node:7777", mgr, &fakeDialer{}, nil, nil, nil, ps)) - resp, err := http.Post(ts.URL+"/v1/sandboxes/sb_1/preview", "application/json", - strings.NewReader(`{"token":"tok","port":0}`)) - if err != nil { - t.Fatalf("preview: %v", err) - } - defer resp.Body.Close() - if resp.StatusCode != http.StatusBadRequest { - t.Errorf("status %d, want 400 for port 0", resp.StatusCode) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/preview", "", `{"token":"tok","port":0}`); code != http.StatusBadRequest { + t.Errorf("status %d, want 400 for port 0", code) } } @@ -2054,10 +1779,7 @@ func TestCheckpointClaimRedirectsToOwner(t *testing.T) { if resp.StatusCode != http.StatusOK { t.Fatalf("status = %d, want 200 carrying a redirect (the mesh redirect is a 200, not a 3xx)", resp.StatusCode) } - var got ClaimResponse - if err := json.UnmarshalRead(resp.Body, &got); err != nil { - t.Fatalf("decode: %v", err) - } + got := decodeJSON[ClaimResponse](t, resp.Body) if len(got.Redirect) != 1 || got.Redirect[0] != "owner-a:7777" { t.Errorf("redirect = %v, want [owner-a:7777]", got.Redirect) } @@ -2110,10 +1832,7 @@ func TestCheckpointClaimFallsBackToHealWhenNoOwner(t *testing.T) { if resp.StatusCode != http.StatusOK { t.Fatalf("status = %d, want 200", resp.StatusCode) } - var got ClaimResponse - if err := json.UnmarshalRead(resp.Body, &got); err != nil { - t.Fatalf("decode: %v", err) - } + got := decodeJSON[ClaimResponse](t, resp.Body) if got.ID != "sb_healed" { t.Errorf("id = %q, want the healed claim", got.ID) } @@ -2137,10 +1856,7 @@ func TestCheckpointClaimNoRedirectGoesStraightToHeal(t *testing.T) { if resp.StatusCode != http.StatusOK { t.Fatalf("status = %d, want 200", resp.StatusCode) } - var got ClaimResponse - if err := json.UnmarshalRead(resp.Body, &got); err != nil { - t.Fatalf("decode: %v", err) - } + got := decodeJSON[ClaimResponse](t, resp.Body) if len(got.Redirect) != 0 { t.Errorf("redirect = %v, want none: no_redirect must not bounce", got.Redirect) } @@ -2215,12 +1931,7 @@ func TestCheckpointBlobHeadUnauthenticated(t *testing.T) { ts := newTestServer(t, "sekret", mgr, nil) head := func(id string) int { - resp, err := http.Head(ts.URL + "/v1/checkpoints/" + id + "/blob") - if err != nil { - t.Fatalf("head: %v", err) - } - defer resp.Body.Close() - return resp.StatusCode + return statusOf(t, http.MethodHead, ts.URL+"/v1/checkpoints/"+id+"/blob", "", "") } if got := head(ckptID); got != http.StatusOK { t.Errorf("HEAD present, no token: status = %d, want 200", got) @@ -2229,13 +1940,8 @@ func TestCheckpointBlobHeadUnauthenticated(t *testing.T) { t.Errorf("HEAD missing, no token: status = %d, want 404", got) } - resp, err := http.Get(ts.URL + "/v1/checkpoints/" + ckptID + "/blob") - if err != nil { - t.Fatalf("get: %v", err) - } - defer resp.Body.Close() - if resp.StatusCode != http.StatusUnauthorized { - t.Errorf("GET no token: status = %d, want 401", resp.StatusCode) + if got := statusOf(t, http.MethodGet, ts.URL+"/v1/checkpoints/"+ckptID+"/blob", "", ""); got != http.StatusUnauthorized { + t.Errorf("GET no token: status = %d, want 401", got) } } @@ -2289,10 +1995,7 @@ func TestCheckpointClaimProbesRealPeerAndRedirects(t *testing.T) { if resp.StatusCode != http.StatusOK { t.Fatalf("status = %d, want 200 carrying a redirect", resp.StatusCode) } - var got ClaimResponse - if err := json.UnmarshalRead(resp.Body, &got); err != nil { - t.Fatalf("decode: %v", err) - } + got := decodeJSON[ClaimResponse](t, resp.Body) if len(got.Redirect) != 1 || got.Redirect[0] != tsB.URL { t.Errorf("redirect = %v, want [%s]", got.Redirect, tsB.URL) } @@ -2317,10 +2020,7 @@ func TestClaimRejectsMetadataOrClaimRefOverABound(t *testing.T) { mgr := &fakeManager{} ts := newTestServer(t, "sekret", mgr, nil) resp := postJSON(t, ts.URL+tt.path, "sekret", tt.body) - var got struct { - Error string `json:"error"` - } - _ = json.UnmarshalRead(resp.Body, &got) + got := decodeJSON[errorReply](t, resp.Body) resp.Body.Close() if resp.StatusCode != http.StatusBadRequest || !strings.Contains(got.Error, tt.want) { t.Errorf("%s: status %d error %q, want 400 naming %q", tt.path, resp.StatusCode, got.Error, tt.want) @@ -2355,10 +2055,8 @@ func TestClaimThreadsMetadataAndEnv(t *testing.T) { mgr := &fakeManager{} tt.setup(mgr) ts := newTestServer(t, "sekret", mgr, nil) - resp := postJSON(t, ts.URL+tt.path, "sekret", tt.body) - resp.Body.Close() - if resp.StatusCode != http.StatusOK || !maps.Equal(mgr.gotMetadata, want) || !mgr.gotEnv.Equal(wantEnv) { - t.Errorf("%s: status %d, manager saw metadata %v env %v, want 200 and %v %v", tt.name, resp.StatusCode, mgr.gotMetadata, mgr.gotEnv, want, wantEnv) + if code := statusOf(t, http.MethodPost, ts.URL+tt.path, "sekret", tt.body); code != http.StatusOK || !maps.Equal(mgr.gotMetadata, want) || !mgr.gotEnv.Equal(wantEnv) { + t.Errorf("%s: status %d, manager saw metadata %v env %v, want 200 and %v %v", tt.name, code, mgr.gotMetadata, mgr.gotEnv, want, wantEnv) } } } @@ -2380,10 +2078,8 @@ func TestSandboxesFilterByMetadata(t *testing.T) { } { mgr := &fakeManager{} ts := newTestServer(t, "root", mgr, nil) - resp := doReq(t, http.MethodGet, ts.URL+"/v1/sandboxes"+tt.query, "root", "") - resp.Body.Close() - if resp.StatusCode != tt.code || !maps.Equal(mgr.gotMetadata, tt.want) { - t.Errorf("%q: status %d, manager saw %v, want %d and %v", tt.query, resp.StatusCode, mgr.gotMetadata, tt.code, tt.want) + if code := statusOf(t, http.MethodGet, ts.URL+"/v1/sandboxes"+tt.query, "root", ""); code != tt.code || !maps.Equal(mgr.gotMetadata, tt.want) { + t.Errorf("%q: status %d, manager saw %v, want %d and %v", tt.query, code, mgr.gotMetadata, tt.code, tt.want) } } } @@ -2410,20 +2106,15 @@ func TestExpireActionThreadsThroughEveryDecoder(t *testing.T) { mgr := &fakeManager{} tt.setup(mgr) ts := newTestServer(t, "sekret", mgr, nil) - resp := postJSON(t, ts.URL+tt.path, "sekret", tt.body) - resp.Body.Close() - if resp.StatusCode != http.StatusOK || mgr.gotOnExpire != types.ExpireArchive { - t.Errorf("%s: status %d, manager saw on_expire %q, want 200 and archive", tt.name, resp.StatusCode, mgr.gotOnExpire) + if code := statusOf(t, http.MethodPost, ts.URL+tt.path, "sekret", tt.body); code != http.StatusOK || mgr.gotOnExpire != types.ExpireArchive { + t.Errorf("%s: status %d, manager saw on_expire %q, want 200 and archive", tt.name, code, mgr.gotOnExpire) } mgr = &fakeManager{} tt.setup(mgr) ts = newTestServer(t, "sekret", mgr, nil) - resp = postJSON(t, ts.URL+tt.path, "sekret", strings.Replace(tt.body, `"archive"`, `"pause"`, 1)) - var got struct { - Error string `json:"error"` - } - _ = json.UnmarshalRead(resp.Body, &got) + resp := postJSON(t, ts.URL+tt.path, "sekret", strings.Replace(tt.body, `"archive"`, `"pause"`, 1)) + got := decodeJSON[errorReply](t, resp.Body) resp.Body.Close() if resp.StatusCode != http.StatusBadRequest || !strings.Contains(got.Error, "on_expire") { t.Errorf("%s with on_expire pause: status %d error %q, want 400 naming on_expire", tt.name, resp.StatusCode, got.Error) @@ -2431,106 +2122,12 @@ func TestExpireActionThreadsThroughEveryDecoder(t *testing.T) { } } -func assertVolumeClaimResponse(t *testing.T, body io.Reader, wantRedirect string, wantRequirePromoted bool) { - t.Helper() - var got ClaimResponse - if err := json.UnmarshalRead(body, &got); err != nil { - t.Fatalf("decode: %v", err) - } - if wantRedirect != "" { - if !slices.Equal(got.Redirect, []string{wantRedirect}) { - t.Errorf("redirect=%v, want [%s]", got.Redirect, wantRedirect) - } - if got.RequirePromoted != wantRequirePromoted { - t.Errorf("require_promoted=%v, want %v", got.RequirePromoted, wantRequirePromoted) - } - return - } - if got.ID == "" || len(got.Redirect) != 0 { - t.Errorf("response=%+v, want local claim", got) - } -} - -func newTestServer(t *testing.T, apiToken string, mgr Manager, dialer Dialer) *httptest.Server { - t.Helper() - return newTenantTestServer(t, apiToken, nil, mgr, dialer) -} - -func newTenantTestServer(t *testing.T, apiToken string, tenants []tenants.TenantSpec, mgr Manager, dialer Dialer) *httptest.Server { - t.Helper() - if dialer == nil { - dialer = &fakeDialer{} - } - if fm, ok := mgr.(*fakeManager); ok { - fm.tenants = tenants - } - return serveTestServer(t, New(apiToken, "node:7777", mgr, dialer, nil, nil, nil, nil)) -} - -func newProbeAuthTestServer(t *testing.T, mgr Manager, probeKey []byte) *httptest.Server { - t.Helper() - return serveTestServer(t, New("", "node:7777", mgr, &fakeDialer{}, nil, nil, probeKey, nil)) -} - -func newMeshTestServer(t *testing.T, apiToken string, mgr Manager, placer Placer, prober CheckpointProber) *httptest.Server { - t.Helper() - return serveTestServer(t, New(apiToken, "node-a:7777", mgr, &fakeDialer{}, placer, prober, nil, nil)) -} - -func newProberTestServer(t *testing.T, apiToken string, mgr Manager, prober CheckpointProber) *httptest.Server { - t.Helper() - return newMeshTestServer(t, apiToken, mgr, nil, prober) -} - -func serveTestServer(t *testing.T, srv *Server) *httptest.Server { - t.Helper() - ts := httptest.NewServer(srv.Handler()) - t.Cleanup(func() { - ts.Close() - srv.CloseRelays() - }) - return ts -} - -func doReq(t *testing.T, method, url, token, body string) *http.Response { - t.Helper() - req, err := http.NewRequestWithContext(t.Context(), method, url, strings.NewReader(body)) - if err != nil { - t.Fatalf("request: %v", err) - } - req.Header.Set("Authorization", "Bearer "+token) - resp, err := http.DefaultClient.Do(req) - if err != nil { - t.Fatalf("do: %v", err) - } - return resp -} - -func postJSON(t *testing.T, url, token, body string) *http.Response { - t.Helper() - return doReq(t, http.MethodPost, url, token, body) +type ownerReply struct { + OwnerAddr string `json:"owner_addr"` } -func credToken(cred pool.Cred) string { - if cred.Operator { - return "" - } - return cred.Token -} - -func fakeClaimProvision(ctx context.Context, f *fakeManager, key types.PoolKey, o pool.ClaimOptions) (*types.Sandbox, error) { - f.provisionCalls++ - f.gotTenant = o.Tenant - f.gotClaimRef = o.ClaimRef - f.gotMetadata = o.Metadata - f.gotEnv = o.Env - f.gotOnExpire = o.OnExpire - f.gotRequirePromoted = o.RequirePromoted - f.gotNoEgress = o.NoEgress - if f.claim == nil { - return &types.Sandbox{ID: "sb_1", Token: "tok", Volumes: slices.Clone(o.Volumes)}, nil - } - return f.claim(ctx, key, o.TTL) +type errorReply struct { + Error string `json:"error"` } type claimFunc func(context.Context, types.PoolKey, time.Duration) (*types.Sandbox, error) @@ -2642,13 +2239,9 @@ func (f *fakeManager) PatchVolumes(_ context.Context, id, tenant string, p types func (f *fakeManager) ClaimWarm(ctx context.Context, key types.PoolKey, o pool.ClaimOptions) (*types.Sandbox, error) { f.warmCalls++ - f.gotTenant, f.gotEgressClass = o.Tenant, o.EgressClass - f.gotClaimRef = o.ClaimRef - f.gotMetadata = o.Metadata - f.gotEnv = o.Env - f.gotOnExpire = o.OnExpire + f.record(o) + f.gotEgressClass = o.EgressClass f.gotWarmVolumes = slices.Clone(o.Volumes) - f.gotNoEgress = o.NoEgress if f.warmClaim == nil { return nil, pool.ErrNoWarm } @@ -2658,7 +2251,13 @@ func (f *fakeManager) ClaimWarm(ctx context.Context, key types.PoolKey, o pool.C func (f *fakeManager) NetRoute(*types.Sandbox) types.NetRoute { return f.netRoute } func (f *fakeManager) ClaimProvision(ctx context.Context, key types.PoolKey, o pool.ClaimOptions) (*types.Sandbox, error) { - return fakeClaimProvision(ctx, f, key, o) + f.provisionCalls++ + f.record(o) + f.gotRequirePromoted = o.RequirePromoted + if f.claim == nil { + return &types.Sandbox{ID: "sb_1", Token: "tok", Volumes: slices.Clone(o.Volumes)}, nil + } + return f.claim(ctx, key, o.TTL) } func (f *fakeManager) Release(_ context.Context, id string, cred pool.Cred) error { @@ -2900,11 +2499,7 @@ func (f *fakeManager) Checkpoint(_ context.Context, id string, cred pool.Cred, n } func (f *fakeManager) ClaimCheckpoint(_ context.Context, ckptID string, o pool.ClaimOptions) (*types.Sandbox, error) { - onExpire, tenant, metadata := o.OnExpire, o.Tenant, o.Metadata - f.gotTenant = tenant - f.gotMetadata = metadata - f.gotEnv = o.Env - f.gotOnExpire = onExpire + f.record(o) if f.claimCheckpoint == nil { return nil, pool.ErrUnknownCheckpoint } @@ -2912,11 +2507,7 @@ func (f *fakeManager) ClaimCheckpoint(_ context.Context, ckptID string, o pool.C } func (f *fakeManager) ClaimCheckpointHeal(_ context.Context, ckptID string, o pool.ClaimOptions) (*types.Sandbox, error) { - onExpire, tenant, metadata := o.OnExpire, o.Tenant, o.Metadata - f.gotTenant = tenant - f.gotMetadata = metadata - f.gotEnv = o.Env - f.gotOnExpire = onExpire + f.record(o) f.healCalls++ if f.healCheckpoint == nil { return nil, pool.ErrUnknownCheckpoint @@ -2989,6 +2580,17 @@ func (f *fakeManager) HasCheckpoint(_ context.Context, ckptID string) bool { return f.hasCheckpoint[ckptID] } +func (f *fakeManager) record(o pool.ClaimOptions) { + f.gotTenant = o.Tenant + f.gotClaimRef = o.ClaimRef + f.gotMetadata = o.Metadata + f.gotEnv = o.Env + f.gotOnExpire = o.OnExpire + f.gotNoEgress = o.NoEgress +} + +type guestFunc func(net.Conn) + type fakeDialer struct { dial func(ctx context.Context, sock string) (net.Conn, error) } @@ -3060,3 +2662,135 @@ type fakeProber struct { func (f *fakeProber) Owners(context.Context, string) []string { return f.owners } func (f *fakeProber) Forget(id string) { f.forgotten = append(f.forgotten, id) } + +func assertVolumeClaimResponse(t *testing.T, body io.Reader, wantRedirect string, wantRequirePromoted bool) { + t.Helper() + got := decodeJSON[ClaimResponse](t, body) + if wantRedirect != "" { + if !slices.Equal(got.Redirect, []string{wantRedirect}) { + t.Errorf("redirect=%v, want [%s]", got.Redirect, wantRedirect) + } + if got.RequirePromoted != wantRequirePromoted { + t.Errorf("require_promoted=%v, want %v", got.RequirePromoted, wantRequirePromoted) + } + return + } + if got.ID == "" || len(got.Redirect) != 0 { + t.Errorf("response=%+v, want local claim", got) + } +} + +func newTestServer(t *testing.T, apiToken string, mgr Manager, dialer Dialer) *httptest.Server { + t.Helper() + return newTenantTestServer(t, apiToken, nil, mgr, dialer) +} + +func newTenantTestServer(t *testing.T, apiToken string, specs []tenants.TenantSpec, mgr Manager, dialer Dialer) *httptest.Server { + t.Helper() + if dialer == nil { + dialer = &fakeDialer{} + } + if fm, ok := mgr.(*fakeManager); ok { + fm.tenants = specs + } + return serveTestServer(t, New(apiToken, "node:7777", mgr, dialer, nil, nil, nil, nil)) +} + +func pipeDialer(guest guestFunc) *fakeDialer { + return &fakeDialer{dial: func(context.Context, string) (net.Conn, error) { + relayEnd, guestEnd := net.Pipe() + go guest(guestEnd) + return relayEnd, nil + }} +} + +func newProbeAuthTestServer(t *testing.T, mgr Manager, probeKey []byte) *httptest.Server { + t.Helper() + return serveTestServer(t, New("", "node:7777", mgr, &fakeDialer{}, nil, nil, probeKey, nil)) +} + +func newMeshTestServer(t *testing.T, apiToken string, mgr Manager, placer Placer, prober CheckpointProber) *httptest.Server { + t.Helper() + return serveTestServer(t, New(apiToken, "node-a:7777", mgr, &fakeDialer{}, placer, prober, nil, nil)) +} + +func newProberTestServer(t *testing.T, apiToken string, mgr Manager, prober CheckpointProber) *httptest.Server { + t.Helper() + return newMeshTestServer(t, apiToken, mgr, nil, prober) +} + +func serveTestServer(t *testing.T, srv *Server) *httptest.Server { + t.Helper() + ts := httptest.NewServer(srv.Handler()) + t.Cleanup(func() { + ts.Close() + srv.CloseRelays() + }) + return ts +} + +func doReq(t *testing.T, method, url, token, body string) *http.Response { + t.Helper() + req, err := http.NewRequestWithContext(t.Context(), method, url, strings.NewReader(body)) + if err != nil { + t.Fatalf("request: %v", err) + } + if token != "" { + req.Header.Set("Authorization", "Bearer "+token) + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatalf("do: %v", err) + } + return resp +} + +func postJSON(t *testing.T, url, token, body string) *http.Response { + t.Helper() + return doReq(t, http.MethodPost, url, token, body) +} + +func scrapeMetrics(t *testing.T, base string) string { + t.Helper() + resp, err := http.Get(base + "/metrics") + if err != nil { + t.Fatalf("metrics: %v", err) + } + defer resp.Body.Close() + raw, _ := io.ReadAll(resp.Body) + return string(raw) +} + +func statusOf(t *testing.T, method, url, token, body string) int { + t.Helper() + resp := doReq(t, method, url, token, body) + _ = resp.Body.Close() + return resp.StatusCode +} + +func bodyOf(t *testing.T, method, url, token, body string) (int, []byte) { + t.Helper() + resp := doReq(t, method, url, token, body) + defer resp.Body.Close() + out, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatalf("read response: %v", err) + } + return resp.StatusCode, out +} + +func decodeJSON[T any](t *testing.T, r io.Reader) T { + t.Helper() + var v T + if err := json.UnmarshalRead(r, &v); err != nil { + t.Fatalf("decode: %v", err) + } + return v +} + +func credToken(cred pool.Cred) string { + if cred.Operator { + return "" + } + return cred.Token +} diff --git a/sandboxd/server/tenants_test.go b/sandboxd/server/tenants_test.go index d48f813d7..a1651ab4e 100644 --- a/sandboxd/server/tenants_test.go +++ b/sandboxd/server/tenants_test.go @@ -3,7 +3,6 @@ package server import ( "context" "fmt" - "io" "net/http" "strings" "testing" @@ -35,47 +34,33 @@ func TestTenantRoutesAreRootOnlyAndNeverServeATokenBack(t *testing.T) { {http.MethodPut, "/v1/tenants/beta", `{"token":"b"}`}, {http.MethodDelete, "/v1/tenants/acme", ""}, } { - resp := doReq(t, route.method, ts.URL+route.path, "acme-tok", route.body) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusForbidden { - t.Errorf("%s %s with a tenant token: %d, want 403", route.method, route.path, resp.StatusCode) + if code := statusOf(t, route.method, ts.URL+route.path, "acme-tok", route.body); code != http.StatusForbidden { + t.Errorf("%s %s with a tenant token: %d, want 403", route.method, route.path, code) } } - resp := doReq(t, http.MethodGet, ts.URL+"/v1/tenants", "root", "") - body, _ := io.ReadAll(resp.Body) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusOK || string(body) != `{"tenants":[{"name":"acme","max_claims":2,"claims":1}],"digest":"set-digest"}` { - t.Errorf("GET /v1/tenants: %d %s", resp.StatusCode, body) + status, body := bodyOf(t, http.MethodGet, ts.URL+"/v1/tenants", "root", "") + if status != http.StatusOK || string(body) != `{"tenants":[{"name":"acme","max_claims":2,"claims":1}],"digest":"set-digest"}` { + t.Errorf("GET /v1/tenants: %d %s", status, body) } - resp = doReq(t, http.MethodPut, ts.URL+"/v1/tenants/beta", "root", `{"token":"beta-tok","max_claims":3,"egress_class":"desk"}`) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusNoContent || put != (tenants.TenantSpec{Name: "beta", Token: "beta-tok", MaxClaims: 3, EgressClass: "desk"}) { - t.Errorf("PUT /v1/tenants/beta: %d, manager got %+v", resp.StatusCode, put) - } - resp = doReq(t, http.MethodPut, ts.URL+"/v1/tenants/u:42%2Fwest", "root", `{"token":"w"}`) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusNoContent || put.Name != "u:42/west" { - t.Errorf("PUT an escaped name: %d, manager got %q", resp.StatusCode, put.Name) - } - resp = doReq(t, http.MethodPut, ts.URL+"/v1/tenants", "root", `{"tenants":[{"name":"acme","max_claims":1}]}`) - body, _ = io.ReadAll(resp.Body) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusOK || len(set) != 1 || set[0].Name != "acme" || strings.Contains(string(body), "tok") { - t.Errorf("PUT /v1/tenants: %d %s, manager got %+v", resp.StatusCode, body, set) + if code := statusOf(t, http.MethodPut, ts.URL+"/v1/tenants/beta", "root", `{"token":"beta-tok","max_claims":3,"egress_class":"desk"}`); code != http.StatusNoContent || put != (tenants.TenantSpec{Name: "beta", Token: "beta-tok", MaxClaims: 3, EgressClass: "desk"}) { + t.Errorf("PUT /v1/tenants/beta: %d, manager got %+v", code, put) + } + if code := statusOf(t, http.MethodPut, ts.URL+"/v1/tenants/u:42%2Fwest", "root", `{"token":"w"}`); code != http.StatusNoContent || put.Name != "u:42/west" { + t.Errorf("PUT an escaped name: %d, manager got %q", code, put.Name) + } + status, body = bodyOf(t, http.MethodPut, ts.URL+"/v1/tenants", "root", `{"tenants":[{"name":"acme","max_claims":1}]}`) + if status != http.StatusOK || len(set) != 1 || set[0].Name != "acme" || strings.Contains(string(body), "tok") { + t.Errorf("PUT /v1/tenants: %d %s, manager got %+v", status, body, set) } for path, want := range map[string]int{"/v1/tenants/acme": http.StatusNoContent, "/v1/tenants/nobody": http.StatusNotFound} { - resp = doReq(t, http.MethodDelete, ts.URL+path, "root", "") - _ = resp.Body.Close() - if resp.StatusCode != want { - t.Errorf("DELETE %s: %d, want %d", path, resp.StatusCode, want) + if code := statusOf(t, http.MethodDelete, ts.URL+path, "root", ""); code != want { + t.Errorf("DELETE %s: %d, want %d", path, code, want) } } - resp = doReq(t, http.MethodPut, ts.URL+"/v1/tenants/beta", "root", `{"token":"b","egress":{}}`) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusBadRequest { - t.Errorf("PUT a tenant with an unknown field: %d, want 400", resp.StatusCode) + if code := statusOf(t, http.MethodPut, ts.URL+"/v1/tenants/beta", "root", `{"token":"b","egress":{}}`); code != http.StatusBadRequest { + t.Errorf("PUT a tenant with an unknown field: %d, want 400", code) } } @@ -87,10 +72,8 @@ func TestRootListsOneTenantsClaimsAndATenantCannotWiden(t *testing.T) { {"root", "?tenant=gone", "gone"}, {"acme-tok", "?tenant=gone", "acme"}, } { - resp := doReq(t, http.MethodGet, ts.URL+"/v1/sandboxes"+tt.query, tt.token, "") - _ = resp.Body.Close() - if resp.StatusCode != http.StatusOK || mgr.gotTenant != tt.want { - t.Errorf("%s %s: %d scope %q, want %q", tt.token, tt.query, resp.StatusCode, mgr.gotTenant, tt.want) + if code := statusOf(t, http.MethodGet, ts.URL+"/v1/sandboxes"+tt.query, tt.token, ""); code != http.StatusOK || mgr.gotTenant != tt.want { + t.Errorf("%s %s: %d scope %q, want %q", tt.token, tt.query, code, mgr.gotTenant, tt.want) } } } @@ -98,10 +81,8 @@ func TestRootListsOneTenantsClaimsAndATenantCannotWiden(t *testing.T) { func TestRenewOfARemovedTenantsClaimIsForbidden(t *testing.T) { mgr := &fakeManager{renew: func(string, string, time.Duration) (time.Time, error) { return time.Time{}, pool.ErrTenantRemoved }} ts := newTestServer(t, "root", mgr, nil) - resp := doReq(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/renew", "sandbox-tok", "") - _ = resp.Body.Close() - if resp.StatusCode != http.StatusForbidden { - t.Errorf("renew of a removed tenant's claim: %d, want 403", resp.StatusCode) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/sandboxes/sb_1/renew", "sandbox-tok", ""); code != http.StatusForbidden { + t.Errorf("renew of a removed tenant's claim: %d, want 403", code) } } @@ -111,18 +92,15 @@ func TestRootFindsARemovedTenantsRemoteTemplate(t *testing.T) { types.TemplateGossipHash(hash, "gone"): {"peer:7777"}, types.TemplateGossipHash(hash, types.TemplateRootScope): {"peer:7777"}, }} - srv := New("root", "node:7777", &fakeManager{}, &fakeDialer{}, placer, nil, nil, nil) - ts := serveTestServer(t, srv) + ts := newMeshTestServer(t, "root", &fakeManager{}, placer, nil) for _, tt := range []struct{ method, path, body string }{ {http.MethodPost, "/v1/claim", `{"template":"tpl"}`}, {http.MethodPost, "/v1/claim", `{"template":"tpl","volumes":[{"name":"imagenet"}]}`}, {http.MethodDelete, "/v1/templates?template=tpl", ""}, } { - resp := doReq(t, tt.method, ts.URL+tt.path, "root", tt.body) - body, _ := io.ReadAll(resp.Body) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusOK || !strings.Contains(string(body), `"redirect":["peer:7777"]`) { - t.Errorf("%s %s: %d %s, want the retained template's owner", tt.method, tt.path, resp.StatusCode, body) + code, body := bodyOf(t, tt.method, ts.URL+tt.path, "root", tt.body) + if code != http.StatusOK || !strings.Contains(string(body), `"redirect":["peer:7777"]`) { + t.Errorf("%s %s: %d %s, want the retained template's owner", tt.method, tt.path, code, body) } } } @@ -145,10 +123,8 @@ func TestConfigReloadIsRootOnlyAndNamesWhatChanged(t *testing.T) { srv.SetConfigReload(reload) ts := serveTestServer(t, srv) - resp := doReq(t, http.MethodPost, ts.URL+"/v1/config/reload", "acme-tok", "") - _ = resp.Body.Close() - if resp.StatusCode != http.StatusForbidden || calls != 0 { - t.Errorf("tenant token: %d after %d reloads, want 403 and none", resp.StatusCode, calls) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/config/reload", "acme-tok", ""); code != http.StatusForbidden || calls != 0 { + t.Errorf("tenant token: %d after %d reloads, want 403 and none", code, calls) } for _, want := range []struct { code int @@ -158,11 +134,9 @@ func TestConfigReloadIsRootOnlyAndNamesWhatChanged(t *testing.T) { {http.StatusConflict, "listen change only at a restart"}, {http.StatusBadRequest, "parse config"}, } { - resp = doReq(t, http.MethodPost, ts.URL+"/v1/config/reload", "root", "") - body, _ := io.ReadAll(resp.Body) - _ = resp.Body.Close() - if resp.StatusCode != want.code || !strings.Contains(string(body), want.body) { - t.Errorf("reload: %d %s, want %d with %q", resp.StatusCode, body, want.code, want.body) + code, body := bodyOf(t, http.MethodPost, ts.URL+"/v1/config/reload", "root", "") + if code != want.code || !strings.Contains(string(body), want.body) { + t.Errorf("reload: %d %s, want %d with %q", code, body, want.code, want.body) } } } @@ -173,48 +147,35 @@ func TestATenantClaimCarriesItsTokensEgressClassAndADownSourceIs503(t *testing.T }} ts := newTenantTestServer(t, "root", []tenants.TenantSpec{{Name: "acme", Token: "acme-tok", EgressClass: "desk"}}, mgr, nil) claim := `{"template":"rt:24.04"}` - resp := doReq(t, http.MethodPost, ts.URL+"/v1/claim", "acme-tok", claim) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusOK || mgr.gotTenant != "acme" || mgr.gotEgressClass != "desk" { - t.Errorf("tenant claim: %d tenant %q class %q, want acme under desk", resp.StatusCode, mgr.gotTenant, mgr.gotEgressClass) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/claim", "acme-tok", claim); code != http.StatusOK || mgr.gotTenant != "acme" || mgr.gotEgressClass != "desk" { + t.Errorf("tenant claim: %d tenant %q class %q, want acme under desk", code, mgr.gotTenant, mgr.gotEgressClass) } - resp = doReq(t, http.MethodPost, ts.URL+"/v1/claim", "root", claim) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusOK || mgr.gotEgressClass != "" { - t.Errorf("root claim: %d class %q, want none", resp.StatusCode, mgr.gotEgressClass) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/claim", "root", claim); code != http.StatusOK || mgr.gotEgressClass != "" { + t.Errorf("root claim: %d class %q, want none", code, mgr.gotEgressClass) } mgr.tenantErr = fmt.Errorf("%w: connection refused", pool.ErrTenantStoreDown) - resp = doReq(t, http.MethodPost, ts.URL+"/v1/claim", "acme-tok", claim) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusServiceUnavailable { - t.Errorf("claim with the tenant source down: %d, want 503", resp.StatusCode) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/claim", "acme-tok", claim); code != http.StatusServiceUnavailable { + t.Errorf("claim with the tenant source down: %d, want 503", code) } mgr.tenantErr = pool.ErrUnknownTenant - resp = doReq(t, http.MethodPost, ts.URL+"/v1/claim", "acme-tok", claim) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusUnauthorized { - t.Errorf("claim with an unknown token: %d, want 401", resp.StatusCode) + if code := statusOf(t, http.MethodPost, ts.URL+"/v1/claim", "acme-tok", claim); code != http.StatusUnauthorized { + t.Errorf("claim with an unknown token: %d, want 401", code) } } func TestTenantListPagesByCursor(t *testing.T) { mgr := &fakeManager{tenantList: []pool.TenantInfo{{Name: "u-2"}}} ts := newTenantTestServer(t, "root", nil, mgr, nil) - resp := doReq(t, http.MethodGet, ts.URL+"/v1/tenants?after=u-1&limit=2", "root", "") - _ = resp.Body.Close() - if resp.StatusCode != http.StatusOK || mgr.gotTenantPage != [2]string{"u-1", "2"} { - t.Errorf("paged list: %d, manager got %v", resp.StatusCode, mgr.gotTenantPage) + if code := statusOf(t, http.MethodGet, ts.URL+"/v1/tenants?after=u-1&limit=2", "root", ""); code != http.StatusOK || mgr.gotTenantPage != [2]string{"u-1", "2"} { + t.Errorf("paged list: %d, manager got %v", code, mgr.gotTenantPage) } - resp = doReq(t, http.MethodGet, ts.URL+"/v1/tenants", "root", "") - _ = resp.Body.Close() + statusOf(t, http.MethodGet, ts.URL+"/v1/tenants", "root", "") if mgr.gotTenantPage != [2]string{"", "1000"} { t.Errorf("default page: manager got %v, want the first 1000", mgr.gotTenantPage) } for _, q := range []string{"limit=0", "limit=10001", "limit=x"} { - resp = doReq(t, http.MethodGet, ts.URL+"/v1/tenants?"+q, "root", "") - _ = resp.Body.Close() - if resp.StatusCode != http.StatusBadRequest { - t.Errorf("%s: %d, want 400", q, resp.StatusCode) + if code := statusOf(t, http.MethodGet, ts.URL+"/v1/tenants?"+q, "root", ""); code != http.StatusBadRequest { + t.Errorf("%s: %d, want 400", q, code) } } } diff --git a/sandboxd/server/usage_test.go b/sandboxd/server/usage_test.go index 0dc09e50c..a58ba3539 100644 --- a/sandboxd/server/usage_test.go +++ b/sandboxd/server/usage_test.go @@ -3,7 +3,6 @@ package server import ( "context" "errors" - "io" "net/http" "net/http/httptest" "strings" @@ -17,11 +16,9 @@ import ( func TestUsageIsRootOnlyAndNamesAMissingStore(t *testing.T) { ts := newTenantTestServer(t, "root", []tenants.TenantSpec{{Name: "acme", Token: "acme-tok"}}, &fakeManager{}, nil) for token, want := range map[string]int{"acme-tok": http.StatusForbidden, "": http.StatusUnauthorized, "root": http.StatusNotImplemented} { - resp := doReq(t, http.MethodGet, ts.URL+"/v1/usage", token, "") - body, _ := io.ReadAll(resp.Body) - _ = resp.Body.Close() - if resp.StatusCode != want { - t.Errorf("GET /v1/usage with %q: %d %s, want %d", token, resp.StatusCode, body, want) + code, body := bodyOf(t, http.MethodGet, ts.URL+"/v1/usage", token, "") + if code != want { + t.Errorf("GET /v1/usage with %q: %d %s, want %d", token, code, body, want) } if want == http.StatusNotImplemented && !strings.Contains(string(body), "no shared usage store configured") { t.Errorf("GET /v1/usage without a store: %s", body) @@ -36,30 +33,24 @@ func TestUsagePassesTheQueryAndPages(t *testing.T) { {Cursor: 9, Node: "n2", Event: usage.Event{Time: t0, Event: "release", ID: "sb_1"}}, }} ts := usageTestServer(t, reader) - resp := doReq(t, http.MethodGet, ts.URL+"/v1/usage?tenant=acme&since=2026-10-30T00:00:00Z&until=2026-10-31T00:00:00.5%2B08:00&after=5&limit=2", "root", "") - body, _ := io.ReadAll(resp.Body) - _ = resp.Body.Close() + code, body := bodyOf(t, http.MethodGet, ts.URL+"/v1/usage?tenant=acme&since=2026-10-30T00:00:00Z&until=2026-10-31T00:00:00.5%2B08:00&after=5&limit=2", "root", "") want := `{"events":[{"cursor":7,"node":"n1","t":"2026-10-30T01:02:03.4Z","ev":"claim","id":"sb_1","template":"rt:24.04","net":"egress","size":"xlarge","tenant":"acme","deadline":"2026-10-30T01:02:03.4Z"},` + `{"cursor":9,"node":"n2","t":"2026-10-30T01:02:03.4Z","ev":"release","id":"sb_1"}],"next":9}` - if resp.StatusCode != http.StatusOK || string(body) != want { - t.Errorf("full page: %d %s, want %s", resp.StatusCode, body, want) + if code != http.StatusOK || string(body) != want { + t.Errorf("full page: %d %s, want %s", code, body, want) } until := time.Date(2026, 10, 31, 0, 0, 0, 500000000, time.FixedZone("", 8*3600)) if q := reader.got; q.Tenant != "acme" || !q.Since.Equal(time.Date(2026, 10, 30, 0, 0, 0, 0, time.UTC)) || !q.Until.Equal(until) || q.After != 5 || q.Limit != 2 { t.Errorf("reader got %+v", q) } - resp = doReq(t, http.MethodGet, ts.URL+"/v1/usage?after=9", "root", "") - body, _ = io.ReadAll(resp.Body) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusOK || strings.Contains(string(body), "next") || reader.got != (usage.Query{After: 9, Limit: 1000}) { - t.Errorf("default partial page: %d %s, reader got %+v", resp.StatusCode, body, reader.got) + code, body = bodyOf(t, http.MethodGet, ts.URL+"/v1/usage?after=9", "root", "") + if code != http.StatusOK || strings.Contains(string(body), "next") || reader.got != (usage.Query{After: 9, Limit: 1000}) { + t.Errorf("default partial page: %d %s, reader got %+v", code, body, reader.got) } reader.recs = nil - resp = doReq(t, http.MethodGet, ts.URL+"/v1/usage", "root", "") - body, _ = io.ReadAll(resp.Body) - _ = resp.Body.Close() + _, body = bodyOf(t, http.MethodGet, ts.URL+"/v1/usage", "root", "") if string(body) != `{"events":[]}` { t.Errorf("empty page: %s", body) } @@ -69,28 +60,17 @@ func TestUsageRefusesABadQueryAndADownStore(t *testing.T) { reader := &fakeUsage{} ts := usageTestServer(t, reader) for _, q := range []string{"after=-1", "after=x", "limit=0", "limit=10001", "limit=x", "since=yesterday", "until=2026-10-30"} { - resp := doReq(t, http.MethodGet, ts.URL+"/v1/usage?"+q, "root", "") - _ = resp.Body.Close() - if resp.StatusCode != http.StatusBadRequest { - t.Errorf("%s: %d, want 400", q, resp.StatusCode) + if code := statusOf(t, http.MethodGet, ts.URL+"/v1/usage?"+q, "root", ""); code != http.StatusBadRequest { + t.Errorf("%s: %d, want 400", q, code) } } reader.err = errors.New("connection refused") - resp := doReq(t, http.MethodGet, ts.URL+"/v1/usage", "root", "") - body, _ := io.ReadAll(resp.Body) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusServiceUnavailable || strings.Contains(string(body), "refused") { - t.Errorf("down store: %d %s, want 503 without the cause", resp.StatusCode, body) + code, body := bodyOf(t, http.MethodGet, ts.URL+"/v1/usage", "root", "") + if code != http.StatusServiceUnavailable || strings.Contains(string(body), "refused") { + t.Errorf("down store: %d %s, want 503 without the cause", code, body) } } -func usageTestServer(t *testing.T, reader UsageReader) *httptest.Server { - t.Helper() - srv := New("root", "node:7777", &fakeManager{}, &fakeDialer{}, nil, nil, nil, nil) - srv.SetUsage(reader) - return serveTestServer(t, srv) -} - type fakeUsage struct { recs []usage.Record err error @@ -101,3 +81,10 @@ func (f *fakeUsage) List(_ context.Context, q usage.Query) ([]usage.Record, erro f.got = q return f.recs, f.err } + +func usageTestServer(t *testing.T, reader UsageReader) *httptest.Server { + t.Helper() + srv := New("root", "node:7777", &fakeManager{}, &fakeDialer{}, nil, nil, nil, nil) + srv.SetUsage(reader) + return serveTestServer(t, srv) +} diff --git a/sandboxd/server/volume_mutation_test.go b/sandboxd/server/volume_mutation_test.go index 8fcab0a8e..cbaad515f 100644 --- a/sandboxd/server/volume_mutation_test.go +++ b/sandboxd/server/volume_mutation_test.go @@ -142,10 +142,8 @@ func TestLiveVolumeRoutesHideOtherTenants(t *testing.T) { if method == http.MethodPatch { body = `{}` } - resp := doReq(t, method, ts.URL+"/v1/sandboxes/sb_1/volumes", "other-token", body) - defer resp.Body.Close() - if resp.StatusCode != http.StatusNotFound || !called { - t.Fatalf("status=%d called=%v", resp.StatusCode, called) + if code := statusOf(t, method, ts.URL+"/v1/sandboxes/sb_1/volumes", "other-token", body); code != http.StatusNotFound || !called { + t.Fatalf("status=%d called=%v", code, called) } }) } diff --git a/sandboxd/store/dir/dir_test.go b/sandboxd/store/dir/dir_test.go index ecc8033f9..1f87e289d 100644 --- a/sandboxd/store/dir/dir_test.go +++ b/sandboxd/store/dir/dir_test.go @@ -11,7 +11,6 @@ import ( "os" "path/filepath" "runtime" - "slices" "testing" "time" @@ -21,43 +20,32 @@ import ( "github.com/cocoonstack/sandbox/sandboxd/store/storetest" ) +const testID = "ck_00000000000000aa" + func TestDirBackendContract(t *testing.T) { - st, err := New(t.TempDir(), store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } + st := newStore(t, t.TempDir()) storetest.RunContract(t, st) } func TestFetchPinsGenerationAcrossStoreInstances(t *testing.T) { - const id = "ck_00000000000000aa" root := t.TempDir() - reader, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("new reader: %v", err) - } - writer, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("new writer: %v", err) - } - firstMeta := []byte(metaJSON("first")) - firstDigest := mustPublishDigested(t, writer, id, "first") - firstDigestPath := filepath.Join(root, id, digestName(firstMeta)) - backdate(t, filepath.Join(root, id, store.ExportGen(firstMeta))) + reader := newStore(t, root) + writer := newStore(t, root) + firstMeta := []byte(storetest.MetaJSON("first")) + firstDigest := mustPublishDigested(t, writer, testID, "first") + firstDigestPath := filepath.Join(root, testID, digestName(firstMeta)) + backdate(t, filepath.Join(root, testID, store.ExportGen(firstMeta))) backdate(t, firstDigestPath) - backdate(t, filepath.Join(root, id, store.MetaFile)) + backdate(t, filepath.Join(root, testID, store.MetaFile)) - dir, meta, digest, err := reader.Fetch(t.Context(), id) - if err != nil { - t.Fatalf("fetch first: %v", err) - } - if string(meta) != metaJSON("first") { + dir, meta, digest := mustFetch(t, reader, testID) + if string(meta) != storetest.MetaJSON("first") { t.Fatalf("fetch meta %q, want first generation", meta) } if digest != firstDigest { t.Fatalf("fetch digest %q, want first generation %q", digest, firstDigest) } - secondDigest := mustPublishDigested(t, writer, id, "second") + secondDigest := mustPublishDigested(t, writer, testID, "second") if _, statErr := os.Stat(filepath.Join(dir, "disk.img")); statErr != nil { t.Fatalf("first generation disturbed by re-publish: %v", statErr) } @@ -65,11 +53,8 @@ func TestFetchPinsGenerationAcrossStoreInstances(t *testing.T) { t.Fatalf("first generation digest disturbed by re-publish: %v", statErr) } - _, meta, digest, err = reader.Fetch(t.Context(), id) - if err != nil { - t.Fatalf("fetch second: %v", err) - } - if string(meta) != metaJSON("second") { + _, meta, digest = mustFetch(t, reader, testID) + if string(meta) != storetest.MetaJSON("second") { t.Fatalf("fetch meta %q, want second generation", meta) } if digest != secondDigest || digest == firstDigest { @@ -78,22 +63,15 @@ func TestFetchPinsGenerationAcrossStoreInstances(t *testing.T) { } func TestPlainPublishReplacesDigestWithEmpty(t *testing.T) { - const id = "ck_00000000000000aa" root := t.TempDir() - st, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } - firstMeta := []byte(metaJSON("digested")) - firstDigest := mustPublishDigested(t, st, id, "digested") - firstDigestPath := filepath.Join(root, id, digestName(firstMeta)) - mustPublish(t, st, id, "plain") - - _, meta, digest, err := st.Fetch(t.Context(), id) - if err != nil { - t.Fatalf("Fetch: %v", err) - } - if string(meta) != metaJSON("plain") || digest != "" { + st := newStore(t, root) + firstMeta := []byte(storetest.MetaJSON("digested")) + firstDigest := mustPublishDigested(t, st, testID, "digested") + firstDigestPath := filepath.Join(root, testID, digestName(firstMeta)) + mustPublish(t, st, testID, "plain") + + _, meta, digest := mustFetch(t, st, testID) + if string(meta) != storetest.MetaJSON("plain") || digest != "" { t.Fatalf("plain replacement meta/digest = %q/%q, want plain/empty", meta, digest) } if firstDigest == "" { @@ -105,7 +83,6 @@ func TestPlainPublishReplacesDigestWithEmpty(t *testing.T) { } func TestPublishDigestedChunkBoundaries(t *testing.T) { - const id = "ck_00000000000000aa" content := bytes.Repeat([]byte{'x'}, int(store.DigestChunkSize)+1) content[len(content)-1] = 'y' tests := []struct { @@ -119,11 +96,8 @@ func TestPublishDigestedChunkBoundaries(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - st, err := New(t.TempDir(), store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } - staging, err := st.Stage(id) + st := newStore(t, t.TempDir()) + staging, err := st.Stage(testID) if err != nil { t.Fatalf("Stage: %v", err) } @@ -134,20 +108,17 @@ func TestPublishDigestedChunkBoundaries(t *testing.T) { if err = os.WriteFile(filepath.Join(export, "boundary.bin"), content[:int(tt.size)], 0o600); err != nil { t.Fatalf("write export: %v", err) } - if err = os.WriteFile(filepath.Join(staging, store.MetaFile), []byte(metaJSON(tt.name)), 0o600); err != nil { + if err = os.WriteFile(filepath.Join(staging, store.MetaFile), []byte(storetest.MetaJSON(tt.name)), 0o600); err != nil { t.Fatalf("write meta: %v", err) } - digest, err := storetest.PublishDigested(t.Context(), st, staging, id) + digest, err := storetest.PublishDigested(t.Context(), st, staging, testID) if err != nil { t.Fatalf("PublishDigested: %v", err) } if digest != tt.want { t.Errorf("digest = %q, want %q", digest, tt.want) } - _, _, fetched, err := st.Fetch(t.Context(), id) - if err != nil { - t.Fatalf("Fetch: %v", err) - } + _, _, fetched := mustFetch(t, st, testID) if fetched != tt.want { t.Errorf("fetched digest = %q, want %q", fetched, tt.want) } @@ -199,12 +170,12 @@ func TestDigestWorkersStoreChunksByOffset(t *testing.T) { func TestHoleyExportDigestEqualsTheReadEverythingDigest(t *testing.T) { root := t.TempDir() - dataTail := writeSparse(t, filepath.Join(root, "disk.img"), 3*store.DigestChunkSize+7, map[int64][]byte{ + dataTail := storetest.WriteSparse(t, filepath.Join(root, "disk.img"), 3*store.DigestChunkSize+7, map[int64][]byte{ store.DigestChunkSize + 5: []byte("data"), 2 * store.DigestChunkSize: make([]byte, 4096), 3*store.DigestChunkSize + 6: []byte("z"), }) - writeSparse(t, filepath.Join(root, "mem"), 2*store.DigestChunkSize+5, map[int64][]byte{0: []byte("head")}) + storetest.WriteSparse(t, filepath.Join(root, "mem"), 2*store.DigestChunkSize+5, map[int64][]byte{0: []byte("head")}) f, err := os.Open(dataTail) if err != nil { t.Fatalf("open: %v", err) @@ -229,76 +200,26 @@ func TestHoleyExportDigestEqualsTheReadEverythingDigest(t *testing.T) { } } -func TestHoleSkippingDigestAB(t *testing.T) { - if os.Getenv("SANDBOX_DIGEST_AB") == "" { - t.Skip("set SANDBOX_DIGEST_AB=1 to run the 10 GiB interleaved A/B") - } - root := t.TempDir() - writes := map[int64][]byte{} - for i := range int64(4) { - block := make([]byte, 50<<20) - for j := range block { - block[j] = byte(j*7 + int(i)) - } - writes[i*(10<<30)/4] = block - } - writeSparse(t, filepath.Join(root, "disk.img"), 10<<30, writes) - want := readEverythingDigest(t, root) - arms := map[string]func() string{ - "read": func() string { return readEverythingDigest(t, root) }, - "seek": func() string { got, _ := hashExport(t.Context(), root); return got }, - } - times := map[string][]time.Duration{} - for round := range 8 { - order := []string{"read", "seek"} - if round%2 == 1 { - order = []string{"seek", "read"} - } - for _, arm := range order { - start := time.Now() - if got := arms[arm](); got != want { - t.Fatalf("%s arm digest %q, want %q", arm, got, want) - } - times[arm] = append(times[arm], time.Since(start)) - } - } - for _, arm := range []string{"read", "seek"} { - slices.Sort(times[arm]) - t.Logf("%s p50 %v over %v", arm, times[arm][len(times[arm])/2], times[arm]) - } -} - func TestPublishDigestedFailureAndFreshRetry(t *testing.T) { - const id = "ck_00000000000000aa" root := t.TempDir() - st, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } - firstDigest := mustPublishDigested(t, st, id, "first") + st := newStore(t, root) + firstDigest := mustPublishDigested(t, st, testID, "first") - failed, err := st.Stage(id) - if err != nil { - t.Fatalf("stage failed publish: %v", err) - } - seedRecord(t, failed, "second") + failed := stagedRecord(t, st, testID, "second") secondMeta, err := os.ReadFile(filepath.Join(failed, store.MetaFile)) if err != nil { t.Fatalf("read second meta: %v", err) } - blocker := filepath.Join(root, id, digestName(secondMeta)) + blocker := filepath.Join(root, testID, digestName(secondMeta)) if err = os.Mkdir(blocker, 0o750); err != nil { t.Fatalf("block digest commit: %v", err) } - if digest, publishErr := storetest.PublishDigested(t.Context(), st, failed, id); publishErr == nil || digest != "" { + if digest, publishErr := storetest.PublishDigested(t.Context(), st, failed, testID); publishErr == nil || digest != "" { t.Fatalf("PublishDigested with blocked sidecar = %q, %v, want empty/error", digest, publishErr) } - dir, meta, digest, err := st.Fetch(t.Context(), id) - if err != nil { - t.Fatalf("Fetch previous generation: %v", err) - } + dir, meta, digest := mustFetch(t, st, testID) content, readErr := os.ReadFile(filepath.Join(dir, "disk.img")) - if string(meta) != metaJSON("first") || digest != firstDigest { + if string(meta) != storetest.MetaJSON("first") || digest != firstDigest { t.Errorf("previous generation meta/digest = %q/%q, want first/%q", meta, digest, firstDigest) } if readErr != nil || string(content) != "first" { @@ -311,13 +232,10 @@ func TestPublishDigestedFailureAndFreshRetry(t *testing.T) { if err = os.RemoveAll(failed); err != nil { t.Fatalf("remove failed staging: %v", err) } - secondDigest := mustPublishDigested(t, st, id, "second") - dir, meta, digest, err = st.Fetch(t.Context(), id) - if err != nil { - t.Fatalf("Fetch replacement generation: %v", err) - } + secondDigest := mustPublishDigested(t, st, testID, "second") + dir, meta, digest = mustFetch(t, st, testID) content, readErr = os.ReadFile(filepath.Join(dir, "disk.img")) - if string(meta) != metaJSON("second") || digest != secondDigest || digest == firstDigest { + if string(meta) != storetest.MetaJSON("second") || digest != secondDigest || digest == firstDigest { t.Errorf("replacement generation meta/digest = %q/%q, want second/%q", meta, digest, secondDigest) } if readErr != nil || string(content) != "second" { @@ -326,29 +244,25 @@ func TestPublishDigestedFailureAndFreshRetry(t *testing.T) { } func TestSweepGenerationsPairsCurrentAndSupersededDigests(t *testing.T) { - const id = "ck_00000000000000aa" root := t.TempDir() - st, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } - firstMeta := []byte(metaJSON("first")) - firstGen := filepath.Join(root, id, store.ExportGen(firstMeta)) - firstSidecar := filepath.Join(root, id, digestName(firstMeta)) - mustPublishDigested(t, st, id, "first") - secondMeta := []byte(metaJSON("second")) - secondGen := filepath.Join(root, id, store.ExportGen(secondMeta)) - secondSidecar := filepath.Join(root, id, digestName(secondMeta)) - secondDigest := mustPublishDigested(t, st, id, "second") - if err = st.SetLabels(t.Context(), id, []byte(`{"tag":"v1"}`)); err != nil { + st := newStore(t, root) + firstMeta := []byte(storetest.MetaJSON("first")) + firstGen := filepath.Join(root, testID, store.ExportGen(firstMeta)) + firstSidecar := filepath.Join(root, testID, digestName(firstMeta)) + mustPublishDigested(t, st, testID, "first") + secondMeta := []byte(storetest.MetaJSON("second")) + secondGen := filepath.Join(root, testID, store.ExportGen(secondMeta)) + secondSidecar := filepath.Join(root, testID, digestName(secondMeta)) + secondDigest := mustPublishDigested(t, st, testID, "second") + if err := st.SetLabels(t.Context(), testID, []byte(`{"tag":"v1"}`)); err != nil { t.Fatalf("SetLabels: %v", err) } - labels := filepath.Join(root, id, store.LabelsFile) + labels := filepath.Join(root, testID, store.LabelsFile) for _, path := range []string{firstGen, firstSidecar, secondGen, secondSidecar, labels} { backdate(t, path) } - if err = st.SweepGenerations(t.Context()); err != nil { + if err := st.SweepGenerations(t.Context()); err != nil { t.Fatalf("SweepGenerations: %v", err) } for _, path := range []string{firstGen, firstSidecar} { @@ -361,37 +275,26 @@ func TestSweepGenerationsPairsCurrentAndSupersededDigests(t *testing.T) { t.Errorf("current entry %s was swept: %v", filepath.Base(path), statErr) } } - _, meta, digest, err := st.Fetch(t.Context(), id) - if err != nil { - t.Fatalf("Fetch current generation: %v", err) - } - if string(meta) != metaJSON("second") || digest != secondDigest { + _, meta, digest := mustFetch(t, st, testID) + if string(meta) != storetest.MetaJSON("second") || digest != secondDigest { t.Fatalf("current meta/digest = %q/%q, want second/%q", meta, digest, secondDigest) } } func TestPublishRetriesAfterExpiredInstall(t *testing.T) { - const id = "ck_00000000000000aa" root := t.TempDir() - st, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } - _, gen := installUncommitted(t, st, root, id, "same") + st := newStore(t, root) + _, gen := installUncommitted(t, st, root, testID, "same") backdate(t, gen) - retry, err := st.Stage(id) - if err != nil { - t.Fatalf("stage retry: %v", err) - } - seedRecord(t, retry, "same") - if publishErr := st.Publish(t.Context(), retry, id, ""); publishErr == nil { + retry := stagedRecord(t, st, testID, "same") + if publishErr := st.Publish(t.Context(), retry, testID, ""); publishErr == nil { t.Fatal("expired installed generation was revived") } if removeErr := os.RemoveAll(retry); removeErr != nil { t.Fatalf("remove failed retry: %v", removeErr) } - if _, readErr := st.ReadMeta(t.Context(), id); !errors.Is(readErr, store.ErrNotFound) { + if _, readErr := st.ReadMeta(t.Context(), testID); !errors.Is(readErr, store.ErrNotFound) { t.Fatalf("expired retry committed meta: %v", readErr) } if sweepErr := st.SweepGenerations(t.Context()); sweepErr != nil { @@ -401,48 +304,38 @@ func TestPublishRetriesAfterExpiredInstall(t *testing.T) { t.Fatalf("expired generation survived sweep: %v", statErr) } - mustPublish(t, st, id, "same") + mustPublish(t, st, testID, "same") backdate(t, gen) - mustPublish(t, st, id, "same") - dir, _, _, err := st.Fetch(t.Context(), id) - if err != nil { - t.Fatalf("fetch after retried publish: %v", err) - } + mustPublish(t, st, testID, "same") + dir, _, _ := mustFetch(t, st, testID) if _, err := os.Stat(filepath.Join(dir, "disk.img")); err != nil { t.Fatalf("export after retried publish: %v", err) } } func TestPublishSweepsGenerationsBySupersessionAge(t *testing.T) { - const id = "ck_00000000000000aa" root := t.TempDir() - st, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } - gen1 := filepath.Join(root, id, store.ExportGen([]byte(metaJSON("first")))) - gen2 := filepath.Join(root, id, store.ExportGen([]byte(metaJSON("second")))) - mustPublish(t, st, id, "first") - mustPublish(t, st, id, "second") - mustPublish(t, st, id, "third") + st := newStore(t, root) + gen1 := filepath.Join(root, testID, store.ExportGen([]byte(storetest.MetaJSON("first")))) + gen2 := filepath.Join(root, testID, store.ExportGen([]byte(storetest.MetaJSON("second")))) + mustPublish(t, st, testID, "first") + mustPublish(t, st, testID, "second") + mustPublish(t, st, testID, "third") if _, statErr := os.Stat(gen1); statErr != nil { t.Fatalf("superseded generation reclaimed inside the grace: %v", statErr) } setAge(t, gen1, store.GenerationGrace+time.Minute) setAge(t, gen2, store.GenerationGrace/2) - mustPublish(t, st, id, "fourth") + mustPublish(t, st, testID, "fourth") if _, statErr := os.Stat(gen1); !errors.Is(statErr, fs.ErrNotExist) { t.Fatalf("generation past its supersession grace survived publish: %v", statErr) } if _, statErr := os.Stat(gen2); statErr != nil { t.Fatalf("generation inside its supersession grace was reclaimed: %v", statErr) } - dir, meta, _, err := st.Fetch(t.Context(), id) - if err != nil { - t.Fatalf("fetch after sweep: %v", err) - } - if string(meta) != metaJSON("fourth") { + dir, meta, _ := mustFetch(t, st, testID) + if string(meta) != storetest.MetaJSON("fourth") { t.Fatalf("fetch meta %q, want the current generation", meta) } if _, err := os.Stat(filepath.Join(dir, "disk.img")); err != nil { @@ -451,20 +344,13 @@ func TestPublishSweepsGenerationsBySupersessionAge(t *testing.T) { } func TestSweepSparesPublishingGeneration(t *testing.T) { - const id = "ck_00000000000000aa" root := t.TempDir() - sweeper, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("new sweeper: %v", err) - } - writer, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("new writer: %v", err) - } - mustPublish(t, writer, id, "old") - backdate(t, filepath.Join(root, id, store.MetaFile)) + sweeper := newStore(t, root) + writer := newStore(t, root) + mustPublish(t, writer, testID, "old") + backdate(t, filepath.Join(root, testID, store.MetaFile)) - meta, gen := installUncommitted(t, writer, root, id, "new") + meta, gen := installUncommitted(t, writer, root, testID, "new") if sweepErr := sweeper.SweepStaging(); sweepErr != nil { t.Fatalf("sweep during publish: %v", sweepErr) } @@ -472,18 +358,15 @@ func TestSweepSparesPublishingGeneration(t *testing.T) { t.Fatalf("publishing generation was swept: %v", statErr) } - tmp := filepath.Join(root, id, store.MetaFile+".tmp") + tmp := filepath.Join(root, testID, store.MetaFile+".tmp") if writeErr := os.WriteFile(tmp, meta, 0o600); writeErr != nil { t.Fatalf("write commit: %v", writeErr) } - if renameErr := os.Rename(tmp, filepath.Join(root, id, store.MetaFile)); renameErr != nil { + if renameErr := os.Rename(tmp, filepath.Join(root, testID, store.MetaFile)); renameErr != nil { t.Fatalf("commit meta: %v", renameErr) } - dir, gotMeta, digest, err := writer.Fetch(t.Context(), id) - if err != nil { - t.Fatalf("fetch committed generation: %v", err) - } - if string(gotMeta) != metaJSON("new") { + dir, gotMeta, digest := mustFetch(t, writer, testID) + if string(gotMeta) != storetest.MetaJSON("new") { t.Fatalf("fetch meta %q, want new generation", gotMeta) } if digest != "" { @@ -500,25 +383,15 @@ func TestSweepReclaimsOnlyAgedUncommittedGenerations(t *testing.T) { publishID = "ck_00000000000000bb" ) root := t.TempDir() - sweeper, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("new sweeper: %v", err) - } - writer, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("new writer: %v", err) - } + sweeper := newStore(t, root) + writer := newStore(t, root) orphan := filepath.Join(root, orphanID, store.ExportDir+"-0011223344556677") if mkdirErr := os.MkdirAll(orphan, 0o750); mkdirErr != nil { t.Fatalf("seed orphan: %v", mkdirErr) } backdate(t, orphan) - staging, err := writer.Stage(publishID) - if err != nil { - t.Fatalf("stage publishing generation: %v", err) - } - seedRecord(t, staging, "fresh") + staging := stagedRecord(t, writer, publishID, "fresh") meta, err := os.ReadFile(filepath.Join(staging, store.MetaFile)) if err != nil { t.Fatalf("read staged meta: %v", err) @@ -550,11 +423,8 @@ func TestSweepReclaimsOnlyAgedUncommittedGenerations(t *testing.T) { t.Fatalf("remove commit blocker: %v", removeErr) } mustPublish(t, writer, publishID, "fresh") - dir, gotMeta, digest, err := writer.Fetch(t.Context(), publishID) - if err != nil { - t.Fatalf("fetch retried publish: %v", err) - } - if string(gotMeta) != metaJSON("fresh") { + dir, gotMeta, digest := mustFetch(t, writer, publishID) + if string(gotMeta) != storetest.MetaJSON("fresh") { t.Fatalf("fetch meta %q, want fresh", gotMeta) } if digest != "" { @@ -567,19 +437,9 @@ func TestSweepReclaimsOnlyAgedUncommittedGenerations(t *testing.T) { func TestSweepStagingSparesAPeersFreshStaging(t *testing.T) { root := t.TempDir() - peer, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("new peer: %v", err) - } - restarted, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("new restarted node: %v", err) - } - staging, err := peer.Stage("ck_00000000000000aa") - if err != nil { - t.Fatalf("stage: %v", err) - } - seedRecord(t, staging, "in-progress") + peer := newStore(t, root) + restarted := newStore(t, root) + staging := stagedRecord(t, peer, testID, "in-progress") if sweepErr := restarted.SweepStaging(); sweepErr != nil { t.Fatalf("sweep: %v", sweepErr) @@ -598,15 +458,8 @@ func TestSweepStagingSparesAPeersFreshStaging(t *testing.T) { } func TestSweepGenerationsRemovesOnlyAgedStaging(t *testing.T) { - st, err := New(t.TempDir(), store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } - aged, err := st.Stage("ck_00000000000000aa") - if err != nil { - t.Fatalf("stage: %v", err) - } - seedRecord(t, aged, "interrupted") + st := newStore(t, t.TempDir()) + aged := stagedRecord(t, st, testID, "interrupted") backdate(t, aged) fresh, err := st.Stage("ck_00000000000000bb") if err != nil { @@ -626,31 +479,27 @@ func TestSweepGenerationsRemovesOnlyAgedStaging(t *testing.T) { } func TestSweepGenerationsToleratesConcurrentDelete(t *testing.T) { - const id = "ck_00000000000000aa" - st, err := New(t.TempDir(), store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } + st := newStore(t, t.TempDir()) ctx := t.Context() done := make(chan struct{}) var g errgroup.Group g.Go(func() error { defer close(done) for range 500 { - staging, err := st.Stage(id) + staging, err := st.Stage(testID) if err != nil { return err } if err := os.MkdirAll(filepath.Join(staging, store.ExportDir), 0o750); err != nil { return err } - if err := os.WriteFile(filepath.Join(staging, store.MetaFile), []byte(metaJSON(id)), 0o600); err != nil { + if err := os.WriteFile(filepath.Join(staging, store.MetaFile), []byte(storetest.MetaJSON(testID)), 0o600); err != nil { return err } - if err := st.Publish(ctx, staging, id, ""); err != nil { + if err := st.Publish(ctx, staging, testID, ""); err != nil { return fmt.Errorf("publish: %w", err) } - if err := st.Delete(ctx, id); err != nil { + if err := st.Delete(ctx, testID); err != nil { return fmt.Errorf("delete: %w", err) } } @@ -663,7 +512,7 @@ func TestSweepGenerationsToleratesConcurrentDelete(t *testing.T) { return nil default: } - if err := st.SweepGenerations(t.Context()); err != nil { + if err := st.SweepGenerations(ctx); err != nil { return fmt.Errorf("sweep: %w", err) } } @@ -674,14 +523,10 @@ func TestSweepGenerationsToleratesConcurrentDelete(t *testing.T) { } func TestMetasSurfacesReadError(t *testing.T) { - const id = "ck_00000000000000aa" root := t.TempDir() - st, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } + st := newStore(t, root) - if err := os.MkdirAll(filepath.Join(root, id, store.MetaFile), 0o750); err != nil { + if err := os.MkdirAll(filepath.Join(root, testID, store.MetaFile), 0o750); err != nil { t.Fatalf("setup: %v", err) } if _, err := st.Metas(t.Context()); err == nil { @@ -691,27 +536,24 @@ func TestMetasSurfacesReadError(t *testing.T) { func TestMetasReadsEveryRecordAndSkipsOnesWithoutMeta(t *testing.T) { root := t.TempDir() - st, err := New(root, store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } + st := newStore(t, root) want := map[string]string{} for i := range 100 { id := fmt.Sprintf("ck_%016x", i) mustPublish(t, st, id, id) - want[metaJSON(id)] = "" + want[storetest.MetaJSON(id)] = "" if i%3 == 0 { labels := `{"n":"` + id + `"}` - if err = st.SetLabels(t.Context(), id, []byte(labels)); err != nil { + if err := st.SetLabels(t.Context(), id, []byte(labels)); err != nil { t.Fatalf("SetLabels: %v", err) } - want[metaJSON(id)] = labels + want[storetest.MetaJSON(id)] = labels } } - if err = os.MkdirAll(filepath.Join(root, "ck_00000000000fffff"), 0o750); err != nil { + if err := os.MkdirAll(filepath.Join(root, "ck_00000000000fffff"), 0o750); err != nil { t.Fatalf("setup: %v", err) } - if _, err = st.Stage("ck_0000000000000000"); err != nil { + if _, err := st.Stage("ck_0000000000000000"); err != nil { t.Fatalf("Stage: %v", err) } recs, err := st.Metas(t.Context()) @@ -728,10 +570,7 @@ func TestMetasReadsEveryRecordAndSkipsOnesWithoutMeta(t *testing.T) { } func BenchmarkMetas(b *testing.B) { - st, err := New(b.TempDir(), store.CheckpointIDRe) - if err != nil { - b.Fatalf("New: %v", err) - } + st := newStore(b, b.TempDir()) for i := range 2000 { id := fmt.Sprintf("ck_%016x", i) mustPublish(b, st, id, id) @@ -744,22 +583,22 @@ func BenchmarkMetas(b *testing.B) { } } -func writeSparse(t testing.TB, path string, size int64, writes map[int64][]byte) string { +func mustFetch(t *testing.T, st *Store, id string) (dir string, meta []byte, digest string) { t.Helper() - f, err := os.Create(path) + dir, meta, digest, err := st.Fetch(t.Context(), id) if err != nil { - t.Fatalf("create %s: %v", path, err) + t.Fatalf("Fetch %s: %v", id, err) } - defer func() { _ = f.Close() }() - if err = f.Truncate(size); err != nil { - t.Fatalf("size %s: %v", path, err) - } - for off, data := range writes { - if _, err = f.WriteAt(data, off); err != nil { - t.Fatalf("write %s: %v", path, err) - } + return dir, meta, digest +} + +func newStore(t testing.TB, root string) *Store { + t.Helper() + st, err := New(root, store.CheckpointIDRe) + if err != nil { + t.Fatalf("New: %v", err) } - return path + return st } func readEverythingDigest(t testing.TB, root string) string { @@ -796,18 +635,10 @@ func readEverythingDigest(t testing.TB, root string) string { return digest } -func metaJSON(metaID string) string { - return `{"id":"` + metaID + `"}` -} - func installUncommitted(t *testing.T, st *Store, root, id, metaID string) (meta []byte, gen string) { t.Helper() - staging, err := st.Stage(id) - if err != nil { - t.Fatalf("stage %s: %v", metaID, err) - } - seedRecord(t, staging, metaID) - meta, err = os.ReadFile(filepath.Join(staging, store.MetaFile)) + staging := stagedRecord(t, st, id, metaID) + meta, err := os.ReadFile(filepath.Join(staging, store.MetaFile)) if err != nil { t.Fatalf("read staged meta: %v", err) } @@ -827,11 +658,7 @@ func installUncommitted(t *testing.T, st *Store, root, id, metaID string) (meta func mustPublish(t testing.TB, st *Store, id, metaID string) { t.Helper() - staging, err := st.Stage(id) - if err != nil { - t.Fatalf("stage %s: %v", metaID, err) - } - seedRecord(t, staging, metaID) + staging := stagedRecord(t, st, id, metaID) if err := st.Publish(t.Context(), staging, id, ""); err != nil { t.Fatalf("publish %s: %v", metaID, err) } @@ -839,11 +666,7 @@ func mustPublish(t testing.TB, st *Store, id, metaID string) { func mustPublishDigested(t *testing.T, st *Store, id, metaID string) string { t.Helper() - staging, err := st.Stage(id) - if err != nil { - t.Fatalf("stage %s: %v", metaID, err) - } - seedRecord(t, staging, metaID) + staging := stagedRecord(t, st, id, metaID) digest, err := storetest.PublishDigested(t.Context(), st, staging, id) if err != nil { t.Fatalf("publish digested %s: %v", metaID, err) @@ -851,6 +674,16 @@ func mustPublishDigested(t *testing.T, st *Store, id, metaID string) string { return digest } +func stagedRecord(t testing.TB, st *Store, id, metaID string) string { + t.Helper() + staging, err := st.Stage(id) + if err != nil { + t.Fatalf("stage %s: %v", metaID, err) + } + seedRecord(t, staging, metaID) + return staging +} + func seedRecord(t testing.TB, dir, metaID string) { t.Helper() if err := os.MkdirAll(filepath.Join(dir, store.ExportDir), 0o750); err != nil { @@ -859,7 +692,7 @@ func seedRecord(t testing.TB, dir, metaID string) { if err := os.WriteFile(filepath.Join(dir, store.ExportDir, "disk.img"), []byte(metaID), 0o600); err != nil { t.Fatalf("write export: %v", err) } - if err := os.WriteFile(filepath.Join(dir, store.MetaFile), []byte(metaJSON(metaID)), 0o600); err != nil { + if err := os.WriteFile(filepath.Join(dir, store.MetaFile), []byte(storetest.MetaJSON(metaID)), 0o600); err != nil { t.Fatalf("write meta: %v", err) } } diff --git a/sandboxd/store/peer/broadcast.go b/sandboxd/store/peer/broadcast.go index 4040cde7e..082cb1ea8 100644 --- a/sandboxd/store/peer/broadcast.go +++ b/sandboxd/store/peer/broadcast.go @@ -12,10 +12,7 @@ import ( "github.com/projecteru2/core/log" ) -const ( - deleteTimeout = 5 * time.Second - defaultDeleteClientTimeout = 10 * time.Second -) +const deleteTimeout = 5 * time.Second // Broadcaster fans a checkpoint delete out to every peer so a healed copy does not outlive it. type Broadcaster struct { @@ -29,12 +26,11 @@ func (b *Broadcaster) Delete(ctx context.Context, id string) { if len(addrs) == 0 { return } - client := &http.Client{Timeout: defaultDeleteClientTimeout} logger := log.WithFunc("peer.Broadcaster.Delete") var wg sync.WaitGroup for _, addr := range addrs { wg.Go(func() { - if err := deleteOn(ctx, client, addr, id, b.Token); err != nil { + if err := deleteOn(ctx, addr, id, b.Token); err != nil { logger.Warnf(ctx, "broadcast delete %s to %s: %v", id, addr, err) } }) @@ -42,7 +38,7 @@ func (b *Broadcaster) Delete(ctx context.Context, id string) { wg.Wait() } -func deleteOn(ctx context.Context, client *http.Client, addr, id, token string) error { +func deleteOn(ctx context.Context, addr, id, token string) error { ctx, cancel := context.WithTimeout(ctx, deleteTimeout) defer cancel() @@ -53,7 +49,7 @@ func deleteOn(ctx context.Context, client *http.Client, addr, id, token string) if token != "" { req.Header.Set("Authorization", "Bearer "+token) } - resp, err := client.Do(req) + resp, err := http.DefaultClient.Do(req) if err != nil { return err } diff --git a/sandboxd/store/peer/peer_test.go b/sandboxd/store/peer/peer_test.go index addf39785..61bea09e8 100644 --- a/sandboxd/store/peer/peer_test.go +++ b/sandboxd/store/peer/peer_test.go @@ -1,17 +1,18 @@ package peer import ( - "bytes" "context" "errors" "os" "path/filepath" + "strings" "sync" "testing" "testing/synctest" "time" "github.com/cocoonstack/sandbox/sandboxd/store" + "github.com/cocoonstack/sandbox/sandboxd/store/storetest" ) const testID = "ck_00000000000000aa" @@ -25,7 +26,7 @@ func TestInertWithoutOwnersOrPuller(t *testing.T) { func TestPullWritesStaging(t *testing.T) { puller := &fakePuller{records: map[string]map[string]string{"peer-a:7777": record()}} - h := NewHealer(func(string) []string { return []string{"peer-a:7777"} }, puller) + h := healerOf(puller, "peer-a:7777") dst := t.TempDir() if err := h.Pull(t.Context(), testID, dst, nil); err != nil { @@ -41,7 +42,7 @@ func TestHealTriesNextOwner(t *testing.T) { records: map[string]map[string]string{"peer-b:7777": record()}, failAddr: "peer-a:7777", } - h := NewHealer(func(string) []string { return []string{"peer-a:7777", "peer-b:7777"} }, puller) + h := healerOf(puller, "peer-a:7777", "peer-b:7777") if err := h.Pull(t.Context(), testID, t.TempDir(), nil); err != nil { t.Fatalf("Pull: %v", err) @@ -53,16 +54,16 @@ func TestHealTriesNextOwner(t *testing.T) { func TestHealAllOwnersMissStaysNotFound(t *testing.T) { puller := &fakePuller{records: map[string]map[string]string{}} - h := NewHealer(func(string) []string { return []string{"peer-a:7777"} }, puller) + h := healerOf(puller, "peer-a:7777") if err := h.Pull(t.Context(), testID, t.TempDir(), nil); !errors.Is(err, store.ErrNotFound) { t.Fatalf("Pull error = %v, want store.ErrNotFound", err) } } -func TestHealNoOwners(t *testing.T) { +func TestHealWithNoOwnersIsNotFoundWithoutAPull(t *testing.T) { puller := &fakePuller{} - h := NewHealer(func(string) []string { return nil }, puller) + h := healerOf(puller) if err := h.Pull(t.Context(), testID, t.TempDir(), nil); !errors.Is(err, store.ErrNotFound) { t.Fatalf("Pull error = %v, want store.ErrNotFound", err) @@ -74,20 +75,20 @@ func TestHealNoOwners(t *testing.T) { func TestHealErrorIsReported(t *testing.T) { puller := &fakePuller{failAddr: "peer-a:7777"} - h := NewHealer(func(string) []string { return []string{"peer-a:7777"} }, puller) + h := healerOf(puller, "peer-a:7777") err := h.Pull(t.Context(), testID, t.TempDir(), nil) if err == nil || errors.Is(err, store.ErrNotFound) { t.Fatalf("Pull error = %v, want the peer failure surfaced", err) } - if !bytes.Contains([]byte(err.Error()), []byte("peer exploded")) { + if !strings.Contains(err.Error(), "peer exploded") { t.Errorf("error = %v, want the underlying peer failure named", err) } } func TestPullDoesNotDedupConcurrentCalls(t *testing.T) { puller := &fakePuller{records: map[string]map[string]string{"peer-a:7777": record()}} - h := NewHealer(func(string) []string { return []string{"peer-a:7777"} }, puller) + h := healerOf(puller, "peer-a:7777") dst1, dst2 := t.TempDir(), t.TempDir() var wg sync.WaitGroup @@ -117,7 +118,7 @@ func TestPullValidateRejectionTriesNextOwner(t *testing.T) { "peer-a:7777": record(), "peer-b:7777": record(), }} - h := NewHealer(func(string) []string { return []string{"peer-a:7777", "peer-b:7777"} }, puller) + h := healerOf(puller, "peer-a:7777", "peer-b:7777") var calls int validate := func(string) error { @@ -137,7 +138,7 @@ func TestPullValidateRejectionTriesNextOwner(t *testing.T) { func TestPullValidateRejectsEveryOwner(t *testing.T) { puller := &fakePuller{records: map[string]map[string]string{"peer-a:7777": record()}} - h := NewHealer(func(string) []string { return []string{"peer-a:7777"} }, puller) + h := healerOf(puller, "peer-a:7777") err := h.Pull(t.Context(), testID, t.TempDir(), func(string) error { return errors.New("bad shape") }) if err == nil || errors.Is(err, store.ErrNotFound) { @@ -147,7 +148,7 @@ func TestPullValidateRejectsEveryOwner(t *testing.T) { func TestPullBudgetBoundsSlowOwners(t *testing.T) { synctest.Test(t, func(t *testing.T) { - h := NewHealer(func(string) []string { return []string{"peer-a:7777", "peer-b:7777"} }, &sleepyPuller{sleep: 2 * healBudget}) + h := healerOf(&sleepyPuller{sleep: 2 * healBudget}, "peer-a:7777", "peer-b:7777") start := time.Now() err := h.Pull(t.Context(), testID, t.TempDir(), nil) if elapsed := time.Since(start); elapsed != healBudget { @@ -204,7 +205,11 @@ func (p *sleepyPuller) Pull(ctx context.Context, _, _, _ string) error { func record() map[string]string { return map[string]string{ - store.MetaFile: `{"id":"` + testID + `"}`, + store.MetaFile: storetest.MetaJSON(testID), filepath.Join(store.ExportDir, "mem"): "guest-pages", } } + +func healerOf(puller Puller, owners ...string) *Healer { + return NewHealer(func(string) []string { return owners }, puller) +} diff --git a/sandboxd/store/peer/probe.go b/sandboxd/store/peer/probe.go index 1b0362275..64e450820 100644 --- a/sandboxd/store/peer/probe.go +++ b/sandboxd/store/peer/probe.go @@ -17,11 +17,11 @@ import ( ) const ( - probeTimeout = time.Second - defaultProbeClientTimeout = 2 * time.Second - maxRedirectOwners = 3 // a redirect decision needs a couple of candidates, not the whole fleet - maxHealOwners = 8 // a heal wants the widest source list a few full/corrupt owners can't hide behind - redirectCacheTTL = 5 * time.Second + probeTimeout = time.Second + fanOutTimeout = 2 * time.Second + maxRedirectOwners = 3 // a redirect decision needs a couple of candidates, not the whole fleet + maxHealOwners = 8 // a heal wants the widest source list a few full/corrupt owners can't hide behind + redirectCacheTTL = 5 * time.Second // probeGrace bounds how long a fan-out waits for more owners once it has its first. probeGrace = 150 * time.Millisecond @@ -93,15 +93,14 @@ func (p *HTTPProber) fanOut(ctx context.Context, id string, maxOwners int, grace if len(addrs) == 0 { return nil } - client := &http.Client{Timeout: defaultProbeClientTimeout} - ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), defaultProbeClientTimeout) + ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), fanOutTimeout) defer cancel() wins := make(chan string, len(addrs)) var wg sync.WaitGroup for _, addr := range addrs { wg.Go(func() { - if probeOwner(ctx, client, p.ProbeKey, addr, id) { + if probeOwner(ctx, p.ProbeKey, addr, id) { wins <- addr } }) @@ -201,7 +200,7 @@ func currentBucket() int64 { } // probeOwner sends a probe MAC when probeKey is set; an empty key falls back to capability-only. -func probeOwner(ctx context.Context, client *http.Client, probeKey []byte, addr, id string) bool { +func probeOwner(ctx context.Context, probeKey []byte, addr, id string) bool { ctx, cancel := context.WithTimeout(ctx, probeTimeout) defer cancel() @@ -212,7 +211,7 @@ func probeOwner(ctx context.Context, client *http.Client, probeKey []byte, addr, if len(probeKey) > 0 { req.Header.Set(ProbeHeader, SignProbe(probeKey, id)) } - resp, err := client.Do(req) + resp, err := http.DefaultClient.Do(req) if err != nil { return false } diff --git a/sandboxd/store/peer/transport_test.go b/sandboxd/store/peer/transport_test.go index 4cb261709..e7537e65e 100644 --- a/sandboxd/store/peer/transport_test.go +++ b/sandboxd/store/peer/transport_test.go @@ -16,6 +16,7 @@ import ( "time" "github.com/cocoonstack/sandbox/sandboxd/store" + "github.com/cocoonstack/sandbox/sandboxd/store/storetest" ) func TestTarUntarRoundTrip(t *testing.T) { @@ -303,29 +304,18 @@ func TestTarRecordUntarRoundTripComplete(t *testing.T) { func TestTarRecordKeepsSparseFilesSparse(t *testing.T) { size := 10*store.DigestChunkSize + 4096 exportDir := t.TempDir() - src := filepath.Join(exportDir, "cow.raw") - f, err := os.Create(src) - if err != nil { - t.Fatalf("create: %v", err) - } - if err = f.Truncate(size); err != nil { - t.Fatalf("truncate: %v", err) - } - for off, data := range map[int64]string{0: "sparse-head", 4*store.DigestChunkSize + 7: "sparse-mid", 8*store.DigestChunkSize + 5: "tail"} { - if _, err = f.WriteAt([]byte(data), off); err != nil { - t.Fatalf("write at %d: %v", off, err) - } - } - if err = f.Close(); err != nil { - t.Fatalf("close: %v", err) - } - if err = os.WriteFile(filepath.Join(exportDir, "snapshot.json"), []byte("{}"), 0o600); err != nil { + src := storetest.WriteSparse(t, filepath.Join(exportDir, "cow.raw"), size, map[int64][]byte{ + 0: []byte("sparse-head"), + 4*store.DigestChunkSize + 7: []byte("sparse-mid"), + 8*store.DigestChunkSize + 5: []byte("tail"), + }) + if err := os.WriteFile(filepath.Join(exportDir, "snapshot.json"), []byte("{}"), 0o600); err != nil { t.Fatalf("write snapshot: %v", err) } want := sha256OfFile(t, src) var buf bytes.Buffer - if err = TarRecord(exportDir, []byte(`{"id":"ck_0000000000000001"}`), &buf); err != nil { + if err := TarRecord(exportDir, []byte(`{"id":"ck_0000000000000001"}`), &buf); err != nil { t.Fatalf("TarRecord: %v", err) } sent := int64(buf.Len()) @@ -333,7 +323,7 @@ func TestTarRecordKeepsSparseFilesSparse(t *testing.T) { t.Errorf("stream carries %d bytes for a %d-byte file with 3 data chunks, want at most %d", sent, size, limit) } dst := t.TempDir() - if err = Untar(&buf, dst); err != nil { + if err := Untar(&buf, dst); err != nil { t.Fatalf("Untar: %v", err) } got := filepath.Join(dst, "export", "cow.raw") diff --git a/sandboxd/store/s3/s3_test.go b/sandboxd/store/s3/s3_test.go index e1fea1f8a..0fae5e5d2 100644 --- a/sandboxd/store/s3/s3_test.go +++ b/sandboxd/store/s3/s3_test.go @@ -28,15 +28,14 @@ import ( ) func TestS3BackendContract(t *testing.T) { - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) + _, st := newFakeStore(t) storetest.RunContract(t, st) } func TestDeleteRetryConverges(t *testing.T) { const id = "ck_00000000000000bb" fake := &fakeS3{objects: map[string][]byte{ - "ck/" + id + "/" + store.MetaFile: []byte(`{"id":"` + id + `"}`), + "ck/" + id + "/" + store.MetaFile: metaJSON(id), "ck/manifests/" + id + "/" + store.ExportDir + "-first.json": []byte(`{"lineage":"` + id + `","files":[]}`), }} st := newTestStore(t, fake) @@ -58,7 +57,7 @@ func TestDeleteRetainsMetaUntilManifestsAreRefreshed(t *testing.T) { "ck/manifests/" + id + "/export-first.json", "ck/manifests/" + id + "/export-second.json", } - for _, tc := range []struct { + for _, tt := range []struct { name string failList bool failDelete bool @@ -66,15 +65,15 @@ func TestDeleteRetainsMetaUntilManifestsAreRefreshed(t *testing.T) { {name: "list failure", failList: true}, {name: "delete failure", failDelete: true}, } { - t.Run(tc.name, func(t *testing.T) { + t.Run(tt.name, func(t *testing.T) { fake := &fakeS3{ objects: map[string][]byte{ - metaKey: []byte(`{"id":"` + id + `"}`), + metaKey: metaJSON(id), manifestKeys[0]: []byte(`{"lineage":"` + id + `","files":[]}`), manifestKeys[1]: []byte(`{"lineage":"` + id + `","files":[]}`), }, - failList: tc.failList, - failDelete: tc.failDelete, + failList: tt.failList, + failDelete: tt.failDelete, } st := newTestStore(t, fake) @@ -130,7 +129,7 @@ func TestDeleteRetainsMetaUntilManifestsAreRefreshed(t *testing.T) { func TestFetchReportsMetaWithoutExportAsNotFound(t *testing.T) { const id = "ck_00000000000000cc" fake := &fakeS3{objects: map[string][]byte{ - "ck/" + id + "/" + store.MetaFile: []byte(`{"id":"` + id + `"}`), + "ck/" + id + "/" + store.MetaFile: metaJSON(id), }} st := newTestStore(t, fake) if _, _, _, err := st.Fetch(t.Context(), id); !errors.Is(err, store.ErrNotFound) { @@ -141,11 +140,8 @@ func TestFetchReportsMetaWithoutExportAsNotFound(t *testing.T) { func TestFetchOutlivesTheCallerThatStartedIt(t *testing.T) { const id = "ck_00000000000000dd" metaKey := "ck/" + id + "/" + store.MetaFile - fake := &fakeS3{objects: map[string][]byte{}, exportGate: make(chan struct{}), exportEntered: make(chan struct{}, 1)} - st := newTestStore(t, fake) - release := sync.OnceFunc(func() { close(fake.exportGate) }) - t.Cleanup(release) - publishRecord(t, st, id, []byte(`{"id":"`+id+`"}`), "payload") + fake, st, release := newGatedStore(t, "") + publishRecord(t, st, id, metaJSON(id), "payload") fake.resetRequests() first, hangUp := context.WithCancel(t.Context()) @@ -182,8 +178,7 @@ func TestFetchOutlivesTheCallerThatStartedIt(t *testing.T) { func TestPublishDigestedRetryAndMetaRequestAccounting(t *testing.T) { const id = "ck_00000000000000dd" - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) + fake, st := newFakeStore(t) meta := []byte(`{"id":"` + id + `","gen":1}`) content := "retry-body" chunk := sha256.Sum256([]byte(content)) @@ -250,8 +245,7 @@ func TestPublishDigestedRetryAndMetaRequestAccounting(t *testing.T) { func TestPublishDigestedFailurePreservesCommittedGeneration(t *testing.T) { const id = "ck_00000000000000ee" - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) + fake, st := newFakeStore(t) oldMeta := []byte(`{"id":"` + id + `","gen":1}`) oldStaging := stageRecord(t, st, id, oldMeta, "old") oldDigest, err := storetest.PublishDigested(t.Context(), st, oldStaging, id) @@ -322,7 +316,7 @@ func TestRepublishRetainsGenerationSelectedByAnotherStore(t *testing.T) { if err = writer.Delete(t.Context(), id); err != nil { t.Fatalf("delete the record: %v", err) } - if _, ok := fake.get("ck/" + id + "/" + store.MetaFile); ok { + if fake.has("ck/" + id + "/" + store.MetaFile) { t.Error("meta survived Delete") } gen := filepath.Join(reader.staging, "selected-first") @@ -337,7 +331,7 @@ func TestRepublishRetainsGenerationSelectedByAnotherStore(t *testing.T) { func TestFetchPrunesSupersededGenerationsIdlePastTheGrace(t *testing.T) { const id = "ck_00000000000000aa" - st := newTestStore(t, &fakeS3{objects: map[string][]byte{}}) + _, st := newFakeStore(t) fetchGen := func(gen int) string { t.Helper() publishRecord(t, st, id, fmt.Appendf(nil, `{"id":"%s","gen":%d}`, id, gen), fmt.Sprint(gen)) @@ -367,7 +361,7 @@ func TestFetchPrunesSupersededGenerationsIdlePastTheGrace(t *testing.T) { func TestFetchHitRefreshesTheGenerationsUse(t *testing.T) { const id = "ck_00000000000000aa" - st := newTestStore(t, &fakeS3{objects: map[string][]byte{}}) + _, st := newFakeStore(t) publishRecord(t, st, id, []byte(`{"id":"`+id+`","gen":1}`), "first") export, _, _, err := st.Fetch(t.Context(), id) if err != nil { @@ -439,11 +433,10 @@ func TestS3BackendContractRealEndpoint(t *testing.T) { func TestSparseExportStaysSparseThroughS3(t *testing.T) { const id = "ck_00000000000000a1" - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) + fake, st := newFakeStore(t) size := 12*store.DigestChunkSize + 4096 - meta := []byte(`{"id":"` + id + `"}`) - staging, src := stageSparseRecord(t, st, id, meta, size, map[int64]string{0: "head", 3*store.DigestChunkSize + 100: "mid", 11*store.DigestChunkSize + 10: "tail"}) + meta := metaJSON(id) + staging, src := stageSparseRecord(t, st, id, meta, size, map[int64][]byte{0: []byte("head"), 3*store.DigestChunkSize + 100: []byte("mid"), 11*store.DigestChunkSize + 10: []byte("tail")}) srcSum := fileSum(t, src) if err := st.Publish(t.Context(), staging, id, ""); err != nil { t.Fatalf("Publish: %v", err) @@ -472,9 +465,8 @@ func TestSparseExportStaysSparseThroughS3(t *testing.T) { func TestAllZeroDataChunkIsNotUploaded(t *testing.T) { const id = "ck_00000000000000a2" - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) - meta := []byte(`{"id":"` + id + `"}`) + fake, st := newFakeStore(t) + meta := metaJSON(id) staging := stageRecord(t, st, id, meta, "") src := filepath.Join(staging, store.ExportDir, "disk.img") dense := make([]byte, 2*store.DigestChunkSize+10) @@ -501,10 +493,10 @@ func TestAllZeroDataChunkIsNotUploaded(t *testing.T) { func TestFetchFailsOnDamagedChunks(t *testing.T) { const id = "ck_00000000000000a3" - meta := []byte(`{"id":"` + id + `"}`) + meta := metaJSON(id) sum := sha256.Sum256([]byte("payload")) chunkKey := "ck/chunks/" + id + "/" + hex.EncodeToString(sum[:]) - for _, tc := range []struct { + for _, tt := range []struct { name string damage func(*fakeS3) want string @@ -512,14 +504,13 @@ func TestFetchFailsOnDamagedChunks(t *testing.T) { {"flipped byte", func(f *fakeS3) { f.objects[chunkKey][0] ^= 0xff }, "checksum"}, {"missing chunk", func(f *fakeS3) { delete(f.objects, chunkKey) }, "is missing"}, } { - t.Run(tc.name, func(t *testing.T) { - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) + t.Run(tt.name, func(t *testing.T) { + fake, st := newFakeStore(t) publishRecord(t, st, id, meta, "payload") - tc.damage(fake) + tt.damage(fake) _, _, _, err := st.Fetch(t.Context(), id) - if err == nil || !strings.Contains(err.Error(), tc.want) { - t.Fatalf("Fetch = %v, want an error containing %q", err, tc.want) + if err == nil || !strings.Contains(err.Error(), tt.want) { + t.Fatalf("Fetch = %v, want an error containing %q", err, tt.want) } if errors.Is(err, store.ErrNotFound) { t.Errorf("a damaged record must not read as absent: %v", err) @@ -530,9 +521,9 @@ func TestFetchFailsOnDamagedChunks(t *testing.T) { func TestPublishDigestedHashesHolesLikeZeros(t *testing.T) { const id = "ck_00000000000000a4" - st := newTestStore(t, &fakeS3{objects: map[string][]byte{}}) + _, st := newFakeStore(t) size := 3*store.DigestChunkSize + 7 - staging, src := stageSparseRecord(t, st, id, []byte(`{"id":"`+id+`"}`), size, map[int64]string{store.DigestChunkSize + 5: "data", size - 1: "z"}) + staging, src := stageSparseRecord(t, st, id, metaJSON(id), size, map[int64][]byte{store.DigestChunkSize + 5: []byte("data"), size - 1: []byte("z")}) sums := denseChunkSums(t, src) digest, err := storetest.PublishDigested(t.Context(), st, staging, id) if err != nil { @@ -549,9 +540,9 @@ func TestPublishDigestedHashesHolesLikeZeros(t *testing.T) { func TestPublishDigestedSplitsChunksAtTheFixedBoundary(t *testing.T) { const id = "ck_00000000000000a5" - st := newTestStore(t, &fakeS3{objects: map[string][]byte{}}) + _, st := newFakeStore(t) content := string(bytes.Repeat([]byte{'x'}, int(store.DigestChunkSize))) + "y" - staging := stageRecord(t, st, id, []byte(`{"id":"`+id+`"}`), content) + staging := stageRecord(t, st, id, metaJSON(id), content) digest, err := storetest.PublishDigested(t.Context(), st, staging, id) if err != nil { t.Fatalf("PublishDigested: %v", err) @@ -570,13 +561,12 @@ func TestPublishDigestedSplitsChunksAtTheFixedBoundary(t *testing.T) { func TestLineageDedupUploadsOnlyChangedChunks(t *testing.T) { const lineage = "sbx_00000000000000a1" - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) + fake, st := newFakeStore(t) cs := store.DigestChunkSize - base := map[int64]string{0: "c0", 2 * cs: "c2", 4 * cs: "c4", 6 * cs: "c6"} + base := map[int64][]byte{0: []byte("c0"), 2 * cs: []byte("c2"), 4 * cs: []byte("c4"), 6 * cs: []byte("c6")} changed := maps.Clone(base) - changed[4*cs] = "changed" - publish := func(id string, gen int, writes map[int64]string, scope string) (puts int, srcSum [sha256.Size]byte) { + changed[4*cs] = []byte("changed") + publish := func(id string, gen int, writes map[int64][]byte, scope string) (puts int, srcSum [sha256.Size]byte) { t.Helper() before, _ := fake.chunkPuts("ck/chunks/") staging, src := stageSparseRecord(t, st, id, fmt.Appendf(nil, `{"id":"%s","gen":%d}`, id, gen), 8*cs, writes) @@ -613,14 +603,13 @@ func TestLineageDedupUploadsOnlyChangedChunks(t *testing.T) { func TestPublishReadsOnlyItsLineage(t *testing.T) { const lineage = "sbx_00000000000000a4" - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) + fake, st := newFakeStore(t) for _, id := range []string{"ck_00000000000000d1", "ck_00000000000000d2"} { - publishRecord(t, st, id, []byte(`{"id":"`+id+`"}`), "unrelated") + publishRecord(t, st, id, metaJSON(id), "unrelated") } publishIn := func(id string) { t.Helper() - staging := stageRecord(t, st, id, []byte(`{"id":"`+id+`"}`), "claim") + staging := stageRecord(t, st, id, metaJSON(id), "claim") if err := st.Publish(t.Context(), staging, id, lineage); err != nil { t.Fatalf("Publish %s: %v", id, err) } @@ -641,12 +630,11 @@ func TestPublishReadsOnlyItsLineage(t *testing.T) { func TestPublishUploadsAnUnreferencedChunkAgain(t *testing.T) { const lineage = "sbx_00000000000000a3" - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) + fake, st := newFakeStore(t) publishIn := func(id, content, scope string) int { t.Helper() before, _ := fake.chunkPuts("ck/chunks/") - staging := stageRecord(t, st, id, []byte(`{"id":"`+id+`"}`), content) + staging := stageRecord(t, st, id, metaJSON(id), content) if err := st.Publish(t.Context(), staging, id, scope); err != nil { t.Fatalf("Publish %s: %v", id, err) } @@ -678,10 +666,9 @@ func TestDeletedRecordLeavesNoPrefixToList(t *testing.T) { gone = "ck_00000000000000b1" live = "ck_00000000000000b2" ) - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) - publishRecord(t, st, gone, []byte(`{"id":"`+gone+`"}`), "gone-payload") - publishRecord(t, st, live, []byte(`{"id":"`+live+`"}`), "live-payload") + fake, st := newFakeStore(t) + publishRecord(t, st, gone, metaJSON(gone), "gone-payload") + publishRecord(t, st, live, metaJSON(live), "live-payload") if err := st.Delete(t.Context(), gone); err != nil { t.Fatalf("Delete: %v", err) } @@ -701,21 +688,20 @@ func TestDeletedRecordLeavesNoPrefixToList(t *testing.T) { } sum := sha256.Sum256([]byte("gone-payload")) - if _, ok := fake.get("ck/chunks/" + gone + "/" + hex.EncodeToString(sum[:])); !ok { + if !fake.has("ck/chunks/" + gone + "/" + hex.EncodeToString(sum[:])) { t.Error("Delete freed the chunk of the deleted record") } } func TestSweepGenerationsLeavesTheBucketAlone(t *testing.T) { const id = "ck_00000000000000a8" - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) + fake, st := newFakeStore(t) publishRecord(t, st, id, []byte(`{"id":"`+id+`","gen":1}`), "first") publishRecord(t, st, id, []byte(`{"id":"`+id+`","gen":2}`), "second") if err := st.Delete(t.Context(), id); err != nil { t.Fatalf("Delete: %v", err) } - fake.age("", 2*store.GenerationGrace) + fake.age(2 * store.GenerationGrace) want := fake.keys() if err := st.SweepGenerations(t.Context()); err != nil { t.Fatalf("SweepGenerations: %v", err) @@ -730,7 +716,7 @@ func TestSweepGenerationsDropsTheCacheOfARecordAPeerDeleted(t *testing.T) { fake := &fakeS3{objects: map[string][]byte{}} owner, fetcher := newTestStore(t, fake), newTestStore(t, fake) for _, id := range []string{deleted, live} { - publishRecord(t, owner, id, []byte(`{"id":"`+id+`"}`), id) + publishRecord(t, owner, id, metaJSON(id), id) if _, _, _, err := fetcher.Fetch(t.Context(), id); err != nil { t.Fatalf("fetch %s: %v", id, err) } @@ -754,11 +740,8 @@ func TestSweepGenerationsDropsTheCacheOfARecordAPeerDeleted(t *testing.T) { func TestSweepGenerationsDropsACacheACanceledFetchInstalledAfterDelete(t *testing.T) { const id = "ck_00000000000000ab" - fake := &fakeS3{objects: map[string][]byte{}, exportGate: make(chan struct{}), exportEntered: make(chan struct{}, 1), gateOn: "/chunks/"} - st := newTestStore(t, fake) - release := sync.OnceFunc(func() { close(fake.exportGate) }) - t.Cleanup(release) - publishRecord(t, st, id, []byte(`{"id":"`+id+`"}`), "payload") + fake, st, release := newGatedStore(t, "/chunks/") + publishRecord(t, st, id, metaJSON(id), "payload") ctx, cancel := context.WithCancel(t.Context()) fetched := make(chan error, 1) go func() { @@ -801,10 +784,9 @@ func TestSweepGenerationsDropsACacheACanceledFetchInstalledAfterDelete(t *testin func TestDeleteRemovesTheLineageMarker(t *testing.T) { const lineage = "sbx_00000000000000a5" - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) + fake, st := newFakeStore(t) for _, id := range []string{"ck_00000000000000f1", "ck_00000000000000f2"} { - staging := stageRecord(t, st, id, []byte(`{"id":"`+id+`"}`), "claim") + staging := stageRecord(t, st, id, metaJSON(id), "claim") if err := st.Publish(t.Context(), staging, id, lineage); err != nil { t.Fatalf("Publish %s: %v", id, err) } @@ -812,10 +794,10 @@ func TestDeleteRemovesTheLineageMarker(t *testing.T) { if err := st.Delete(t.Context(), "ck_00000000000000f1"); err != nil { t.Fatalf("Delete: %v", err) } - if _, ok := fake.get("ck/lineages/" + lineage + "/ck_00000000000000f1"); ok { + if fake.has("ck/lineages/" + lineage + "/ck_00000000000000f1") { t.Error("Delete left the lineage marker of the deleted record") } - if _, ok := fake.get("ck/lineages/" + lineage + "/ck_00000000000000f2"); !ok { + if !fake.has("ck/lineages/" + lineage + "/ck_00000000000000f2") { t.Error("Delete removed the marker of another record") } } @@ -829,9 +811,8 @@ func TestTransferBudgetStaysBelowTheReuseWindow(t *testing.T) { func TestFetchFileDownloadsOnlyTheRequestedFile(t *testing.T) { const id = "ck_00000000000000aa" - meta := []byte(`{"id":"` + id + `"}`) - fake := &fakeS3{objects: map[string][]byte{}} - st := newTestStore(t, fake) + meta := metaJSON(id) + fake, st := newFakeStore(t) staging := stageRecord(t, st, id, meta, "disk-bytes") if err := os.WriteFile(filepath.Join(staging, store.ExportDir, "memory.bin"), []byte("memory-bytes"), 0o600); err != nil { t.Fatalf("write memory: %v", err) @@ -860,11 +841,8 @@ func TestFetchFileDownloadsOnlyTheRequestedFile(t *testing.T) { func TestFetchFileInstallsOnlyACompleteDownload(t *testing.T) { const id = "ck_00000000000000aa" - meta := []byte(`{"id":"` + id + `"}`) - fake := &fakeS3{objects: map[string][]byte{}, exportGate: make(chan struct{}), exportEntered: make(chan struct{}, 1), gateOn: "/chunks/"} - st := newTestStore(t, fake) - release := sync.OnceFunc(func() { close(fake.exportGate) }) - t.Cleanup(release) + meta := metaJSON(id) + fake, st, release := newGatedStore(t, "/chunks/") publishRecord(t, st, id, meta, "disk-bytes") dst := filepath.Join(t.TempDir(), "disk.img") @@ -890,9 +868,9 @@ func TestFetchFileInstallsOnlyACompleteDownload(t *testing.T) { func TestFetchFileKeepsHoles(t *testing.T) { const id = "ck_00000000000000a2" - st := newTestStore(t, &fakeS3{objects: map[string][]byte{}}) + _, st := newFakeStore(t) size := 12*store.DigestChunkSize + 4096 - staging, src := stageSparseRecord(t, st, id, []byte(`{"id":"`+id+`"}`), size, map[int64]string{0: "head", 11*store.DigestChunkSize + 10: "tail"}) + staging, src := stageSparseRecord(t, st, id, metaJSON(id), size, map[int64][]byte{0: []byte("head"), 11*store.DigestChunkSize + 10: []byte("tail")}) srcSum := fileSum(t, src) if err := st.Publish(t.Context(), staging, id, ""); err != nil { t.Fatalf("Publish: %v", err) @@ -913,144 +891,6 @@ func TestFetchFileKeepsHoles(t *testing.T) { } } -func ageCache(t *testing.T, dir string) { - t.Helper() - entries, err := os.ReadDir(dir) - if err != nil { - t.Fatalf("read cache %s: %v", dir, err) - } - old := time.Now().Add(-2 * store.GenerationGrace) - for _, e := range entries { - if err := os.Chtimes(filepath.Join(dir, e.Name()), old, old); err != nil { - t.Fatalf("age cache: %v", err) - } - } -} - -func publishRecord(t *testing.T, st *Store, id string, meta []byte, content string) { - t.Helper() - staging := stageRecord(t, st, id, meta, content) - if err := st.Publish(t.Context(), staging, id, ""); err != nil { - t.Fatalf("publish record: %v", err) - } -} - -func stageRecord(t *testing.T, st *Store, id string, meta []byte, content string) string { - t.Helper() - staging, err := st.Stage(id) - if err != nil { - t.Fatalf("stage record: %v", err) - } - export := filepath.Join(staging, store.ExportDir) - if err := os.MkdirAll(export, 0o750); err != nil { - t.Fatalf("mkdir export: %v", err) - } - if err := os.WriteFile(filepath.Join(export, "disk.img"), []byte(content), 0o600); err != nil { - t.Fatalf("write export: %v", err) - } - if err := os.WriteFile(filepath.Join(staging, store.MetaFile), meta, 0o600); err != nil { - t.Fatalf("write meta: %v", err) - } - return staging -} - -func stageSparseRecord(t *testing.T, st *Store, id string, meta []byte, size int64, writes map[int64]string) (staging, path string) { - t.Helper() - staging = stageRecord(t, st, id, meta, "") - path = filepath.Join(staging, store.ExportDir, "disk.img") - f, err := os.OpenFile(path, os.O_WRONLY, 0) - if err != nil { - t.Fatalf("open export: %v", err) - } - defer func() { _ = f.Close() }() - if err = f.Truncate(size); err != nil { - t.Fatalf("truncate export: %v", err) - } - for off, data := range writes { - if _, err = f.WriteAt([]byte(data), off); err != nil { - t.Fatalf("write export at %d: %v", off, err) - } - } - return staging, path -} - -func fileSum(t *testing.T, path string) [sha256.Size]byte { - t.Helper() - f, err := os.Open(path) - if err != nil { - t.Fatalf("open %s: %v", path, err) - } - defer func() { _ = f.Close() }() - h := sha256.New() - if _, err = io.Copy(h, f); err != nil { - t.Fatalf("hash %s: %v", path, err) - } - return [sha256.Size]byte(h.Sum(nil)) -} - -func allocatedBytes(t *testing.T, path string) int64 { - t.Helper() - info, err := os.Stat(path) - if err != nil { - t.Fatalf("stat %s: %v", path, err) - } - return info.Sys().(*syscall.Stat_t).Blocks * 512 -} - -func denseChunkSums(t *testing.T, path string) [][sha256.Size]byte { - t.Helper() - f, err := os.Open(path) - if err != nil { - t.Fatalf("open %s: %v", path, err) - } - defer func() { _ = f.Close() }() - var sums [][sha256.Size]byte - buf := make([]byte, store.DigestChunkSize) - for { - n, err := io.ReadFull(f, buf) - if n > 0 { - sums = append(sums, sha256.Sum256(buf[:n])) - } - if err != nil { - return sums - } - } -} - -func newTestStore(t *testing.T, fake *fakeS3) *Store { - t.Helper() - return newTestStoreAt(t, fake, false) -} - -func newTestStoreAt(t *testing.T, endpoint http.Handler, unsignedAcceptEncoding bool) *Store { - t.Helper() - ts := httptest.NewServer(endpoint) - t.Cleanup(ts.Close) - t.Setenv("AWS_ACCESS_KEY_ID", "test") - t.Setenv("AWS_SECRET_ACCESS_KEY", "test") - - t.Setenv("AWS_REQUEST_CHECKSUM_CALCULATION", "when_required") - t.Setenv("AWS_RESPONSE_CHECKSUM_VALIDATION", "when_required") - st, err := New(t.Context(), store.S3Config{ - Bucket: "testbucket", Prefix: "ck/", Endpoint: ts.URL, - Region: "us-east-1", ForcePathStyle: true, UnsignedAcceptEncoding: unsignedAcceptEncoding, - }, t.TempDir(), store.CheckpointIDRe) - if err != nil { - t.Fatalf("New: %v", err) - } - return st -} - -func assertSingleMetaGet(t *testing.T, fake *fakeS3, metaKey string) { - t.Helper() - if got := fake.requestCount(http.MethodGet, metaKey); got != 1 { - t.Errorf("meta GET requests = %d, want 1", got) - } - if got := fake.requestCount(http.MethodHead, metaKey); got != 0 { - t.Errorf("meta HEAD requests = %d, want 0", got) - } -} - type fakeRequest struct { method string key string @@ -1095,12 +935,8 @@ func (f *fakeS3) ServeHTTP(w http.ResponseWriter, r *http.Request) { f.handleDeleteObjects(w, r) case r.Method == http.MethodPut: f.handlePutObject(w, r, key) - case r.Method == http.MethodHead: - f.handleHeadObject(w, key) case r.Method == http.MethodGet: - f.handleGetObject(w, r, key) - case r.Method == http.MethodDelete: - f.handleDeleteObject(w, key) + f.handleGetObject(w, key) default: http.Error(w, "unsupported", http.StatusNotImplemented) } @@ -1133,25 +969,25 @@ func (f *fakeS3) objectMetadata(key string) map[string]string { return maps.Clone(f.metadata[key]) } -func (f *fakeS3) chunkPuts(prefix string) (count int, bytes int64) { +func (f *fakeS3) chunkPuts(prefix string) (count int, size int64) { f.mu.Lock() defer f.mu.Unlock() for key, lengths := range f.putLengths { if strings.HasPrefix(key, prefix) { count += len(lengths) for _, n := range lengths { - bytes += n + size += n } } } - return count, bytes + return count, size } -func (f *fakeS3) get(key string) ([]byte, bool) { +func (f *fakeS3) has(key string) bool { f.mu.Lock() defer f.mu.Unlock() - body, ok := f.objects[key] - return body, ok + _, ok := f.objects[key] + return ok } func (f *fakeS3) keys() []string { @@ -1160,13 +996,11 @@ func (f *fakeS3) keys() []string { return slices.Sorted(maps.Keys(f.objects)) } -func (f *fakeS3) age(prefix string, by time.Duration) { +func (f *fakeS3) age(by time.Duration) { f.mu.Lock() defer f.mu.Unlock() for key := range f.objects { - if strings.HasPrefix(key, prefix) { - f.modified[key] = time.Now().Add(-by) - } + f.modified[key] = time.Now().Add(-by) } } @@ -1188,7 +1022,9 @@ func (f *fakeS3) handlePutObject(w http.ResponseWriter, r *http.Request, key str f.putLengths[key] = append(f.putLengths[key], r.ContentLength) if f.failPuts[key] > 0 { f.failPuts[key]-- - writeFakeError(w, http.StatusInternalServerError, "InternalError", "injected put failure") + w.Header().Set("Content-Type", "application/xml") + w.WriteHeader(http.StatusInternalServerError) + _, _ = io.WriteString(w, `InternalErrorinjected put failure`) return } if f.objects == nil { @@ -1257,17 +1093,7 @@ func (f *fakeS3) handleListObjects(w http.ResponseWriter, r *http.Request) { _ = xml.NewEncoder(w).Encode(result) } -func (f *fakeS3) handleHeadObject(w http.ResponseWriter, key string) { - body, ok := f.objects[key] - if !ok { - http.Error(w, "NoSuchKey", http.StatusNotFound) - return - } - writeMetadata(w.Header(), f.metadata[key]) - w.Header().Set("Content-Length", strconv.Itoa(len(body))) -} - -func (f *fakeS3) handleGetObject(w http.ResponseWriter, r *http.Request, key string) { +func (f *fakeS3) handleGetObject(w http.ResponseWriter, key string) { body, ok := f.objects[key] if !ok { w.WriteHeader(http.StatusNotFound) @@ -1275,20 +1101,6 @@ func (f *fakeS3) handleGetObject(w http.ResponseWriter, r *http.Request, key str return } writeMetadata(w.Header(), f.metadata[key]) - if mod := f.modified[key]; !mod.IsZero() { - w.Header().Set("Last-Modified", mod.UTC().Format(http.TimeFormat)) - } - if rng := r.Header.Get("Range"); rng != "" { - var start, end int - if _, err := fmt.Sscanf(rng, "bytes=%d-%d", &start, &end); err == nil && start < len(body) { - end = min(end, len(body)-1) - w.Header().Set("Content-Length", strconv.Itoa(end-start+1)) - w.Header().Set("Content-Range", fmt.Sprintf("bytes %d-%d/%d", start, end, len(body))) - w.WriteHeader(http.StatusPartialContent) - _, _ = w.Write(body[start : end+1]) - return - } - } w.Header().Set("Content-Length", strconv.Itoa(len(body))) _, _ = w.Write(body) } @@ -1338,10 +1150,133 @@ func (f *fakeS3) handleDeleteObjects(w http.ResponseWriter, r *http.Request) { _ = xml.NewEncoder(w).Encode(result) } -func (f *fakeS3) handleDeleteObject(w http.ResponseWriter, key string) { - delete(f.objects, key) - delete(f.metadata, key) - w.WriteHeader(http.StatusNoContent) +func ageCache(t *testing.T, dir string) { + t.Helper() + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatalf("read cache %s: %v", dir, err) + } + old := time.Now().Add(-2 * store.GenerationGrace) + for _, e := range entries { + if err := os.Chtimes(filepath.Join(dir, e.Name()), old, old); err != nil { + t.Fatalf("age cache: %v", err) + } + } +} + +func publishRecord(t *testing.T, st *Store, id string, meta []byte, content string) { + t.Helper() + staging := stageRecord(t, st, id, meta, content) + if err := st.Publish(t.Context(), staging, id, ""); err != nil { + t.Fatalf("publish record: %v", err) + } +} + +func stageRecord(t *testing.T, st *Store, id string, meta []byte, content string) string { + t.Helper() + return storetest.StageRecord(t, st, id, string(meta), "disk.img", content) +} + +func stageSparseRecord(t *testing.T, st *Store, id string, meta []byte, size int64, writes map[int64][]byte) (staging, path string) { + t.Helper() + staging = stageRecord(t, st, id, meta, "") + return staging, storetest.WriteSparse(t, filepath.Join(staging, store.ExportDir, "disk.img"), size, writes) +} + +func fileSum(t *testing.T, path string) [sha256.Size]byte { + t.Helper() + f, err := os.Open(path) + if err != nil { + t.Fatalf("open %s: %v", path, err) + } + defer func() { _ = f.Close() }() + h := sha256.New() + if _, err = io.Copy(h, f); err != nil { + t.Fatalf("hash %s: %v", path, err) + } + return [sha256.Size]byte(h.Sum(nil)) +} + +func allocatedBytes(t *testing.T, path string) int64 { + t.Helper() + info, err := os.Stat(path) + if err != nil { + t.Fatalf("stat %s: %v", path, err) + } + return info.Sys().(*syscall.Stat_t).Blocks * 512 +} + +func denseChunkSums(t *testing.T, path string) [][sha256.Size]byte { + t.Helper() + f, err := os.Open(path) + if err != nil { + t.Fatalf("open %s: %v", path, err) + } + defer func() { _ = f.Close() }() + var sums [][sha256.Size]byte + buf := make([]byte, store.DigestChunkSize) + for { + n, err := io.ReadFull(f, buf) + if n > 0 { + sums = append(sums, sha256.Sum256(buf[:n])) + } + if err != nil { + return sums + } + } +} + +func newFakeStore(t *testing.T) (*fakeS3, *Store) { + t.Helper() + fake := &fakeS3{objects: map[string][]byte{}} + return fake, newTestStore(t, fake) +} + +func newTestStore(t *testing.T, fake *fakeS3) *Store { + t.Helper() + return newTestStoreAt(t, fake, false) +} + +func metaJSON(id string) []byte { + return []byte(storetest.MetaJSON(id)) +} + +func newGatedStore(t *testing.T, gateOn string) (*fakeS3, *Store, func()) { + t.Helper() + fake := &fakeS3{objects: map[string][]byte{}, exportGate: make(chan struct{}), exportEntered: make(chan struct{}, 1), gateOn: gateOn} + st := newTestStore(t, fake) + release := sync.OnceFunc(func() { close(fake.exportGate) }) + t.Cleanup(release) + return fake, st, release +} + +func newTestStoreAt(t *testing.T, endpoint http.Handler, unsignedAcceptEncoding bool) *Store { + t.Helper() + ts := httptest.NewServer(endpoint) + t.Cleanup(ts.Close) + t.Setenv("AWS_ACCESS_KEY_ID", "test") + t.Setenv("AWS_SECRET_ACCESS_KEY", "test") + + t.Setenv("AWS_REQUEST_CHECKSUM_CALCULATION", "when_required") + t.Setenv("AWS_RESPONSE_CHECKSUM_VALIDATION", "when_required") + st, err := New(t.Context(), store.S3Config{ + Bucket: "testbucket", Prefix: "ck/", Endpoint: ts.URL, + Region: "us-east-1", ForcePathStyle: true, UnsignedAcceptEncoding: unsignedAcceptEncoding, + }, t.TempDir(), store.CheckpointIDRe) + if err != nil { + t.Fatalf("New: %v", err) + } + return st +} + +func assertSingleMetaGet(t *testing.T, fake *fakeS3, metaKey string) { + t.Helper() + if got := fake.requestCount(http.MethodGet, metaKey); got != 1 { + t.Errorf("meta GET requests = %d, want 1", got) + } + if got := fake.requestCount(http.MethodHead, metaKey); got != 0 { + t.Errorf("meta HEAD requests = %d, want 0", got) + } } func requestMetadata(header http.Header) map[string]string { @@ -1360,9 +1295,3 @@ func writeMetadata(header http.Header, metadata map[string]string) { header.Set("x-amz-meta-"+key, value) } } - -func writeFakeError(w http.ResponseWriter, status int, code, message string) { - w.Header().Set("Content-Type", "application/xml") - w.WriteHeader(status) - _, _ = fmt.Fprintf(w, `%s%s`, code, message) -} diff --git a/sandboxd/store/sparse_test.go b/sandboxd/store/sparse_test.go index d2f5ac5c3..c9228c7b8 100644 --- a/sandboxd/store/sparse_test.go +++ b/sandboxd/store/sparse_test.go @@ -15,16 +15,12 @@ import ( func TestScanFileEmitsEachNonZeroChunkOnce(t *testing.T) { size := 4*DigestChunkSize + 5 path := filepath.Join(t.TempDir(), "disk.img") - f, err := os.Create(path) - if err != nil { - t.Fatalf("create: %v", err) - } - defer func() { _ = f.Close() }() - if err = f.Truncate(size); err != nil { + f := createFile(t, path) + if err := f.Truncate(size); err != nil { t.Fatalf("truncate: %v", err) } for off, data := range map[int64]string{DigestChunkSize - 2: "abcd", DigestChunkSize + 10: "zz", DigestChunkSize + 1<<20: "yy", 4*DigestChunkSize + 1: "t"} { - if _, err = f.WriteAt([]byte(data), off); err != nil { + if _, err := f.WriteAt([]byte(data), off); err != nil { t.Fatalf("write at %d: %v", off, err) } } @@ -56,12 +52,8 @@ func TestScanFileEmitsEachNonZeroChunkOnce(t *testing.T) { func TestScanFileDoesNotReadHoles(t *testing.T) { size := 6 * DigestChunkSize - f, err := os.Create(filepath.Join(t.TempDir(), "holes.img")) - if err != nil { - t.Fatalf("create: %v", err) - } - defer func() { _ = f.Close() }() - if err = f.Truncate(size); err != nil { + f := createFile(t, filepath.Join(t.TempDir(), "holes.img")) + if err := f.Truncate(size); err != nil { t.Fatalf("truncate: %v", err) } reads := 0 @@ -75,8 +67,13 @@ func TestScanFileDoesNotReadHoles(t *testing.T) { if reads != 0 { t.Errorf("a hole-only file asked for %d read buffers, want 0", reads) } - if len(digest.Chunks) != 6 || digest.Chunks[0] != zeroSum(DigestChunkSize) { - t.Errorf("hole chunks do not carry the zero-chunk sum: %d chunks", len(digest.Chunks)) + if len(digest.Chunks) != 6 { + t.Fatalf("%d chunks, want 6", len(digest.Chunks)) + } + for i, sum := range digest.Chunks { + if sum != zeroSum(DigestChunkSize) { + t.Errorf("hole chunk %d does not carry the zero-chunk sum", i) + } } } @@ -84,7 +81,7 @@ func TestWriteChunkRejectsWrongLengthAndChecksum(t *testing.T) { data := "payload" sum := sha256.Sum256([]byte(data)) badSum := "0000000000000000000000000000000000000000000000000000000000000000" - for _, tc := range []struct { + for _, tt := range []struct { name string size int64 index int64 @@ -97,24 +94,26 @@ func TestWriteChunkRejectsWrongLengthAndChecksum(t *testing.T) { {"wrong sum", int64(len(data)), 0, badSum, data, "checksum"}, {"index outside file", int64(len(data)), 1, hex.EncodeToString(sum[:]), data, "outside"}, } { - t.Run(tc.name, func(t *testing.T) { - f, err := os.Create(filepath.Join(t.TempDir(), "chunk")) - if err != nil { - t.Fatalf("create: %v", err) - } - defer func() { _ = f.Close() }() - err = WriteChunk(f, tc.size, tc.index, tc.sha, strings.NewReader(tc.body)) - if err == nil || !strings.Contains(err.Error(), tc.want) { - t.Errorf("WriteChunk = %v, want an error containing %q", err, tc.want) + t.Run(tt.name, func(t *testing.T) { + f := createFile(t, filepath.Join(t.TempDir(), "chunk")) + err := WriteChunk(f, tt.size, tt.index, tt.sha, strings.NewReader(tt.body)) + if err == nil || !strings.Contains(err.Error(), tt.want) { + t.Errorf("WriteChunk = %v, want an error containing %q", err, tt.want) } }) } - f, err := os.Create(filepath.Join(t.TempDir(), "ok")) + f := createFile(t, filepath.Join(t.TempDir(), "ok")) + if err := WriteChunk(f, int64(len(data)), 0, hex.EncodeToString(sum[:]), strings.NewReader(data)); err != nil { + t.Errorf("WriteChunk of a matching chunk: %v", err) + } +} + +func createFile(t *testing.T, path string) *os.File { + t.Helper() + f, err := os.Create(path) if err != nil { t.Fatalf("create: %v", err) } - defer func() { _ = f.Close() }() - if err = WriteChunk(f, int64(len(data)), 0, hex.EncodeToString(sum[:]), strings.NewReader(data)); err != nil { - t.Errorf("WriteChunk of a matching chunk: %v", err) - } + t.Cleanup(func() { _ = f.Close() }) + return f } diff --git a/sandboxd/store/store.go b/sandboxd/store/store.go index f1763ff7f..fec7e0555 100644 --- a/sandboxd/store/store.go +++ b/sandboxd/store/store.go @@ -56,6 +56,7 @@ type S3Config struct { UnsignedAcceptEncoding bool `json:"unsigned_accept_encoding,omitzero"` } +// CommitFunc makes a prepared generation the record's current one. type CommitFunc func(ctx context.Context) error // Store is one record backend. @@ -72,6 +73,7 @@ type Store interface { FetchFile(ctx context.Context, id, rel, dst string) error // ReadMeta returns a record's metadata, or an error when the record does not exist. ReadMeta(ctx context.Context, id string) ([]byte, error) + // ReadDigest returns a record's export digest, empty when it was published undigested. ReadDigest(ctx context.Context, id string) (string, error) // Metas lists the metadata and digest of every record in this instance's id namespace. Metas(ctx context.Context) ([]Record, error) diff --git a/sandboxd/store/storetest/storetest.go b/sandboxd/store/storetest/storetest.go index 3751bddf1..121e13110 100644 --- a/sandboxd/store/storetest/storetest.go +++ b/sandboxd/store/storetest/storetest.go @@ -1,6 +1,4 @@ -// Package storetest is the backend contract: every store.Store -// implementation must pass RunContract, so the pool sees identical -// semantics regardless of what sits underneath. +// Package storetest is the backend contract every store.Store implementation must pass. package storetest import ( @@ -20,10 +18,10 @@ func RunContract(t *testing.T, st store.Store) { ctx := t.Context() const id = "ck_00000000000000aa" - publishRecord(t, st, id, `{"id":"`+id+`"}`, "disk.img", "snapshot-bytes") + publishRecord(t, st, id, MetaJSON(id), "disk.img", "snapshot-bytes") raw, err := st.ReadMeta(ctx, id) - if err != nil || string(raw) != `{"id":"`+id+`"}` { + if err != nil || string(raw) != MetaJSON(id) { t.Fatalf("ReadMeta: %q, %v", raw, err) } @@ -31,7 +29,7 @@ func RunContract(t *testing.T, st store.Store) { if err != nil { t.Fatalf("Fetch: %v", err) } - if string(meta) != `{"id":"`+id+`"}` { + if string(meta) != MetaJSON(id) { t.Fatalf("Fetch meta: %q, want the published record", meta) } if digest != "" { @@ -52,7 +50,7 @@ func RunContract(t *testing.T, st store.Store) { if err != nil { t.Fatalf("Metas: %v", err) } - if !slices.ContainsFunc(metas, func(r store.Record) bool { return string(r.Meta) == `{"id":"`+id+`"}` }) || len(metas) != 1 { + if !slices.ContainsFunc(metas, func(r store.Record) bool { return string(r.Meta) == MetaJSON(id) }) || len(metas) != 1 { t.Fatalf("Metas: %d records %q, want exactly the published one", len(metas), metas) } @@ -99,6 +97,7 @@ func RunContract(t *testing.T, st store.Store) { runLabelsContract(t, st) } +// PublishDigested prepares and commits a digested record in one call. func PublishDigested(ctx context.Context, st store.Store, staging, id string) (string, error) { digest, commit, err := st.PrepareDigested(ctx, staging, id, "") if err != nil { @@ -110,6 +109,44 @@ func PublishDigested(ctx context.Context, st store.Store, staging, id string) (s return digest, nil } +// MetaJSON is the bare record meta the contract and its callers publish. +func MetaJSON(id string) string { + return `{"id":"` + id + `"}` +} + +// StageRecord stages one export file and its meta, ready to publish. +func StageRecord(t *testing.T, st store.Store, id, meta, file, content string) string { + t.Helper() + staging, err := st.Stage(id) + if err != nil { + t.Fatalf("Stage %s: %v", id, err) + } + writeExport(t, staging, file, content) + if err = os.WriteFile(filepath.Join(staging, store.MetaFile), []byte(meta), 0o600); err != nil { + t.Fatalf("write meta: %v", err) + } + return staging +} + +// WriteSparse creates path at size with only writes allocated. +func WriteSparse(t testing.TB, path string, size int64, writes map[int64][]byte) string { + t.Helper() + f, err := os.Create(path) //nolint:gosec // test path + if err != nil { + t.Fatalf("create %s: %v", path, err) + } + defer func() { _ = f.Close() }() + if err = f.Truncate(size); err != nil { + t.Fatalf("size %s: %v", path, err) + } + for off, data := range writes { + if _, err = f.WriteAt(data, off); err != nil { + t.Fatalf("write %s: %v", path, err) + } + } + return path +} + func runFetchFileContract(t *testing.T, st store.Store, id string) { t.Helper() ctx := t.Context() @@ -141,7 +178,7 @@ func runDigestContract(t *testing.T, st store.Store) { } writeExport(t, staging, "z.bin", "z") writeExport(t, staging, "nested/a.bin", "a") - if err = os.WriteFile(filepath.Join(staging, store.MetaFile), []byte(`{"id":"`+id+`"}`), 0o600); err != nil { + if err = os.WriteFile(filepath.Join(staging, store.MetaFile), []byte(MetaJSON(id)), 0o600); err != nil { t.Fatalf("write digested meta: %v", err) } digest, commit, err := st.PrepareDigested(ctx, staging, id, "") @@ -200,7 +237,7 @@ func runLabelsContract(t *testing.T, st store.Store) { t.Helper() ctx := t.Context() const id = "ck_00000000000000dd" - publishRecord(t, st, id, `{"id":"`+id+`"}`, "disk.img", "labeled") + publishRecord(t, st, id, MetaJSON(id), "disk.img", "labeled") labelsOf := func() []byte { t.Helper() recs, metasErr := st.Metas(ctx) @@ -271,7 +308,7 @@ func rejectNonRegularReplacement(t *testing.T, st store.Store, id, wantDigest st if err != nil { t.Fatalf("Fetch after rejected replacement: %v", err) } - if string(meta) != `{"id":"`+id+`"}` || digest != wantDigest { + if string(meta) != MetaJSON(id) || digest != wantDigest { t.Errorf("committed meta/digest after rejection = %q/%q, want original/%q", meta, digest, wantDigest) } content, err := os.ReadFile(filepath.Join(dir, "nested", "a.bin")) //nolint:gosec // test path @@ -282,15 +319,8 @@ func rejectNonRegularReplacement(t *testing.T, st store.Store, id, wantDigest st func publishRecord(t *testing.T, st store.Store, id, meta, file, content string) { t.Helper() - staging, err := st.Stage(id) - if err != nil { - t.Fatalf("Stage %s: %v", id, err) - } - writeExport(t, staging, file, content) - if err = os.WriteFile(filepath.Join(staging, store.MetaFile), []byte(meta), 0o600); err != nil { - t.Fatalf("write meta: %v", err) - } - if err = st.Publish(t.Context(), staging, id, ""); err != nil { + staging := StageRecord(t, st, id, meta, file, content) + if err := st.Publish(t.Context(), staging, id, ""); err != nil { t.Fatalf("Publish %s: %v", id, err) } } diff --git a/sandboxd/types/env_test.go b/sandboxd/types/env_test.go index edf76f624..89a561943 100644 --- a/sandboxd/types/env_test.go +++ b/sandboxd/types/env_test.go @@ -37,7 +37,7 @@ func TestEnvValidate(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { err := tt.env.Validate() - if (tt.want == "") != (err == nil) || (err != nil && !strings.Contains(err.Error(), tt.want)) { + if !errMatches(err, tt.want) { t.Errorf("Validate() = %v, want %q", err, tt.want) } if err != nil && strings.Contains(err.Error(), "B=b") { @@ -64,7 +64,7 @@ func TestEnvPatchAppliesAndValidates(t *testing.T) { {EnvPatch{"A-B": nil}, "env name"}, {EnvPatch{"A": {Value: "a\nB=b"}}, "control characters"}, } { - if err := tt.patch.Validate(); (tt.want == "") != (err == nil) || (err != nil && !strings.Contains(err.Error(), tt.want)) { + if err := tt.patch.Validate(); !errMatches(err, tt.want) { t.Errorf("Validate(%v) = %v, want %q", tt.patch, err, tt.want) } } @@ -133,7 +133,7 @@ func TestEnvInjectValidates(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { err := tt.env.Validate() - if (tt.want == "") != (err == nil) || (err != nil && !strings.Contains(err.Error(), tt.want)) { + if !errMatches(err, tt.want) { t.Errorf("Validate = %v, want %q", err, tt.want) } }) @@ -216,3 +216,10 @@ func TestEnvSplitsGuestAndHostOnlyEntries(t *testing.T) { t.Errorf("GuestFile = %q", got) } } + +func errMatches(err error, want string) bool { + if want == "" { + return err == nil + } + return err != nil && strings.Contains(err.Error(), want) +} diff --git a/sandboxd/types/types.go b/sandboxd/types/types.go index 7e6459b1b..035fda8ea 100644 --- a/sandboxd/types/types.go +++ b/sandboxd/types/types.go @@ -805,6 +805,7 @@ func VolumeRWNames(volumes []Volume) []string { return names } +// ValidateClaimRef bounds a caller reference named field like a metadata value. func ValidateClaimRef(field, ref string) error { if len(ref) > maxMetadataValueBytes || !validMetadataValue(ref) { return fmt.Errorf("%s must be at most %d bytes of printable UTF-8", field, maxMetadataValueBytes) diff --git a/sdk/go/proc_test.go b/sdk/go/proc_test.go index 6688dcad3..ac0d87d53 100644 --- a/sdk/go/proc_test.go +++ b/sdk/go/proc_test.go @@ -75,7 +75,7 @@ func TestProcVerbsUnknownPid(t *testing.T) { } } -func procServe(script map[string][]string) func(net.Conn) { +func procServe(script map[string][]string) connHandler { return func(conn net.Conn) { defer func() { _ = conn.Close() }() line, err := bufio.NewReader(conn).ReadString('\n') diff --git a/sdk/go/sandbox_test.go b/sdk/go/sandbox_test.go index d5b596d12..2521d2719 100644 --- a/sdk/go/sandbox_test.go +++ b/sdk/go/sandbox_test.go @@ -69,7 +69,7 @@ func TestRunStreamsStdin(t *testing.T) { } } -func TestRunContextCancel(t *testing.T) { +func TestRunReturnsAtItsContextDeadline(t *testing.T) { sb := testSandbox(t, newAgentServer(t, silkdtest.ServeConn)) ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond) @@ -213,8 +213,8 @@ func TestDialAgentRejectsControlChars(t *testing.T) { {"sb\r\nInjected: 1", "tok"}, {"sb_1", "tok\r\nInjected: 1"}, } { - if _, err := c.dialAgent(t.Context(), c.addr, bad.id, bad.token); err == nil { - t.Errorf("dialAgent(%q, %q) succeeded, want control-character error", bad.id, bad.token) + if _, err := c.dialAgent(t.Context(), c.addr, bad.id, bad.token); err == nil || !strings.Contains(err.Error(), "control character") { + t.Errorf("dialAgent(%q, %q) = %v, want control-character error", bad.id, bad.token, err) } } } @@ -277,7 +277,16 @@ func TestRenewRefusalLeavesTheDeadline(t *testing.T) { } } -func newAgentServer(t testing.TB, serve func(net.Conn)) *httptest.Server { +type connHandler func(net.Conn) + +type onWrite func() + +func (f onWrite) Write(p []byte) (int, error) { + f() + return len(p), nil +} + +func newAgentServer(t testing.TB, serve connHandler) *httptest.Server { t.Helper() mux := http.NewServeMux() mux.HandleFunc("GET /v1/sandboxes/{id}/agent", func(w http.ResponseWriter, r *http.Request) { @@ -311,10 +320,3 @@ func legacySandbox(t *testing.T, ts *httptest.Server) *Sandbox { sb.proto.Store(1) return sb } - -type onWrite func() - -func (f onWrite) Write(p []byte) (int, error) { - f() - return len(p), nil -} diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index f796c33fc..2379ae0d9 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -2,7 +2,7 @@ import threading import time -from http.server import HTTPServer, ThreadingHTTPServer +from http.server import ThreadingHTTPServer import pytest from conftest import FakeNode, sandbox_at, wait_until @@ -31,16 +31,12 @@ def _reply(self, code, payload): @pytest.fixture -def node(): - server = HTTPServer(("127.0.0.1", 0), FakeNode) +def node(spawn_node): FakeNode.routes = {} - thread = threading.Thread(target=server.serve_forever, daemon=True) - thread.start() - yield f"127.0.0.1:{server.server_port}" - server.shutdown() + return spawn_node(FakeNode.routes) -def test_claim_happy_path(node): +def test_claim_returns_the_sandbox_with_its_owner_and_digest(node): FakeNode.routes[("POST", "/v1/claim")] = lambda body, path: ( 200, {"id": "sb_1", "token": "tok", "owner_addr": node, "template_digest": "sha256:task"}, @@ -351,8 +347,11 @@ def test_claim_deadline_bounds_the_redirect_walk(node, black_hole): def test_claim_deadline_bounds_the_entry_node(black_hole): client = Client(black_hole, timeout=2.0) + started = time.monotonic() with pytest.raises(TimeoutError): - client.new("rt:24.04", deadline=time.monotonic() + 0.3) + client.new("rt:24.04", deadline=started + 0.3) + elapsed = time.monotonic() - started + assert elapsed < 1.5, elapsed def recording_claim(reply): diff --git a/sdk/python/tests/test_fixtures.py b/sdk/python/tests/test_fixtures.py index 2d30b607e..f34cb9c69 100644 --- a/sdk/python/tests/test_fixtures.py +++ b/sdk/python/tests/test_fixtures.py @@ -1,7 +1,4 @@ -"""Round-trips the shared protocol fixture corpus, the same drift guard the -Rust and Go implementations run: every request fixture must re-encode -byte-for-byte (up to key order) through frames.encode_request, and every -response fixture must decode.""" +"""Round-trips the protocol fixture corpus, the same drift guard the Rust and Go implementations run.""" import base64 import json diff --git a/sdk/python/tests/test_hardening.py b/sdk/python/tests/test_hardening.py index 0b3b017cb..2686aa89e 100644 --- a/sdk/python/tests/test_hardening.py +++ b/sdk/python/tests/test_hardening.py @@ -56,21 +56,21 @@ def test_send_on_a_dead_socket_raises_a_typed_error(): conn.close() -def test_watcher_propagates_silkd_error(): - client_sock, guest_sock = socket.socketpair() +def test_watcher_propagates_silkd_error(conn_pair): + conn, guest_sock = conn_pair guest_sock.sendall(json.dumps({"type": "error", "kind": "not_found", "message": "gone"}).encode() + b"\n") guest_sock.close() - watcher = Watcher(Conn(client_sock, client_sock.makefile("rb"))) + watcher = Watcher(conn) with pytest.raises(SilkdError): for _ in watcher: pass -def test_watcher_ends_cleanly_on_garbage_frame(): - client_sock, guest_sock = socket.socketpair() +def test_watcher_ends_cleanly_on_garbage_frame(conn_pair): + conn, guest_sock = conn_pair guest_sock.sendall(b"not json at all\n") guest_sock.close() - watcher = Watcher(Conn(client_sock, client_sock.makefile("rb"))) + watcher = Watcher(conn) assert list(watcher) == [] @@ -103,19 +103,19 @@ def test_empty_claim_reply_is_an_api_error(spawn_node): Client(addr).new("rt:24.04") -def test_pty_read_returns_empty_after_the_exit_frame(): - client_sock, guest_sock = socket.socketpair() +def test_pty_read_returns_empty_after_the_exit_frame(conn_pair): + conn, guest_sock = conn_pair guest_sock.sendall(json.dumps({"type": "exit", "code": 3}).encode() + b"\n") guest_sock.close() - pty = Pty(sandbox_at("127.0.0.1:1"), Conn(client_sock, client_sock.makefile("rb")), 1) + pty = Pty(sandbox_at("127.0.0.1:1"), conn, 1) assert pty.read() == b"" assert pty.exit_code == 3 assert pty.read() == b"" -def test_pty_write_chunks_a_large_buffer(): - client_sock, guest_sock = socket.socketpair() - pty = Pty(sandbox_at("127.0.0.1:1"), Conn(client_sock, client_sock.makefile("rb")), 1) +def test_pty_write_chunks_a_large_buffer(conn_pair): + conn, guest_sock = conn_pair + pty = Pty(sandbox_at("127.0.0.1:1"), conn, 1) frames = [] def drain(): @@ -153,10 +153,10 @@ def broken(op, **fields): Session(sb, "s1").close() -def test_watcher_error_stays_none_after_close(): - client_sock, guest_sock = socket.socketpair() +def test_watcher_error_stays_none_after_close(conn_pair): + conn, guest_sock = conn_pair guest_sock.sendall(json.dumps({"type": "ready"}).encode() + b"\n") - watcher = Watcher(Conn(client_sock, client_sock.makefile("rb"))) + watcher = Watcher(conn) events = [] consumer = threading.Thread(target=lambda: events.extend(watcher), daemon=True) consumer.start() @@ -202,10 +202,18 @@ def slow_headers() -> None: threading.Thread(target=slow_headers, daemon=True).start() addr = f"127.0.0.1:{server.getsockname()[1]}" started = time.monotonic() - with pytest.raises(SandboxTimeout), dial_agent(addr, "sb_x", "tok", timeout=0.2) as conn: + with server, pytest.raises(SandboxTimeout), dial_agent(addr, "sb_x", "tok", timeout=0.2) as conn: conn.send("info") conn.recv() assert time.monotonic() - started < 0.6, ( "each header arrived under the per-read timeout; only a whole-dial budget stops this" ) - server.close() + + +@pytest.fixture +def conn_pair(): + client_sock, guest_sock = socket.socketpair() + conn = Conn(client_sock, client_sock.makefile("rb")) + yield conn, guest_sock + conn.close() + guest_sock.close() diff --git a/sdk/python/tests/test_keepalive.py b/sdk/python/tests/test_keepalive.py index 3b39e81a2..dffbf9cd8 100644 --- a/sdk/python/tests/test_keepalive.py +++ b/sdk/python/tests/test_keepalive.py @@ -74,87 +74,55 @@ def _answer(self, op: str) -> list: return [{"type": "error", "kind": "unimplemented", "message": op}] -def test_calls_share_one_connection(fd_floor): - agent = FakeAgent() - sb = sandbox_at(agent.addr) - try: - for _ in range(3): - assert sb.stat("/")["kind"] == "dir" - assert sb.exec("echo", "42") == "42\n" - assert sb._pool._parked[0][1]._sock.fileno() >= fd_floor - finally: - sb._pool.drain() - agent.stop() +def test_calls_share_one_connection(fd_floor, relay): + agent, sb = relay() + for _ in range(3): + assert sb.stat("/")["kind"] == "dir" + assert sb.exec("echo", "42") == "42\n" + assert sb._pool._parked[0][1]._sock.fileno() >= fd_floor assert agent.upgrades == 1, agent.upgrades -def test_a_refused_stream_open_keeps_its_connection(): - agent = FakeAgent() - sb = sandbox_at(agent.addr) - try: - for _ in range(3): - with pytest.raises(SilkdError): - sb.watch("/missing") - finally: - sb._pool.drain() - agent.stop() +def test_a_refused_stream_open_keeps_its_connection(relay): + agent, sb = relay() + for _ in range(3): + with pytest.raises(SilkdError): + sb.watch("/missing") assert agent.upgrades == 1 -def test_old_daemon_dials_per_call(): - agent = FakeAgent(proto=1) - sb = sandbox_at(agent.addr) - try: - for _ in range(3): - assert sb.exec("echo", "42") == "42\n" - finally: - agent.stop() +def test_old_daemon_dials_per_call(relay): + agent, sb = relay(proto=1) + for _ in range(3): + assert sb.exec("echo", "42") == "42\n" assert agent.upgrades == 4, "the proto probe plus one dial per call" -def test_old_daemon_that_closes_after_info_gets_the_request_on_a_redial(): - agent = FakeAgent(proto=1, slam=True) - sb = sandbox_at(agent.addr) - try: - for _ in range(2): - assert sb.stat("/")["kind"] == "dir" - finally: - agent.stop() +def test_old_daemon_that_closes_after_info_gets_the_request_on_a_redial(relay): + agent, sb = relay(proto=1, slam=True) + for _ in range(2): + assert sb.stat("/")["kind"] == "dir" assert agent.upgrades == 3, "the proto probe plus one dial per call" -def test_keep_alive_off_dials_per_call(): - agent = FakeAgent() - sb = sandbox_at(agent.addr, keep_alive=0) - try: - for _ in range(3): - sb.stat("/") - finally: - agent.stop() +def test_keep_alive_off_dials_per_call(relay): + agent, sb = relay(keep_alive=0) + for _ in range(3): + sb.stat("/") assert agent.upgrades == 3 -def test_idle_connection_closes(): - agent = FakeAgent() - sb = sandbox_at(agent.addr, keep_alive=0.05) - try: - sb.stat("/") - wait_until(lambda: agent.closed == 1, "idle connection still open after the keep-alive window") - finally: - sb._pool.drain() - agent.stop() +def test_idle_connection_closes(relay): + agent, sb = relay(keep_alive=0.05) + sb.stat("/") + wait_until(lambda: agent.closed == 1, "idle connection still open after the keep-alive window") -def test_peer_hang_up_is_noticed_before_reuse(fd_floor): - agent = FakeAgent(hang_up_after=2) - sb = sandbox_at(agent.addr) - try: - sb.stat("/") - wait_until(lambda: agent.hangups == 1, "the fake never hung up") - assert sb.stat("/")["kind"] == "dir" - finally: - sb._pool.drain() - agent.stop() +def test_peer_hang_up_is_noticed_before_reuse(fd_floor, relay): + agent, sb = relay(hang_up_after=2) + sb.stat("/") + wait_until(lambda: agent.hangups == 1, "the fake never hung up") + assert sb.stat("/")["kind"] == "dir" assert agent.upgrades == 2 @@ -181,23 +149,31 @@ def serve(): server.close() -def test_close_drains_the_parked_connection(monkeypatch): - agent = FakeAgent() - sb = sandbox_at(agent.addr) +def test_close_drains_the_parked_connection(monkeypatch, relay): + agent, sb = relay() monkeypatch.setattr(sb._client, "_request", lambda *args, **kwargs: {}) - try: - sb.stat("/") - sb.close() - wait_until(lambda: agent.closed == 1, "parked connection survived close") - finally: - agent.stop() + sb.stat("/") + sb.close() + wait_until(lambda: agent.closed == 1, "parked connection survived close") -def test_an_upload_an_old_daemon_rejects_raises_its_error_frame(): - agent = FakeAgent(proto=1, slam=True) - sb = sandbox_at(agent.addr) - try: - with pytest.raises(SilkdError): - sb.write_file("/missing/f", bytes(64 << 20)) - finally: +def test_an_upload_an_old_daemon_rejects_raises_its_error_frame(relay): + _, sb = relay(proto=1, slam=True) + with pytest.raises(SilkdError): + sb.write_file("/missing/f", bytes(64 << 20)) + + +@pytest.fixture +def relay(): + started = [] + + def start(proto=KEEP_ALIVE_PROTO, hang_up_after=0, slam=False, **client_kwargs): + agent = FakeAgent(proto, hang_up_after, slam) + sb = sandbox_at(agent.addr, **client_kwargs) + started.append((agent, sb)) + return agent, sb + + yield start + for agent, sb in started: + sb._pool.drain() agent.stop() diff --git a/sdk/python/tests/test_proc.py b/sdk/python/tests/test_proc.py index e3418488a..53585f0f3 100644 --- a/sdk/python/tests/test_proc.py +++ b/sdk/python/tests/test_proc.py @@ -1,5 +1,4 @@ -"""Process-management verbs against a scripted fake conn: pure frame -plumbing, so the fake pins the framing and terminal semantics.""" +"""Process-management verbs against a scripted fake conn, which pins the framing and terminal semantics.""" import threading @@ -69,11 +68,11 @@ def test_run_pumps_stdin_while_reading_output(monkeypatch): ) def test_a_text_stdin_is_refused_before_dialing(monkeypatch, call): sb = sandbox_at("127.0.0.1:1", keep_alive=0) - conn = StdinGatedConn([{"type": "exit", "code": 0}]) - monkeypatch.setattr(sb, "_dial", lambda deadline=None: conn) + dials = [] + monkeypatch.setattr(sb, "_dial", lambda deadline=None: dials.append(deadline)) with pytest.raises(TypeError, match="stdin must be bytes"): call(sb) - assert conn.sent == [] + assert dials == [] class FakeConn: @@ -107,8 +106,7 @@ def close(self): class BlockingStdinConn(FakeConn): - """A guest that stops draining stdin until its output is read: send blocks - past the buffer, exactly as the real socket pair does.""" + """A guest that stops draining stdin until its output is read, so send blocks past the buffer.""" def __init__(self, frames, buffer_frames): super().__init__(frames) @@ -126,23 +124,6 @@ def recv(self): return super().recv() -class StdinGatedConn(FakeConn): - """A guest that answers only after stdin closes, as cat does.""" - - def __init__(self, frames): - super().__init__(frames) - self._closed = threading.Event() - - def send(self, op, **fields): - super().send(op, **fields) - if op == "stdin_close": - self._closed.set() - - def recv(self): - assert self._closed.wait(2), "stdin_close never sent" - return super().recv() - - def fake_sandbox(monkeypatch, frames): sb = sandbox_at("127.0.0.1:1", keep_alive=0) conn = FakeConn(frames) diff --git a/silkd/src/find.rs b/silkd/src/find.rs index d991c7270..b0c596013 100644 --- a/silkd/src/find.rs +++ b/silkd/src/find.rs @@ -21,7 +21,7 @@ const MATCH_QUEUE: usize = 256; /// Bytes of match content in flight: above one size-bound match, so a reserve never blocks forever, far below 256 of them. const MATCH_QUEUE_BYTES: usize = 2 * FIND_MAX_FILE as usize; -const LINE_ANCHORS: [&str; 4] = ["^", "$", "\\A", "\\z"]; +const ANCHORS: [&str; 4] = ["^", "$", "\\A", "\\z"]; struct MatchBudget { bytes: Semaphore, @@ -228,7 +228,7 @@ where Some(Ok(re)) => Some(re), Some(Err(e)) => return proto::error_frame(w, ErrorKind::BadRequest, e.to_string()).await, }; - let prefilter = !LINE_ANCHORS.iter().any(|a| pattern.contains(a)); + let prefilter = !ANCHORS.iter().any(|a| pattern.contains(a)); // one blocking-pool dispatch for the whole tree, not three per file. let (tx, mut rx) = mpsc::channel::(MATCH_QUEUE); let budget = Arc::new(MatchBudget::new(budget_bytes, Handle::current())); diff --git a/silkd/src/proto.rs b/silkd/src/proto.rs index 981ad5c04..bd2e9ca29 100644 --- a/silkd/src/proto.rs +++ b/silkd/src/proto.rs @@ -567,25 +567,21 @@ pub async fn write_feed_error(w: &mut W, err: FeedError) #[inline] pub(crate) fn parse_request(frame: &[u8], buf: &mut Vec) -> serde_json::Result { - if let Some(n) = decode_bulk_frame(frame, DATA_FRAME_HEAD, buf) { - buf.truncate(n); - return Ok(Request::Data { - data: std::mem::take(buf), - }); + if let Some(data) = decode_bulk_frame(frame, DATA_FRAME_HEAD, buf) { + return Ok(Request::Data { data }); } - if let Some(n) = decode_bulk_frame(frame, STDIN_FRAME_HEAD, buf) { - buf.truncate(n); - return Ok(Request::Stdin { - data: std::mem::take(buf), - }); + if let Some(data) = decode_bulk_frame(frame, STDIN_FRAME_HEAD, buf) { + return Ok(Request::Stdin { data }); } serde_json::from_slice(frame) } -fn decode_bulk_frame(frame: &[u8], head: &[u8], out: &mut Vec) -> Option { +fn decode_bulk_frame(frame: &[u8], head: &[u8], out: &mut Vec) -> Option> { let b64 = frame.strip_prefix(head)?.strip_suffix(b"\"}")?; out.resize(base64::decoded_len_estimate(b64.len()), 0); - STANDARD.decode_slice(b64, out).ok() + let n = STANDARD.decode_slice(b64, out).ok()?; + out.truncate(n); + Some(std::mem::take(out)) } fn cap_check(len: usize) -> io::Result<()> { diff --git a/silkd/src/tree.rs b/silkd/src/tree.rs index cdb041d43..cfa34134d 100644 --- a/silkd/src/tree.rs +++ b/silkd/src/tree.rs @@ -116,7 +116,7 @@ where return proto::write_feed_error(w, fail).await; } if !status.success() { - return proto::subprocess_result(w, status.success(), "tar extract", &msg).await; + return proto::subprocess_result(w, false, "tar extract", &msg).await; } // spawn_blocking: a big tree is thousands of local fs syscalls. let merged = tokio::task::spawn_blocking(move || merge_tree(&staging, &dest)) diff --git a/silkd/tests/common/mod.rs b/silkd/tests/common/mod.rs index 40cb63341..f3566985f 100644 --- a/silkd/tests/common/mod.rs +++ b/silkd/tests/common/mod.rs @@ -75,6 +75,16 @@ async fn request_on(state: &Arc, lines: &[String]) -> Vec { frames } +pub async fn request(frame: Value) -> Vec { + roundtrip(&frame.to_string()).await +} + +#[track_caller] +pub fn assert_error(frame: &Value, kind: &str) { + assert_eq!(type_of(frame), "error", "{frame}"); + assert_eq!(frame["kind"], kind, "{frame}"); +} + pub async fn one(state: &Arc, request_line: &str) -> Vec { request_on(state, &[request_line.to_string()]).await } diff --git a/silkd/tests/exec_e2e.rs b/silkd/tests/exec_e2e.rs index 695798322..b9d560d95 100644 --- a/silkd/tests/exec_e2e.rs +++ b/silkd/tests/exec_e2e.rs @@ -11,7 +11,8 @@ use silkd::server::State; use tokio::io::{AsyncReadExt, AsyncWriteExt, BufReader}; use common::{ - b64, connect, exchange, frames_until, next_frame, one, roundtrip, send, stdout_body, type_of, + assert_error, b64, connect, exchange, frames_until, next_frame, one, roundtrip, send, + stdout_body, type_of, }; async fn detached_pid(state: &Arc, script: &str) -> u64 { @@ -89,8 +90,7 @@ async fn nonzero_exit_is_reported() { #[tokio::test] async fn empty_argv_is_a_bad_request() { let frames = roundtrip(r#"{"op":"exec","argv":[]}"#).await; - assert_eq!(type_of(&frames[0]), "error"); - assert_eq!(frames[0]["kind"], "bad_request"); + assert_error(&frames[0], "bad_request"); } #[tokio::test] @@ -104,8 +104,7 @@ async fn info_reports_version_and_proto() { #[tokio::test] async fn kill_unknown_pid_is_not_found() { let frames = roundtrip(r#"{"op":"kill","pid":999999}"#).await; - assert_eq!(type_of(&frames[0]), "error"); - assert_eq!(frames[0]["kind"], "not_found"); + assert_error(&frames[0], "not_found"); } #[tokio::test] @@ -212,8 +211,7 @@ async fn attach_keeps_a_tail_emitted_during_the_replay() { #[tokio::test] async fn attach_unknown_pid_is_not_found() { let frames = roundtrip(r#"{"op":"attach","pid":999999}"#).await; - assert_eq!(type_of(&frames[0]), "error"); - assert_eq!(frames[0]["kind"], "not_found"); + assert_error(&frames[0], "not_found"); } #[tokio::test] @@ -230,23 +228,20 @@ async fn kill_actually_terminates_a_live_process() { async fn exec_unknown_user_is_rejected() { let frames = roundtrip(r#"{"op":"exec","argv":["true"],"user":"definitely-not-a-user-xyz"}"#).await; - assert_eq!(type_of(&frames[0]), "error"); - assert_eq!(frames[0]["kind"], "bad_request"); + assert_error(&frames[0], "bad_request"); } #[tokio::test] async fn exec_missing_binary_is_a_bad_request() { let frames = roundtrip(r#"{"op":"exec","argv":["/no/such/binary-xyz"]}"#).await; - assert_eq!(type_of(&frames[0]), "error"); - assert_eq!(frames[0]["kind"], "bad_request"); + assert_error(&frames[0], "bad_request"); assert!(frames[0]["message"].as_str().unwrap().contains("spawn")); } #[tokio::test] async fn exec_missing_cwd_is_a_bad_request() { let frames = roundtrip(r#"{"op":"exec","argv":["true"],"cwd":"/no/such/dir-xyz"}"#).await; - assert_eq!(type_of(&frames[0]), "error"); - assert_eq!(frames[0]["kind"], "bad_request"); + assert_error(&frames[0], "bad_request"); } #[tokio::test] @@ -258,8 +253,7 @@ async fn exec_non_executable_file_is_a_bad_request() { &json!({"op":"exec","argv":[file.path()]}).to_string(), ) .await; - assert_eq!(type_of(&frames[0]), "error"); - assert_eq!(frames[0]["kind"], "bad_request"); + assert_error(&frames[0], "bad_request"); assert!(frames[0]["message"].as_str().unwrap().contains("spawn")); } diff --git a/silkd/tests/find_e2e.rs b/silkd/tests/find_e2e.rs index 9625ea3a7..66edc3bb3 100644 --- a/silkd/tests/find_e2e.rs +++ b/silkd/tests/find_e2e.rs @@ -10,7 +10,7 @@ use serde_json::{Value, json}; use silkd::server::State; use tokio::io::AsyncWriteExt; -use common::{connect, exchange, frames_until, send, type_of}; +use common::{assert_error, connect, frames_until, request, send, type_of}; async fn find_frames(req: Value) -> Vec { let (mut cw, mut out, handle) = connect(&Arc::new(State::new())); @@ -127,8 +127,7 @@ async fn find_rejects_bad_pattern() { "pattern": "(" })) .await; - assert_eq!(type_of(&frames[0]), "error"); - assert_eq!(frames[0]["kind"], "bad_request"); + assert_error(&frames[0], "bad_request"); } #[tokio::test] @@ -139,13 +138,12 @@ async fn replace_rewrites_and_counts() { tokio::fs::write(&a, "foo foo bar\n").await.unwrap(); tokio::fs::write(&b, "no match here\n").await.unwrap(); - let frames = exchange(&[json!({ + let frames = request(json!({ "op": "fs_replace", "files": [a.to_str().unwrap(), b.to_str().unwrap()], "pattern": "foo", "replacement": "baz" - }) - .to_string()]) + })) .await; let replaced: Vec<_> = frames.iter().filter(|f| type_of(f) == "replaced").collect(); @@ -170,13 +168,12 @@ async fn replace_skips_a_file_over_the_size_bound() { let body = format!("foo\n{}", "x".repeat(silkd::find::FIND_MAX_FILE as usize)); tokio::fs::write(&big, &body).await.unwrap(); - let frames = exchange(&[json!({ + let frames = request(json!({ "op": "fs_replace", "files": [big.to_str().unwrap()], "pattern": "foo", "replacement": "bar" - }) - .to_string()]) + })) .await; let replaced: Vec<_> = frames.iter().filter(|f| type_of(f) == "replaced").collect(); @@ -219,13 +216,12 @@ async fn replace_preserves_exec_bit() { .await .unwrap(); - exchange(&[json!({ + request(json!({ "op": "fs_replace", "files": [script.to_str().unwrap()], "pattern": "foo", "replacement": "bar" - }) - .to_string()]) + })) .await; let mode = tokio::fs::metadata(&script) @@ -249,13 +245,12 @@ async fn replace_keeps_the_destination_owner_under_root() { std::fs::write(&file, "foo\n").unwrap(); std::os::unix::fs::chown(&file, Some(1000), Some(1000)).unwrap(); - exchange(&[json!({ + request(json!({ "op": "fs_replace", "files": [file.to_str().unwrap()], "pattern": "foo", "replacement": "bar" - }) - .to_string()]) + })) .await; let meta = std::fs::metadata(&file).unwrap(); diff --git a/silkd/tests/fs_e2e.rs b/silkd/tests/fs_e2e.rs index fdee6b886..4987d3484 100644 --- a/silkd/tests/fs_e2e.rs +++ b/silkd/tests/fs_e2e.rs @@ -8,7 +8,7 @@ use std::os::unix::fs::{MetadataExt, PermissionsExt}; use nix::sys::stat::Mode; use serde_json::{Value, json}; -use common::{b64, data_frames, exchange, payload, type_of}; +use common::{assert_error, b64, data_frames, exchange, payload, request, type_of}; async fn fs_write(req: Value, body: &[u8]) -> Vec { let mut lines = vec![req.to_string()]; @@ -33,7 +33,7 @@ async fn write_then_read_roundtrips_bytes() { "write should succeed: {wrote:?}" ); - let read = exchange(&[json!({"op":"fs_read","path":path}).to_string()]).await; + let read = request(json!({"op":"fs_read","path":path})).await; assert_eq!(payload(&read, "data"), b"silk file body"); assert_eq!(type_of(read.last().unwrap()), "done"); } @@ -48,8 +48,7 @@ async fn list_reports_written_entry() { ) .await; - let listed = - exchange(&[json!({"op":"fs_list","path":dir.path().to_str().unwrap()}).to_string()]).await; + let listed = request(json!({"op":"fs_list","path":dir.path().to_str().unwrap()})).await; assert_eq!(type_of(listed.last().unwrap()), "done"); let a = listed .iter() @@ -69,8 +68,7 @@ async fn list_batches_large_directories() { std::fs::write(dir.path().join(format!("f{i}")), b"").unwrap(); } - let listed = - exchange(&[json!({"op":"fs_list","path":dir.path().to_str().unwrap()}).to_string()]).await; + let listed = request(json!({"op":"fs_list","path":dir.path().to_str().unwrap()})).await; let frames: Vec<_> = listed.iter().filter(|f| type_of(f) == "entries").collect(); assert_eq!(frames.len(), 2, "one full batch + the remainder"); assert_eq!( @@ -95,7 +93,7 @@ async fn stat_reports_kind_and_size() { ) .await; - let st = exchange(&[json!({"op":"fs_stat","path":path.to_str().unwrap()}).to_string()]).await; + let st = request(json!({"op":"fs_stat","path":path.to_str().unwrap()})).await; assert_eq!(type_of(&st[0]), "stat"); assert_eq!(st[0]["info"]["kind"], "file"); assert_eq!(st[0]["info"]["size"], 10); @@ -110,7 +108,7 @@ async fn stat_mode_excludes_type_bits() { b"x", ) .await; - let st = exchange(&[json!({"op":"fs_stat","path":path.to_str().unwrap()}).to_string()]).await; + let st = request(json!({"op":"fs_stat","path":path.to_str().unwrap()})).await; assert_eq!( st[0]["info"]["mode"], 0o640, "mode must be masked to permission bits" @@ -123,7 +121,7 @@ async fn mkdir_rename_rm_lifecycle() { let sub = dir.path().join("d1"); let moved = dir.path().join("d2"); - let mk = exchange(&[json!({"op":"fs_mkdir","path":sub.to_str().unwrap()}).to_string()]).await; + let mk = request(json!({"op":"fs_mkdir","path":sub.to_str().unwrap()})).await; assert_eq!(type_of(&mk[0]), "done"); let mv = exchange(&[ @@ -150,7 +148,7 @@ async fn read_streams_files_around_the_first_chunk_bound() { let path = dir.path().join(format!("f{len}")); let body: Vec = (0..len).map(|i| (i % 251) as u8).collect(); std::fs::write(&path, &body).unwrap(); - let read = exchange(&[json!({"op":"fs_read","path":path}).to_string()]).await; + let read = request(json!({"op":"fs_read","path":path})).await; let data = read.iter().filter(|f| type_of(f) == "data").count(); assert_eq!(data, frames, "len {len}"); assert!(payload(&read, "data") == body, "len {len}"); @@ -161,7 +159,7 @@ async fn read_streams_files_around_the_first_chunk_bound() { #[cfg(target_os = "linux")] #[tokio::test] async fn read_returns_a_proc_file_that_reports_zero_length() { - let read = exchange(&[json!({"op":"fs_read","path":"/proc/self/status"}).to_string()]).await; + let read = request(json!({"op":"fs_read","path":"/proc/self/status"})).await; assert!( String::from_utf8(payload(&read, "data")) .unwrap() @@ -181,7 +179,7 @@ async fn read_of_a_non_regular_file_is_bad_request() { "/dev/null", fifo.to_str().unwrap(), ] { - let frames = exchange(&[json!({"op":"fs_read","path":path}).to_string()]).await; + let frames = request(json!({"op":"fs_read","path":path})).await; assert_eq!(type_of(&frames[0]), "error", "{path}: {frames:?}"); assert_eq!(frames[0]["kind"], "bad_request", "{path}"); } @@ -189,9 +187,8 @@ async fn read_of_a_non_regular_file_is_bad_request() { #[tokio::test] async fn read_missing_path_is_not_found() { - let frames = exchange(&[json!({"op":"fs_read","path":"/no/such/file/xyz"}).to_string()]).await; - assert_eq!(type_of(&frames[0]), "error"); - assert_eq!(frames[0]["kind"], "not_found"); + let frames = request(json!({"op":"fs_read","path":"/no/such/file/xyz"})).await; + assert_error(&frames[0], "not_found"); } #[tokio::test] @@ -264,16 +261,15 @@ async fn rm_plain_file_and_missing_and_nonempty_dir() { let dir = tempfile::tempdir().unwrap(); let f = dir.path().join("f"); std::fs::write(&f, b"x").unwrap(); - let r = exchange(&[json!({"op":"fs_rm","path":f.to_str().unwrap()}).to_string()]).await; + let r = request(json!({"op":"fs_rm","path":f.to_str().unwrap()})).await; assert_eq!(type_of(&r[0]), "done"); assert!(!f.exists()); - let m = exchange(&[json!({"op":"fs_rm","path":"/no/such/xyz"}).to_string()]).await; - assert_eq!(type_of(&m[0]), "error"); - assert_eq!(m[0]["kind"], "not_found"); + let m = request(json!({"op":"fs_rm","path":"/no/such/xyz"})).await; + assert_error(&m[0], "not_found"); let sub = dir.path().join("d"); std::fs::create_dir(&sub).unwrap(); std::fs::write(sub.join("inner"), b"y").unwrap(); - let ne = exchange(&[json!({"op":"fs_rm","path":sub.to_str().unwrap()}).to_string()]).await; + let ne = request(json!({"op":"fs_rm","path":sub.to_str().unwrap()})).await; assert_eq!(type_of(&ne[0]), "error"); assert!( sub.exists(), diff --git a/silkd/tests/git_e2e.rs b/silkd/tests/git_e2e.rs index f9e2fcb94..364f54b80 100644 --- a/silkd/tests/git_e2e.rs +++ b/silkd/tests/git_e2e.rs @@ -5,9 +5,9 @@ mod common; use serde_json::json; -use common::{exchange, type_of}; +use common::{assert_error, exchange, request, type_of}; -fn git(dir: &std::path::Path, args: &[&str]) { +fn git(dir: &std::path::Path, args: &[&str]) -> String { let out = std::process::Command::new("git") .arg("-C") .arg(dir) @@ -23,6 +23,7 @@ fn git(dir: &std::path::Path, args: &[&str]) { "git {args:?} failed: {}", String::from_utf8_lossy(&out.stderr) ); + String::from_utf8_lossy(&out.stdout).into_owned() } fn init_repo() -> tempfile::TempDir { @@ -49,7 +50,7 @@ async fn add_commit_status_reports_structured_state() { let p = repo.path().to_str().unwrap(); std::fs::write(repo.path().join("a.txt"), "hello\n").unwrap(); - let st = exchange(&[json!({"op":"git_status","path":p}).to_string()]).await; + let st = request(json!({"op":"git_status","path":p})).await; assert_eq!(type_of(&st[0]), "git_status_result"); assert_eq!(st[0]["branch"], "main"); let files = st[0]["files"].as_array().unwrap(); @@ -59,17 +60,16 @@ async fn add_commit_status_reports_structured_state() { .any(|f| f["path"] == "a.txt" && f["staged"] == "?") ); - let add = exchange(&[json!({"op":"git_add","path":p,"files":["a.txt"]}).to_string()]).await; + let add = request(json!({"op":"git_add","path":p,"files":["a.txt"]})).await; assert_eq!(type_of(add.last().unwrap()), "done"); - let commit = exchange(&[json!({ + let commit = request(json!({ "op":"git_commit","path":p,"message":"first","author":"Dev " - }) - .to_string()]) + })) .await; assert_eq!(type_of(&commit[0]), "git_commit_result"); assert_eq!(commit[0]["hash"].as_str().unwrap().len(), 40); - let st2 = exchange(&[json!({"op":"git_status","path":p}).to_string()]).await; + let st2 = request(json!({"op":"git_status","path":p})).await; assert!( st2[0]["files"].as_array().unwrap().is_empty(), "expected clean: {st2:?}" @@ -90,7 +90,7 @@ async fn branch_create_list_checkout() { .await; assert_eq!(type_of(created.last().unwrap()), "done"); - let list = exchange(&[json!({"op":"git_branch","path":p,"action":"list"}).to_string()]).await; + let list = request(json!({"op":"git_branch","path":p,"action":"list"})).await; assert_eq!(type_of(&list[0]), "git_branches"); assert_eq!(list[0]["current"], "main"); let branches: Vec<&str> = list[0]["branches"] @@ -112,10 +112,9 @@ async fn branch_create_list_checkout() { async fn commit_failure_surfaces_git_error() { let repo = init_repo(); let p = repo.path().to_str().unwrap(); - let frames = exchange(&[json!({ + let frames = request(json!({ "op":"git_commit","path":p,"message":"empty","author":"Dev " - }) - .to_string()]) + })) .await; assert_eq!(type_of(&frames[0]), "error"); assert_eq!(frames[0]["kind"], "internal", "{frames:?}"); @@ -124,13 +123,11 @@ async fn commit_failure_surfaces_git_error() { #[tokio::test] async fn clone_lane_behavior() { silkd::net::override_egress_for_tests(Some(false)); - let none = exchange(&[json!({ + let none = request(json!({ "op":"git_clone","url":"https://example.com/x.git","path":"/tmp/silkd-x" - }) - .to_string()]) + })) .await; - assert_eq!(type_of(&none[0]), "error"); - assert_eq!(none[0]["kind"], "unimplemented"); + assert_error(&none[0], "unimplemented"); assert!(none[0]["message"].as_str().unwrap().contains("fs.push")); let src = init_repo(); @@ -140,10 +137,9 @@ async fn clone_lane_behavior() { let dst = tempfile::tempdir().unwrap(); let target = dst.path().join("clone"); silkd::net::override_egress_for_tests(Some(true)); - let ok = exchange(&[json!({ + let ok = request(json!({ "op":"git_clone", "url": src.path().to_str().unwrap(), "path": target.to_str().unwrap() - }) - .to_string()]) + })) .await; silkd::net::override_egress_for_tests(None); assert_eq!(type_of(ok.last().unwrap()), "done", "clone failed: {ok:?}"); @@ -164,7 +160,7 @@ async fn status_handles_rename_and_spaces() { .unwrap(); git(repo.path(), &["add", "-A"]); - let st = exchange(&[json!({"op":"git_status","path":p}).to_string()]).await; + let st = request(json!({"op":"git_status","path":p})).await; let files = st[0]["files"].as_array().unwrap(); let paths: Vec<&str> = files.iter().map(|f| f["path"].as_str().unwrap()).collect(); assert!( @@ -172,7 +168,7 @@ async fn status_handles_rename_and_spaces() { "rename path with spaces mangled: {paths:?}" ); assert!( - !paths.iter().any(|p| p.contains("R100")), + !paths.iter().any(|path| path.contains("R100")), "score leaked: {paths:?}" ); } @@ -182,26 +178,19 @@ async fn commit_works_without_preconfigured_identity() { let dir = init_repo(); std::fs::write(dir.path().join("f"), "x").unwrap(); let p = dir.path().to_str().unwrap(); - let add = exchange(&[json!({"op":"git_add","path":p,"files":["f"]}).to_string()]).await; + let add = request(json!({"op":"git_add","path":p,"files":["f"]})).await; assert_eq!(type_of(add.last().unwrap()), "done"); - let commit = exchange(&[json!({ + let commit = request(json!({ "op":"git_commit","path":p,"message":"m","author":"Dev " - }) - .to_string()]) + })) .await; assert_eq!( type_of(&commit[0]), "git_commit_result", "commit failed: {commit:?}" ); - let log = std::process::Command::new("git") - .arg("-C") - .arg(dir.path()) - .args(["log", "-1", "--format=%cn <%ce>"]) - .output() - .unwrap(); assert_eq!( - String::from_utf8_lossy(&log.stdout).trim(), + git(dir.path(), &["log", "-1", "--format=%cn <%ce>"]).trim(), "Dev ", "the committer must come from the author, not the host's git config" ); @@ -215,9 +204,9 @@ async fn git_runs_as_the_repo_owner_under_root() { std::fs::write(repo.path().join("a.txt"), "hello\n").unwrap(); chown_tree(repo.path(), 65534); - let st = exchange(&[json!({"op":"git_status","path":p}).to_string()]).await; + let st = request(json!({"op":"git_status","path":p})).await; assert_eq!(type_of(&st[0]), "git_status_result", "{st:?}"); - let add = exchange(&[json!({"op":"git_add","path":p,"files":["a.txt"]}).to_string()]).await; + let add = request(json!({"op":"git_add","path":p,"files":["a.txt"]})).await; assert_eq!(type_of(add.last().unwrap()), "done", "{add:?}"); let index = std::fs::metadata(repo.path().join(".git/index")).unwrap(); assert_eq!(std::os::unix::fs::MetadataExt::uid(&index), 65534); diff --git a/silkd/tests/lsp_e2e.rs b/silkd/tests/lsp_e2e.rs index 199e1f995..904e4c465 100644 --- a/silkd/tests/lsp_e2e.rs +++ b/silkd/tests/lsp_e2e.rs @@ -7,13 +7,13 @@ use std::io::Write; use std::os::unix::fs::PermissionsExt; use std::sync::Arc; -use serde_json::json; +use serde_json::{Value, json}; use silkd::server::State; use tempfile::TempDir; use tokio::io::AsyncWriteExt; use tokio::time::timeout; -use common::{DEADLINE, b64, connect, decode, next_frame, one, send, type_of}; +use common::{DEADLINE, assert_error, b64, connect, decode, next_frame, one, send, type_of}; const FAKE_SERVER: &str = r#"#!/bin/sh IFS= read -r line @@ -28,8 +28,7 @@ fn manifest_env(server_body: &str) -> TempDir { let mut f = std::fs::File::create(&bin).unwrap(); f.write_all(server_body.as_bytes()).unwrap(); std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).unwrap(); - // SAFETY: ENV_LOCK, held by the caller for its whole test, serializes every - // writer and reader of SILKD_LSP_DIR in this binary. + // SAFETY: ENV_LOCK, held by the caller for its whole test, serializes all access to SILKD_LSP_DIR. unsafe { std::env::set_var("SILKD_LSP_DIR", dir.path()) }; dir } @@ -55,12 +54,16 @@ async fn started_server(lang: &str, server_body: &str) -> (TempDir, Arc, } async fn assert_stopped(state: &Arc, server_id: &str) { - let gone = one( + let gone = lsp_stop(state, server_id).await; + assert_eq!(gone.last().unwrap()["kind"], "not_found", "{gone:?}"); +} + +async fn lsp_stop(state: &Arc, server_id: &str) -> Vec { + one( state, &json!({"op":"lsp_stop","server_id":server_id}).to_string(), ) - .await; - assert_eq!(gone.last().unwrap()["kind"], "not_found", "{gone:?}"); + .await } #[tokio::test] @@ -76,8 +79,7 @@ async fn lsp_start_missing_manifest_is_not_found() { .await .expect("deadline"); let last = frames.last().unwrap(); - assert_eq!(type_of(last), "error"); - assert_eq!(last["kind"], "not_found"); + assert_error(last, "not_found"); } #[tokio::test] @@ -97,11 +99,7 @@ async fn lsp_start_language_name_cannot_escape() { assert_eq!(type_of(last), "error", "{bad} should be rejected"); assert_eq!(last["kind"], "not_found", "{bad}"); } - one( - &state, - &json!({"op":"lsp_stop","server_id":server_id}).to_string(), - ) - .await; + lsp_stop(&state, &server_id).await; } #[tokio::test] @@ -119,7 +117,7 @@ async fn lsp_broker_relays_to_the_server() { assert_eq!(decode(&frame), b"reply:ping\n"); assert_eq!(type_of(&next_frame(&mut out).await), "done"); cw.shutdown().await.unwrap(); - let _ = handle.await; + handle.await.unwrap().unwrap(); assert_stopped(&state, &server_id).await; } @@ -128,11 +126,7 @@ async fn lsp_broker_relays_to_the_server() { async fn lsp_stop_kills_an_idle_server() { let _env_lock = ENV_LOCK.lock().await; let (_env, state, server_id) = started_server("idletest", "#!/bin/sh\nsleep 60\n").await; - let stop = one( - &state, - &json!({"op":"lsp_stop","server_id":server_id}).to_string(), - ) - .await; + let stop = lsp_stop(&state, &server_id).await; assert_eq!(type_of(stop.last().unwrap()), "done"); assert_stopped(&state, &server_id).await; } @@ -173,5 +167,5 @@ async fn lsp_data_end_half_closes_stdin() { assert_eq!(decode(&frame), b"ate:5\n"); assert_eq!(type_of(&next_frame(&mut out).await), "done"); cw.shutdown().await.unwrap(); - let _ = handle.await; + handle.await.unwrap().unwrap(); } diff --git a/silkd/tests/pty_e2e.rs b/silkd/tests/pty_e2e.rs index bc984146e..75214088c 100644 --- a/silkd/tests/pty_e2e.rs +++ b/silkd/tests/pty_e2e.rs @@ -17,7 +17,9 @@ use silkd::server::State; use tokio::io::{AsyncWrite, AsyncWriteExt}; use tokio::sync::mpsc; -use common::{FrameLines, b64, connect, decode, exchange, frames_until, one, send, type_of}; +use common::{ + FrameLines, assert_error, b64, connect, decode, frames_until, one, request, send, type_of, +}; struct FailingSink(Vec); @@ -114,10 +116,8 @@ async fn pty_appears_in_ps_and_resizes() { #[tokio::test] async fn resize_unknown_pid_is_not_found() { - let frames = - exchange(&[json!({"op":"pty_resize","pid":999999,"cols":80,"rows":24}).to_string()]).await; - assert_eq!(type_of(&frames[0]), "error"); - assert_eq!(frames[0]["kind"], "not_found"); + let frames = request(json!({"op":"pty_resize","pid":999999,"cols":80,"rows":24})).await; + assert_error(&frames[0], "not_found"); } #[tokio::test] diff --git a/silkd/tests/session_e2e.rs b/silkd/tests/session_e2e.rs index b601c1f61..bc1563649 100644 --- a/silkd/tests/session_e2e.rs +++ b/silkd/tests/session_e2e.rs @@ -10,7 +10,7 @@ use serde_json::{Value, json}; use silkd::server::State; use tokio::task::JoinHandle; -use common::{one, stdout_body, type_of}; +use common::{assert_error, one, stdout_body, type_of}; async fn create(state: &Arc) -> String { create_with(state, json!({"op":"session_create"})).await @@ -141,8 +141,7 @@ async fn session_list_and_rm() { assert_eq!(type_of(&rm[0]), "done"); let gone = sh(&state, &id, &["true"]).await; - assert_eq!(type_of(&gone[0]), "error"); - assert_eq!(gone[0]["kind"], "not_found"); + assert_error(&gone[0], "not_found"); } #[tokio::test] @@ -253,8 +252,7 @@ async fn empty_argv_in_session_is_a_bad_request() { &json!({"op":"exec","argv":[],"session":id}).to_string(), ) .await; - assert_eq!(type_of(&f[0]), "error"); - assert_eq!(f[0]["kind"], "bad_request"); + assert_error(&f[0], "bad_request"); } #[tokio::test] @@ -266,16 +264,14 @@ async fn detach_with_session_is_a_bad_request() { &json!({"op":"exec","argv":["true"],"session":id,"detach":true}).to_string(), ) .await; - assert_eq!(type_of(&f[0]), "error"); - assert_eq!(f[0]["kind"], "bad_request"); + assert_error(&f[0], "bad_request"); } #[tokio::test] async fn exec_in_unknown_session_is_not_found() { let state = Arc::new(State::new()); let f = one(&state, r#"{"op":"exec","argv":["true"],"session":"nope"}"#).await; - assert_eq!(type_of(&f[0]), "error"); - assert_eq!(f[0]["kind"], "not_found"); + assert_error(&f[0], "not_found"); } #[tokio::test] diff --git a/silkd/tests/tree_e2e.rs b/silkd/tests/tree_e2e.rs index 100778b4b..f094aaca6 100644 --- a/silkd/tests/tree_e2e.rs +++ b/silkd/tests/tree_e2e.rs @@ -10,7 +10,7 @@ use std::process::Command; use serde_json::json; -use common::{b64, data_frames, exchange, payload, type_of}; +use common::{assert_error, b64, data_frames, exchange, payload, request, type_of}; fn sys_tar_create(dir: &Path) -> Vec { let out = Command::new("tar") @@ -66,8 +66,7 @@ async fn pull_streams_a_tar_of_the_path() { std::fs::create_dir(&proj).unwrap(); std::fs::write(proj.join("x.txt"), b"hello-pull").unwrap(); - let frames = - exchange(&[json!({"op":"fs_pull","path":proj.to_str().unwrap()}).to_string()]).await; + let frames = request(json!({"op":"fs_pull","path":proj.to_str().unwrap()})).await; assert_eq!( type_of(frames.last().unwrap()), "done", @@ -97,8 +96,7 @@ async fn push_then_pull_roundtrips_a_tree() { lines.extend(data_frames(&archive)); assert_eq!(type_of(&exchange(&lines).await[0]), "done"); - let pulled = - exchange(&[json!({"op":"fs_pull","path":dest.to_str().unwrap()}).to_string()]).await; + let pulled = request(json!({"op":"fs_pull","path":dest.to_str().unwrap()})).await; let out_archive = payload(&pulled, "data"); let out = tempfile::tempdir().unwrap(); sys_tar_extract(&out_archive, out.path()); @@ -130,8 +128,7 @@ async fn push_bad_archive_reports_tar_failure() { json!({"op":"data_end"}).to_string(), ]) .await; - assert_eq!(type_of(frames.last().unwrap()), "error"); - assert_eq!(frames.last().unwrap()["kind"], "internal"); + assert_error(frames.last().unwrap(), "internal"); } #[tokio::test] @@ -159,8 +156,7 @@ async fn push_large_bad_archive_reports_tar_not_the_pipe() { async fn pull_missing_path_errors() { let dir = tempfile::tempdir().unwrap(); let missing = dir.path().join("xyz"); - let frames = - exchange(&[json!({"op":"fs_pull","path":missing.to_str().unwrap()}).to_string()]).await; + let frames = request(json!({"op":"fs_pull","path":missing.to_str().unwrap()})).await; assert_eq!( type_of(frames.last().unwrap()), "error", @@ -171,7 +167,7 @@ async fn pull_missing_path_errors() { #[tokio::test] async fn pull_missing_parent_errors_not_found() { - let frames = exchange(&[json!({"op":"fs_pull","path":"/no/such/dir/xyz"}).to_string()]).await; + let frames = request(json!({"op":"fs_pull","path":"/no/such/dir/xyz"})).await; assert_eq!( type_of(frames.last().unwrap()), "error", @@ -186,8 +182,7 @@ async fn pull_dangling_symlink_archives_the_link() { let link = src.path().join("dangling"); symlink("/no/such/target", &link).unwrap(); - let frames = - exchange(&[json!({"op":"fs_pull","path":link.to_str().unwrap()}).to_string()]).await; + let frames = request(json!({"op":"fs_pull","path":link.to_str().unwrap()})).await; assert_eq!( type_of(frames.last().unwrap()), "done", diff --git a/silkd/tests/watch_e2e.rs b/silkd/tests/watch_e2e.rs index 593795729..baf9067ba 100644 --- a/silkd/tests/watch_e2e.rs +++ b/silkd/tests/watch_e2e.rs @@ -9,7 +9,7 @@ use std::time::Duration; use serde_json::json; use silkd::server::State; -use common::{connect, exchange, next_frame, send, type_of}; +use common::{connect, next_frame, request, send, type_of}; #[tokio::test] async fn watch_streams_events_until_disconnect() { @@ -45,9 +45,7 @@ async fn watch_streams_events_until_disconnect() { #[tokio::test] async fn watch_missing_path_is_error() { - let frames = - exchange(&[json!({"op": "fs_watch", "path": "/no/such/dir/silkd-watch"}).to_string()]) - .await; + let frames = request(json!({"op": "fs_watch", "path": "/no/such/dir/silkd-watch"})).await; assert_eq!(type_of(&frames[0]), "error"); assert_eq!(frames[0]["kind"], "not_found", "{frames:?}"); }