Repository navigation
perf: remaining performance ledger after the 2026-09 SOCKS5 batch #172
Description
Successor to #30, which closes with this issue open. Every #30 item was re-verified against main on 2026-09-14 (ledger in the closing comment there): the 2026-07 hotspot work, the refill admission re-eval, the digest and generation-store rework, the XFS/direct-IO storm, the staging fsync recipe, the VM-name suffix widening and the SOCKS5 opt-in are done or superseded; wire v2 and the bulk-frame buffer live in #34. What remains is below, each with the trigger that reopens it. Numbers of record live in docs/benchmarks.md and docs/performance.md; the C1M campaign measurements stay in #30's thread.
Open — unowned lever
- Cocoon clone-serving / CLI-per-op. Every engine operation is one fork+exec of the cocoon binary; measured 155–180 clones/s through the CLI against 257–274/s direct. The cocoon supervisor issue closed without a daemon or library mode. Largest remaining fill-rate lever; needs a cocoon-side decision.
Deferred — reopen triggers
| item | where | trigger |
|---|---|---|
| claimStore whole-map marshal + rename per commit | sandboxd/pool/claims.go |
>120 live claims per node |
| idleOnce/archiveOnce O(n) scans under the manager mutex | sandboxd/pool/hibernate.go, archive.go |
same |
| checkpoint TTL sweep re-lists the full store per tick | sandboxd/pool/checkpoint.go |
checkpoint count grows 10x |
| resyncEgress serial lock pass at restart | sandboxd/pool/reconcile.go |
many egress claims + a restart-time SLA |
| probeReady engine-list fallback poll | sandboxd/pool/refill.go |
the fallback branch shows up in practice |
| recLock global serialization on template records | sandboxd/pool/template.go |
recLock reaching the claim path |
| ownership probing per distinct cold id (5 s positive cache + singleflight) | sandboxd/store/peer/probe.go |
sustained distinct-id branch rate |
Micro items — unchanged since 2026-07-26, all below the noise floor today
silkd: 64 KB inbound vsock buffer (folds into #34), tagged double-pass decode in proto.rs, regex recompiled per call in find.rs, 256 KB ring / 300 s detached linger with no aggregate cap in proc.rs/exec.rs. sandboxd: two random draws per claim id, eager key hash in server.go, quota precheck and warm-pop as separate lock takes, refill catch-up rescan under the mutex, 1 s gossip push/pull, intercept leaf-cache full clear. SDKs: fresh connection per call and in-memory archives (Python), client rebuilt per op (openai adapter), double marshal (mcp), unbounded append in sdk/go/files.go.
Rejected with data — do not re-litigate without new evidence
Pre-dialed spare connection; fs_push chunk 256K→1M; nft netlink connection merge; silkd conn scanner-buffer pooling; encodeTagged rework; silkd cold-start slimming; twice-registered cancel hook.
Activity
CMGS commentedon Sep 14, 2026
Two rows from the 2026-09-14 Rust ownership audit (all of silkd and boot/init read in full), kept out of the review PR #173 as their own change:
silkd/src/proc.rsTable::registercallsbroadcast::channel(OUTPUT_FANOUT)(256) for every exec and every pty open, and tokio allocates the whole slot buffer at construction: measuredChunk32 B,Slot48 B,Mutex<Slot>56 B on Linux → about 14 KB of heap per registered process, for a live fan-out most execs never attach to. Present since silkd's first commit (ddeddc7, 2026-07-05), where the ring backedattach/logson detached processes and the concern was reaping and pid reuse, not per-exec memory. CPU is under one percent of a fork+exec; the real cost is retention: a detached exec keeps its entry forREAP_DELAY(300 s), so a thousand detached execs pin about 14 MB of empty rings on top of the log ring. Fix shape: create the sender lazily on the firstattach_stream(OnceLock<Sender>;emitalready checksreceiver_count() > 0), or cut the capacity. Trigger: detached-exec-heavy workloads, or any memory budget per guest.silkd/src/git.rsbuilds theformat!("git {verb}")label beforesubprocess_resultdecides whether it needs it (only on failure), and every session command allocates two strings for its sentinel (session.rs/sysutil::rand_token). Noise next to a fork and a shell round trip; recorded so nobody re-finds them.pool: bind a warm clone's egress doors at refill, not at claim
pool: bind a warm clone's egress doors at refill, not at claim (#177)
CMGS commentedon Sep 14, 2026
Two rows closed on 2026-09-14:
Table::registerno longer allocates the 256-slot channel;attach_streambuilds it on the first attacher under the ring lock. cocoon-test1, same sandboxd, guests differing only in silkd, 500 timed execs per run, A B B A × 3: exec p50 726 → 699 µs, p90 780 → 754 (n=3000 each, every round lower); silkd VmRSS after 1000 detached spawns 5040 → 20768 kB before, 4920 → 5636 kB after. The 256 KB log ring and the 300 s detached linger in the micro list are a different item and stay.Also landed from the same batch, not perf rows: #176 (idle sweep vs live connections, #146) and #174 (spawn error kinds, #145).
CMGS commentedon Sep 14, 2026
Rows from the whole-repo review round (PR #179), measured or estimated by the readers, none changed in that PR:
fastBulkdecodes each data frame into a fresh buffer and copies it once more into the caller's slice; append-decode into the caller's buffer would drop one copy per 256 KiB chunk on every file read.json.Marshalplus an unbufferedwrite(2)per event, shared by the claim path (recordUsageinfinalizeBatch) and every guarded-egress decision; ~2–4 µs marshal plus 1–3 µs syscall per claim, and at 10k egress requests/s the claim path queues behind the data plane on one mutex. A size- and time-bounded buffer or a single writer goroutine.m.muacross three full scans ofm.claimed(reapOnce, idleOnce, archiveOnce) and hashes a PoolKey per claim in two of them; ≈185 µs of global-lock hold per 5 s at 1000 live claims. One scan collecting all three victim sets.maps.Clone+ marshal + whole-file write underwriteMu); ≈1.7 ms at 1000 live claims. Per-claim records or an append-only op log.reached()probe takessyscall.ForkLockon every guarded-egress claim; a claim that overlaps a refill'scocoonexec waits for that fork window.Accept4(SOCK_CLOEXEC)(Linux-only file) drops the lock; needs its own A/B.CMGS commentedon Sep 14, 2026
One more row from the same round (PR #179), unchanged there:
armEgressrunsnetfilter.Lock(tap)on the claim path: two netlink connections and two flush round trips per guarded-egress claim (the ~36 µs measured in review: batch-end round on the SOCKS5 opt-in, ports and lane-file state #173). The tap is already known at refill (startVMrecords it), so the lock could move next to the door pre-bind of pool: bind a warm clone's egress doors at refill, not at claim #177 and leave the claim path with the map write; a trimmed warm VM would unlock indestroy. Constrained by the hibernate rule (cocoon resumes before the lock can be re-applied), so it needs its own A/B and a look at what a warm VM may reach before its claim.CMGS commentedon Sep 17, 2026
Ledger addition: #213 — a redirected claim counts as an arrival on the entry node as well as on the node that serves it, so a burst against one node raises the warm target across the mesh. It rode on the back of #181: until the shrink lands, every inflated pool keeps its high-water mark, so the cluster strands N times the single-node residue. #211 bounds that in time; what gets counted is still open, and the reopen trigger is a mesh whose total warm VMs exceed its peak concurrent claims.
CMGS commentedon Sep 22, 2026
Re-verified every row of this ledger against
mainat24d9993(2026-09-22). All twelve are still open — nothing here has been fixed since the ledger was written, and no named file or symbol has moved. The only two structural changes touching these rows,netfilter.LockedTaps()prefetch inresyncEgressand theseq/writtencoalescing inclaimStore, both landed ind19b677on 2026-09-03, before this ledger, so neither counts against it.Two corrections to the wording, so nobody optimizes against a shape the code does not have:
The reap tick does not hold
m.muacross the three scans.reapOnce,idleOnceandarchiveOnceare three separate calls in the tick (sandboxd/pool/pool.go:497-503), and each takes and releasesm.muon its own (sandboxd/pool/claim.go:400-428,sandboxd/pool/hibernate.go:181-192,sandboxd/pool/archive.go:46-61). The cost stands — three O(n) passes overm.claimedper tick, two of them hashing a PoolKey per claim — but there is no single critical section spanning all three.fastBulkis not where the extra copy is. It allocates one buffer per data frame and returnsout[:n](protocol/wire/frame.go:738-744); there is no second copy inside it. The second copy is at the consumers —sandboxd/engine/silkd.go:109(data = append(data, resp.Data...)) andsdk/go/utils.go:172-175. The row's cost claim holds; the fix belongs on the consumer side, or in an append-decode API that lets the caller supply the buffer.Current file:line for each row, for whoever picks one up:
sandboxd/pool/claims.go:130-142sandboxd/pool/hibernate.go:181-192,sandboxd/pool/archive.go:46-61sandboxd/pool/checkpoint.go:339→checkpoint.go:91sandboxd/pool/reconcile.go:170-199sandboxd/pool/refill.go:429-439,:461sandboxd/pool/reclock.go:7-15,sandboxd/pool/template.go:256-269sandboxd/store/peer/probe.go:62-73sandboxd/pool/journal.go:49-64sandboxd/pool/claim.go:352→sandboxd/pool/egress.go:139→sandboxd/netfilter/nft_linux.go:19-53,77-85sandboxd/engine/engine.go:342-353#213 was re-checked too and is unchanged:
p.noteArrival(start)still runs before the warm check atsandboxd/pool/claim.go:46.CMGS commentedon Oct 17, 2026
Closed since 09-22:
claims.jsoncommit encodes only changed claims (pool: a claims.json commit encodes only the changed claims #305);sandboxd/engine/silkd.go:118still appends;Still open: one cocoon CLI process per engine op; the archive scan; the checkpoint TTL full list; the serial
resyncEgresspass at restart; theprobeReadyfallback poll;recLockserialization on template records; the peer ownership probe; the journal mutex with an unbuffered write; egress usage written per request; theForkLockinoutbound/doors.go:119; the nft lock taken on the claim path; #213.CMGS commentedon Oct 26, 2026
Addition, off the claim path: the archive checkpoint pins.
DeleteCheckpoint, once per delete marker in the retry sweep,dropStaleRow) is now a lookup. It went from 162 µs, 218 KB and 18 allocs to 25 ns and 0 allocs. It no longer copies the whole set under the manager mutex.Measured on a mac M2 Max with
-cpu 1, both arms interleaved in one run. The cost on the claim path is one extra map lookup per journal set/del, about 25 ns, with no allocation.CMGS commentedon Nov 2, 2026
Backlog from the 2026-11-02 hardware round: VM density limit in
kvm_irqfd, plus one row to downgrade.Density limit. On a 384-core bare-metal node, kernel 6.12.100, none lane, 64 concurrent clones up to 1000 VMs: every span is flat except CH
vm.restore, which climbs with the VM count, and the fill rate falls with it.vm.restorep50 msKernel stacks of the CH threads during the storm: 625 of 648 D-state samples are in
irq_bypass_register_consumer<-kvm_irqfd. In v6.12 that function takes a global mutex and walks every consumer on the host (virt/lib/irqbypass.c), so each irqfd costs O(irqfds on the host). v6.15 and v6.16 still use the lists, v6.18 uses an xarray, v6.17 not checked.Measured per VM with kprobes, 1 vCPU, no NIC, 21
kvm_irqfdcalls per VM (kvm_irqfdsum ofvm.restore): 1 clone 5.5 of 13.1 ms, 16 clones 125.6 of 146 ms, 64 clones 921.9 of 1035 ms. Upstream 8394b32faecd makes the lookup O(1); the global mutex stays, and no kernel with it has been run here.Options, none tried:
kvm_arch_has_irq_bypass()is true, which isenable_apicv && irq_remapping_cap(IRQ_POSTING_CAP). Turning off posted interrupts in the IOMMU at boot (amd_iommu_intr=legacy,intremap=noposton Intel; check the kernel docs, these names are from memory) should skip it and only affects devices assigned through VFIO. Needs a reboot.kvm_amd.avic=0also skips it but gives up AVIC. Needs a module reload with no VMs running.Blocked on a host that can be rebooted. Whether this is the only limit is unknown.
Open question. A 2026-07-29 run reached 1000 warm at 201/s on a 384-core node with
restore_mode: mmap,no_direct_io, a tmpfs run dir, cocoon v0.5.8 and kernel 6.12.90. This round reached 50/s at n=1000.no_direct_ioalone did not change it (48/s). The rest of that recipe was not tried. This round also ran a CH build from 2026-08-12, older than the fork'sdevand without 0295e241e, while the July run used the fork'sdevof that day. So this may be the recipe, the CH build or a regression, and the table above should be re-measured on currentdevfirst.Downgrade: cocoon clone-serving / CLI-per-op. With
GOMAXPROCScapped (#413), an in-process clone on cocoon v0.7.1 buys less than this ledger assumed. One clone: 32 ms through the CLI vs 17 ms in-process (51 ms before #413). Ready rate at 8 workers: 166-173/s vs 180-182/s (121-123/s before #413). At 64 workers all arms tie at about 200/s, limited by the row above. It is no longer the largest fill-rate lever.CMGS commentedon Nov 3, 2026
Update from the audit behind #420-#432. Numbers are from a 384-core bare-metal node, cocoon v0.7.5, Cloud Hypervisor v54,
restore_modemmap, with both arms interleaved (the audited tree againstmainbefore it).Density, the next limits.
/v1/infostays at 0.25-0.30 ms.kernel.pty.max: each VM holds a console pty, and the default of 4096 is host-wide. A node hit it at 4057 VMs:vm.restoreanswers 500 withNo space left on device.docs/deploy.mdstates this (review: the audit's final tidy: layout, docs, shared test helpers #432). It is a sysctl, not yet tested above 4096.vm cloneaveraged 4.29 s at the top, with the last 50 at 7.33 s, whilevm inspectstayed at 18 ms. This is probably thekvm_irqfdrow above, but no kernel stacks were taken in this run, so that is not confirmed.Closed since the 2026-10-17 list.
ForkLockinoutbound/doors.gois gone (outbound: probe a pre-bound door with poll, not accept #414).recLockis held only for a re-promote's commit, not across hashing and upload, so a claim waits at most for the commit (pool, store: hold a template's record lock only for the re-promote commit #430).Not rechecked in this round: the archive scan, the checkpoint TTL full list,
resyncEgress, theprobeReadyfallback poll, the peer ownership probe, the journal mutex and egress usage per request.Decided, not planned: a delta log for
claims.json. A commit took 0.45-0.57 ms at 3000 claims in this run, on a store that encodes only changed claims since #305. That is fast enough.Deferred, with triggers.
ScanFilehashinglseekskip in the hole scanRejected with data.
Content-Lengthwith no furtherRead, so a reader returned fromReadis never reused (12000 of 12000 requests allocated a new one). Returning it onCloseraces an in-flightReadand needs an atomic perReadon the relay path. pool, store: dead warm VMs, shared staging, cache reclaim, volume unmount budget #427 grows the reader lazily instead.CMGS commentedon Nov 4, 2026
Path update after #438, no row changed.
probeReadyis now insandboxd/pool/provision.go, golden builds ingolden.go,reapOnceinreap.go, and the record locks are therecordLockstype inreclock.go. The latest warm and clone A/B is on #419.CMGS commentedon Nov 5, 2026
Update after #459-#462, from the eight perf leads #457 listed as unmeasured. Measured on 2026-11-05: x86 test host, 2 pinned cores, ABBA order, 4 rounds, spread under 6%, except where marked mac or modeled.
Done:
m.muMetasread every record's labelsreplace_allon a runtime worker\w{1,200}find 47-70 ms → 2.4-2.8 ms (mac)Open, each with its trigger: #463 (a live move hashes every chunk on both ends; decided by the boundary-scale move test) and #464 (an archived wake waits for the store delete; reopen on an archived-wake latency target).
Still open in #458: the silkd
feed_clientdecode buffer, the usage and audit append on every egress decision under contention, claims racing at the last free slot, and the promoted-templatemeta.jsonGET on S3.Benchmark sources are kept with the round's evidence, not in the repository.
CMGS commentedon Nov 8, 2026
Status against main 46fd28c (2026-11-08).
#458 closed on 2026-11-05:
meta.jsonGET moved to store/s3: a promoted-template claim GETs meta.json on every warm miss #466.feed_clientdecode buffer and the per-decision usage/audit append were measured and dropped. Revisit past about 50k egress decisions/s per node.Closed from the original table:
Still open:
sandboxd/pool/archive.go);sandboxd/pool/checkpoint.go);resyncEgresspass at restart (sandboxd/pool/reconcile.go);probeReadyfallback, now avm inspectpoll (sandboxd/pool/provision.go);sandboxd/store/peer/probe.go; sandboxd: skip the checkpoint owner probe on a shared s3 store #460 removed it on s3);kvm_irqfddensity limit, blocked on a host that can be rebooted;The body's table and micro list are out of date; the list above is the current set. The micro list was not re-measured item by item.
CMGS commentedon Nov 10, 2026
One row narrowed since 11-08: #536 (v0.1.28) moved archive and replica images to
ar_ids, andCheckpointslists only theck_prefix (an S3ListObjectsV2prefix, a name filter on dir). The checkpoint TTL sweep andGET /v1/checkpointsno longer list or read wake images. With 3000 archived claims per node on 10 nodes, that removes ~30k meta GETs per call. The sweep still reads every user checkpoint's meta per tick. Every other row of the 11-08 list is unchanged.