Add support for on-disk SQLite store in computerd

.changeset/on-disk-computerd-store.md

@@ -0,0 +1,5 @@
+---
+"@cloudflare/dofs": minor
+---
+
+Let `computerd` keep its workspace on disk instead of in memory, so it survives a restart. Set `COMPUTERD_DB` to a file path — see [the `computerd` README](../packages/computerd/README.md#on-disk-store).

docs/02_sync_protocol.md

@@ -444,9 +444,9 @@ needed; pure DO-side optimisation.
 ### Push backpressure
 
 A long-running exec can dirty container state faster than the DO can
-pull. Today's process-lifetime container VFS caps this by OOMing, which
-is a bad answer. Once a disk-backed container mirror lands the bound
-shifts to path count, but the same problem persists. Likely shape: a
+pull. An in-memory container VFS caps this by running out of memory.
+Setting `COMPUTERD_DB` to a file path shifts the limit to path count,
+but the same problem remains. Likely shape: a
 soft cap on the dirty set (say, 256 MiB pending bytes or 100k paths)
 above which FUSE write replies are delayed (real backpressure into the
 writer), or the container opportunistically initiates a push to the DO

docs/11_lifecycle.md

@@ -30,14 +30,14 @@ capnweb WebSocket session.
 │  └──────────┬──────────┘  │                  │  └────────┬─────────┘  │
 │             │             │                  │           │            │
 │  ┌──────────▼──────────┐  │                  │  ┌────────▼─────────┐  │
-│  │ SQLite (ctx.storage)│  │                  │  │ In-memory VFS    │  │
-│  │  _vfs_watermark     │  │                  │  │  (process-       │  │
-│  │  vfs_blobs / nodes  │  │                  │  │   lifetime DB)   │  │
+│  │ SQLite (ctx.storage)│  │                  │  │ SQLite VFS       │  │
+│  │  _vfs_watermark     │  │                  │  │  (in memory, or  │  │
+│  │  vfs_blobs / nodes  │  │                  │  │   COMPUTERD_DB)  │  │
 │  └─────────────────────┘  │                  │  └──────────────────┘  │
 └───────────────────────────┘                  └────────────────────────┘
         |                                                  |
-        | source of truth                  process-lifetime |
-        | (durable across restarts)        (lost on restart)|
+        | source of truth                    mirror, in memory
+        | (durable across restarts)          or on the container disk
 ```
 
 The 1:1 mapping is load-bearing for several reasons:
@@ -91,8 +91,8 @@ an incarnation boundary. What survives is:
 On every new incarnation `Workspace.ready()` re-runs `#connect()`,
 which re-enters the backend's bootstrap sequence. If the container is
 still alive, the backend's `POST /connect` + `/api` handshake produces
-a fresh capnweb session against the same in-memory VFS on the
-container side. If the container died too (e.g. host OOM took both),
+a fresh capnweb session against the same container-side VFS. If the
+container died too,
 the next sync round is a rev-0 baseline rebuild from the DO's store.
 
 ### Wake triggers
@@ -127,11 +127,22 @@ lifetime policy. From the DO's perspective:
 exits, and the backend's `#monitoring` flag drops the cached handle at
 that point so the next call rebuilds from scratch (see the container host and backend implementations under `packages/computer/src/backends/container/`).
 
-The critical asymmetry: the **container's VFS is process-lifetime
-in-memory**, while the **DO's VFS is durable SQLite**. A container
-restart loses container-side state. The durable object drives sync
-across the capnweb session it opens through `POST /connect`, and that
-is what brings state back on the next push/pull round.
+When `computerd` runs with its default in-memory store, the two sides
+differ: the **container's VFS lasts only as long as the process**,
+while the **DO's VFS is durable SQLite**. A container restart loses
+container-side state. The durable object drives sync across the
+capnweb session it opens through `POST /connect`, and that is what
+brings state back on the next push/pull round.
+
+Setting `COMPUTERD_DB` to a path changes this for a process restart.
+The container-side store is written to disk, sync cursors included,
+and reopened on the next start. The durable object then finds a peer
+that still knows what it was sent, and only sends the difference.
+
+The container's disk does not survive a container restart, so this
+helps a `computerd` crash inside a live container today. It will help
+a container restart once the platform offers disk snapshots. See the
+`computerd` README for the setting and its limits.
 
 ## Capnweb lifecycle
 

docs/19_performance.md

@@ -48,6 +48,69 @@ computerd is ~2x slower than the container's ext4 disk for the full
 `npm install`, and ~3.6x slower than tmpfs. The disk comparison is
 the more realistic baseline for general usage.
 
+## In-memory store versus on-disk store
+
+`computerd` keeps its SQLite store in memory by default. Set
+`COMPUTERD_DB` to a path and it goes on the container's disk instead.
+
+These numbers come from `script/store-compare.mjs`, which uses the
+dofs filesystem directly with 2,000 files in one directory. There is
+no FUSE mount involved, so any difference is down to the store.
+
+| Operation | memory | on disk (64 MiB cache) | ratio |
+|---|---:|---:|---:|
+| create 2000 files | 180.9 ms | 659.1 ms | 3.64x |
+| stat 2000 paths, cold | 1580.4 ms | 1558.7 ms | **0.99x** |
+| stat 2000 paths, warm | 12.1 ms | 15.1 ms | 1.25x |
+| readdir x50 | 155.4 ms | 143.1 ms | **0.92x** |
+
+Reads are not slower on disk. That holds even when the cache is far
+too small for the tree: a 2 MiB cache against a 3.8 MiB database still
+reads at 0.99x. Two reasons. Most of the time goes on walking the path
+rather than fetching pages, and the operating system caches whatever
+SQLite drops.
+
+Writes are slower, and the reason is the cost of flushing to disk.
+Creating 1,000 files takes 445 ms at `synchronous = full`, 292 ms at
+`normal` (what we ship), and 148 ms at `off` — which matches the
+in-memory store's 181 ms.
+
+Through a real FUSE mount the write cost shrinks, because the mount
+itself is already the bigger expense:
+
+| Scenario | memory store | file store | baseline |
+|---|---:|---:|---:|
+| stat 1000 files | 2777.3 ms (1.10x) | 3114.9 ms (1.22x) | ~2540 ms |
+| create 1000 files | 989.4 ms (0.98x) | 1178.7 ms (1.17x) | ~1010 ms |
+| write 64 MiB | 238.1 ms (11.14x) | 221.9 ms (12.69x) | ~19 ms |
+| overwrite 64 MiB | 294.3 ms (26.16x) | 304.6 ms (29.30x) | ~11 ms |
+
+## Restore time
+
+This is what the on-disk store buys. `script/restore-time.mjs` times
+what a host waits after a restart: connect, compare sync positions,
+and send whatever the other side is missing.
+
+| Tree | memory store | file store |
+|---|---:|---:|
+| 500 files | 480 ms (502 entries sent) | 26 ms (0 sent) |
+| 3,000 files | 3749 ms (3002 entries sent) | 23 ms (0 sent) |
+
+An in-memory store sends the whole workspace again after every
+restart, so its cost grows with the tree. A file store sends nothing,
+because the sync positions came back along with the files. Restoring
+takes about 25 ms whatever the size, so the saving grows too: 18x at
+500 files, 161x at 3,000.
+
+The trade: small-file work costs 10 to 20 percent more, and a restart
+costs a flat 25 ms instead of resending everything.
+
+Two things these numbers do not cover. They come from one Linux
+container, not from Cloudflare Containers hardware, and a full
+`cloudflare/sandbox-sdk` `npm install` has not been run. The restore
+figures also drive the sync protocol in process, so they show the work
+avoided but not the network round trips a real host would also skip.
+
 ## Where computerd is faster than the disk baseline
 
 The in-memory inode store beats real disk on metadata-heavy work:

packages/computerd/README.md

@@ -26,7 +26,8 @@ Current endpoints:
 
 - `GET /health` returns `200 OK` with `ok\n` once the HTTP server is up (it does not currently block on FUSE readiness).
 - `GET /__computerd/info` returns JSON with the selected FUSE backend, mount point, and bound port.
-- `GET /__computerd/stats` returns JSON with DOFS table row counts, total inline and blob byte sizes, the orphan-blob subset, and process resident memory. Useful for watching how the store grows under load.
+- `GET /__computerd/stats` returns JSON with DOFS table row counts, total inline and blob byte sizes, the orphan-blob subset, process resident memory, and the store's own size and free-page count. Useful for watching how the store grows under load.
+- `POST /__computerd/checkpoint` folds the store's write-ahead log back into the database file and returns `{ walFrames, sizeBytes, durationMs }`. For a host about to take a disk snapshot. Any other method returns `405`.
 - `GET /` returns `200 OK` with an empty JSON object: `{}`.
 - `GET /api` upgrades to a WebSocket carrying the capnweb RPC surface backed by `@cloudflare/computer-rpc`. This is the container's only RPC carrier. A request without an `Upgrade` header returns `400`; a handshake naming an unsupported `Sec-WebSocket-Version` returns `426` along with the versions the server speaks.
 - `GET /api/watermarks` returns JSON with `currentRev`, `pushRev`, and `fetchCursor`, read through the same `watermarks()` the wire serves. For samplers that want a few numbers without opening a session. It sits under `/api` because it reads the workspace surface; `/__computerd` is for daemon introspection.
@@ -45,7 +46,41 @@ Current filesystem support:
 - Unsupported FUSE operations return `ENOSYS` to the kernel; the binding logs a one-shot warning per operation.
 - capnweb RPC over `/api` exposes the workspace database and an `exec` runner to clients.
 - Synchronization is driven by whoever holds the other end of the session. The daemon serves `SyncRPC`; it does not run a sync loop of its own.
-- No on-disk persistence yet — the in-memory VFS is rebuilt on each start, and the host pushes state back after a restart.
+- Optional on-disk storage through `COMPUTERD_DB`. Unset, the in-memory store is rebuilt at each start and the host sends its state back. Set to a path, the store survives a restart and the host sends only what changed. See [On-disk store](#on-disk-store).
+
+## On-disk store
+
+`COMPUTERD_DB` picks where the workspace lives:
+
+```sh
+COMPUTERD_DB=memory                      # default: in-memory, rebuilt on every start
+COMPUTERD_DB=/var/lib/computerd/state.db # on-disk, survives a restart
+```
+
+The path must be absolute and must not sit inside `MOUNT_POINT`. A database file that the FUSE mount also shows would feed its own writes back to itself. `computerd` refuses to start on either mistake.
+
+Keeping the store on disk saves more than the files. The sync positions live in the same database (`_vfs_watermark`, `_vfs_fetch_cursor`, `_vfs_push_cursor`). Without them a restarted daemon looks further behind than it is, so the durable object sends every path in the workspace again. With them it sends only what changed. On a large workspace that is the difference between resending everything and doing nothing.
+
+The exec log does not persist. `computerd_exec_log` and `computerd_exec_meta` are cleared at every start, because the processes they describe are gone.
+
+### Settings
+
+A file store opens with write-ahead logging, `synchronous = normal`, a 64 MiB page cache, a 256 MiB memory map, temporary tables in memory, and a five-second busy timeout.
+
+`synchronous = normal` flushes to disk when the log is folded back rather than on every commit. That is safe here: the durable object holds the real copy, so a host crash that loses the last few writes costs a resend, not data.
+
+### Checkpointing
+
+- `POST /__computerd/checkpoint` folds the write-ahead log back into the database file and returns `{ walFrames, sizeBytes, durationMs }`. Call it before taking a disk snapshot, so the snapshot holds one file rather than a file plus a log.
+- The same thing happens on `SIGTERM` and `SIGINT`, after the FUSE unmount. The order matters: the FUSE driver writes buffered bytes to the database when it releases a file, so unmounting first is what gets those bytes in.
+- `GET /__computerd/stats` reports `store_size_bytes` and `store_freelist_count` next to the table counts, so you can watch the file grow.
+
+### Limits
+
+- Take snapshots between commands, not during one. A checkpoint keeps the database itself valid, but a snapshot taken mid-command catches a half-written workspace. A half-finished `npm install` is still half-finished after a restore.
+- An older `computerd` exits with `EIO` rather than open a store written by a newer one. Restoring onto an older release fails loudly, which is intended.
+- Mount rows (`_vfs_mounts`) come back with the store and may be out of date until the durable object rebuilds them.
+- If the store is further ahead than the durable object, the daemon cannot fix it: it answers sync requests but never starts one. Begin from a fresh disk instead.
 
 ## FUSE write model
 
@@ -115,6 +150,7 @@ Additional environment variables:
 EXEC_LOG_MAX_BYTES=1048576        # cap the in-memory exec log buffer (bytes)
 RPC_CLIENT_SECRET=<secret>        # require Authorization: Bearer <secret> on every route but /health
 COMPUTER_VAR_NODE_ENV=production  # forwarded into exec as NODE_ENV
+COMPUTERD_DB=/var/lib/computerd/state.db  # on-disk store; "memory" or unset keeps it in memory
 ```
 
 `FUSE_MOUNT=auto` is the friendly default: if `/dev/fuse` (or macFUSE) is available `computerd` mounts a real FUSE filesystem, otherwise it transparently falls back to the userspace shim. Pin the value (`fuse` / `macfuse` / `shim` / `none`) when a test needs to assert a specific code path.

packages/computerd/bench-results.md

@@ -1,4 +1,119 @@
-# FUSE mount option benchmarks
+# Store and FUSE mount option benchmarks
+
+## In-memory store versus file-backed store
+
+Numbers from `script/store-compare.mjs`, which
+drives the dofs filesystem directly against both storage backends.
+It deliberately skips FUSE so a difference here is the store and
+nothing else. 2,000 files in one directory, Node 24 on a Linux
+container.
+
+| Store | create 2000 | stat cold | stat warm | readdir x50 |
+|---|---:|---:|---:|---:|
+| memory | 180.9 ms | 1580.4 ms | 12.1 ms | 155.4 ms |
+| file, 64 MiB cache | 659.1 ms (3.64x) | 1558.7 ms (0.99x) | 15.1 ms (1.25x) | 143.1 ms (0.92x) |
+| file, 256 MiB cache | 650.0 ms (3.59x) | 1509.8 ms (0.96x) | 13.8 ms (1.14x) | 146.9 ms (0.95x) |
+
+Two findings, one of which contradicts what we assumed when writing
+the plan.
+
+**Metadata reads do not regress.** Cold `stat` of 2,000 paths is a
+wash (0.96x to 0.99x), and `readdir` is if anything slightly faster
+on the file store. The plan predicted this was where a file-backed
+store would hurt. It does not, because the working set here is about
+1.4 MiB — small enough to sit entirely in SQLite's page cache, so the
+reads never reach the disk. Warm `stat` is 1.14x to 1.25x slower,
+which is the resolve cache doing its job in both cases and the
+remaining difference being page-cache lookup overhead rather than
+input or output.
+
+**Writes are the real cost, and the cause is fsync.** Creating 2,000
+files is 3.6x slower on the file store. Varying `synchronous` isolates
+it:
+
+| `synchronous` | create 1000 files |
+|---|---:|
+| `full` | 444.6 ms |
+| `normal` | 291.9 ms |
+| `off` | 148.1 ms |
+
+`off` matches the in-memory store, so the gap is entirely the cost of
+flushing to disk. `normal` is the shipped default and already buys
+back a third of `full`. Anything faster trades durability for speed,
+which is defensible here because the durable object is the source of
+truth, but `off` risks a corrupt database on host loss rather than
+merely losing recent transactions, so it stays off the table.
+
+Sweeping the cache budget changes almost nothing. At 6,000 files the
+database is 3.8 MiB; squeezing the cache to 2 MiB, so the working set
+genuinely cannot fit, still leaves cold `stat` at 0.99x:
+
+| Store | create 6000 | stat cold | stat warm | readdir x50 |
+|---|---:|---:|---:|---:|
+| memory | 391.9 ms | 14389.1 ms | 34.4 ms | 437.1 ms |
+| file, 2 MiB cache | 1682.0 ms (4.29x) | 14177.7 ms (0.99x) | 55.4 ms (1.61x) | 474.4 ms (1.09x) |
+
+That is the interesting result. The prediction was that a cache too
+small for the tree would turn every resolve into a `pread` and wreck
+the metadata numbers. It does not, because cold `stat` is dominated by
+the resolve walk itself rather than by fetching pages, and the
+operating system's own page cache absorbs what SQLite evicts. Warm
+`stat` is where the difference shows, and it is 20 microseconds per
+operation on a path that is already cheap.
+
+## Through a real FUSE mount
+
+The numbers above isolate the storage layer. These run the same
+comparison through `script/fs-bench.sh` against a real kernel FUSE
+mount, with `computerd` started on the host (`FUSE_MOUNT=fuse`), and
+`/tmp` as the baseline. REPS=2, WARMUP=1.
+
+| Scenario | memory store | file store | baseline |
+|---|---:|---:|---:|
+| stat 1000 files | 2777.3 ms (1.10x) | 3114.9 ms (1.22x) | ~2540 ms |
+| create 1000 files | 989.4 ms (0.98x) | 1178.7 ms (1.17x) | ~1010 ms |
+| write 64 MiB | 238.1 ms (11.14x) | 221.9 ms (12.69x) | ~19 ms |
+| overwrite 64 MiB | 294.3 ms (26.16x) | 304.6 ms (29.30x) | ~11 ms |
+
+Large-file input and output is unchanged between the two stores, which
+is what the storage-layer numbers predicted: those paths are dominated
+by chunking and the FUSE round trip, so the store barely registers.
+The small-file scenarios cost 10 to 20 percent more on disk. That is a
+real regression, and smaller than the 3.6x the storage-layer create
+number would suggest on its own, because FUSE overhead dilutes it.
+
+## Restore time
+
+What the on-disk store buys, measured by `script/restore-time.mjs`. It
+times the interval a host actually waits: from a healthy daemon to a
+workspace the peer agrees is current, meaning connect, reconcile
+watermarks, and push whatever the peer believes is missing.
+
+| Tree | store | first boot | restart |
+|---|---|---:|---:|
+| 500 files | memory | 454 ms (502 pushed) | 480 ms (502 pushed) |
+| 500 files | file | 451 ms (502 pushed) | **26 ms (0 pushed)** |
+| 3,000 files | memory | 3725 ms (3002 pushed) | 3749 ms (3002 pushed) |
+| 3,000 files | file | 4063 ms (3002 pushed) | **23 ms (0 pushed)** |
+
+An in-memory store re-ships the whole tree on every restart, so its
+restart cost tracks the tree size. A file store ships nothing, because
+the sync cursors came back with the files and the peer can see there
+is no difference to send. The saving is 18x at 500 files and 161x at
+3,000, and it keeps growing: the restore side stays flat at roughly
+25 ms while the memory side climbs with the workspace.
+
+This is the trade in one line. Small-file work costs 10 to 20 percent
+more, and a restart costs a fixed 25 ms instead of a full replay.
+
+Caveats. These run on one Linux container, not on Cloudflare
+Containers hardware. The restore measurement drives the sync protocol
+directly rather than through a real durable object over a real
+network, so it captures the work avoided but not the round-trip
+latency a real host would also save. The full `cloudflare/sandbox-sdk`
+`npm install` comparison has not been run.
+
+## FUSE mount option benchmarks
 
 Numbers from running `script/run-fs-bench.sh` against the linux-x64
 `computerd` binary in a privileged docker container, with the bench's pure