silkd: advanced network mode, tun0 over a vsock packet stream

.github/workflows/silkd.yml

@@ -36,4 +36,4 @@ jobs:
 
       - name: root-only tests
         working-directory: silkd
-        run: sudo -E env "PATH=$PATH" "HOME=$HOME" cargo test --locked --test git_e2e --test fs_e2e --test find_e2e -- --ignored
+        run: sudo -E env "PATH=$PATH" "HOME=$HOME" cargo test --locked --test git_e2e --test fs_e2e --test find_e2e --test tun_e2e -- --ignored --skip throughput

docs/advanced-network.md

@@ -0,0 +1,143 @@
+# Advanced network mode
+
+The **default network mode** of a none-lane guest is the proxy path: silkd
+relays `127.0.0.1:3128` and `127.0.0.1:1080` to the host's
+[guarded-egress](egress.md) proxy over vsock. The **advanced network mode**
+adds a full L3 path beside it for clients the proxy cannot carry (UDP, QUIC,
+WebRTC, clients that ignore proxy variables): silkd creates `tun0` in the
+guest and carries raw IP packets over one vsock stream to the host. The guest
+still has no NIC, so hibernate, archive, wake and move work as before.
+
+This page is the contract between silkd and the host. The host side design is
+[#471](https://github.com/cocoonstack/sandbox/issues/471).
+
+## Guest side
+
+| Item | Value |
+| --- | --- |
+| Device | `tun0`, `IFF_TUN \| IFF_NO_PI \| IFF_VNET_HDR`, owned by silkd |
+| Address | `100.64.0.2/30` in every guest; the host tells guests apart by their vsock stream, never by address |
+| Gateway | `100.64.0.1`, default route via `tun0` |
+| MTU | 1500 until the host hello names another |
+| IPv6 | none; silkd turns IPv6 off on `tun0` |
+| Packet stream | guest dials host vsock port **2051** (`<vsock_socket>_2051` on the host) |
+
+Both modes run at once. silkd keeps the proxy variables in every exec, so a
+client that honours them still reaches `127.0.0.1:3128` over loopback; all
+other traffic takes the default route through `tun0`. Loopback never enters
+`tun0`, so the guest DNS stub of #472 is unaffected; the tun path does not
+intercept DNS.
+
+### Turning it on and off
+
+- **`tun_up`** (control channel, port 2048): creates `tun0` with its address
+  and route, starts the packet stream, answers `done` or an `error` saying why
+  `tun0` could not be created. On a guest whose `tun0` is already up it ends
+  the current stream and dials again. sandboxd sends it once a claim of a
+  tun-enabled pool is live, and again after every wake, unarchive and move,
+  so the guest drops its old stream at once whatever the VMM did with it (on
+  Cloud Hypervisor the guest also notices by itself within about 200 ms of a
+  restore). Warm VMs never get it, so goldens are unchanged.
+- **`tun_down`** (control channel): stops the packet stream and removes `tun0`
+  with its routes before answering `done`, so clients fail at once with
+  "network unreachable".
+- **Boot and reconnect**: silkd dials 2051 at boot and after every stream it
+  sees end, and lets the host hello decide. At boot a refused dial or a failed
+  hello means off. Once `tun0` is up, a refused dial or a failed hello is
+  retried with backoff (50 ms doubling to 1 s); an off answer removes `tun0`.
+
+The guest switch is advisory. The guest user can be root, create its own TUN
+and dial 2051 directly, so the host enforces: it accepts a packet stream only
+for a tun-enabled claim, rewrites every source address, and closes the stream
+to revoke.
+
+## Wire protocol (version 1)
+
+All integers are little-endian.
+
+### Hello
+
+Each side sends one 12-byte hello first; the guest sends first, the host
+answers.
+
+| Offset | Size | Field | Guest | Host |
+| --- | --- | --- | --- | --- |
+| 0 | 4 | magic | `SLKT` | `SLKT` |
+| 4 | 2 | version | highest it speaks (1) | the version both use; ≤ the guest's |
+| 6 | 2 | offloads | offloads its tun accepts on write | offloads its tun accepts on write |
+| 8 | 2 | mtu | 0 | MTU for `tun0`; 0 keeps the current one |
+| 10 | 2 | flags | 0 | bit 0 `ON`: packet mode is on for this box |
+
+Flag bits 1 to 15 are reserved and sent as 0; a peer ignores a bit it does
+not know. A later revision uses one to open several streams per guest, one
+per `tun0` queue.
+
+A host that answers without `ON` closes the stream after the hello; the guest
+removes `tun0`. A host that refuses a guest answers without `ON` rather than
+closing silently. A guest that does not speak the host's version closes the
+stream. The guest gives up on a hello that does not arrive in 5 seconds.
+
+Offload bits are the kernel's `TUN_F_*` values: `0x01` CSUM, `0x02` TSO4,
+`0x04` TSO6, `0x08` TSO_ECN, `0x20` USO4, `0x40` USO6. Each side probes its
+kernel (USO needs Linux 6.2) and then enables, with `TUNSETOFFLOAD`, only the
+offloads both hellos carry; segmentation offloads are dropped when CSUM is not
+common. A side never builds a frame whose GSO type the peer did not offer;
+packets its kernel queued before a reconnect keep their old type, and a
+receiver may drop one it cannot take.
+
+### Frames
+
+After the hellos, both directions carry frames back to back:
+
+```
+u32 len | virtio_net_hdr (10 bytes) | IP packet
+```
+
+`len` counts the header and the packet: 10 to 65545 bytes. A length outside
+that range is a protocol error and ends the stream. The header is the plain
+10-byte `struct virtio_net_hdr` (`flags`, `gso_type`, `hdr_len`, `gso_size`,
+`csum_start`, `csum_offset`) exactly as `IFF_VNET_HDR` reads and writes it, so
+a GSO packet of up to 64 KiB crosses vsock as one frame and is segmented, if
+at all, by the receiving kernel. A packet the receiving kernel refuses is
+dropped like a packet lost on a link.
+
+Writers batch: every frame ready at the moment goes out in one stream write.
+Readers parse every complete frame one stream read returns; the io_uring path
+hands them to the TUN in batched submissions.
+
+### Reconnect
+
+There is no session state beyond the hello. After any stream loss the guest
+dials again and both sides renegotiate offloads and MTU, so a move to a host
+with a different kernel or MTU is safe. Packets in flight during the gap are
+lost; TCP retransmits.
+
+## Host side
+
+The host side is sandboxd; its design is
+[#471](https://github.com/cocoonstack/sandbox/issues/471). Whatever it does
+with the packets, it owes the guest this contract:
+
+- **One stream per claim.** The packet stream arrives on the claim's own
+  vsock socket, so the stream, not any address in a packet, names the claim.
+  Every guest uses `100.64.0.2`; the host must not trust a source address the
+  guest wrote and must deliver only the claim's own return traffic on its
+  stream, with the destination set back to `100.64.0.2`.
+- **Enforcement on the host.** The host answers `ON` only for a claim that is
+  allowed packet mode, and answers off, closes the stream or refuses the dial
+  otherwise; revocation closes the stream. Nothing depends on the guest
+  obeying `tun_down`.
+- **Checksums after a rewrite.** A host that changes an address updates the
+  IPv4 header checksum and the L4 checksum incrementally. For a frame with
+  `VIRTIO_NET_HDR_F_NEEDS_CSUM` the L4 checksum sits at
+  `csum_start + csum_offset` and holds the uncomplemented pseudo-header sum;
+  a UDP checksum of 0 stays 0. With GSO this happens once per frame.
+- **Offloads it offered.** The host sends only GSO types the guest's hello
+  offered and segments the rest itself.
+
+## Limits
+
+- IPv4 only.
+- One stream per guest; a guest kernel without io_uring, or `SILKD_TUN_IO=epoll`
+  in silkd's environment, uses the epoll path.
+- Windows guests run no silkd and get no advanced network mode.

docs/index.md

@@ -59,6 +59,8 @@ vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC.
 - [Security model](security.md) — trust boundaries, what a compromised
   guest can reach, the deployment assumptions the guarantees rest on, and
   the honest limitations
+- [Advanced network mode](advanced-network.md) — the L3 `tun0` path beside the
+  proxies, and its vsock packet protocol
 - [silkd](silkd.md) — the in-guest daemon: protocol, verb reference, lanes
   and limits
 - [Performance](performance.md) — measured latencies and the environments

docs/silkd.md

@@ -58,6 +58,7 @@ suites — a frame only one side can parse fails CI.
 | git | `git_clone {url, path, branch?, depth?, auth?}` / `git_status {path}` / `git_add {path, files}` / `git_commit {path, message, author}` / `git_push {path, auth?}` / `git_pull {path, auth?}` / `git_branch {path, action, name?}` | `git_status_result` (porcelain v2), `git_commit_result{hash}`, `git_branches{current, branches}` for `git_branch {action: "list"}`; every other verb terminates with `done`. A status past about 1 MiB of entries carries `truncated: true` with the head of the list, so one frame never exceeds the cap. `auth` is injected as an in-memory header, never written to guest disk |
 | port | `port_forward {port}` | relays guest TCP 127.0.0.1:port over this connection: `ready` once connected, then `data` both ways. Each direction ends on its own: `data_end` half-closes the guest socket, `done` reports the guest closing its write side, and the RPC ends once both have. A client that only drops the connection leaves the relay open until the guest side closes or writes. Works on both lanes — the no-network lane's only way in |
 | lsp | `lsp_start {language, root?}` / `lsp_request {server_id}` / `lsp_stop {server_id}` | a broker for the language server a flavor image ships: `lsp_start` spawns the argv named in `/etc/silkd/lsp.d/<language>` (absent on the base image → `not_found` naming the flavor) and answers `lsp_started{server_id}`; `lsp_request` attaches the JSON-RPC byte stream (`ready`, then `data` both ways — silkd pipes bytes, it never parses LSP; `data_end` half-closes the server's stdin) and the stream ending reaps the server (v1 single-shot); `lsp_stop` kills it early, and a server nothing attaches within 5 minutes is reaped |
+| tun | `tun_up` / `tun_down` | the [advanced network mode](advanced-network.md) switch: `tun_up` creates `tun0` and starts its packet stream on vsock 2051, `tun_down` removes `tun0` and its routes; both answer `done`, `tun_up` an `error` when `tun0` cannot be created. Advisory: the host enforces |
 | misc | `info` | `info{version, uptime_secs, procs, sessions}` — sandboxd uses it for readiness; there is no public SDK method. Distinct from the control plane's `GET /v1/info`, which reports node pools and claims |
 
 ## Error kinds
@@ -94,6 +95,11 @@ forwards the image-baked proxy variables (`http_proxy` and friends) into every
 exec, so unconfigured clients use the relay without being told; a lane with its
 own routed network never gets them.
 
+In the [advanced network mode](advanced-network.md) silkd also owns `tun0`
+(`100.64.0.2/30`, default route) and carries its packets to the host over
+vsock (`CID2:2051`); it asks the host at boot and after every lost stream, and
+the proxy variables stay set beside it.
+
 ## Claim env
 
 The host writes a claim's [`guest` env entries](sandboxd-api.md#getputpatch-v1sandboxesidenv)

protocol/wire/fixtures/v1/req_tun_down.json

@@ -0,0 +1 @@
+{"v":1,"op":"tun_down"}

protocol/wire/fixtures/v1/req_tun_up.json

@@ -0,0 +1 @@
+{"v":1,"op":"tun_up"}

protocol/wire/frame.go

@@ -90,6 +90,8 @@ var (
 		"lsp_start":      decodeReq[LspStart],
 		"lsp_request":    decodeReq[LspRequest],
 		"lsp_stop":       decodeReq[LspStop],
+		"tun_up":         decodeReq[TunUp],
+		"tun_down":       decodeReq[TunDown],
 		"data":           decodeReq[Data], //nolint:goconst // wire tag shared with the response type by design
 		"data_end":       decodeReq[DataEnd],
 	}
@@ -435,6 +437,16 @@ type LspStop struct {
 
 func (LspStop) Op() string { return "lsp_stop" }
 
+// TunUp brings tun0 up and starts its packet stream on vsock port 2051; the host hello may still turn it off.
+type TunUp struct{}
+
+func (TunUp) Op() string { return "tun_up" }
+
+// TunDown removes tun0 and its routes; the host, not the guest, enforces packet mode.
+type TunDown struct{}
+
+func (TunDown) Op() string { return "tun_down" }
+
 // Data carries one chunk of an upload stream (FsWrite/FsPush payloads).
 type Data struct {
 	Data []byte `json:"data"`

protocol/wire/frame_test.go

@@ -58,8 +58,8 @@ func TestFixtureCorpusRoundTrips(t *testing.T) {
 		seen++
 	}
 
-	if seen != 61 {
-		t.Fatalf("fixture corpus: %d frames, want exactly 61", seen)
+	if seen != 63 {
+		t.Fatalf("fixture corpus: %d frames, want exactly 63", seen)
 	}
 }
 

silkd/Cargo.toml

@@ -18,9 +18,12 @@ serde_json = "1"
 tokio = { version = "1", features = ["fs", "rt-multi-thread", "macros", "io-util", "net", "process", "sync", "time"] }
 
 [target.'cfg(target_os = "linux")'.dependencies]
+io-uring = "0.7"
 tokio-vsock = "0.7"
+vsock = "0.5"
 
 [dev-dependencies]
+nix = { version = "0.31", features = ["sched"] }
 tempfile = "3"
 
 [lints.clippy]

silkd/src/lib.rs

@@ -16,5 +16,8 @@ pub mod server;
 pub mod session;
 pub mod sysutil;
 pub mod tree;
+pub mod tun;
+#[cfg(target_os = "linux")]
+mod tun_uring;
 pub mod vsock;
 pub mod watch;

silkd/src/main.rs

@@ -33,6 +33,8 @@ async fn main() {
     let dns = SocketAddrV4::new(net_egress::DNS_ADDR, net_egress::DNS_PORT);
     tokio::spawn(relay(dns, net_egress::DNS_HOST_VSOCK_PORT));
     tokio::spawn(datagrams(dns, net_egress::DNS_DATAGRAM_HOST_VSOCK_PORT));
+    #[cfg(target_os = "linux")]
+    silkd::tun::ask_at_boot();
     tokio::spawn(async {
         if let Err(e) = imds::serve().await {
             eprintln!("silkd imds: {e}");

silkd/src/proto.rs

@@ -169,6 +169,8 @@ pub enum Request {
     LspStop {
         server_id: String,
     },
+    TunUp,
+    TunDown,
     Data {
         #[serde(with = "b64")]
         data: Vec<u8>,
@@ -845,7 +847,7 @@ mod tests {
             seen += 1;
         }
         assert_eq!(
-            seen, 61,
+            seen, 63,
             "fixture corpus: adding a verb means adding its fixture"
         );
     }

silkd/src/server.rs

@@ -9,7 +9,7 @@ use tokio::task::JoinHandle;
 
 use crate::proc::{Chunk, Table, write_chunk};
 use crate::proto::{self, ErrorKind, ExecReq, ProcInfo, Request, Response};
-use crate::{exec, find, forward, fs, git, lsp, pty, session, tree, watch};
+use crate::{exec, find, forward, fs, git, lsp, pty, session, tree, tun, watch};
 
 /// Shared daemon state handed to every connection.
 pub struct State {
@@ -153,6 +153,14 @@ impl State {
                 return finish_feeder(feeder, res).await;
             }
             Request::LspStop { server_id } => self.lsp.stop(writer, &server_id).await,
+            Request::TunUp => tun_switch(writer, "tun up", tun::up).await,
+            Request::TunDown => {
+                tun_switch(writer, "tun down", || {
+                    tun::down();
+                    Ok(())
+                })
+                .await
+            }
             Request::PortForward { port } => {
                 let (rx, feeder) = spawn_feeder(reader);
                 let res = forward::run(port, rx, writer).await;
@@ -366,6 +374,21 @@ where
     (reader, None)
 }
 
+/// Runs a blocking tun switch off the async workers and answers `done` or the error.
+async fn tun_switch<W: AsyncWrite + Unpin>(
+    w: &mut W,
+    what: &str,
+    switch: fn() -> std::io::Result<()>,
+) -> std::io::Result<()> {
+    match tokio::task::spawn_blocking(switch)
+        .await
+        .unwrap_or_else(|e| Err(std::io::Error::other(e)))
+    {
+        Ok(()) => proto::write_frame(w, &Response::Done).await,
+        Err(e) => proto::error_frame(w, ErrorKind::Internal, format!("{what}: {e}")).await,
+    }
+}
+
 fn now_secs() -> u64 {
     SystemTime::now()
         .duration_since(UNIX_EPOCH)