sandboxd: advanced network mode, an L3 tun path over VXLAN for none-lane claims (design) #471

Motivation

A none-lane guest can only leave through the HTTP proxy (3128) and the SOCKS5 door (1080, CONNECT only, opt-in per pool). Two classes of workload cannot run on that:

• UDP: QUIC-only clients, WebRTC media and ICE, games, VoIP, peer-to-peer hole punching, user-run VPNs.
• Clients that ignore the proxy variables and take no --proxy-server style flag (some Electron apps). On a NIC-less guest they have no network at all.

The egress lane does not help: its NIC is locked default-deny, and an egress-lane claim cannot hibernate, archive, fork or move.

This issue proposes an L3 path beside the proxies. The guest gets a tun0, silkd carries raw IP packets over vsock, and sandboxd sends them over VXLAN to an exit, which is plain Linux configuration on the same host or on a separate gateway machine. There is no NIC and no per-claim host device, so hibernate, archive, wake and move keep working, and a live move can keep connections open when the exit is off the node. The companion silkd PR covers the guest side and the vsock packet protocol; this issue covers the host side. Name resolution is #472 (PR #473).

Naming: the proxy path stays the default network mode and should keep covering as much as it can. The tun path is an advanced network mode that a product turns on for the pools or claims that need full L3. sandboxd only provides the mechanism (a pool switch, a claim-level switch and an exit table); which users get it is product policy.

Shape

Two modes side by side. The HTTP proxy and the SOCKS5 door stay byte-identical. Clients that honour the proxy variables keep using 127.0.0.1:3128, which goes over loopback, not tun0; everything else takes the default route through tun0.

guest kernel -> tun0 -> silkd --vsock--> sandboxd --VXLAN--> exit (vxlan0, forward, NAT) -> uplink
                                             ^                   |
                                             +----VXLAN reply----+

sandboxd needs no TUN, no kernel routes and no NAT rules. It sends and receives VXLAN on an ordinary UDP socket.

Addresses

• Guest address: every guest's tun0 uses the same fixed address. The guest never learns its overlay address. Identity comes from the vsock socket path, as it does for the proxy.
• Overlay address: each tun-enabled claim gets a unique address from one configured overlay subnet. Give it a large block (a /8, or 240.0.0.0/4); it never appears on the internet, since every exit NATs. It must be unique because the exit's conntrack and NAT see only inner addresses and ports.
• MAC: derived from the overlay address (a locally administered prefix plus the IPv4 bytes), so no extra state.
• Allocation:
  • Single node (no mesh): sandboxd allocates from the subnet locally and stores the address in the claim record (claims.json), so a restart gets the same address back.
  • Multiple nodes (mesh set): tun mode requires meta_store with a cell. The address is allocated in a meta-store table (cell, address, claim, allocated and released times, unique on cell and address), unique across the cell. A config that enables tun with a mesh and no meta_store.cell refuses to load.
  • The address is allocated at claim time, kept while the claim is hibernated or archived, kept across a move, and freed at release. Reuse waits until conntrack entries for it can have expired at any exit; reusing earlier would hand a new claim the replies meant for the old one.
  • Fail closed: when the meta store cannot be reached, sandboxd allocates nothing and the claim gets no tun path (the default mode still works). It never falls back to a local allocation on a multi-node config.

Data path in sandboxd

• Outbound: overwrite the source address with the claim's overlay address, whatever the guest wrote; add an Ethernet header (source the claim's MAC, destination the exit's MAC) and a VXLAN header; send to the exit's VTEP address picked from the exit table. Checksums are updated incrementally; with GSO this happens once per segment, not per MSS.
• Inbound: VXLAN from an exit; map the inner destination address to its claim, rewrite it back to the fixed guest address, and hand the packet to the claim's vsock connection. Frames for an address this node does not host are dropped.
• Exit table: node config, reloadable, mapping a profile label to an exit VTEP address. A claim carries a profile label (set like a claim env entry, accepted on hibernated and archived claims); a claim without one uses default. sandboxd looks the label up and does not interpret it.
• Guest L2 is never forwarded: tun0 is an L3 device and sandboxd only wraps IP packets. Every Ethernet frame on the overlay, ARP included, is built by sandboxd.

ARP

• Responder: sandboxd answers ARP requests from an exit for the overlay addresses of claims on this node, and only those. After a claim moves away, the source stops answering for it.
• Gratuitous ARP on arrival: on claim, wake and move-in, sandboxd sends a gratuitous ARP for the claim to its exit, so the exit's VXLAN forwarding table and neighbour table point at this node at once, without waiting for the guest to send a packet.
• Periodic refresh: sandboxd re-sends a gratuitous ARP for each live tun claim every 30 to 60 seconds, so an exit never has to broadcast ARP. A learning VXLAN device has nowhere to flood a broadcast.

Exits are plain Linux configuration

An exit is not a sandboxd component. On a single host it runs on that host; on a fleet it can be a separate gateway machine with a fixed public address. Both use the same configuration, which deployment owns:

• a VXLAN device with learning on, carrying the overlay subnet and a fixed MAC that the exit table points at;
• net.ipv4.conf.<dev>.arp_accept=1, so a gratuitous ARP creates or updates a neighbour entry;
• neighbour timers longer than sandboxd's refresh interval;
• ip_forward, and one MASQUERADE or SNAT rule from the overlay subnet to the uplink;
• a forward rule that drops traffic from the VXLAN device back into it, so claims cannot reach each other through the exit;
• an input rule that drops everything from the VXLAN device addressed to the exit itself except ARP, so a claim cannot reach the exit's own services (sshd, metrics, management);
• a conntrack table sized for the claims it serves, with its fill level monitored;
• a firewall that accepts VXLAN only from node addresses, because any sender that reaches the VXLAN port could claim an address with a gratuitous ARP.

On a single host the exit's VXLAN device and sandboxd both need the VXLAN UDP port, and Linux binds the VXLAN socket to the wildcard address, so the local exit lives in its own network namespace connected by a veth pair. Verify this layout during implementation and document it as the reference single-host setup.

MTU: VXLAN with the inner Ethernet header adds 50 bytes. On a 1460-byte cloud underlay the guest's tun0 MTU is 1410; on a single host the underlay is local and this does not apply.

Policy

Filtering happens in sandboxd, not at the exit, so it reuses the existing evaluator and its hot reload.

• A flow table keyed by 5-tuple. The first packet of a flow is evaluated once, and later packets cost one lookup.
• EvalFlow(addr, port, proto) matches only bare host: "*" rules with no methods, secret or intercept, for the same reason EvalStream exists: a host-specific rule, a method filter or an injected credential has nothing to act on at L3, and honouring it there would silently downgrade it. Allowed ports on the tun path are therefore the union of the bare wildcard rules' ports. A pool without such a rule carries nothing on tun0.
• The internal-address guard applies unchanged (special-purpose ranges, the node's own addresses, egress_internal_allow), and additionally refuses destinations in the overlay subnet and the fixed guest address range, so a guest cannot address another claim. Nodes behind a 1:1 cloud NAT also need that public address configured, since the NIC does not carry it.
• Per-claim limits: byte-rate and packet-rate token buckets, new-flow rate, concurrent flows, and a total byte quota whose counters persist with the claim.

Enforcement is host-side; the guest is advisory

The guest user can be root. A guest can create its own TUN and dial the packet port directly, so tun_down only removes the guest route to make clients fail fast. sandboxd enforces:

1. Accept-time check: the packet port accepts a connection only when the claim is tun-enabled: the pool switch (tun: true, config-owned, reloadable) and the claim not disabled. Otherwise sandboxd closes the connection without reading a packet.
2. Live revocation: disabling a claim closes its packet connections, drops its flow-table entries and stops answering ARP for it.
3. A claim-level switch settable through the API, accepted on hibernated and archived claims without waking them. The switch and the profile label are root-token only: the sandbox's own token can read them but never change them, so whoever holds a claim's token cannot turn on its tun path or pick another exit.
4. Durable state: the disabled flag and quota counters live in the claim record and, under a meta store, in the cell's row, so a restart, a wake or a move does not reopen a disabled claim.
5. Source rewrite is unconditional: a forged source never leaves; whatever the guest writes, the packet leaves with the claim's own overlay address.

Guest control: the host turns tun0 on with a tun_up verb on the existing control channel and off with tun_down. silkd also asks on the packet port at boot and on every reconnect. Warm VMs are never tun-enabled; claiming one triggers tun_up. Goldens stay unchanged.

Audit

• One flow record per flow, at its end, in the existing audit stream: claim, tenant, path: "tun", protocol, destination address and port, decision, bytes each way, start and end, exit profile. Proxy records keep their current shape; a new event kind avoids breaking current consumers of egress.
• Destination name, best effort: the per-claim name map from silkd, sandboxd: a guest DNS stub over vsock, shared by the default and advanced network modes #472, plus a peek at the TLS SNI or HTTP Host of a flow's first payload packet without terminating it. DoH, ECH and VPN tunnels hide the name; the address and port are always recorded.
• Abuse attribution: the exit's NAT mapping (overlay address and 5-tuple to public address and port) must be recordable, so a complaint about a public address, port and time resolves to a claim through its overlay address.

Performance

• io_uring as the default packet path, epoll as the fallback when io_uring setup is refused.
• GSO/GRO on the guest TUN and the vsock stream, UDP GSO/GRO on the VXLAN socket, batched reads and writes.
• Measure on hardware before merging: throughput (TCP bulk and UDP small packets), per-packet cost, CPU per Gbps, with a local exit and with an exit on another machine, and the same numbers on the 3128 path as the baseline.
• Zero cost on the claim path for pools without tun: no address, no socket work, no flow table.
• VXLAN entropy: set the outer UDP source port from a hash of the inner 5-tuple, so NIC RSS and any ECMP on the underlay spread flows across queues and paths.
• Per-hop counters (silkd, vsock, sandboxd, VXLAN socket) so a later change shows where time goes; record the numbers per commit.
• Fairness: a claim's packet-rate limit applies before any costly work on its packets, and a worker serving several claims schedules them fairly, so one flooding claim cannot starve the others on its worker.

Untrusted input

The packet port and the DNS port (#472) parse bytes the guest fully controls: frame headers, virtio_net_hdr fields (gso_size, hdr_len, checksum offsets), IPv4 headers and DNS messages.

• Every length and field is bounds-checked before use. Packets this design does not handle (non-IPv4, IP options, fragments, a GSO type it does not expect) are dropped by an explicit rule, in the first version, and counted.
• Buffers per claim are capped, so a claim cannot grow sandboxd's memory.
• The frame, virtio_net_hdr, IPv4 and DNS parsers each get a fuzz target, run in CI.

Performance roadmap

The first version keeps two user-space hops (silkd in the guest, sandboxd on the host) and a vsock stream between them. Keep the transport separable from the rest (identity, policy, rewrite, ARP, VXLAN, audit), so each step below replaces one piece without touching exits or audit:

1. Batching and offloads first (this issue): io_uring, GSO/GRO end to end, UDP GSO/GRO on the VXLAN socket, one worker per CPU sharded by claim. Measure before anything else.
2. Kernel encapsulation: hand inner frames to one shared kernel VXLAN device in metadata mode instead of building VXLAN in user space, so the NIC's tunnel segmentation and checksum offloads apply. One shared device, never per claim.
3. AF_XDP on the underlay: if packet rate on the VXLAN side is the limit, move sandboxd's VXLAN traffic to an AF_XDP socket on a dedicated NIC queue.
4. Exit fast path: nftables flowtables on exits, so established flows skip the full netfilter path (hardware flowtable offload where the NIC supports it); more exits sharded by profile or hash when one machine's bandwidth is the limit.
5. More vsock streams per claim: one packet stream per guest TUN queue, so silkd and sandboxd work a claim's traffic in parallel, with full-size GSO frames on each.

Every step stays inside the none-lane guarantee that the guest has no network device. Adding a guest device (a virtio-net device with a user-space backend, for example) is not on this roadmap: it changes the none-lane security model and cocoon's device, snapshot and migration behaviour, and would need its own cocoon-level design and security review.

Lifecycle

Open questions

• IPv6 on tun0: none in the first version.
• QUIC Initial SNI for audit.
• The API shape of the claim-level switch, the profile label and the quota.

Out of scope

• Gateway machines themselves (provisioning, fixed public addresses, failover) and what each profile means (direct, residential, dedicated addresses). sandboxd provides the overlay address, the VXLAN encapsulation, ARP and the exit table.
• Windows guests: no silkd there.

Acceptance

• A tun-enabled claim reaches a TCP and a UDP echo service through tun0 with no proxy variables, through a local exit and through an exit on another machine; a non-tun pool's claim gets no packet port.
• Guest-side spoofing (raw socket with a forged source) leaves with the claim's own overlay address; a packet to another claim's overlay address is dropped by sandboxd and by the exit.
• A guest that recreates tun0 after tun_down on a disabled claim gets its packet-port dial refused; revocation kills an established flow within a second; the disabled state survives a sandboxd restart and a hibernate/wake.
• Hibernate, wake and move still pass on a tun-enabled claim. With an exit on another machine, an established TCP connection and a UDP flow survive a live move.
• Two nodes cannot hand out the same overlay address; a multi-node config without meta_store.cell refuses to load.
• Performance numbers as listed above, compared with the 3128 path.
• Fuzz targets for the frame, virtio_net_hdr and IPv4 parsers run without findings for a set budget; malformed frames (bad lengths, absurd gso_size, truncated headers) are dropped and counted, never crash the pump.
• With the meta store unreachable on a multi-node config, a new tun claim gets no address and no tun path, and the default mode still works.
• The sandbox's own token is refused when it tries to change the tun switch or the profile label.
• A claim flooding its packet port does not raise the latency of another claim on the same worker beyond a stated bound.