diff --git a/.github/workflows/sandboxd.yml b/.github/workflows/sandboxd.yml index 3cc13e983..47463cbf9 100644 --- a/.github/workflows/sandboxd.yml +++ b/.github/workflows/sandboxd.yml @@ -81,6 +81,22 @@ jobs: GOWORK=off go test -race -count=1 -v -run 'TestTwoNodesOnOneMetaStore|TestNodesOfACell|TestAMetaStoreNodeWithoutACell' ./pool/ | tee -a "$RUNNER_TEMP/pg-test.log" ! grep -- '--- SKIP' "$RUNNER_TEMP/pg-test.log" + - name: Test the tun path through a kernel exit + working-directory: sandboxd + shell: bash + run: | + sudo apt-get update -qq && sudo apt-get install -y -qq --no-install-recommends nftables conntrack ethtool + GOWORK=off go test -c -o "$RUNNER_TEMP/udp.test" ./tunnet/udp/ + sudo SANDBOXD_TUN_EXIT_TEST=1 "$RUNNER_TEMP/udp.test" -test.run TestTrafficCrossesAKernelExit -test.v | tee "$RUNNER_TEMP/exit-test.log" + ! grep -- '--- SKIP' "$RUNNER_TEMP/exit-test.log" + + - name: Fuzz the tun parsers + working-directory: sandboxd + run: | + for t in FuzzFrameLen FuzzParseAndRewrite FuzzSegment FuzzVXLANAndARP; do + GOWORK=off go test -run '^$' -fuzz "^$t\$" -fuzztime 30s ./tunnet/ + done + - name: Set up Python uses: actions/setup-python@v7.1.0 with: diff --git a/docs/advanced-network.md b/docs/advanced-network.md index e32880dc7..837ab32b3 100644 --- a/docs/advanced-network.md +++ b/docs/advanced-network.md @@ -114,26 +114,157 @@ lost; TCP retransmits. ## Host side -The host side is sandboxd; its design is -[#471](https://github.com/cocoonstack/sandbox/issues/471). Whatever it does -with the packets, it owes the guest this contract: - -- **One stream per claim.** The packet stream arrives on the claim's own - vsock socket, so the stream, not any address in a packet, names the claim. - Every guest uses `100.64.0.2`; the host must not trust a source address the - guest wrote and must deliver only the claim's own return traffic on its - stream, with the destination set back to `100.64.0.2`. -- **Enforcement on the host.** The host answers `ON` only for a claim that is - allowed packet mode, and answers off, closes the stream or refuses the dial - otherwise; revocation closes the stream. Nothing depends on the guest - obeying `tun_down`. -- **Checksums after a rewrite.** A host that changes an address updates the - IPv4 header checksum and the L4 checksum incrementally. For a frame with - `VIRTIO_NET_HDR_F_NEEDS_CSUM` the L4 checksum sits at - `csum_start + csum_offset` and holds the uncomplemented pseudo-header sum; - a UDP checksum of 0 stays 0. With GSO this happens once per frame. -- **Offloads it offered.** The host sends only GSO types the guest's hello - offered and segments the rest itself. +sandboxd needs no TUN, no kernel routes and no NAT rules on the node. It +carries each claim's packets over VXLAN, on an ordinary UDP socket, to an +**exit**: plain Linux configuration on the same host or on a gateway machine. +The design is [#471](https://github.com/cocoonstack/sandbox/issues/471). + +``` +guest -> tun0 -> silkd --vsock 2051--> sandboxd --VXLAN--> exit (vxlan0, forward, NAT) -> uplink +``` + +- **Overlay address**: each tun-enabled claim gets a unique address from + `tun.subnet`, kept across hibernate, archive and move and freed at release. + A single node allocates it in memory and keeps it in `claims.json`, with no + disk write on the claim path; each release is appended to + `/tun-freed.log` so its reuse window outlives a restart. A node + with a `mesh` must have `meta_store.cell` and allocates it in the cell's + table, one round trip per claim, and stays off (proxy mode only) while the + meta store is unreachable. A freed address is reused after + `tun.reuse_after_seconds`, so the subnet's size over that window bounds the + sustained claim rate: a `/8` with the 5-day default carries about 38 new tun + claims a second across the cell. The claim's MAC is `02:00:` followed by the + address bytes. +- **Outbound**: the source address becomes the claim's overlay address + whatever the guest wrote; a GSO frame is cut into wire-sized packets with + full checksums; each gets an Ethernet header (claim MAC to the exit's MAC) + and a VXLAN header and leaves for the exit picked by the claim's profile + label (`default` when it has none). The outer source port follows a hash of + the inner 5-tuple. +- **Inbound**: a VXLAN packet from an exit's VTEP whose inner destination is + a claim on this node goes back to `100.64.0.2` and onto that claim's stream; + anything else is dropped. Inbound passes only for a flow the claim opened, + except that a TCP segment other than a bare SYN reopens its flow when the + claim's policy allows the peer, so a stream crosses a move, a wake or a + restart. In-order TCP segments of one flow whose checksums + verify are merged into one GSO frame for the guest. An ICMP error gets the + quoted packet's source rewritten to `100.64.0.2` too, so path MTU discovery + and port-unreachable reach the guest's socket. +- **ARP**: sandboxd answers ARP from the exits for its own claims only, + announces a claim with a gratuitous ARP when it is armed (claim, wake, + move-in), after a reload of `tun.exits` and every 45 seconds, and never + forwards a frame the guest built. +- **Policy**: a flow table keyed by 5-tuple; the first packet of a flow is + judged by `EvalFlow` (bare `host: "*"` rules only) and the internal-address + guard, which also refuses the overlay subnet and `100.64.0.0/30`. + Per-claim byte, packet and new-flow rates, a flow cap and a byte quota apply + before any further work on a packet; a GSO frame is charged for every packet + and header it puts on the wire. The limits and flows belong to the claim, so + a guest that reconnects its stream starts with what it had used. +- **Untrusted input**: frames, virtio-net headers and IPv4 headers are bounds + checked; non-IPv4, IP options, fragments, a packet or GSO segment larger + than the MTU the stream's hello announced, a GSO frame of more than 128 + packets, a GSO type that does not match the protocol and a partial checksum + outside the L4 checksum field are dropped and counted per reason. A stream + whose guest stops reading for 10 seconds is closed, freeing its queue. +- **Enforcement**: the packet port answers `ON` only for an armed, + tun-enabled claim; disabling a claim ends its stream, drops its flows and + stops its ARP answers. A guest of a `tun` pool that wakes, moves or reloads + into a state without a door (disabled, no address, the pool's `tun` off) is + sent `tun_down` in the background. The claim switch and the profile label change only + with the operator token, through + [`PUT /v1/sandboxes/{id}/tun`](sandboxd-api.md#put-v1sandboxesidtun). +- **Audit and usage**: with `audit_log` on, each flow writes one + `op:"tun_flow"` record when it ends ([fields](deploy.md#configuration)); with + `egress_usage_bytes` on, each allowed flow also appends an `egress_bytes` + usage event with its IP bytes both ways, the destination `ip:port` as its + reference, and `upstream` `tun/