tun: host packet loop throughput leads after #478 #479

Activity

1. CMGS commented on Nov 5, 2026

   

   CMGS

   on Nov 5, 2026

   ContributorAuthor

   More actions

   Measured three send-path variants against the #478 head on two bare-metal hosts. Arms were interleaved by swapping only the sandboxd binary, 4 GiB per leg, on a VM restored on a test CH build with a faster vsock backend (the build where the host side limits the tun path).

   variant	local-exit up Gbit/s	remote-exit up Gbit/s	local / remote down Gbit/s
   #478 (serial read → sendmsg)	2.57–2.61	2.56–2.60, and 4.20–4.23 in the last run	2.55–2.60 / 1.60–1.84
   pipelined sender goroutine + sendmmsg of all ready frames	2.21–2.23	4.11–4.24	2.42–2.53 / 1.53–1.62
   pipelined, one frame per send	2.19–2.22	2.56–2.60	2.46–2.51 / 1.60–1.68
   serial, batching frames already in the reader's buffer	2.60–2.62	2.56–2.59	2.57–2.60 / 1.56–1.72
   serial, batching while the socket has queued bytes (SIOCINQ)	2.11–2.14	2.54–4.40	2.54–2.56 / 1.60–1.71

   Findings:

   • Remote-exit upload is bimodal, about 2.6 or about 4.2 Gbit/s, independent of the build. The unchanged sandboxd: advanced network mode host side (VXLAN to exits, overlay addresses, root-only switch) #478 binary hit 4.20/4.23 in one run and 2.56–2.60 in others. Each connection's flow hash picks the outer source port and with it the NIC queue and CPU at the receiving exit. The exit host was also shared with another workload during part of the runs. So the earlier +60% for the pipelined variant is not attributable to it.
   • Every variant that overlaps or batches sends lowers local-exit upload by about 15% (2.6 → 2.1–2.2 Gbit/s), and costs sandboxd more CPU (4.4 → 5.4–6.0 s/GiB). Pipelining alone does it too.
   • Downloads did not move.

   Nothing from this is adopted. Next steps if this is picked up again:

   • pin the outer source port, or measure per-queue at the exit, to make remote upload reproducible before comparing builds;
   • measure on a gateway exit, not a local veth one;
   • look at the guest side: with a 16 MiB guest send buffer the queue fills and upload stays at about 2.8, so the per-stream limit may sit in silkd/CH rather than in sandboxd.
2. CMGS commented on Nov 6, 2026

   

   CMGS

   on Nov 6, 2026

   ContributorAuthor

   More actions

   More leads from the static audit of 2026-11-06 (PR #481), unmeasured:

   5. One extra 64 KiB copy per guest frame (tunnet/router.go read path): io.ReadFull copies each frame from the bufio reader into the link's buffer; Peek/Discard would forward from the bufio buffer directly. The docs' "one copy" holds only for the payload rewrite.
   6. USO is offered as USO4 only (router.go segmentable): the kernel enables UDP segmentation offload on tun0 only with both USO4 and USO6, so a guest never sends a UDP GSO frame and the host's UDP GSO path never runs. One token, but it needs a QUIC upload A/B.
   7. Per-packet activity stamp (fix: count TUN traffic as sandbox activity #480): every admitted inbound packet and successful send stores the claim's last-activity time; it now reuses the flow check's timestamp, but the atomic store on a shared cache line is per packet in both directions.
   8. Queue bound by frames, not bytes (sendQueue): each queued inbound frame holds a 64 KiB pool buffer; MTU-sized buffers for frames the coalescer does not merge would cut the worst case.
3. CMGS commented on Nov 10, 2026

   

   CMGS

   on Nov 10, 2026

   ContributorAuthor

   More actions

   Lead 6 is done: #539 (v0.1.28) offers USO6 next to USO4, so the guest kernel enables UDP segmentation offload on tun0. Hardware A/B (8 runs per arm): guest UDP_SEGMENT to a server netns went from 972 to 2438 Mbit/s, and to the host veth from 878 to 1897 Mbit/s. Plain UDP did not change, and no frames were dropped. A guest frame now carries 42 datagrams.
   Still open: leads 1–5, 7 and 8. Lead 5 (the extra 64 KiB copy per guest frame through io.ReadFull) is still in tunnet/router.go.