Skip to content

perf(linux): batch TUN reads and trim message cleanup#139

Open
tionis wants to merge 2 commits into
encodeous:mainfrom
tionis:linux-performance-1
Open

perf(linux): batch TUN reads and trim message cleanup#139
tionis wants to merge 2 commits into
encodeous:mainfrom
tionis:linux-performance-1

Conversation

@tionis

@tionis tionis commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Improve Linux packet-rate performance by preserving batches across the TUN and
UDP send paths.

The Linux TUN reader now blocks for the first packet and then drains immediately
available packets until the caller's batch is full or the descriptor returns
EAGAIN. This introduces no batching timer.

The standard network bind also resets only the pooled message descriptors that
were used by the current operation instead of walking all 128 entries after
every send.

Changes

  • Refactor Linux TUN packet decoding into a reusable helper.
  • Block normally for the first TUN packet.
  • Drain additional immediately available packets with nonblocking reads.
  • Bound each drain to the caller-provided batch capacity.
  • Coordinate raw descriptor access through RawConn.Control so the descriptor
    remains valid across concurrent close operations.
  • Preserve virtio-net GSO splitting, offsets, packet ordering, and partial-batch
    error behavior.
  • Add a Linux socket-based test covering one-packet and multi-packet reads.
  • Reset only the message descriptors used by each standard-bind operation.
  • Track the maximum number of descriptors touched across the UDP GSO fallback
    retry path.

Performance

Tests ran over a gigabit Ethernet switch from Normandy, an x86_64 Debian laptop,
to Natanus, a Raspberry Pi 5 running Debian, using a direct Nylon route.

Each UDP workload used four iperf3 streams. Delivered throughput accounts for
reported packet loss.

Workload Baseline Final candidate
TCP, 4 streams 844–882 Mbit/s 877 Mbit/s
UDP, 1200-byte payload, 900 Mbit/s offered 437–505 Mbit/s 890–891 Mbit/s
UDP, 64-byte payload, 100 Mbit/s offered 30.0–30.9 Mbit/s 97.1–98.2 Mbit/s
UDP, 64-byte loss 69–70% 1.8–2.8%

For the 64-byte workload, delivered throughput improved by approximately 221%.
Normal-packet UDP improved by approximately 89%. TCP remained within the
bracketing baseline range.

The changes were also tested separately:

  • timer-free TUN draining raised 64-byte UDP delivery to approximately
    82–93 Mbit/s;
  • limiting message cleanup to used descriptors raised it further to
    approximately 97–98 Mbit/s.

Latency and CPU

The TUN drain has no timer: after the first blocking read it only consumes
packets already queued by the kernel.

Repeated idle probes remained in the same sub-millisecond range. Stable samples
had average RTTs around 0.5–0.7 ms, with no consistent candidate regression.

The baseline processed only about 30 Mbit/s of the offered small-packet traffic
and used roughly 30% of one CPU core. The optimized path processed approximately
87 Mbit/s during profiling and used roughly 88% of one core. CPU use therefore
scaled with useful delivered traffic rather than remaining idle while packets
were dropped.

The optimized profile showed:

  • Linux syscalls as the primary remaining cost;
  • the Linux TUN read path as the next packet-rate boundary;
  • metrics overhead reduced from approximately 9.3% to 3.5%;
  • pooled-message reset overhead absent from sampled CPU time;
  • encryption remaining a minor cost.

Multiqueue TUN support may be useful beyond the tested packet rate, but it is
more invasive and is intentionally outside this PR.

Validation

  • Full go test ./... suite passes.
  • go test -race ./polyamide/tun passes.
  • Complete Windows amd64 cross-build passes.
  • The focused Linux test verifies that one blocking read drains additional
    available packet boundaries without waiting for future packets.
  • git diff --check passes.

Copilot AI review requested due to automatic review settings July 24, 2026 15:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves Linux packet-rate performance by batching TUN reads (blocking for the first packet, then draining immediately available packets) and by reducing per-send cleanup work in the standard UDP bind’s pooled message descriptors.

Changes:

  • Refactors Linux TUN packet decoding into a reusable helper and adds a drain loop using RawConn.Control + nonblocking unix.Read.
  • Adds a Linux-only socket-based test to validate one- and multi-packet drain behavior.
  • Optimizes StdNetBind pooled message cleanup to reset only descriptors used by the operation (including across UDP GSO fallback retries).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
polyamide/tun/tun_linux.go Adds reusable packet read helper and drains additional immediately-available packets via nonblocking reads.
polyamide/tun/tun_linux_test.go Adds Linux test validating the “read then drain” batching behavior.
polyamide/conn/bind_std.go Limits pooled message cleanup to only the descriptors used by the current send/receive operation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread polyamide/tun/tun_linux_test.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants