Skip to content

fix(envd): open multiple TCP connections to the NFS proxy per mount - #3833

Open
tunsuy wants to merge 4 commits into
e2b-dev:mainfrom
tunsuy:contrib/oss-radar-3818
Open

tunsuy wants to merge 4 commits into
e2b-dev:mainfrom
tunsuy:contrib/oss-radar-3818

Conversation

@tunsuy

@tunsuy tunsuy commented Nov 17, 2026 •

Copy link
Copy Markdown

Fixes #3818

Problem

The NFS client multiplexes every request of a mount over a single TCP connection by default, and the nfs proxy serves each connection's requests in turn. One slow operation — a READDIR over a directory with a very large number of files, or a synchronous write — therefore blocks every other request queued behind it on that connection. From inside the sandbox, the whole volume looks unresponsive until the slow operation completes.

Change

Add nconnect=8 to the default NFS mount options in envd, so the guest kernel spreads one mount's requests over up to 8 TCP connections to the nfs proxy:

  • The proxy already serves every connection in its own goroutine, so a handful of extra connections costs it little. The extra sockets are also established lazily, only while requests are outstanding.
  • The kernel accepts nconnect values of 1–16; an out-of-range value fails the whole mount, hence the guard test below.
  • The proxy resolves NFSv3 file handles per mount, not per connection, so the extra connections need no additional MOUNT.

Tests

  • New TestNFSOptionsCarryAValidNconnect asserts nconnect is present in nfsOptions and within the kernel-accepted 1–16 range (an out-of-range value would fail the mount).
  • go test ./internal/api/ -count=1 from packages/envd — the package passes, including the existing TestShouldRemountNFS.

8 sits in the middle of the kernel-accepted 1–16 range and covers the typical fan-out of the parallel workloads that hit this bug (builds, make -j, npm, multi-process agents), while the worst-case per-mount cost stays negligible: the kernel establishes the extra connections on demand, and each one costs the proxy ~25KB (two goroutines plus the bufio pair).

Without nconnect the guest kernel multiplexes every request of an NFS
mount over a single TCP connection, and the proxy serves each
connection's requests in turn. One slow operation -- a READDIR of a big
directory, a synchronous write -- therefore blocks every other request
queued behind it on that connection, stalling unrelated work on the
whole volume.

Mount volumes with nconnect=8 so the kernel spreads outstanding requests
over several connections; the go-nfs server already serves each accepted
connection in its own goroutine, so the proxy needs no change. The
connections are established lazily and only while requests are
outstanding, and the proxy resolves NFSv3 handles per mount rather than
per connection, so the extra connections need no additional MOUNT.

Fixes e2b-dev#3818

Co-Authored-By: Claude Code <noreply@anthropic.com>
@cla-bot

cla-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

We require contributors to sign our Contributor License Agreement, and we don't have @tunsuy on file. You can sign our CLA at https://e2b.dev/docs/cla . Once you've signed, post a comment here that says '@cla-bot check'

@tunsuy tunsuy changed the title fix: 同上:E2B 开源沙箱运行时,envd 与控制面为 Go 编写,近一周 Issue/PR 持续更新。 fix(envd): open multiple TCP connections to the NFS proxy per mount Nov 17, 2026
@tunsuy

tunsuy commented Nov 17, 2026

Copy link
Copy Markdown
Author

@cla-bot check

@cla-bot cla-bot Bot added the cla-signed label Nov 17, 2026
@cla-bot

cla-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

The cla-bot has been summoned, and re-checked this pull request!

@tunsuy

tunsuy commented Nov 17, 2026

Copy link
Copy Markdown
Author

Hi @ryan-sana, this PR is ready for review — it fixes #3818 (a slow NFS operation, e.g. a READDIR of a huge directory, blocks every other request on the mount due to head-of-line blocking) by adding nconnect=8 to the default NFS mount options in envd.

Status:

  • CLA signed, verification/cla-signed passing
  • Branch up to date with main (no conflicts)
  • go test ./internal/api/ passes in packages/envd, including the new TestNFSOptionsCarryAValidNconnect guard

No server-side change is needed — the nfs proxy already serves each connection in its own goroutine. Happy to adjust the connection count or add more tests if preferred.

@tunsuy

tunsuy commented Nov 17, 2026

Copy link
Copy Markdown
Author

@ValentaTomas @jakubno @dobrac Looping in the code owners as well (following up on my ping to @ryan-sana above) — this PR is ready for review:

  • Fixes Slow NFS Operations Block Subsequent Requests #3818: one slow NFS operation (e.g. READDIR on a huge directory) blocks every other request on the mount, because the guest NFS client multiplexes the whole mount over a single TCP connection by default.
  • Change: adds nconnect=8 to the default mount options in envd, so the kernel spreads requests over up to 8 connections; the go-nfs proxy already serves each connection in its own goroutine, so this costs it little.
  • Tests: new TestNFSOptionsCarryAValidNconnect pins the option and the kernel-accepted 1–16 range; go test ./internal/api/ passes.
  • Status: CLA check green, branch up to date with main, MERGEABLE.

Happy to adjust the value or add benchmarks if you'd like data for other values in the 2–16 range.

@ryan-sana

Copy link
Copy Markdown
Contributor

@tunsuy

Have you explored the cost of running 8 connections of various layers? In an overload scenario what how would you avoid performance collapse? We are introducing more concurrent work and assuming there is always more capacity.

I understand your HOL case, and it is a nice find, my concern is the consequences of increasing to 8. In fact, why not 4? or 2? how did you pick 8?

Cheers
Ryan

@tunsuy

tunsuy commented Nov 18, 2026 •

Copy link
Copy Markdown
Author

Hi @ryan-sana — thanks, fair questions. Here's the cost model (numbers from the go-nfs fork and the proxy code):

What the extra connections cost

  • The proxy already serves every connection with a dedicated goroutine and has no connection cap (go-nfs/server.go: go c.serve() per accept). Per connection: 2 goroutines (serve + write serializer, 2KB initial stacks) plus 4KB bufio read/write buffers — order of 25KB. Even the worst case stays small: 200 sandboxes (max-sandboxes-per-node) × 8 = 1600 connections ≈ 40MB per node.
  • Guest side: the kernel establishes nconnect connections on demand — an idle or lightly-loaded mount holds one or two, not eight. The value is a concurrency ceiling, not a fixed cost: a mount only pays for the connections its workload actually uses.
  • The proxy issues plain syscalls into a host directory tree (PERSISTENT_VOLUME_MOUNTS), i.e. local storage in the standard deployment — the same storage that already absorbs every sandbox's IO on the node concurrently.

Why overload behavior doesn't change

  • nconnect adds no work; it only lets requests the guest was already generating overlap. Total request volume is workload-driven and identical.
  • go-nfs processes each connection strictly serially, so a mount's maximum in-flight FS operations is exactly nconnect — bounded concurrency, with the kernel capping the option at 16 (out of range fails the whole mount; the guard test pins this).
  • Back-pressure is per-connection and unchanged: each connection carries its own sunrpc slot table and TCP congestion window. If the proxy stalls, every connection stops advancing exactly like the single connection does today — nothing accumulates unboundedly.
  • Shared proxy state (mountcache RWMutex, dirverifier LRU) is already node-wide concurrent-safe; connections from one mount don't introduce a new contention class.

Why 8

  • The failure mode being fixed (Slow NFS Operations Block Subsequent Requests #3818) is one slow op blocking everything else; at nconnect=2 just two concurrent slow ops recreate it. The workloads that hit this — parallel builds, make -j, npm, multi-process agents — routinely fan out to 4–16 concurrent ops, so 8 covers typical fan-out with headroom.
  • Because connections are on-demand and each costs ~25KB, a ceiling of 8 vs 4 costs an idle mount nothing, and only pays off exactly when the workload needs the parallelism — which is when the HOL bug bites.

@ryan-sana

Copy link
Copy Markdown
Contributor

Thanks for you clarification, I am not worried that much on the memory side, but on the network bandwidth mainly.

Can you also elaborate what would be the benefits in case of bandwidth limitations, where we throttle bandwidth? Would this still be helpful? If not what would be the impact you reckon?

@tunsuy

tunsuy commented Nov 19, 2026

Copy link
Copy Markdown
Author

Hi @ryan-sana — sure, here is how the two interact:

The throttle is per-NIC and connection-count-agnostic. The runtime's bandwidth limit is the Firecracker VMM-level token bucket on the guest NIC (tcpfirewall-egress-throttle-config → setTxRateLimit in fc/client.go — a bytes bucket plus a packets/ops bucket, default off). It caps the aggregate leaving the interface, not per-flow. nconnect spreads the same request stream over more sockets — it adds no bytes and no operations — so with the bucket on, the aggregate cap holds exactly as before: all connections draw from the same bucket, and there is no way to parallelize past it.

Under throttle the fix is if anything more valuable, not less. Throttling stretches the drain time of any large op: a 200MB WRITE that takes 2s at line rate takes 20s through a 10MB/s bucket. On a single connection that 20s is precisely the head-of-line window during which every other op on the mount is stuck — the #3818 failure, amplified. With the requests spread over connections, small ops draw interleaved tokens between the bulk op's packets and stay responsive; the bulk op finishes in the same total time. The bucket delays packets rather than dropping them, so the guest TCP just sees the same RTT inflation one connection sees today — no new loss/retransmit dynamics.

Direction: the configured limiter is Tx-only (only TxRateLimiter is ever set on the interface). NFS WRITE payloads and RPC requests are guest Tx — throttled; READ payloads return as guest Rx — untouched by this limiter. READ-side HOL exists on a single connection regardless (the proxy answers each connection's requests strictly in turn), and nconnect addresses that part independently of throttling.

Net effect when the throttle is on and saturated: total throughput unchanged; bulk ops share the capped bandwidth with the interleaved small ops instead of monopolizing it while they starve — which is the intended semantics of the fix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Slow NFS Operations Block Subsequent Requests

2 participants