Repository navigation
Conversation
Without nconnect the guest kernel multiplexes every request of an NFS mount over a single TCP connection, and the proxy serves each connection's requests in turn. One slow operation -- a READDIR of a big directory, a synchronous write -- therefore blocks every other request queued behind it on that connection, stalling unrelated work on the whole volume. Mount volumes with nconnect=8 so the kernel spreads outstanding requests over several connections; the go-nfs server already serves each accepted connection in its own goroutine, so the proxy needs no change. The connections are established lazily and only while requests are outstanding, and the proxy resolves NFSv3 handles per mount rather than per connection, so the extra connections need no additional MOUNT. Fixes e2b-dev#3818 Co-Authored-By: Claude Code <noreply@anthropic.com>
|
We require contributors to sign our Contributor License Agreement, and we don't have @tunsuy on file. You can sign our CLA at https://e2b.dev/docs/cla . Once you've signed, post a comment here that says '@cla-bot check' |
|
@cla-bot check |
|
The cla-bot has been summoned, and re-checked this pull request! |
|
Hi @ryan-sana, this PR is ready for review — it fixes #3818 (a slow NFS operation, e.g. a READDIR of a huge directory, blocks every other request on the mount due to head-of-line blocking) by adding Status:
No server-side change is needed — the nfs proxy already serves each connection in its own goroutine. Happy to adjust the connection count or add more tests if preferred. |
|
@ValentaTomas @jakubno @dobrac Looping in the code owners as well (following up on my ping to @ryan-sana above) — this PR is ready for review:
Happy to adjust the value or add benchmarks if you'd like data for other values in the 2–16 range. |
|
Have you explored the cost of running 8 connections of various layers? In an overload scenario what how would you avoid performance collapse? We are introducing more concurrent work and assuming there is always more capacity. I understand your HOL case, and it is a nice find, my concern is the consequences of increasing to 8. In fact, why not 4? or 2? how did you pick 8? Cheers |
|
Hi @ryan-sana — thanks, fair questions. Here's the cost model (numbers from the go-nfs fork and the proxy code): What the extra connections cost
Why overload behavior doesn't change
Why 8
|
|
Thanks for you clarification, I am not worried that much on the memory side, but on the network bandwidth mainly. Can you also elaborate what would be the benefits in case of bandwidth limitations, where we throttle bandwidth? Would this still be helpful? If not what would be the impact you reckon? |
|
Hi @ryan-sana — sure, here is how the two interact: The throttle is per-NIC and connection-count-agnostic. The runtime's bandwidth limit is the Firecracker VMM-level token bucket on the guest NIC ( Under throttle the fix is if anything more valuable, not less. Throttling stretches the drain time of any large op: a 200MB WRITE that takes 2s at line rate takes 20s through a 10MB/s bucket. On a single connection that 20s is precisely the head-of-line window during which every other op on the mount is stuck — the #3818 failure, amplified. With the requests spread over connections, small ops draw interleaved tokens between the bulk op's packets and stay responsive; the bulk op finishes in the same total time. The bucket delays packets rather than dropping them, so the guest TCP just sees the same RTT inflation one connection sees today — no new loss/retransmit dynamics. Direction: the configured limiter is Tx-only (only Net effect when the throttle is on and saturated: total throughput unchanged; bulk ops share the capped bandwidth with the interleaved small ops instead of monopolizing it while they starve — which is the intended semantics of the fix. |
Fixes #3818
Problem
The NFS client multiplexes every request of a mount over a single TCP connection by default, and the nfs proxy serves each connection's requests in turn. One slow operation — a
READDIRover a directory with a very large number of files, or a synchronous write — therefore blocks every other request queued behind it on that connection. From inside the sandbox, the whole volume looks unresponsive until the slow operation completes.Change
Add
nconnect=8to the default NFS mount options in envd, so the guest kernel spreads one mount's requests over up to 8 TCP connections to the nfs proxy:nconnectvalues of 1–16; an out-of-range value fails the whole mount, hence the guard test below.Tests
TestNFSOptionsCarryAValidNconnectassertsnconnectis present innfsOptionsand within the kernel-accepted 1–16 range (an out-of-range value would fail the mount).go test ./internal/api/ -count=1frompackages/envd— the package passes, including the existingTestShouldRemountNFS.8 sits in the middle of the kernel-accepted 1–16 range and covers the typical fan-out of the parallel workloads that hit this bug (builds,
make -j,npm, multi-process agents), while the worst-case per-mount cost stays negligible: the kernel establishes the extra connections on demand, and each one costs the proxy ~25KB (two goroutines plus the bufio pair).