Skip to content

fix(envd): give the guest a routable IPv4 address so Tailscale sees it online - #3835

Open
tunsuy wants to merge 3 commits into
e2b-dev:mainfrom
tunsuy:contrib/oss-radar-3780
Open

tunsuy wants to merge 3 commits into
e2b-dev:mainfrom
tunsuy:contrib/oss-radar-3780

Conversation

@tunsuy

@tunsuy tunsuy commented Nov 17, 2026

Copy link
Copy Markdown

Fixes #3780

Problem

A sandbox's only IPv4 address is the link-local 169.254.0.21/30 configured by the kernel's ip= argument. Software that requires a non-link-local address to consider the host online therefore treats the guest as having no network at all. The reported case is Tailscale: its network monitor does not count 169.254.0.0/16 as usable, so tailscaled pauses its control client before login — auth keys silently never reach the control plane, even though outbound connectivity is fine (curl to the control plane returns 200 from the same guest).

Change

As proposed in #3780: when envd starts in a Firecracker guest (!isNotFC), add the RFC 7600 IPv4 dummy address 192.0.0.8/32 to eth0 with IFA_F_NOPREFIXROUTE.

  • Why 192.0.0.8: RFC 7600 reserves it for exactly this situation (a host with no other usable IPv4 address) and it is never routed, so — unlike 100.64.0.0/10 (Tailscale's own CGNAT range) or RFC 1918 ranges — it cannot clash with user networks or VPNs.
  • Why noprefixroute: no prefix route is created, so egress keeps selecting 169.254.0.21 as its source and nothing routes to the dummy address: no inbound effect, no routing change.
  • Why envd: it runs in every sandbox regardless of distro/init system, and already touches guest networking (host.PinMMDSRoute). The ip= kernel argument only configures one address, so the orchestrator cannot add a second one.
  • Why raw netlink instead of the vishvananda/netlink library or ip addr: some base images ship no iproute2, and envd (unlike the orchestrator) deliberately has no netlink dependency — golang.org/x/sys/unix is already one, and envd already talks to a netlink socket directly (the uevent watcher in internal/services/memlimit/events_linux.go). The RTM_NEWADDR request is built to the same wire format ip addr add … noprefixroute produces (NOPREFIXROUTE rides in IFA_FLAGS, since it does not fit the legacy 8-bit field), with zero new dependencies and no go.mod/go.sum churn.
  • Lifecycle: idempotent both ways — a cold boot re-runs envd and a repeated add ignores EEXIST; a memory-snapshot resume keeps the address because it lives in kernel state. A failure only logs a warning; startup never blocks on it.

Tests

  • TestBuildAddrMsgEncodesNoPrefixRoute decodes the built request and asserts the wire format: IFA_LOCAL/IFA_ADDRESS = the dummy address, IFA_FLAGS = NOPREFIXROUTE, no IFA_BROADCAST (a /32 must not get one), legacy flags field zero.
  • TestEnsureRoutableAddrRoundTrip exercises the real netlink path (root-gated, skipped otherwise): add → verify via a netlink dump → re-add hits the EEXIST branch → delete → verify gone.
  • TestEnsureRoutableAddrRejectsBadInput covers bad CIDR, missing link, and IPv6 input.
  • TestNlaPad pins the attribute alignment arithmetic.
  • Full packages/envd/internal/host package passes on linux/amd64; go build/go vet clean for linux, and the !linux stub keeps darwin builds working.

Manual verification (Docker, privileged alpine)

Running EnsureRoutableAddr against the container's eth0 reproduces the issue's verified workaround state exactly:

inet 192.0.0.8/32 scope global noprefixroute eth0     # added
ip route get 1.1.1.1 → via <gw> dev eth0 src <original addr>   # egress source unchanged
ip route show | grep 192.0.0.8 → (no route)           # noprefixroute: nothing to route
second call → no error                                # EEXIST treated as success

Notes

  • Per the issue's rollout section: existing templates only pick this up when rebuilt with the new envd, and tools that report all interface addresses (e.g. hostname -I) will additionally list 192.0.0.8; tools using the default route's source address keep seeing the link-local one.
  • The change is small and additive: one new linux file + stub + tests in internal/host, and an 11-line hook in main.go.

…t online

The sandbox's only IPv4 address is the link-local 169.254.0.21/30 from the
kernel's ip= argument, and software that requires a non-link-local address to
consider the host online reads the guest as having no network at all. The
reported case is Tailscale: its network monitor does not count 169.254.0.0/16
as usable, so tailscaled pauses its control client before login and auth keys
never reach the control plane — while plain egress (curl to the control plane
itself) works fine.

Add the RFC 7600 IPv4 dummy address 192.0.0.8/32 to eth0 when envd starts in
a Firecracker guest. RFC 7600 reserves that address for hosts with no other
usable IPv4 address and it is never routed, so it cannot clash with user
networks or VPN ranges the way RFC 1918 or Tailscale's own 100.64.0.0/10
would. It is added with IFA_F_NOPREFIXROUTE, so no prefix route is created:
egress keeps selecting 169.254.0.21 as its source and nothing routes to the
dummy address, leaving inbound traffic unchanged.

The request is built and sent over NETLINK_ROUTE directly rather than by
running `ip addr`, because some base images ship no iproute2 and envd (unlike
the orchestrator) has no netlink library dependency — golang.org/x/sys/unix is
already one. Adding is idempotent, which covers both lifecycle paths: a cold
boot runs envd again and ignores EEXIST, while a memory-snapshot resume keeps
the address because it lives in kernel state. A failure only warns, since
egress does not depend on it.

Fixes e2b-dev#3780

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tailscale can't connect from sandboxes: guest has only a link-local IPv4 address

1 participant