Repository navigation
Conversation
The main-table /128 redirect for a backend was deleted by destination alone. When two VPCs carry the same backend address, as when IPAM hands a deleted Network's subnet to the next one, the live VPC's RouteReplace takes over the route and the deleted VPC's teardown then removes it. Nothing reinstalls it, so connects to the live backend fail until the slice or datapath changes. Name the VRF device in the delete so the kernel removes the route only while it still points at the tearing-down VPC, and treat a missing VRF as nothing left to remove. Refs datum-cloud/engineering#467
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Scot Schuchert-Wells · project thread
Summary
The ingress sidecar deleted a backend's main-table
/128redirect by destination alone. Two VPCs can carry the same backend address: IPAM hands a deleted Network's subnet to the next Network, which the production e2e runs do every time (fd20:0:1e::/64). When the new VPC's slice is applied before the deleted VPC's slice finishes its teardown grace,ensureRedirectRoute'sRouteReplacepoints the route at the new VPC's VRF, and the deleted VPC's teardown then deletes it. Nothing reinstalls it until the slice or the eBPF datapath changes, so every connect from Envoy to the live backend fails while Envoy still lists it as a healthy endpoint.This matches cause 2 of datum-cloud/engineering#467 (503 with
upstream_cx_connect_failthrough the us-central-1 edge, intermittent, only on runs whose backend reuses the previous run's address).The store's shared-prefix guard (
prefixClaimedElsewhereLocked) only covers slices of the same VPC, so it does not catch this case.Change
removeRedirectRoutenow takes the VRF table, resolves its VRF device and deletes withLinkIndexset. The kernel then removes the route only while it still leaves through this VPC's VRF; a route another VPC replaced it with is kept (ESRCH, already treated as success).Testing
TestRemoveRedirectRoute_KeepsAnotherVPCsRoute: two VRFs, same backend address, old VPC's removal keeps the new VPC's route, removal for a table with no VRF keeps it, owner's removal deletes it. It fails without the change (redirect via ifindex 0, want live VPC's 3).go test -race ./internal/ingresssidecar/as root in a privileged Linux container with thevrfmodule: pass.golangci-lint run ./internal/ingresssidecar/...: 0 issues.task ci(e2e).