Repository navigation
bpf: vendor the libbpf headers, so BTF ordering depends on the repo (#117) - #182
Merged
Merged
Conversation
…117) BPF objects were not byte-reproducible across machines that agreed on every visible input. The issue eliminated the compiler, the strip tool, the package version and the header contents by measurement, and identified the remaining variable as the headers coming from whatever /usr/include/bpf the generating machine happened to hold. Its proposed fix was to vendor them. That fix had already been half-applied, inertly. cpu/gen.go passed -I../bpf/libbpf, but bpf/libbpf did not exist -- clang ignores a missing -I without complaint -- and the other four packages did not pass it at all. profile/, which holds the object the issue names as divergent, was one of those four. cpu/gen.go also carried -I../bpf/vmlinux/, equally dead, and that pair is what made the first flag look like vendoring had been done. The dead one is removed; the sources find vmlinux.h relative to themselves. Vendored are the four headers clang actually opens, determined with clang -H over the three this repo includes directly rather than guessed: bpf_helpers.h, bpf_core_read.h, bpf_tracing.h, and bpf_helper_defs.h transitively. Taken from libbpf-dev 1:1.3.0-2build2 in the CI image, NOT from the local system: this machine ships 1.6.3, and vendoring that would have "fixed" reproducibility by breaking it against CI. bpf/libbpf/README records the trap and the command to update them correctly. Four objects move once, and the shape of the move is the issue's own signature. Comparing cpu/cpu_x86_bpfel.o before and after: .text same <- no codegen change .BTF.ext same .strtab same .BTF DIFFERS <- 247->250, 251->247, 250->251 Values exchanged rather than changed, confined to .BTF's type section. That a header's PATH alone permutes the ordering is the clearest evidence yet for the pointer-keyed-map explanation, and it is the same mechanism by which any difference in a machine's /usr/include/bpf could permute it -- which is the invisible variable this issue could not pin down. From here the ordering depends on files in the repo. generate-check continues to verify it, and #178's privileged unit pass now actually loads the DWARF object rather than skipping it, so a BTF change that broke loading would surface. Claude-Session: https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #117, if CI agrees — and CI is the only place this claim can be settled, since reproducibility across machines is the property in question.
The fix was already half-applied, inertly
#117 proposed vendoring the libbpf headers. That had been started and never finished:
cpu/gen.gopassed-I../bpf/libbpf— butbpf/libbpf/did not exist, and clang ignores a missing-Iwithout complaintprofile/, which holdsperf_dwarf_x86_bpfel.o, the object the issue names as divergentcpu/gen.goalso carried-I../bpf/vmlinux/, equally dead; that pair is what made the first flag look realSo every object has always been built against whatever
/usr/include/bpfthe generating machine held. The deadvmlinuxflag is removed — the sources findvmlinux.hrelative to themselves.What's vendored, and from where
The four headers clang actually opens, determined with
clang -Hover the three included directly rather than guessed:bpf_helpers.h,bpf_core_read.h,bpf_tracing.h, andbpf_helper_defs.htransitively.Taken from
libbpf-dev 1:1.3.0-2build2in the CI image, not from the local system. This machine ships 1.6.3; vendoring that would have "fixed" reproducibility by breaking it against CI.bpf/libbpf/README.mdrecords that trap and the command to update them correctly.Four objects move once, and the shape of the move is the issue's own signature
Comparing
cpu/cpu_x86_bpfel.obefore and after:.text.BTF.ext.strtab.BTF247→250,251→247,250→251Values exchanged rather than changed, confined to
.BTF's type section — exactly the "same values, exchanged positions" the issue describes.That a header's path alone permutes the ordering is the clearest evidence yet for the pointer-keyed-map explanation, and it is the same mechanism by which any difference in a machine's
/usr/include/bpfcould permute it. Which is precisely the invisible variable #117 could not pin down.What this does and does not prove
From here, BTF ordering depends on files in the repo rather than on the generating machine's headers.
generate-checkcontinues to verify it, and #178's privileged unit pass now actually loads the DWARF object instead of skipping it, so a BTF change that broke loading would surface rather than sit.It does not eliminate the build directory, which clang's pointer-keyed map also follows — that is already pinned via
CI_BUILD_PATH. Headers were the unpinned half.I could not verify idempotency locally:
make generate-checkruns with this machine's clang 22 and libbpf 1.6.3, so 14 objects move for reasons unrelated to vendoring. The objects here come frommake generate-container, which uses CI's clang-18 and build path. If this job's "Verify generated objects are up to date" step is clean, that is the proof.https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im