Skip to content

bpf: vendor the libbpf headers, so BTF ordering depends on the repo (#117) - #182

Merged
dpsoft merged 1 commit into
mainfrom
fix/vendor-libbpf-headers
Oct 3, 2026
Merged

dpsoft merged 1 commit into
mainfrom
fix/vendor-libbpf-headers

Conversation

@dpsoft

@dpsoft dpsoft commented Oct 3, 2026

Copy link
Copy Markdown
Owner

Closes #117, if CI agrees — and CI is the only place this claim can be settled, since reproducibility across machines is the property in question.

The fix was already half-applied, inertly

#117 proposed vendoring the libbpf headers. That had been started and never finished:

  • cpu/gen.go passed -I../bpf/libbpf — but bpf/libbpf/ did not exist, and clang ignores a missing -I without complaint
  • the other four packages didn't pass it at all — including profile/, which holds perf_dwarf_x86_bpfel.o, the object the issue names as divergent
  • cpu/gen.go also carried -I../bpf/vmlinux/, equally dead; that pair is what made the first flag look real

So every object has always been built against whatever /usr/include/bpf the generating machine held. The dead vmlinux flag is removed — the sources find vmlinux.h relative to themselves.

What's vendored, and from where

The four headers clang actually opens, determined with clang -H over the three included directly rather than guessed: bpf_helpers.h, bpf_core_read.h, bpf_tracing.h, and bpf_helper_defs.h transitively.

Taken from libbpf-dev 1:1.3.0-2build2 in the CI image, not from the local system. This machine ships 1.6.3; vendoring that would have "fixed" reproducibility by breaking it against CI. bpf/libbpf/README.md records that trap and the command to update them correctly.

Four objects move once, and the shape of the move is the issue's own signature

Comparing cpu/cpu_x86_bpfel.o before and after:

section
.text same — no codegen change
.BTF.ext same
.strtab same
.BTF DIFFERS — 247→250, 251→247, 250→251

Values exchanged rather than changed, confined to .BTF's type section — exactly the "same values, exchanged positions" the issue describes.

That a header's path alone permutes the ordering is the clearest evidence yet for the pointer-keyed-map explanation, and it is the same mechanism by which any difference in a machine's /usr/include/bpf could permute it. Which is precisely the invisible variable #117 could not pin down.

What this does and does not prove

From here, BTF ordering depends on files in the repo rather than on the generating machine's headers. generate-check continues to verify it, and #178's privileged unit pass now actually loads the DWARF object instead of skipping it, so a BTF change that broke loading would surface rather than sit.

It does not eliminate the build directory, which clang's pointer-keyed map also follows — that is already pinned via CI_BUILD_PATH. Headers were the unpinned half.

I could not verify idempotency locally: make generate-check runs with this machine's clang 22 and libbpf 1.6.3, so 14 objects move for reasons unrelated to vendoring. The objects here come from make generate-container, which uses CI's clang-18 and build path. If this job's "Verify generated objects are up to date" step is clean, that is the proof.

https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im

…117)

BPF objects were not byte-reproducible across machines that agreed on
every visible input. The issue eliminated the compiler, the strip tool,
the package version and the header contents by measurement, and
identified the remaining variable as the headers coming from whatever
/usr/include/bpf the generating machine happened to hold. Its proposed
fix was to vendor them.

That fix had already been half-applied, inertly. cpu/gen.go passed
-I../bpf/libbpf, but bpf/libbpf did not exist -- clang ignores a missing
-I without complaint -- and the other four packages did not pass it at
all. profile/, which holds the object the issue names as divergent, was
one of those four. cpu/gen.go also carried -I../bpf/vmlinux/, equally
dead, and that pair is what made the first flag look like vendoring had
been done. The dead one is removed; the sources find vmlinux.h relative
to themselves.

Vendored are the four headers clang actually opens, determined with
clang -H over the three this repo includes directly rather than guessed:
bpf_helpers.h, bpf_core_read.h, bpf_tracing.h, and bpf_helper_defs.h
transitively. Taken from libbpf-dev 1:1.3.0-2build2 in the CI image, NOT
from the local system: this machine ships 1.6.3, and vendoring that would
have "fixed" reproducibility by breaking it against CI. bpf/libbpf/README
records the trap and the command to update them correctly.

Four objects move once, and the shape of the move is the issue's own
signature. Comparing cpu/cpu_x86_bpfel.o before and after:

    .text      same      <- no codegen change
    .BTF.ext   same
    .strtab    same
    .BTF       DIFFERS   <- 247->250, 251->247, 250->251

Values exchanged rather than changed, confined to .BTF's type section.
That a header's PATH alone permutes the ordering is the clearest evidence
yet for the pointer-keyed-map explanation, and it is the same mechanism
by which any difference in a machine's /usr/include/bpf could permute it
-- which is the invisible variable this issue could not pin down.

From here the ordering depends on files in the repo. generate-check
continues to verify it, and #178's privileged unit pass now actually
loads the DWARF object rather than skipping it, so a BTF change that
broke loading would surface.

Claude-Session: https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im
@dpsoft
dpsoft merged commit 96d7fc2 into main Oct 3, 2026
16 checks passed
@dpsoft
dpsoft deleted the fix/vendor-libbpf-headers branch October 5, 2026 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BPF objects are not byte-reproducible across machines that agree on every visible input

1 participant