From 85a22c246e53530fab9324975879defd22eda736 Mon Sep 17 00:00:00 2001 From: Denys Fedoryshchenko Date: Fri, 18 Sep 2026 11:58:52 +0300 Subject: [PATCH 1/2] docker: bump pahole to 1.31 pahole 1.29 deadlocks when its threaded BTF encoder meets a type it cannot represent. The worker that hits the error terminates without signalling its peers, so the main thread and one remaining worker park on futexes and never return. Any -j >= 2 triggers it, provided there are enough compilation units for the pool to run concurrently, and vmlinux always has enough. The result is a build that hangs forever instead of failing. ARC reaches this because it links libgcc, whose libgcc2.c is built with debug info and carries complex float base types in its DWARF. pahole cannot encode those in BTF and says so: Complex, interval and imaginary float types are not supported Error while encoding BTF. Under 1.29 that message is followed by the deadlock. Builds seen in production sat Running for more than six days, each holding a whole node until it was reaped by hand. 1.31 reports the same unsupported-type error and exits cleanly, so the build fails normally. This reproduces without a kernel tree: 1000 trivial translation units, a few containing a _Complex float member, linked with -g. 1.29 exits 0 at -j1 but hangs at -j2 and -j8, while 1.31 exits 0 at every -j. Note this only stops the hang. ARC still produces no BTF and so still fails, later and visibly, in resolve_btfids. Signed-off-by: Denys Fedoryshchenko --- config/docker/base/host-tools.jinja2 | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/config/docker/base/host-tools.jinja2 b/config/docker/base/host-tools.jinja2 index 433cd4d08d..21ab216a47 100644 --- a/config/docker/base/host-tools.jinja2 +++ b/config/docker/base/host-tools.jinja2 @@ -100,22 +100,22 @@ RUN CFLAGS="-DPyString_FromString=PyUnicode_FromString \ -DPyInt_AsLong=PyLong_AsLong" \ pip3 install dtschema --break-system-packages -# Download and build pahole v1.29 +# Download and build pahole v1.31 # # Build in Release mode: pahole's CMakeLists defaults to Debug, which adds # -Werror (and -O0). glibc 2.43 in forky makes strchr()/bsearch() and # friends preserve the constness of their argument, so pahole's assignments # of their results to non-const pointers now trip -Wdiscarded-qualifiers and # fail the build. Release drops -Werror and builds with -O2. -RUN wget -c https://storage.kernelci.org/pahole-1.29.tar.gz && \ - tar -xzf pahole-1.29.tar.gz && \ - cd pahole-1.29 && \ +RUN wget -c https://storage.kernelci.org/pahole-1.31.tar.gz && \ + tar -xzf pahole-1.31.tar.gz && \ + cd pahole-1.31 && \ mkdir build && cd build && \ cmake .. -DCMAKE_BUILD_TYPE=Release -DLIBBPF_EMBEDDED=OFF && \ make && \ make install && \ echo "/usr/local/lib" > /etc/ld.so.conf.d/dwarves.conf && \ ldconfig && \ - cd ../.. && rm -rf pahole-1.29 + cd ../.. && rm -rf pahole-1.31 {% endblock %} From 501d72cf9403f840ec4ed967fdc82287e4d92e59 Mon Sep 17 00:00:00 2001 From: Denys Fedoryshchenko Date: Fri, 18 Sep 2026 11:59:01 +0300 Subject: [PATCH 2/2] config/runtime: bound Kubernetes job runtime and fix backoffLimit backoffLimit is a Job-level field, but it was written under spec.template.spec. It is not part of the PodSpec schema there, so the API server drops it and the jobs ran with the default limit of 6: every failing build was retried seven times over, multiplying one bad build into a pile of dead pods. Move it to the Job spec, where the intended limit of 1 takes effect. Also add activeDeadlineSeconds, so a job cannot run indefinitely. A build whose toolchain wedges keeps its pod Running and holds a whole node, and nothing reclaims it: ttlSecondsAfterFinished only applies once a job has finished, and the timeout the API tracks on the node is advisory and is never enforced against the Job. Builds have been observed Running for over six days this way, with six of the nine nodes in one cluster held by them. Default to 21600 seconds, matching the six hour timeout the API already records, and allow a job to override it. Signed-off-by: Denys Fedoryshchenko --- config/runtime/base/kubernetes.jinja2 | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/config/runtime/base/kubernetes.jinja2 b/config/runtime/base/kubernetes.jinja2 index e4a6443f26..ea6017a127 100644 --- a/config/runtime/base/kubernetes.jinja2 +++ b/config/runtime/base/kubernetes.jinja2 @@ -8,12 +8,20 @@ metadata: spec: completions: 1 + # backoffLimit is a Job-level field. This used to sit under + # spec.template.spec, where the API server drops it silently, so jobs + # ran with the default of 6 retries and every failure was retried + # seven times over. + backoffLimit: 1 + # Cap the wall-clock runtime of a job. Without it a wedged build keeps + # its pod Running indefinitely and holds a whole node: the node timeout + # tracked by the API is advisory and is not enforced against the Job. + activeDeadlineSeconds: {{ k8s_active_deadline_seconds | default(21600) }} # During development we keep logs a bit longer # TODO: Reduce on production ttlSecondsAfterFinished: 1800 template: spec: - backoffLimit: 1 restartPolicy: Never terminationGracePeriodSeconds: 10