run-vmtest: fail selftest runs that produce kernel splats

.github/workflows/kernel-test.yml

@@ -96,3 +96,12 @@ jobs:
           name: tmon-logs-${{ inputs.arch }}-${{ inputs.toolchain_full }}-${{ inputs.test }}
           if-no-files-found: ignore
           path: /tmp/tmon_pcap/*
+
+      # Written by check-kernel-splats.sh inside the VM, into the bind-mounted
+      # workspace. Needed to triage a splat, and to audit the scan itself.
+      - if: ${{ always() }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: kernel-log-${{ inputs.arch }}-${{ inputs.toolchain_full }}-${{ inputs.test }}
+          if-no-files-found: ignore
+          path: dmesg.txt

ci/vmtest/configs/SPLAT_ALLOWLIST

@@ -0,0 +1,21 @@
+# Kernel splats that check-kernel-splats.sh must not fail the run on.
+#
+# One extended regex per line, matched against the dmesg line. `#` comments
+# and blank lines are ignored.
+#
+# Every entry is a hole in the scan, so give each one a reason and a link, and
+# delete it once the fix reaches the CI kernel. An entry with no expiry is a
+# bug that CI has agreed to stop reporting.
+#
+# The selftest config leaves unprivileged BPF on (CONFIG_BPF_UNPRIV_DEFAULT_OFF
+# is not set in tools/testing/selftests/bpf/config), so the Spectre v2 code
+# warns about it on every boot on x86_64, and whenever a test writes the sysctl
+# on arm64. It states a config choice, not a kernel bug.
+WARNING: Unprivileged eBPF is enabled
+
+# arm64 stack unwinder writes past the entry array while KASAN saves a free
+# stack, so any kfree() under test_progs reports stack-out-of-bounds. Not a BPF
+# bug; BPF only walks that path often. Fix posted 2026-08-12:
+# https://lore.kernel.org/all/20260812-hello_world-v1-1-c3c2ddcb362d@meta.com/
+# Delete this entry once that fix reaches the CI kernel.
+BUG: KASAN: stack-out-of-bounds in (stack_trace_consume_entry|filter_irq_stacks)\+

ci/vmtest/configs/SPLAT_DENYLIST

@@ -0,0 +1,24 @@
+# Kernel splat signatures for check-kernel-splats.sh. A dmesg line matching
+# any of these fails the run.
+#
+# One extended regex per line. `#` comments and blank lines are ignored. This
+# file is required: with no patterns there is no check, so the action fails the
+# run rather than reporting a clean log.
+#
+# `^(\[[^]]*\] *)*` eats the timestamp and the CONFIG_PRINTK_CALLER field, so a
+# pattern holds for `dmesg`, for `dmesg -t` and for both configs.
+
+# `BUG:` covers KASAN, KCSAN, KMSAN and KFENCE. `WARNING:` covers __warn(), and
+# with it lockdep, refcount_t and list corruption, which all print through
+# WARN(). An oops panics the VM (PANIC_ON_OOPS plus panic=-1), which fails the
+# job on its own, so it needs no pattern here.
+^(\[[^]]*\] *)*(BUG|WARNING|UBSAN|Oops)[: ]
+^(\[[^]]*\] *)*kernel BUG at
+
+# kernel/watchdog.c sets pr_fmt, so a lockup line starts with `watchdog: `,
+# not with `BUG:`.
+^(\[[^]]*\] *)*watchdog: .*(soft lockup|hard LOCKUP)
+
+# kernel/rcu/tree.c sets pr_fmt to "rcu: ", which hides the INFO: from an
+# anchored match.
+^(\[[^]]*\] *)*(rcu: )?INFO: (task .* blocked for more than|[_a-z]+ (self-)?detected stall)

ci/vmtest/configs/run-vmtest.env

@@ -41,3 +41,10 @@ DENYLIST_FILES=(
 # Export pipe-separated strings, because bash doesn't support array export
 export SELFTESTS_BPF_ALLOWLIST_FILES=$(IFS="|"; echo "${ALLOWLIST_FILES[*]}")
 export SELFTESTS_BPF_DENYLIST_FILES=$(IFS="|"; echo "${DENYLIST_FILES[*]}")
+
+# Kernel splat matching for check-kernel-splats.sh. The denylist says what a
+# splat is, the allowlist drops the matches that are benign for this CI. The
+# action carries no patterns of its own, so a change to either costs one PR to
+# this repo, and no libbpf/ci sync.
+export SPLAT_DENYLIST_FILE="${VMTEST_CONFIGS}/SPLAT_DENYLIST"
+export SPLAT_ALLOWLIST_FILE="${VMTEST_CONFIGS}/SPLAT_ALLOWLIST"

run-vmtest/README.md

@@ -39,6 +39,30 @@ environment variables.
 * `kbuild-output` (default: `./kbuild-output`) - path to Linux Kernel binaries, aka `$KBUILD_OUTPUT`
 * `vmtest-release` - release version name of the vmtest tool
 
+## Kernel splats
+
+`check-kernel-splats.sh` runs after the selftests and greps the kernel log for
+splats that leave the VM alive: WARN, KASAN and friends, lockdep, RCU stalls,
+hung tasks and lockups. A hit adds a `kernel_splats` row to `exitstatus`, which
+turns the run red like any other failing test group. Anything fatal panics the
+VM instead, and fails the job on its own.
+
+The action carries no patterns. What a splat is, and what is benign, is policy
+that changes per arch and per kernel, so it lives with the configs. Two files
+of extended regexes, one per line, `#` comments and blank lines ignored, named
+by `run-vmtest.env` next to the allow and denylists:
+
+* `SPLAT_DENYLIST_FILE` - a matching dmesg line is a splat. **Required.**
+* `SPLAT_ALLOWLIST_FILE` - a matching splat line is ignored. Optional.
+
+The denylist is required on purpose: with no patterns there is no check, so a
+missing or empty file fails the run rather than reporting a clean log.
+
+See `ci/vmtest/configs/SPLAT_DENYLIST` for the set BPF CI uses.
+
+`dmesg.txt` is written to the output dir and uploaded as the `kernel-log-*`
+artifact, so the full log is always one click away.
+
 ## run-vmtest.env
 
 There are a couple of scripts, as well as code in the

run-vmtest/check-kernel-splats.sh

@@ -0,0 +1,82 @@
+#!/bin/bash
+# Fail the selftest run if the kernel logged a splat. A WARN, a KASAN report,
+# a lockdep report or a hung task leaves the VM running, so the test binary
+# still exits 0 and the job goes green on a kernel that hit a real bug.
+#
+# What counts as a splat, and what is benign, is policy, so both live in the
+# config repo rather than here:
+#
+#   SPLAT_DENYLIST_FILE   extended regexes; a matching dmesg line is a splat.
+#                         Required: with no patterns there is no check, so a
+#                         missing file fails the run instead of passing it.
+#   SPLAT_ALLOWLIST_FILE  extended regexes; a matching splat line is ignored.
+#                         Optional: no file means no exceptions.
+#
+# `#` comments and blank lines are ignored in both. See run-vmtest.env in the
+# repo that owns $VMTEST_CONFIGS.
+#
+# The whole log is written to dmesg.txt, which the workflow uploads.
+#
+# $1 - log file to scan instead of running dmesg (used by the unit tests)
+
+set -euo pipefail
+
+source "$(cd "$(dirname "$0")" && pwd)/helpers.sh"
+
+STATUS_FILE=${STATUS_FILE:-/mnt/vmtest/exitstatus}
+OUTPUT_DIR=${OUTPUT_DIR:-/mnt/vmtest}
+SPLAT_DENYLIST_FILE=${SPLAT_DENYLIST_FILE:-}
+SPLAT_ALLOWLIST_FILE=${SPLAT_ALLOWLIST_FILE:-}
+
+# Fail the row and stop. Also used when the scan cannot run: a check that is
+# not working must not look like a clean log.
+fail_check() {
+    echo "$1"
+    echo "kernel_splats:1" >> "${STATUS_FILE}"
+    foldable end kernel_splats
+    exit 0
+}
+
+# `grep -f` has no comment syntax, and to grep an empty line is a regex that
+# matches every line, so drop both before either file can hide a splat.
+read_patterns() {
+    grep -vE '^[[:space:]]*(#|$)' "$1" || [ $? = 1 ]
+}
+
+log=${1:-}
+if [ -z "${log}" ]; then
+    log="${OUTPUT_DIR}/dmesg.txt"
+    dmesg > "${log}"
+fi
+
+foldable start kernel_splats "Checking kernel log for splats"
+
+[ -s "${SPLAT_DENYLIST_FILE}" ] || \
+    fail_check "No splat denylist: SPLAT_DENYLIST_FILE=${SPLAT_DENYLIST_FILE:-<unset>}"
+echo "Splat denylist: ${SPLAT_DENYLIST_FILE}"
+deny=$(read_patterns "${SPLAT_DENYLIST_FILE}")
+[ -n "${deny}" ] || fail_check "Splat denylist holds no patterns"
+
+# `|| [ $? = 1 ]` lets "no match" through, but a grep error trips set -e, so a
+# scan that cannot run fails the job instead of reporting a clean log. Two
+# greps must not share a pipeline: pipefail reports the rightmost non-zero
+# status, so "no match" from one would mask an error from the other.
+hits=$(grep -E -f <(printf '%s\n' "${deny}") "${log}" || [ $? = 1 ])
+
+if [ -s "${SPLAT_ALLOWLIST_FILE}" ]; then
+    echo "Splat allowlist: ${SPLAT_ALLOWLIST_FILE}"
+    allow=$(read_patterns "${SPLAT_ALLOWLIST_FILE}")
+    [ -z "${allow}" ] || hits=$(printf '%s' "${hits}" \
+        | grep -vE -f <(printf '%s\n' "${allow}") || [ $? = 1 ])
+fi
+foldable end kernel_splats
+
+if [ -z "${hits}" ]; then
+    echo "No kernel splats detected"
+    echo "kernel_splats:0" >> "${STATUS_FILE}"
+    exit 0
+fi
+
+echo "kernel_splats:1" >> "${STATUS_FILE}"
+grep -nC 30 -E -f <(printf '%s\n' "${deny}") "${log}" || true
+echo "::error title=kernel_splats::kernel splat detected, see the dmesg.txt artifact"

run-vmtest/run-bpf-selftests.sh

@@ -14,7 +14,8 @@
 
 set -xeuo pipefail
 
-source "$(cd "$(dirname "$0")" && pwd)/helpers.sh"
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "${SCRIPT_DIR}/helpers.sh"
 
 ARCH=$(uname -m)
 
@@ -176,3 +177,5 @@ else
 		"${test_name}"
 	done
 fi
+
+"${SCRIPT_DIR}/check-kernel-splats.sh"

run-vmtest/run-scx-selftests.sh

@@ -37,5 +37,7 @@ fi
 
 echo "selftests/sched_ext:$failed" >> "${STATUS_FILE}"
 
+"$(cd "$(dirname "$0")" && pwd)/check-kernel-splats.sh"
+
 exit 0
 

run-vmtest/run.sh

@@ -96,7 +96,7 @@ cat > $VMTEST_TOML <<EOF
 [[target]]
 name = "run-vmtest"
 kernel = "${VMLINUZ}"
-kernel_args = "panic=-1 sysctl.vm.panic_on_oom=1 hardlockup_all_cpu_backtrace=1 softlockup_all_cpu_backtrace=1 kasan_multi_shot no5lvl"
+kernel_args = "panic=-1 sysctl.vm.panic_on_oom=1 hardlockup_all_cpu_backtrace=1 softlockup_all_cpu_backtrace=1 kasan_multi_shot no5lvl log_buf_len=16M"
 command = """\
 ${GITHUB_ACTION_PATH}/vmtest-init.sh && \
 cd ${GITHUB_WORKSPACE} && \

run-vmtest/tests/check_kernel_splats/cases/base_allowlist/dmesg.txt

@@ -0,0 +1,3 @@
+[ 0.1] Linux version 6.5.0
+[ 0.2] WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
+[ 1.0] test_progs: PASS

run-vmtest/tests/check_kernel_splats/cases/base_allowlist/expected-output.txt

@@ -0,0 +1,5 @@
+::group::kernel_splats - Checking kernel log for splats
+Splat denylist: $VMTEST_CONFIGS/SPLAT_DENYLIST
+Splat allowlist: $VMTEST_CONFIGS/SPLAT_ALLOWLIST
+::endgroup::
+No kernel splats detected

run-vmtest/tests/check_kernel_splats/cases/base_allowlist/expected-status.txt

@@ -0,0 +1 @@
+kernel_splats:0

run-vmtest/tests/check_kernel_splats/cases/case_allowlist/allowlist.txt

@@ -0,0 +1,2 @@
+# Silence the WARN this case produces.
+WARNING: CPU: 0 PID: 100 at kernel/sched/core\.c:1234

run-vmtest/tests/check_kernel_splats/cases/case_allowlist/dmesg.txt

@@ -0,0 +1,14 @@
+[    0.000000] Linux version 6.5.0
+[    0.001234] CPU: 0
+[   12.345678] ------------[ cut here ]------------
+[   12.345679] WARNING: CPU: 0 PID: 100 at kernel/sched/core.c:1234 do_thing+0x10/0x20
+[   12.345680] Modules linked in:
+[   12.345681] CPU: 0 PID: 100 Comm: test_progs
+[   12.345682] RIP: 0010:do_thing+0x10/0x20
+[   12.345683] Call Trace:
+[   12.345684]  <TASK>
+[   12.345685]  ? __warn+0x80/0xc0
+[   12.345686]  ? report_bug+0xa0/0xc0
+[   12.345687]  </TASK>
+[   12.345688] ---[ end trace ]---
+[   13.000000] test_progs: PASS

run-vmtest/tests/check_kernel_splats/cases/case_allowlist/env

@@ -0,0 +1 @@
+export SPLAT_ALLOWLIST_FILE="${CASE_DIR}/allowlist.txt"

run-vmtest/tests/check_kernel_splats/cases/case_allowlist/expected-output.txt

@@ -0,0 +1,5 @@
+::group::kernel_splats - Checking kernel log for splats
+Splat denylist: $VMTEST_CONFIGS/SPLAT_DENYLIST
+Splat allowlist: $CASE_DIR/allowlist.txt
+::endgroup::
+No kernel splats detected

run-vmtest/tests/check_kernel_splats/cases/case_allowlist/expected-status.txt

@@ -0,0 +1 @@
+kernel_splats:0

run-vmtest/tests/check_kernel_splats/cases/clean/dmesg.txt

@@ -0,0 +1,4 @@
+[    0.000000] Linux version 6.5.0
+[    0.001234] CPU: 0
+[    0.002345] Memory: 4G
+[    1.000000] vmtest selftests complete

run-vmtest/tests/check_kernel_splats/cases/clean/expected-output.txt

@@ -0,0 +1,5 @@
+::group::kernel_splats - Checking kernel log for splats
+Splat denylist: $VMTEST_CONFIGS/SPLAT_DENYLIST
+Splat allowlist: $VMTEST_CONFIGS/SPLAT_ALLOWLIST
+::endgroup::
+No kernel splats detected

run-vmtest/tests/check_kernel_splats/cases/clean/expected-status.txt

@@ -0,0 +1 @@
+kernel_splats:0

run-vmtest/tests/check_kernel_splats/cases/kasan/dmesg.txt

@@ -0,0 +1,13 @@
+[    0.000000] Linux version 6.5.0
+[   20.000000] ==================================================================
+[   20.000001] BUG: KASAN: slab-use-after-free in foo+0x42/0x100
+[   20.000002] Read of size 8 at addr ffff888100001234 by task test_progs/123
+[   20.000003]
+[   20.000004] CPU: 0 PID: 123 Comm: test_progs
+[   20.000005] Call Trace:
+[   20.000006]  <TASK>
+[   20.000007]  dump_stack_lvl+0x60/0x80
+[   20.000008]  print_report+0xc4/0x630
+[   20.000009]  kasan_report+0xb6/0xf0
+[   20.000010]  </TASK>
+[   20.000011] ==================================================================

run-vmtest/tests/check_kernel_splats/cases/kasan/expected-output.txt

@@ -0,0 +1,18 @@
+::group::kernel_splats - Checking kernel log for splats
+Splat denylist: $VMTEST_CONFIGS/SPLAT_DENYLIST
+Splat allowlist: $VMTEST_CONFIGS/SPLAT_ALLOWLIST
+::endgroup::
+1-[    0.000000] Linux version 6.5.0
+2-[   20.000000] ==================================================================
+3:[   20.000001] BUG: KASAN: slab-use-after-free in foo+0x42/0x100
+4-[   20.000002] Read of size 8 at addr ffff888100001234 by task test_progs/123
+5-[   20.000003]
+6-[   20.000004] CPU: 0 PID: 123 Comm: test_progs
+7-[   20.000005] Call Trace:
+8-[   20.000006]  <TASK>
+9-[   20.000007]  dump_stack_lvl+0x60/0x80
+10-[   20.000008]  print_report+0xc4/0x630
+11-[   20.000009]  kasan_report+0xb6/0xf0
+12-[   20.000010]  </TASK>
+13-[   20.000011] ==================================================================
+::error title=kernel_splats::kernel splat detected, see the dmesg.txt artifact

run-vmtest/tests/check_kernel_splats/cases/kasan/expected-status.txt

@@ -0,0 +1 @@
+kernel_splats:1

run-vmtest/tests/check_kernel_splats/cases/multi/dmesg.txt

@@ -0,0 +1,14 @@
+[    0.100000] Linux version 6.5.0
+[   10.000000] WARNING: CPU: 0 PID: 100 at kernel/sched/core.c:1 first+0x1/0x2
+[   10.000001] Call Trace:
+[   10.000002]  ? __warn+0x80/0xc0
+[   10.000003] ---[ end trace ]---
+[   10.500000] test_progs: PASS one
+[   11.000000] test_progs: running two
+[   11.000001] BUG: KASAN: slab-out-of-bounds in second+0x3/0x4
+[   11.000002] Read of size 8 at addr ffff888000000000 by task test_progs/9
+[   11.000003] ---[ end trace ]---
+[  199.000000] test_progs: running three
+[  200.000000] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [test_progs:9]
+[  200.000001] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [test_progs:9]
+[  201.000000] test_progs: PASS

run-vmtest/tests/check_kernel_splats/cases/multi/expected-output.txt

@@ -0,0 +1,19 @@
+::group::kernel_splats - Checking kernel log for splats
+Splat denylist: $VMTEST_CONFIGS/SPLAT_DENYLIST
+Splat allowlist: $VMTEST_CONFIGS/SPLAT_ALLOWLIST
+::endgroup::
+1-[    0.100000] Linux version 6.5.0
+2:[   10.000000] WARNING: CPU: 0 PID: 100 at kernel/sched/core.c:1 first+0x1/0x2
+3-[   10.000001] Call Trace:
+4-[   10.000002]  ? __warn+0x80/0xc0
+5-[   10.000003] ---[ end trace ]---
+6-[   10.500000] test_progs: PASS one
+7-[   11.000000] test_progs: running two
+8:[   11.000001] BUG: KASAN: slab-out-of-bounds in second+0x3/0x4
+9-[   11.000002] Read of size 8 at addr ffff888000000000 by task test_progs/9
+10-[   11.000003] ---[ end trace ]---
+11-[  199.000000] test_progs: running three
+12:[  200.000000] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [test_progs:9]
+13:[  200.000001] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [test_progs:9]
+14-[  201.000000] test_progs: PASS
+::error title=kernel_splats::kernel splat detected, see the dmesg.txt artifact

run-vmtest/tests/check_kernel_splats/cases/multi/expected-status.txt

@@ -0,0 +1 @@
+kernel_splats:1

run-vmtest/tests/check_kernel_splats/cases/no_denylist/dmesg.txt

@@ -0,0 +1,14 @@
+[    0.000000] Linux version 6.5.0
+[    0.001234] CPU: 0
+[   12.345678] ------------[ cut here ]------------
+[   12.345679] WARNING: CPU: 0 PID: 100 at kernel/sched/core.c:1234 do_thing+0x10/0x20
+[   12.345680] Modules linked in:
+[   12.345681] CPU: 0 PID: 100 Comm: test_progs
+[   12.345682] RIP: 0010:do_thing+0x10/0x20
+[   12.345683] Call Trace:
+[   12.345684]  <TASK>
+[   12.345685]  ? __warn+0x80/0xc0
+[   12.345686]  ? report_bug+0xa0/0xc0
+[   12.345687]  </TASK>
+[   12.345688] ---[ end trace ]---
+[   13.000000] test_progs: PASS

run-vmtest/tests/check_kernel_splats/cases/no_denylist/env

@@ -0,0 +1 @@
+export SPLAT_DENYLIST_FILE=/nonexistent/SPLAT_DENYLIST

run-vmtest/tests/check_kernel_splats/cases/no_denylist/expected-output.txt

@@ -0,0 +1,3 @@
+::group::kernel_splats - Checking kernel log for splats
+No splat denylist: SPLAT_DENYLIST_FILE=/nonexistent/SPLAT_DENYLIST
+::endgroup::

run-vmtest/tests/check_kernel_splats/cases/no_denylist/expected-status.txt

@@ -0,0 +1 @@
+kernel_splats:1

run-vmtest/tests/check_kernel_splats/cases/warn/dmesg.txt

@@ -0,0 +1,14 @@
+[    0.000000] Linux version 6.5.0
+[    0.001234] CPU: 0
+[   12.345678] ------------[ cut here ]------------
+[   12.345679] WARNING: CPU: 0 PID: 100 at kernel/sched/core.c:1234 do_thing+0x10/0x20
+[   12.345680] Modules linked in:
+[   12.345681] CPU: 0 PID: 100 Comm: test_progs
+[   12.345682] RIP: 0010:do_thing+0x10/0x20
+[   12.345683] Call Trace:
+[   12.345684]  <TASK>
+[   12.345685]  ? __warn+0x80/0xc0
+[   12.345686]  ? report_bug+0xa0/0xc0
+[   12.345687]  </TASK>
+[   12.345688] ---[ end trace ]---
+[   13.000000] test_progs: PASS