From 210df8c3b832d5887d531a6e4ebc35ec3f75ee25 Mon Sep 17 00:00:00 2001 From: Umar Sabirin Date: Mon, 7 Sep 2026 09:54:43 +0700 Subject: [PATCH 1/4] feat: measure the kernel attack surface of the creation path Count distinct host kernel functions a runtime traverses while it holds root, which is the window where every privileged operation happens and the one axis published runtime comparisons do not cover. mars reaches 1559 functions to start a container against crun's 2062 and runc's 2351, and 484 against 920 and 1140 on exec. Namespace, pivot_root, cgroup and capability work appears at parity, so the gap is not skipped work: of the 843 functions runc reaches and mars does not, 250 are file and /proc traversal and only 10 are thread or scheduler functions. Widening the workload to cover seccomp exposed a blocker: mars refuses any profile naming a syscall libseccomp cannot resolve, so the profile Podman and CRI-O ship needs 59 names removed before it will start. runc and crun skip unresolvable names and accept it unchanged. The script asserts rather than trusts, because three separate methodology faults each produced a plausible wrong answer first: per-PID trace filters lose crun's child process, global tracing charges a runtime for its neighbours, and a runtime that exits non-zero still yields a number. --- README.md | 2 + docs/attack-surface.md | 129 ++++++++++++++++++ scripts/hap-bench.sh | 289 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 420 insertions(+) create mode 100644 docs/attack-surface.md create mode 100755 scripts/hap-bench.sh diff --git a/README.md b/README.md index 12dfdaf..5bc72be 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,8 @@ created and wrote itself. | Integration suite | 128 assertions, all reading kernel state rather than the runtime's own claims | | Docker drop-in | `run`, `run -it`, `exec`, `stop`, `--memory` — [details](docs/06-docker-and-tracing.md) | | Startup tracing | OTLP spans per phase, accepted by Tempo | +| Kernel attack surface | **1559** distinct host kernel functions to start a container, against `crun`'s 2062 and `runc`'s 2351 — [details](docs/attack-surface.md) | +| Known defect | rejects standard seccomp profiles: 59 syscall names must be removed before it will start, where `runc` and `crun` remove none — [details](docs/attack-surface.md#what-the-benchmark-found-in-mars) | | Not implemented | rootless without privilege, CNI networking, image pulling, cgroup v1, systemd cgroup driver | ## Why build this when runc exists? diff --git a/docs/attack-surface.md b/docs/attack-surface.md new file mode 100644 index 0000000..15668e2 --- /dev/null +++ b/docs/attack-surface.md @@ -0,0 +1,129 @@ +# The kernel surface of starting a container + +A runtime spends its whole life as root. It holds `CAP_SYS_ADMIN`, it is the thing that creates +namespaces and writes cgroups, and when it is finished it exits and hands a much less privileged +process the machine. Everything dangerous it will ever do, it does in that window. + +Nobody measures that window. Published comparisons of `runc`, `crun` and `youki` measure how *fast* +they start a container and how much memory the result costs. Both are real, and both are largely +settled: `youki` reached parity with `runc` and moved nothing, and per-container memory belongs to +whoever removes the shim. + +This measures something else — how much of the kernel a runtime touches while it is privileged. + +## The metric + +James Bottomley's *horizontal attack profile* approximates exposure as the amount of kernel code a +workload traverses, on the assumption that bug density is roughly uniform, so more code reached means +more chance of reaching a bug. He counts distinct kernel functions entered, traced with `ftrace`. + +Bottomley applied it to *isolation platforms* at steady state: a container versus a VM, running the +same application. Later work (van Rijn and Rellermeyer, Middleware '21) extended it by weighting each +function with an exploitability score. + +Applying it at steady state to OCI runtimes would measure nothing. Once a container is running, the +runtime has already exited; `runc`, `crun` and `mars` all leave behind the same namespaces and the +same process. The difference lives entirely in the setup path, which is exactly the part that runs as +root. + +## Running it + +```sh +sudo -E scripts/hap-bench.sh # all four workloads +sudo -E RUNS=9 scripts/hap-bench.sh exec # one workload, more repetitions +``` + +The workload is a static C program that returns immediately, so what is counted is the runtime's own +work rather than an application's. That choice follows Jansen et al., who use a minimal C service for +the same reason. + +## Three ways to get a wrong number + +Every one of these produced a plausible, wrong result before it was caught. They are the reason the +script asserts rather than trusts. + +**Per-PID trace filters lose children.** `trace-cmd -F -c` is documented to follow forks, and it does +follow `runc`'s. It silently failed to follow `crun`'s container process, which made `crun` look three +times leaner than `runc`. The script traces globally and attributes afterwards by process name. + +**Global tracing catches the neighbours.** With the filter removed, `containerd` and `udisksd` land in +the same trace. One `runc` run picked up 875 functions that were not its own, including 74 OverlayFS +functions that a plain-directory bundle cannot possibly reach. Attribution by process name is what +separates the runtime's work from the machine's. + +**A runtime that fails still produces a number.** `runc spec` writes `ociVersion: 1.3.0`; `crun` +1.14.1 rejects it with `unknown version specified` and exits 1. Measured with stderr hidden, that +failure looked like an extremely efficient runtime. Every run is now checked for exit status and for +the container process appearing in the trace; runs that fail either check are discarded and counted. + +Three idle traces are taken and their union subtracted, so kernel background work is not charged to +anyone. + +## Results + +aarch64, Linux 6.8, five runs per cell, median, idle subtracted, all runtimes sharing one seccomp +profile. `tty` is a detached start with a pty passed over a console socket, so it covers creation +only and is not comparable in absolute terms to the columns that run to container exit. + +| runtime | `run` | `vol` | `exec` | `tty` | +|---|---|---|---|---| +| `mars` | **1559** | **1560** | **484** | **1544** | +| `crun` 1.14.1 | 2062 | 2065 | 920 | 2017 | +| `runc` 1.5.1 | 2351 | 2354 | 1140 | 2366 | + +`mars` reaches 24% less kernel than `crun` and 34% less than `runc` on a plain start, and 47% / 58% +less on `exec` — the operation a Kubernetes exec probe repeats for the lifetime of a pod. + +Three things fall out of the matrix: + +**It is not that `mars` skips work.** Namespace creation, `pivot_root`, cgroup setup and capability +handling all appear at parity; on cgroups `mars` matches `runc` and touches eight times what `crun` +does. Of the 843 functions `runc` reaches and `mars` does not, only 10 are thread, futex or scheduler +functions — the Go runtime is not the explanation. 250 are file, path and `/proc` traversal. + +**A bind mount is nearly free.** `vol` costs one to three functions more than `run` for every runtime. +The mount machinery has already been walked to assemble the rootfs. + +**A pty costs almost nothing.** `tty` lands within a few percent of `run` despite allocating a +terminal and passing a descriptor over a socket. + +The 40 functions `mars` reaches that `runc` does not are its own instrumentation: +`cgroup_events_show`, `memory_events_show`, `cpu_stat_show`, `css_task_iter_*` — reading the cgroup +event and statistics files that [`failure-modes.md`](failure-modes.md) is built on. + +## What the benchmark found in `mars` + +Widening the workload to cover seccomp turned up a defect that the narrow one hid. + +`mars` will not start a container whose profile names a syscall `libseccomp` cannot resolve. It fails +on the first one: + +``` +config.json is invalid: add a seccomp rule for bdflush: +The library doesn't permit the particular operation +``` + +`runc` and `crun` accept the same profile unchanged. Against the profile Podman and CRI-O ship, +**59 syscall names have to be removed before `mars` will start**, among them `bpf`, `setns`, +`chroot`, `init_module`, `perf_event_open`, `userfaultfd` and `kexec_load`. Both other runtimes skip +unresolvable names deliberately; `mars` propagates the error as fatal. + +This is a blocker rather than a rough edge. Every real platform ships a profile carrying legacy x86 +syscall names, so `mars` would refuse to start under all of them. + +The fix has to be narrower than "ignore errors from `seccomp_rule_add`". Skipping a name is only safe +because these profiles deny by default; under a profile with an `SCMP_ACT_ALLOW` default, silently +dropping a name would open a syscall that was meant to be closed. Only the arch-unavailable case may +be skipped, and every other libseccomp error must stay fatal. + +## Limits + +The number is an approximation, and Bottomley says so: counting function entries cannot see control +flow *inside* a function, so a ten-line function and a five-hundred-line one count the same. +Basic-block coverage through `kcov` is the honest version, and Ubuntu's generic kernel ships with +`CONFIG_KCOV` off — it needs a kernel built for the purpose. + +Beyond that: one architecture, so these figures cannot be lined up against published x86 results; one +kernel version; and a guest kernel under Apple's hypervisor rather than bare metal. The comparison +between runtimes is sound because all three meet identical conditions. The absolute values are not +portable. diff --git a/scripts/hap-bench.sh b/scripts/hap-bench.sh new file mode 100755 index 0000000..5057962 --- /dev/null +++ b/scripts/hap-bench.sh @@ -0,0 +1,289 @@ +#!/usr/bin/env bash +set -uo pipefail + +usage() { + cat >&2 <<'EOF' +usage: hap-bench.sh [WORKLOAD ...] + +Measure the horizontal attack profile of the container creation path: how many +distinct host kernel functions a runtime traverses while it holds root, for one +container carrying a minimal static C program. + +Workloads (default: all four) + run foreground run to container exit + vol same, with a read-only bind mount + exec exec into an already-running container + tty detached run with a pty handed over a console socket + +Environment + RUNTIMES space separated NAME=PATH pairs + (default: "runc=runc crun=crun mars=mars") + RUNS traced repetitions per cell (default: 5) + OUT working directory (default: /tmp/hap-bench) + BUF trace-cmd ring buffer, KB per CPU (default: 60000) + PROFILE seccomp profile URL to adapt + (default: the containers/common profile Podman and CRI-O ship) + +Requires root, and: gcc jq curl python3 trace-cmd runc. +EOF + exit 2 +} + +[[ "${1:-}" == "-h" || "${1:-}" == "--help" ]] && usage + +RUNTIMES="${RUNTIMES:-runc=runc crun=crun mars=mars}" +RUNS="${RUNS:-5}" +OUT="${OUT:-/tmp/hap-bench}" +BUF="${BUF:-60000}" +PROFILE="${PROFILE:-https://raw.githubusercontent.com/containers/common/main/pkg/seccomp/seccomp.json}" +WORKLOADS=("$@") +[[ ${#WORKLOADS[@]} -eq 0 ]] && WORKLOADS=(run vol exec tty) + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +RECVTTY="$REPO_ROOT/tests/recvtty.py" + +die() { echo "error: $*" >&2; exit 1; } + +[[ "$(uname -s)" == "Linux" ]] || die "this measures Linux kernel functions; run it in a Linux VM" +[[ "$EUID" -eq 0 ]] || die "tracing and container creation need root; use sudo -E" +for t in gcc jq curl python3 trace-cmd runc; do + command -v "$t" >/dev/null || die "$t is not on PATH" +done +[[ -r /sys/kernel/tracing/available_tracers ]] || + die "ftrace is not exposed at /sys/kernel/tracing" +grep -qw function /sys/kernel/tracing/available_tracers || + die "the kernel has no function tracer; CONFIG_FUNCTION_TRACER is off" +[[ -r "$RECVTTY" ]] || die "$RECVTTY not found" + +rm -rf "$OUT"; mkdir -p "$OUT" + +declare -a RT_NAMES RT_BINS +for pair in $RUNTIMES; do + n=${pair%%=*}; b=${pair#*=} + command -v "$b" >/dev/null || { echo "skipping $n: $b is not on PATH" >&2; continue; } + RT_NAMES+=("$n"); RT_BINS+=("$b") +done +[[ ${#RT_NAMES[@]} -gt 0 ]] || die "no runtime under test is installed" + +build_programs() { + printf 'int main(void){return 0;}\n' > "$OUT/app.c" + printf '#include \nint main(void){sleep(300);return 0;}\n' > "$OUT/sleeper.c" + gcc -static -O2 -o "$OUT/app" "$OUT/app.c" || die "cannot build a static test program" + gcc -static -O2 -o "$OUT/sleeper" "$OUT/sleeper.c" || die "cannot build the sleeper" +} + +fetch_profile() { + curl -sSL -o "$OUT/profile-src.json" "$PROFILE" || die "cannot fetch $PROFILE" + jq -e . "$OUT/profile-src.json" >/dev/null 2>&1 || + die "$PROFILE did not return JSON; the upstream path may have moved" +} + +adapt_profile() { + local caps_json=$1 arch=$2 scmp_json=$3 + jq --argjson caps "$caps_json" --arg arch "$arch" --argjson scmp "$scmp_json" ' + { + defaultAction: .defaultAction, + architectures: $scmp, + syscalls: [ .syscalls[] + | select( + ((.includes.arches // null) == null or (.includes.arches | any(. == $arch))) + and ((.includes.caps // null) == null + or (.includes.caps | any(. as $c | $caps | index($c)))) + and ((.excludes.caps // null) == null + or ((.excludes.caps | map(. as $c | $caps | index($c)) + | map(select(. != null)) | length) == 0)) + ) + | {names, action} + + (if (.args // null) != null and (.args | length) > 0 then {args} else {} end) + + (if (.errnoRet // null) != null then {errnoRet} else {} end) + ] + }' "$OUT/profile-src.json" > "$OUT/profile-oci.json" +} + +drop_syscall() { + jq --arg s "$1" ' + .syscalls = [ .syscalls[] + | .names = [ .names[] | select(. != $s) ] + | select((.names | length) > 0) ]' "$OUT/profile.json" > "$OUT/p.tmp" + mv "$OUT/p.tmp" "$OUT/profile.json" +} + +converge_profile() { + cp "$OUT/profile-oci.json" "$OUT/profile.json" + : > "$OUT/profile-rejections.txt" + local i n bad + for ((i = 0; i < ${#RT_NAMES[@]}; i++)); do + n=${RT_NAMES[i]} + local removed=0 + while true; do + write_bundle "$OUT/probe" "$OUT/app" '["/app"]' false + "${RT_BINS[i]}" delete -f "happrobe-$n" >/dev/null 2>&1 + local err; err=$("${RT_BINS[i]}" run --bundle "$OUT/probe" "happrobe-$n" 2>&1) + local rc=$? + "${RT_BINS[i]}" delete -f "happrobe-$n" >/dev/null 2>&1 + [[ $rc -eq 0 ]] && break + bad=$(printf '%s' "$err" \ + | grep -oE 'seccomp rule for [a-z0-9_]+' | head -1 | awk '{print $NF}') + [[ -z "$bad" ]] && die "$n fails on the probe bundle for a non-seccomp reason: $err" + echo "$n $bad" >> "$OUT/profile-rejections.txt" + drop_syscall "$bad" + removed=$((removed + 1)) + [[ $removed -gt 200 ]] && die "$n rejected more than 200 syscalls; giving up" + done + echo " $n accepts the profile after $removed removals" + done +} + +write_bundle() { + local dir=$1 prog=$2 args=$3 terminal=$4 mount=${5:-} + rm -rf "$dir"; mkdir -p "$dir/rootfs" + cp "$prog" "$dir/rootfs/$(basename "$prog")" + [[ "$prog" != "$OUT/app" ]] && cp "$OUT/app" "$dir/rootfs/app" + ( cd "$dir" && runc spec ) + local extra='.' + [[ -n "$mount" ]] && extra='.mounts += [{"destination":"/data","type":"bind","source":"'"$mount"'","options":["rbind","ro"]}]' + jq --slurpfile sc "$OUT/profile.json" --argjson args "$args" --argjson tty "$terminal" " + .ociVersion = \"1.0.2\" + | .process.args = \$args + | .process.terminal = \$tty + | .root.readonly = true + | .linux.seccomp = \$sc[0] + | $extra" "$dir/config.json" > "$dir/c.json" + mv "$dir/c.json" "$dir/config.json" +} + +record() { + local label=$1; shift + trace-cmd record -p function -b "$BUF" -o "$OUT/$label.dat" "$@" \ + >"$OUT/$label.cmdout" 2>&1 + trace-cmd report -i "$OUT/$label.dat" 2>/dev/null > "$OUT/$label.rep" + rm -f "$OUT/$label.dat" +} + +attribute() { + local label=$1 rt=$2 + : > "$OUT/$label.own" + awk -v rt="$rt" ' + /function:/ { + p = $1; sub(/-[0-9]+$/, "", p) + if (p == rt || p == "app" || p == "sleeper" || index(p, rt ":") == 1) + print $NF > OWN + }' OWN="$OUT/$label.own" "$OUT/$label.rep" + awk '/function:/ { p = $1; sub(/-[0-9]+$/, "", p); print p }' "$OUT/$label.rep" \ + | sort -u > "$OUT/$label.procs" + sort -u -o "$OUT/$label.own" "$OUT/$label.own" + rm -f "$OUT/$label.rep" +} + +measure_idle() { + local i + for i in 1 2 3; do + record "idle-$i" sleep 1 + awk '/function:/ {print $NF}' "$OUT/idle-$i.rep" | sort -u > "$OUT/idle-$i.funcs" + rm -f "$OUT/idle-$i.rep" + done + sort -u "$OUT"/idle-*.funcs > "$OUT/idle.union" + echo " idle union: $(wc -l < "$OUT/idle.union") functions" +} + +echo "== preparing ==" +build_programs +fetch_profile +mkdir -p "$OUT/probe-spec" && ( cd "$OUT/probe-spec" && runc spec ) +CAPS=$(jq -c '.process.capabilities.bounding // []' "$OUT/probe-spec/config.json") +case "$(uname -m)" in + aarch64) ARCH=arm64; SCMP='["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM"]' ;; + x86_64) ARCH=amd64; SCMP='["SCMP_ARCH_X86_64","SCMP_ARCH_X86","SCMP_ARCH_X32"]' ;; + *) die "no seccomp architecture mapping for $(uname -m)" ;; +esac +adapt_profile "$CAPS" "$ARCH" "$SCMP" +echo " profile: $(jq '[.syscalls[].names[]] | length' "$OUT/profile-oci.json") syscall names, arch $ARCH" +converge_profile +echo " shared profile: $(jq '[.syscalls[].names[]] | length' "$OUT/profile.json") syscall names" + +write_bundle "$OUT/b-run" "$OUT/app" '["/app"]' false +write_bundle "$OUT/b-tty" "$OUT/app" '["/app"]' true +write_bundle "$OUT/b-exec" "$OUT/sleeper" '["/sleeper"]' false +mkdir -p "$OUT/voldir" && echo present > "$OUT/voldir/marker" +write_bundle "$OUT/b-vol" "$OUT/app" '["/app"]' false "$OUT/voldir" + +echo "== idle baseline ==" +measure_idle + +echo "== measuring ==" +: > "$OUT/results" +for ((i = 0; i < ${#RT_NAMES[@]}; i++)); do + name=${RT_NAMES[i]}; bin=${RT_BINS[i]} + for wl in "${WORKLOADS[@]}"; do + counts=(); dropped=0 + for ((r = 1; r <= RUNS; r++)); do + id="hap-$name-$wl-$r"; lbl="$name-$wl-$r"; need_app=1; rp="" + "$bin" delete -f "$id" >/dev/null 2>&1 + case $wl in + run) record "$lbl" "$bin" run --bundle "$OUT/b-run" "$id" ;; + vol) record "$lbl" "$bin" run --bundle "$OUT/b-vol" "$id" ;; + exec) + "$bin" run -d --bundle "$OUT/b-exec" "$id" >/dev/null 2>&1 + sleep 1 + record "$lbl" "$bin" exec "$id" /app + ;; + tty) + python3 "$RECVTTY" "$OUT/con-$name.sock" "$OUT/con-$name.out" \ + >"$OUT/con-$name.log" 2>&1 & + rp=$! + for _ in $(seq 1 50); do + grep -q listening "$OUT/con-$name.log" 2>/dev/null && break + sleep 0.1 + done + record "$lbl" "$bin" run -d --bundle "$OUT/b-tty" \ + --console-socket "$OUT/con-$name.sock" "$id" + need_app=0 + ;; + *) die "unknown workload: $wl" ;; + esac + attribute "$lbl" "$name" + [[ -n "$rp" ]] && kill -9 "$rp" >/dev/null 2>&1 + "$bin" delete -f "$id" >/dev/null 2>&1 + if ! grep -qx "$name" "$OUT/$lbl.procs" || + { [[ $need_app -eq 1 ]] && ! grep -qx app "$OUT/$lbl.procs"; }; then + dropped=$((dropped + 1)); rm -f "$OUT/$lbl.own"; continue + fi + counts+=( "$(comm -23 "$OUT/$lbl.own" "$OUT/idle.union" | wc -l)" ) + done + if [[ ${#counts[@]} -eq 0 ]]; then + printf '%s|%s|-|-|%s\n' "$name" "$wl" "$dropped" >> "$OUT/results" + echo " $name/$wl: every run discarded" + continue + fi + sort -u "$OUT/$name-$wl"-*.own > "$OUT/$name-$wl.raw" + comm -23 "$OUT/$name-$wl.raw" "$OUT/idle.union" > "$OUT/$name-$wl.union" + med=$(printf '%s\n' "${counts[@]}" | sort -n \ + | awk '{a[NR]=$1} END {print a[int((NR + 1) / 2)]}') + printf '%s|%s|%s|%s|%s\n' "$name" "$wl" "$med" \ + "$(wc -l < "$OUT/$name-$wl.union")" "$dropped" >> "$OUT/results" + echo " $name/$wl: median $med, union $(wc -l < "$OUT/$name-$wl.union"), $dropped discarded" + done +done + +echo +echo "distinct host kernel functions, median of $RUNS runs, idle subtracted" +printf '%-8s' runtime; printf ' %8s' "${WORKLOADS[@]}"; printf '\n' +for name in "${RT_NAMES[@]}"; do + printf '%-8s' "$name" + for wl in "${WORKLOADS[@]}"; do + printf ' %8s' "$(awk -F'|' -v r="$name" -v w="$wl" \ + '$1 == r && $2 == w {print $3}' "$OUT/results")" + done + printf '\n' +done + +if [[ -s "$OUT/profile-rejections.txt" ]]; then + echo + echo "syscalls each runtime refused to build a filter for" + awk '{c[$1]++} END {for (r in c) printf " %-8s %d\n", r, c[r]}' \ + "$OUT/profile-rejections.txt" + echo " full list: $OUT/profile-rejections.txt" +fi + +echo +echo "raw function sets: $OUT/*.union" From 4556fada4afa3fc61d3349b6a01ef08118e62fbd Mon Sep 17 00:00:00 2001 From: Umar Sabirin Date: Mon, 7 Sep 2026 10:51:30 +0700 Subject: [PATCH 2/4] fix: accept a seccomp rule that restates the filter's default action MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mars refused to start under the profile Podman and CRI-O ship: 59 syscall names had to be removed first, where runc and crun removed none. The names that failed — bdflush, vm86, oldfstat, pciconfig_* — all look like syscalls absent on aarch64, so the first reading was that libseccomp could not resolve them. It already skipped unresolvable names; the failure came from seccomp_rule_add, whose "the library doesn't permit the particular operation" is libseccomp's wording for EACCES, returned when a rule's action equals the filter's default action. These profiles deny by default and then spell out denials for privileged syscalls, so every such rule is redundant by construction. All 59 came from SCMP_ACT_ERRNO entries under an SCMP_ACT_ERRNO default; none was an architecture problem. Tolerate EACCES only when the rule's verdict equals the default, which cannot change what the filter permits. Every other libseccomp error stays fatal, and a deny rule under an allow-by-default filter carries the whole policy, differs from the default and so never takes the tolerant path — asserted by test. Splitting filter construction from load makes this testable without applying seccomp to the test process. All three runtimes now share the unmodified 442-name profile, and the surface figures are restated from that run. --- README.md | 3 +- docs/attack-surface.md | 75 +++++++++++++---------- src/container/seccomp.rs | 125 +++++++++++++++++++++++++++++++++++---- 3 files changed, 160 insertions(+), 43 deletions(-) diff --git a/README.md b/README.md index 5bc72be..7af6cdc 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,7 @@ created and wrote itself. | Integration suite | 128 assertions, all reading kernel state rather than the runtime's own claims | | Docker drop-in | `run`, `run -it`, `exec`, `stop`, `--memory` — [details](docs/06-docker-and-tracing.md) | | Startup tracing | OTLP spans per phase, accepted by Tempo | -| Kernel attack surface | **1559** distinct host kernel functions to start a container, against `crun`'s 2062 and `runc`'s 2351 — [details](docs/attack-surface.md) | -| Known defect | rejects standard seccomp profiles: 59 syscall names must be removed before it will start, where `runc` and `crun` remove none — [details](docs/attack-surface.md#what-the-benchmark-found-in-mars) | +| Kernel attack surface | **1542** distinct host kernel functions to start a container, against `crun`'s 2030 and `runc`'s 2361 — [details](docs/attack-surface.md) | | Not implemented | rootless without privilege, CNI networking, image pulling, cgroup v1, systemd cgroup driver | ## Why build this when runc exists? diff --git a/docs/attack-surface.md b/docs/attack-surface.md index 15668e2..c583371 100644 --- a/docs/attack-surface.md +++ b/docs/attack-surface.md @@ -67,54 +67,69 @@ only and is not comparable in absolute terms to the columns that run to containe | runtime | `run` | `vol` | `exec` | `tty` | |---|---|---|---|---| -| `mars` | **1559** | **1560** | **484** | **1544** | -| `crun` 1.14.1 | 2062 | 2065 | 920 | 2017 | -| `runc` 1.5.1 | 2351 | 2354 | 1140 | 2366 | +| `mars` | **1542** | **1542** | **494** | **1545** | +| `crun` 1.14.1 | 2030 | 2030 | 889 | 1984 | +| `runc` 1.5.1 | 2361 | 2355 | 1134 | 2370 | -`mars` reaches 24% less kernel than `crun` and 34% less than `runc` on a plain start, and 47% / 58% +`mars` reaches 24% less kernel than `crun` and 35% less than `runc` on a plain start, and 44% / 56% less on `exec` — the operation a Kubernetes exec probe repeats for the lifetime of a pod. -Three things fall out of the matrix: +Four things fall out of the matrix: **It is not that `mars` skips work.** Namespace creation, `pivot_root`, cgroup setup and capability handling all appear at parity; on cgroups `mars` matches `runc` and touches eight times what `crun` -does. Of the 843 functions `runc` reaches and `mars` does not, only 10 are thread, futex or scheduler -functions — the Go runtime is not the explanation. 250 are file, path and `/proc` traversal. +does. Of the 1000 functions `runc` reaches on a plain start and `mars` does not, only 10 are thread, +futex or scheduler functions — the Go runtime is not the explanation, which is the opposite of what +the shim's memory cost would suggest. 301 are file, path and `/proc` traversal, and 116 are +networking. -**A bind mount is nearly free.** `vol` costs one to three functions more than `run` for every runtime. -The mount machinery has already been walked to assemble the rootfs. +**A bind mount is free.** `vol` costs the same as `run` to the function for `mars` and `crun`, and six +fewer for `runc`. The mount machinery has already been walked to assemble the rootfs. -**A pty costs almost nothing.** `tty` lands within a few percent of `run` despite allocating a -terminal and passing a descriptor over a socket. +**A pty costs almost nothing.** `tty` lands within a few percent of `run` for all three despite +allocating a terminal and passing a descriptor over a socket. -The 40 functions `mars` reaches that `runc` does not are its own instrumentation: -`cgroup_events_show`, `memory_events_show`, `cpu_stat_show`, `css_task_iter_*` — reading the cgroup -event and statistics files that [`failure-modes.md`](failure-modes.md) is built on. +**The ordering is stable across every workload.** Four different operations, the same ranking and +roughly the same ratios. A measurement artefact would have to survive all four to explain that. -## What the benchmark found in `mars` +The 29 functions `mars` reaches that `runc` does not are its own instrumentation: +`cgroup_events_show`, `memory_events_show`, `cpu_stat_show`, `cgroup_base_stat_cputime_show`, +`__arm64_sys_ppoll` — reading the cgroup event and statistics files that +[`failure-modes.md`](failure-modes.md) is built on. -Widening the workload to cover seccomp turned up a defect that the narrow one hid. +## What the benchmark found in `mars` -`mars` will not start a container whose profile names a syscall `libseccomp` cannot resolve. It fails -on the first one: +Widening the workload to cover seccomp turned up a defect the narrow one hid. Against the profile +Podman and CRI-O ship, `mars` refused to start at all: ``` config.json is invalid: add a seccomp rule for bdflush: The library doesn't permit the particular operation ``` -`runc` and `crun` accept the same profile unchanged. Against the profile Podman and CRI-O ship, -**59 syscall names have to be removed before `mars` will start**, among them `bpf`, `setns`, -`chroot`, `init_module`, `perf_event_open`, `userfaultfd` and `kexec_load`. Both other runtimes skip -unresolvable names deliberately; `mars` propagates the error as fatal. - -This is a blocker rather than a rough edge. Every real platform ships a profile carrying legacy x86 -syscall names, so `mars` would refuse to start under all of them. - -The fix has to be narrower than "ignore errors from `seccomp_rule_add`". Skipping a name is only safe -because these profiles deny by default; under a profile with an `SCMP_ACT_ALLOW` default, silently -dropping a name would open a syscall that was meant to be closed. Only the arch-unavailable case may -be skipped, and every other libseccomp error must stay fatal. +59 syscall names had to be removed before it would run — `bpf`, `setns`, `chroot`, `init_module`, +`perf_event_open`, `userfaultfd`, `kexec_load` among them. `runc` and `crun` accepted the same +profile unchanged, removing none. + +The first diagnosis was wrong, and worth recording because it was plausible. Those names look like +syscalls that do not exist on aarch64, so the obvious reading was that `libseccomp` could not resolve +them. But `mars` already skipped unresolvable names; the failure came one step later, from +`seccomp_rule_add`. + +That message is `libseccomp`'s wording for `EACCES`, and `seccomp_rule_add` returns `EACCES` for a +rule whose action equals the filter's *default* action — a rule that asks for what the filter already +does. These profiles deny by default and then spell out denials for the privileged syscalls, so every +such rule is redundant by construction. All 59 rejected names came from `SCMP_ACT_ERRNO` entries +under an `SCMP_ACT_ERRNO` default; not one was an architecture problem. + +Getting the cause right made the fix narrow. `mars` now tolerates `EACCES` **only** when the rule's +verdict equals the default action, which cannot change what the filter permits: the syscall was +already denied by the default and stays denied. Every other `libseccomp` error is still fatal, and a +deny rule under an `SCMP_ACT_ALLOW` default — where the rule carries the whole policy and dropping it +would open the syscall it was written to close — has a different action from the default, so it never +takes the tolerant path. A unit test asserts exactly that case. + +All three runtimes now share the unmodified 442-name profile. ## Limits diff --git a/src/container/seccomp.rs b/src/container/seccomp.rs index 99a7581..f5285ff 100644 --- a/src/container/seccomp.rs +++ b/src/container/seccomp.rs @@ -1,6 +1,6 @@ use libseccomp::{ - ScmpAction, ScmpArch, ScmpArgCompare, ScmpCompareOp, ScmpFilterAttr, ScmpFilterContext, - ScmpSyscall, + error::SeccompErrno, ScmpAction, ScmpArch, ScmpArgCompare, ScmpCompareOp, ScmpFilterAttr, + ScmpFilterContext, ScmpSyscall, }; use oci_spec::runtime::{ Arch, LinuxSeccomp, LinuxSeccompAction, LinuxSeccompFilterFlag, LinuxSeccompOperator, @@ -9,6 +9,16 @@ use oci_spec::runtime::{ use crate::error::{Error, Result}; pub fn apply(spec: &LinuxSeccomp) -> Result { + let (filter, rules) = build(spec)?; + + filter + .load() + .map_err(|error| Error::Invalid(format!("load the seccomp filter: {error}")))?; + + Ok(rules) +} + +fn build(spec: &LinuxSeccomp) -> Result<(ScmpFilterContext, usize)> { let default = action(spec.default_action(), spec.default_errno_ret())?; let mut filter = ScmpFilterContext::new(default) .map_err(|error| Error::Invalid(format!("create a seccomp filter: {error}")))?; @@ -70,22 +80,34 @@ pub fn apply(spec: &LinuxSeccomp) -> Result { .map(comparator) .collect::>>()?; - if comparators.is_empty() { + let added = if comparators.is_empty() { filter.add_rule(verdict, syscall) } else { filter.add_rule_conditional(verdict, syscall, &comparators) - } - .map_err(|error| Error::Invalid(format!("add a seccomp rule for {name}: {error}")))?; + }; - rules += 1; + match added { + Ok(_) => rules += 1, + Err(error) + if verdict == default && error.errno() == Some(SeccompErrno::EACCES) => + { + tracing::debug!( + syscall = %name, + "the spec restates the filter's default action for this syscall, so \ + libseccomp refuses the rule as redundant; dropping it cannot change \ + what the filter permits" + ); + } + Err(error) => { + return Err(Error::Invalid(format!( + "add a seccomp rule for {name}: {error}" + ))); + } + } } } - filter - .load() - .map_err(|error| Error::Invalid(format!("load the seccomp filter: {error}")))?; - - Ok(rules) + Ok((filter, rules)) } fn action(requested: LinuxSeccompAction, errno: Option) -> Result { @@ -214,4 +236,85 @@ mod tests { ScmpSyscall::from_name("chmod").unwrap(); assert!(ScmpSyscall::from_name("definitely_not_a_syscall").is_err()); } + + fn seccomp(json: &str) -> LinuxSeccomp { + serde_json::from_str(json).expect("the test's seccomp fragment should deserialize") + } + + #[test] + fn a_rule_restating_the_default_action_is_tolerated() { + let spec = seccomp( + r#"{ + "defaultAction": "SCMP_ACT_ERRNO", + "architectures": ["SCMP_ARCH_NATIVE"], + "syscalls": [ + { "names": ["bpf", "setns", "init_module"], "action": "SCMP_ACT_ERRNO" } + ] + }"#, + ); + + let (_, rules) = build(&spec).expect( + "libseccomp refuses these as redundant, and refusing the container over that is \ + what stopped every standard profile from loading", + ); + + assert_eq!(rules, 0); + } + + #[test] + fn a_rule_that_differs_from_the_default_is_still_added() { + let spec = seccomp( + r#"{ + "defaultAction": "SCMP_ACT_ERRNO", + "architectures": ["SCMP_ARCH_NATIVE"], + "syscalls": [ + { "names": ["write", "read"], "action": "SCMP_ACT_ALLOW" } + ] + }"#, + ); + + let (_, rules) = build(&spec).unwrap(); + + assert_eq!(rules, 2); + } + + #[test] + fn tolerating_the_redundant_rule_does_not_tolerate_an_inverted_one() { + let spec = seccomp( + r#"{ + "defaultAction": "SCMP_ACT_ALLOW", + "architectures": ["SCMP_ARCH_NATIVE"], + "syscalls": [ + { "names": ["bpf"], "action": "SCMP_ACT_ERRNO" } + ] + }"#, + ); + + let (_, rules) = build(&spec).unwrap(); + + assert_eq!( + rules, 1, + "under an allow-by-default filter a deny rule carries the whole policy; silently \ + dropping it would open the syscall it was written to close" + ); + } + + #[test] + fn the_standard_profile_shape_loads_end_to_end() { + let spec = seccomp( + r#"{ + "defaultAction": "SCMP_ACT_ERRNO", + "architectures": ["SCMP_ARCH_NATIVE"], + "syscalls": [ + { "names": ["read", "write", "exit_group"], "action": "SCMP_ACT_ALLOW" }, + { "names": ["bdflush", "vm86", "uselib"], "action": "SCMP_ACT_ERRNO" }, + { "names": ["definitely_not_a_syscall"], "action": "SCMP_ACT_ALLOW" } + ] + }"#, + ); + + let (_, rules) = build(&spec).unwrap(); + + assert_eq!(rules, 3); + } } From 72ca0f3174225745b727a2289fb605d6dc4ee8c8 Mon Sep 17 00:00:00 2001 From: Umar Sabirin Date: Mon, 7 Sep 2026 11:00:02 +0700 Subject: [PATCH 3/4] style: apply rustfmt to the seccomp rule-add match --- src/container/seccomp.rs | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/src/container/seccomp.rs b/src/container/seccomp.rs index f5285ff..12e8bd2 100644 --- a/src/container/seccomp.rs +++ b/src/container/seccomp.rs @@ -1,6 +1,6 @@ use libseccomp::{ - error::SeccompErrno, ScmpAction, ScmpArch, ScmpArgCompare, ScmpCompareOp, ScmpFilterAttr, - ScmpFilterContext, ScmpSyscall, + ScmpAction, ScmpArch, ScmpArgCompare, ScmpCompareOp, ScmpFilterAttr, ScmpFilterContext, + ScmpSyscall, error::SeccompErrno, }; use oci_spec::runtime::{ Arch, LinuxSeccomp, LinuxSeccompAction, LinuxSeccompFilterFlag, LinuxSeccompOperator, @@ -88,9 +88,7 @@ fn build(spec: &LinuxSeccomp) -> Result<(ScmpFilterContext, usize)> { match added { Ok(_) => rules += 1, - Err(error) - if verdict == default && error.errno() == Some(SeccompErrno::EACCES) => - { + Err(error) if verdict == default && error.errno() == Some(SeccompErrno::EACCES) => { tracing::debug!( syscall = %name, "the spec restates the filter's default action for this syscall, so \ From f5f42fa78d52ebfa16953f0472c08b030cf99094 Mon Sep 17 00:00:00 2001 From: Umar Sabirin Date: Mon, 7 Sep 2026 11:23:37 +0700 Subject: [PATCH 4/4] chore: point the repository field at MarStack-Labs/marstack-container --- Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Cargo.toml b/Cargo.toml index 1b7bbef..bbb9d92 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -5,7 +5,7 @@ edition = "2024" rust-version = "1.85" license = "Apache-2.0" description = "An OCI-compliant container runtime written from scratch in Rust" -repository = "https://github.com/umars28/mars-container-runtime" +repository = "https://github.com/MarStack-Labs/marstack-container" [lib] name = "mars"