perf + security: what the fast path is worth, and eleven defects the audit of the cores found - #38
Merged
Merged
Conversation
scripts/bench-latency.sh has been in the repo since 0.4.0 and could not answer the question it exists for. The comparison that matters needs CFC armed - the fail-closed queue rule loaded, the daemon deciding, the fast path granting - and arming that on a development machine takes its network with it if anything is wrong. scripts/vm-bench boots a guest for one run instead. It assembles an initramfs out of this machine's own kernel modules, nftables, iproute2, python3 and the release binaries, boots it under KVM, and runs the bench there against a real daemon. Nothing is downloaded, nothing outside target/vm-bench is written, and the guest exists for one boot. One sudo, once, to read the host's kernel image. The states differ from each other in one thing at a time, so each difference isolates one cost: no firewall, the queue, the queue with a daemon built with a shorter idle beat, the fast path. ALT_DAEMON carries that second daemon into the same image, so a constant can be attributed by changing it and measuring both in one boot rather than by reasoning about a distribution. Two facts are recorded beside every measurement rather than assumed: what `cfc status` says the fast path is, and `id_sequence` from /proc/net/netfilter/nfnetlink_queue - one increment per packet the kernel really handed to userspace. A state calling itself `fast` whose queue saw one packet per connect did not take the fast path, and no latency figure says that on its own. Both directions run everywhere, because `in` never meets the queue and so measures the eBPF hooks alone. Three defects of its own were found by running it, all of the class this repo keeps meeting: a pipeline whose legitimate empty answer killed the build under pipefail (a module this kernel builds in rather than builds), a `/lib` created as a directory before being made a symlink, and `modprobe --show-depends` silently omitting nf_conntrack because Arch ships an install directive for it - which booted a guest with no conntrack and refused `ct state new queue num 0` with a bare ENOENT.
TODO 1a has said since the fast path landed that the number justifying it
was not measured. It is now. Per new outbound TCP flow, median, Linux
7.2.2 under KVM:
300 flows 3000 flows
no firewall 0.0158 ms 0.0162 ms
the fast path 0.0268 ms 0.0269 ms
the NFQUEUE queue 5.6745 ms 7.6083 ms
The fast path saves 5.6 ms per new flow at 300 flows and 7.6 ms at 3000,
and costs 0.011 ms over having no firewall at all. Its cost does not grow
with load, because those flows never reach the daemon; the queue's does.
The 0.28 ms this file used to quote came from a different bench and
understated the case by two orders of magnitude.
Attributing that cost rather than only recording it found two statements
in this repo that the measurement contradicts, and both are corrected
here rather than left standing beside the number that disproves them.
nfqueue.rs predicted "up to one RECV_POLL_INTERVAL (mean: half that)".
Half is right for arrivals independent of the beat and wrong for a client
connecting in series: each connect lands just after the worker observed
an empty queue and committed to a fresh wait, so it pays close to a whole
interval every time. Proved the way this file's own memory demands -
build the same daemon with the constant at 200 us, run both in one guest
- and not from the shape of a distribution, which is how this path has
been misread before: 4.90 ms of the 5.67 at 300 flows, 5.24 of the 7.61
at 3000.
docs/ARCHITECTURE.md still described the design nfqueue.rs replaced: a
worker blocking in `recv` with "no polling, no added latency" whenever no
prompt was outstanding. That has not been true since the blocking recv
became the ninety-second hang on every daemon stop.
What is left open, in TODO 1a: the remaining queued cost grows with the
number of live sockets - 0.77 ms at 300 flows against 2.36 at 3000, with
the beat removed - and that growth is the daemon's own per-packet work,
not the kernel's, because the floor moved 0.0003 ms across the same
range. Attribution is the suspect and is not yet proven.
…e my own reading did An audit of the cores - the control socket, the packet parser and the refusal path, the rule engine, DNS and provenance - returned findings I then verified against the code myself, one at a time, before writing a line. Two of them were the same shape and it is the shape that matters: a rule the user wrote was not the rule being applied. **The fast path could grant what the packet path denies.** The two deciders read different uids - exec-time on one side, live on the other - so for a program that drops privileges a uid-scoped deny above a broad allow was answered both ways, and the grant is process-wide. The grant side now stands aside wherever a uid-scoped rule could reach the program, which is exactly what the deny side already did. Picking a uid instead would have changed what every existing rule means. **A rule that could not be decided was walked past.** A deny scoped to a digest the daemon cannot compute handed the flow to a lower allow. `lookup` is three-valued now and stops there. The connection half is tested first so a rule its own destination excludes still abstains for nothing - the trap in the obvious version of this fix. **A hostname nobody confirmed could admit traffic.** The kernel gate for an observed DNS answer is "source port 53" and nothing else: no transaction id, no resolver address, no question-to-answer check. Such a name may now refuse but not admit. One direction, deliberately: refusing it both ways is the obvious change and would have quietly disarmed every `deny --dst-host` on a machine running the DNS observer. The rest, each verified before and after: ICMP refusals forged with a multicast source for mDNS and DHCP traffic; the same refusals unbudgeted off the machine, which is a reflector; four raw sockets receiving a copy of every TCP segment on the host for the daemon's life; `enforcing` latched true after one packet, including after the ruleset was removed under it; an unbounded `exe_path` from the wire; an unbounded event offset; a hostname cache whose bound one insert path did not honour. One performance change rides with them because it is the same file as a security one: attribution opened and closed a netlink socket per queued packet, measured at 0.28 ms of every queued flow. It is now one socket per thread - which is only safe because each request carries its own sequence number and the reply is checked against it. Every request used to carry seq 1, so a late answer to a timed-out request was indistinguishable from the next one's: a wrong attribution, not a slow one. The socket is discarded on anything but a cleanly-sequenced answer. CHANGELOG has the full account. Every fix carries the test that fails without it, including one that attaches the send-only filter to a UDP socket and proves the kernel really drops - the raw-socket version needs privileges the test suite does not have. Two mistakes of my own, caught by verifying rather than by assuming, are worth recording because both were the fix being wrong rather than the finding: refusing an unconfirmed hostname in both directions (it would have disarmed every deny by name), and reporting an absent nftables table verbatim (the shipped units load it *after* the daemon, so that called every boot unprotected for its first minute).
MotherSphere
changed the title
…e it has The section on observed answers framed a forgery as something an attacker must win against the resolver - "the same attacker who could also forge the forward lookup FCrDNS depends on". That is a race for a transaction id and an ephemeral port, and it is not what the code allows. The kernel gate is `source port == 53` and nothing else: any peer the host sends a UDP datagram to can answer from that port and assert any name for any address, with no spoofing, no guessing, and no involvement from the resolving library at all. Rewritten to say that, and to say what now follows from it: such a name may refuse but not admit. The `dst_host` warning above it gains the same note, because that advice - do not lean on a hostname allow rule - is now enforced rather than only given. Also here, found by reading the whole diff back rather than by a test: the fast path's own event consumer named its flows with `with_host`, which marks a name unverified whatever it is. Those connections never reach the matcher, so nothing was decided on it, but a field that says "nobody confirmed this" about a name that was confirmed is a trap for the next reader. It uses the same seam the packet path does.
Merged
MotherSphere
added a commit
that referenced
this pull request
Sep 6, 2026
Carries the measurement and the security round from #38. Minor rather than patch because behaviour changes: a hostname the daemon did not confirm against the address no longer satisfies an `allow --dst-host`, `enforcing` now answers about the nftables table rather than about whether a packet was ever seen, and a rule scoped to a digest that cannot be computed abstains instead of falling through to a lower allow. docs/HARDENING.md explains the first. Every hardcoded version moves together, the kernel crate's own lock included - which check-versions.sh learned to guard this round, after the 0.3.0 and 0.4.0 bumps had both silently skipped it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two rounds in one branch: the measurement
TODO.md1a asked for, and the fixes an audit of CFC's cores turned up while the harness that produced it was being built.The measurement
Per new outbound TCP flow, median, Linux 7.2.2 under KVM:
The fast path saves 5.6 ms per new flow at 300 flows and 7.6 ms at 3000, and costs 0.011 ms over having no firewall.
scripts/vm-benchboots a throwaway guest from this machine's own kernel and binaries so CFC can be measured armed.Two documentation claims the attribution disproved are corrected:
nfqueue.rspredicted half an idle beat per queued flow and it is a whole one, anddocs/ARCHITECTURE.mdstill described the blocking-recvdesign that was replaced.The security round
Eleven defects, every one verified against the code before a line was written. The two that matter most are the same shape - a rule the user wrote was not the rule being applied:
denyscoped to a digest the daemon cannot compute handed the flow to a lowerallow.lookupis three-valued now.deny --dst-host.Also: ICMP refusals forged with a multicast source for ordinary mDNS and DHCP traffic; those refusals unbudgeted off the machine, which is a reflector; four raw sockets receiving a copy of every TCP segment on the host for the daemon's life;
enforcinglatched true after one packet, including after the ruleset was removed under it; an unboundedexe_pathfrom the wire; an unbounded event offset; a hostname cache whose bound one insert path did not honour.One performance change rides along: attribution opened a netlink socket per queued packet (0.28 ms of every queued flow), now one per thread - safe only because each request carries its own sequence number and the reply is checked against it.
Verification
402 daemon tests, 17 suites, clippy clean with
-D warningsin both feature configurations,fmtclean, version guard green. Every fix carries the test that fails without it, including one that attaches the send-only filter to a UDP socket and proves the kernel really drops. Validated end to end in the VM: the daemon comes up, arms, filters, the fast path takes zero packets through the queue, and the daemon logs no warning.Two mistakes of my own were caught by that verification rather than by assuming, and both were the fix being wrong rather than the finding: refusing an unconfirmed hostname in both directions, and reporting an absent nftables table verbatim when the shipped units load it after the daemon.
Left open, deliberately
The full DNS fix is to check the answer's source against a configured resolver, which needs the source address in the ring record and therefore an ABI bump. The mitigation here removes the security consequence without it. Also open: the queued cost that is not the idle beat grows with the live socket count, and that growth is the daemon's own per-packet work.