Skip to content

Stop faking SIGCHLD on a CLONE_THREAD exit - #361

Merged
jserv merged 1 commit into
sysprog21:mainfrom
alanhc:futex-phantom-eintr-v2
Sep 5, 2026
Merged

Stop faking SIGCHLD on a CLONE_THREAD exit#361
jserv merged 1 commit into
sysprog21:mainfrom
alanhc:futex-phantom-eintr-v2

Conversation

@alanhc

@alanhc alanhc commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

When the last CLONE_THREAD worker exits, forkipc.c raises the futex
interrupt, which makes the next blocking call in the surviving thread return
EINTR. The comment gave the reason: "In real Linux, child exit delivers
SIGCHLD which interrupts futex_wait with -EINTR."

Linux does not. clone(2) is explicit that a thread created with
CLONE_THREAD sends no signal to its parent when it terminates; the thread
group's exit signal goes to the process's parent, not to a sibling. Nothing
is delivered, so the sibling's wait runs to its timeout.

Reproduction

Clone a CLONE_THREAD worker that exits at once, wait for the
CLONE_CHILD_CLEARTID store, then park for 300 ms with nothing to wake the
caller. tests/test-futex-no-phantom-eintr.c is that program; the three
rows below are what it observes.

call Linux 6.18.44 elfuse before
futex(FUTEX_WAIT) ETIMEDOUT, 303ms EINTR, 101ms
ppoll 0, 303ms EINTR, 0ms
epoll_pwait 0, 311ms EINTR, 302ms
nanosleep 0, 309ms 0, 328ms

No signal is pending on either side. nanosleep agrees because it does not
read the flag, which identifies the flag as the mechanism rather than
anything process-wide.

Why the request can go

The interrupt is not what wakes anybody. The futex waiter wakes on its own
100 ms quantum and re-checks thread_stop_requested, its woken flag,
expired itimers and queued signals; futex_interrupt_consume is one reason
among several, not the wake. wakeup_pipe_signal and thread_interrupt_all
stay, so threads parked on the shared pipe and inside hv_vcpu_run still
get the nudge this site sends. Teardown keeps its interrupt: the four
callers that mean it go through thread_wake_all_blocked, where every
thread really is leaving. This site was the only one raising it while the
process carried on running.

The one-shot consume from 520568c is untouched. That fixed the flag staying
set, which is why foot spun on EINTR forever, and left open whether the
flag should have been set at all. The premise dates to the initial import.

Two clean matrix runs bound the liveness risk without closing it. A lost
wake that the interrupt had been masking would surface as a hang rather
than a failure, and the argument above is what rules that out: every
re-check the waiter performs survives the change.

Test

tests/test-futex-no-phantom-eintr.c fails 3 of 7 before the change, passes
after, and passes unchanged on the reference kernel. It is registered in
test-matrix.sh under run_unit_tests and not in tests/manifest.txt, per
that file's scope note, since every assertion is a timeout the guest can
observe.

Its CLONE_CHILD_CLEARTID wait is spelled out rather than reusing
raw_futex_wait, which carries FUTEX_PRIVATE_FLAG. The exit-time wake is
a plain futex wake, so a private wait never matches its key; the test hung
on the reference kernel until that was a plain wait with a timeout.

Environment and status

MacBookPro18,3, Apple M1 Pro, macOS 26.5.2 (25F84), SDK 26.5, Apple clang
21.0.0. Reference kernel is Linux 6.18.44 through the qemu-aarch64 lane,
QEMU 11.1.1. Rebased on main at 4388e7c.

make check stops at build/test-hello with aarch64-none-elf-as: No such file or directory; the bare-metal toolchain needs a sudo installer this
machine cannot run, so test-hello and the full test-matrix lanes are
unbuilt rather than passing. What did run: tests/driver.sh passes 98 of
98, and tests/test-matrix.sh elfuse-aarch64 passes 280 with 0 failed and
10 skipped, twice. That lane is the real-application check, since it runs
busybox, coreutils, dash, lua and jq under the patched binary rather than
unit tests alone. The new test passes 5 of 5 under elfuse and 3 of 3 on the
reference kernel under QEMU_ACCEL=tcg, where timing is slowest.
check-format, check-ascii, check-eintr-contract, check-lock-order,
check-atomics, check-skill-refs, check-syscall-coverage,
check-svc-tails and .ci/check-matrix-lists.sh are clean.

cubic-dev-ai[bot]

This comment was marked as resolved.

@jserv jserv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refine tests/test-futex-no-phantom-eintr.c to be more consistent with existing naming schemes.

@alanhc
alanhc force-pushed the futex-phantom-eintr-v2 branch from 0fb5df7 to f707291 Compare September 5, 2026 03:55
@alanhc

alanhc commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator Author

Naming: renamed ctid/child_stack/burn_worker to child_tid/child_stack_buf/spawn_and_reap_worker, matching test-thread.c and test-exec-handoff.c's conventions.

cubic's findings:

  1. tests/test-futex-no-phantom-eintr.c:89 — the reap loop's own FUTEX_WAIT on the CLEARTID word could consume the one-shot phantom EINTR before the calls under test run, passing the test even with the bug back. Replaced with an nanosleep-based poll (same pattern as msleep in test-exit-group-teardown.c), which never touches the futex path. Verified by temporarily reintroducing futex_interrupt_request() in forkipc.c: the old FUTEX_WAIT-based reap loop would have been a live blind spot, and this version still fails the same 3 of 7 cases as before the fix.
  2. src/runtime/futex.c:587 — reworded to say the atomic interrupt is consumed by one waiter while teardown state marks every thread as leaving, not "an EINTR each of them has to see."
  3. src/runtime/futex.c:66-68 — updated the stale file-header comment; it no longer claims to simulate SIGCHLD on CLONE_THREAD exit.

Rebuilt, ran on elfuse and cross-checked on the qemu reference kernel (Linux 6.18.44) — both pass 7/7 unchanged.

@jserv

jserv commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Rename tests/test-futex-no-phantom-eintr.c to tests/test-futex-ops.c to comply with the file-naming scheme, where ops refers to the futex operations being validated.

When the last CLONE_THREAD worker exited, forkipc.c raised the futex
interrupt, which makes the next blocking call in the surviving thread
return EINTR. The comment gave the reason: "In real Linux, child exit
delivers SIGCHLD which interrupts futex_wait with -EINTR."

Linux does not. clone(2) is explicit that a thread created with
CLONE_THREAD sends no signal to its parent when it terminates; the
thread group's exit signal goes to the process's parent, not to a
sibling. Nothing is delivered, so a sibling's wait is not interrupted
and its timeout is what ends it.

Measured on Linux 6.18.44 through the qemu lane, a worker exits and the
main thread then parks for 300 ms with nothing to wake it:

  call                  Linux            elfuse before
  futex(FUTEX_WAIT)     ETIMEDOUT 303ms  EINTR 101ms
  ppoll                 0         303ms  EINTR   0ms
  epoll_pwait           0         311ms  EINTR 302ms
  nanosleep             0         309ms  0     328ms

No signal is pending on either side. nanosleep agrees because it does
not read the flag, which is what identifies the flag as the mechanism
rather than anything process-wide.

Removing the request drops the fabricated errno and nothing else. The
futex waiter is not woken by the interrupt: it wakes on its own 100 ms
quantum and re-checks thread_stop_requested, its woken flag, expired
itimers and queued signals, of which futex_interrupt_consume is one
reason among several. wakeup_pipe_signal and thread_interrupt_all stay,
so threads parked on the shared pipe and inside hv_vcpu_run still get
the nudge this site exists to send.

Teardown keeps its interrupt. All four callers that mean it go through
thread_wake_all_blocked, where teardown state marks every thread as
leaving but the atomic interrupt itself is consumed by only one waiter.
This site was the only one that raised it while the process carried on
running.

The one-shot consume in 520568c is untouched. That fixed the flag
staying set, which is why foot spun on EINTR forever; it did not ask
whether the flag should have been set at all.

tests/test-futex-ops.c parks in each of the three calls above after a
worker exit and fails on an early return. It fails three of seven
before this change, passes after, and passes unchanged on the reference
kernel. It lives in test-matrix.sh's run_unit_tests and not in
tests/manifest.txt, per that file's scope note: every assertion is a
timeout the guest can observe, so it is cross-checkable.

The worker's own reap loop polls the CLEARTID store with nanosleep
rather than a futex wait: a futex wait on any address, including that
one, is a chance to consume the one-shot interrupt before the calls
under test run, which would pass the test even with the bug back.
Reconfirmed against the reintroduced bug: the reap loop's own wait no
longer hides it, and the same three cases fail again. Naming follows
test-thread.c and test-exec-handoff.c: child_tid, child_stack_buf,
spawn_and_reap_worker.

Filed as test-futex-no-phantom-eintr.c, which named the regression;
renamed to test-futex-ops.c to match the file-naming scheme, where ops
names the futex operations under test (test-file-ops.c is the existing
precedent).
@alanhc
alanhc force-pushed the futex-phantom-eintr-v2 branch from f707291 to e7dc395 Compare September 5, 2026 04:10
@alanhc

alanhc commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator Author

Renamed to tests/test-futex-ops.c (matching test-file-ops.c's precedent), and updated the reference in test-matrix.sh. Rebuilt and reran; 7/7 pass unchanged.

@jserv
jserv merged commit 8a91493 into sysprog21:main Sep 5, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants