Skip to content

test(subprocess): sample one process tree idle and loaded - #2992

Open
DeusData wants to merge 1 commit into
mainfrom
fix/test-subprocess-tree-rss-one-tree
Open

DeusData wants to merge 1 commit into
mainfrom
fix/test-subprocess-tree-rss-one-tree

Conversation

@DeusData

Copy link
Copy Markdown
Owner

What does this PR do?

Makes subprocess_tree_rss_measures_contained_descendants (tests/test_subprocess.c) deterministic. The test checks that the process-tree memory probe (cbm_subprocess_tree_rss_bytes) counts the contained descendants of a supervised process, not only its root.

Why it was flaky

The test sampled two independently spawned process trees, one idle and one whose descendant touched 128 MiB, and required the loaded total to be at least 96 MiB above the idle total. Two separate trees need not start from the same footprint. On the hosted Windows CLANG64 leg they did not, and the test failed twice in two days on PRs that do not touch this code:

The diagnosis that the two separate trees are the cause comes from #2965 (commit ce65401 by Bowen-Xue-ac2). Thank you for tracking it down.

What changed

One tree, sampled twice, with the test and the descendant moving in lock step over two pipes:

  1. The test creates two pipes, "up" (tree to test) and "down" (test to tree): FIFOs inside the test's private temporary directory on POSIX, and on Windows named pipes named after that directory's unique leaf (FILE_FLAG_FIRST_PIPE_INSTANCE, PIPE_REJECT_REMOTE_CLIENTS).
  2. The runner's __cbm_rss_tree_root role connects both ends as its first action, then starts ONE __cbm_rss_tree_hold descendant that inherits the connected ends (fork on POSIX; on Windows an explicit two-handle PROC_THREAD_ATTRIBUTE_HANDLE_LIST, so nothing else is inherited). The root then drops its own copies, so the descendant alone holds the tree's side.
  3. The descendant writes I; the test takes the idle sample and writes L; the descendant touches every page of a 192 MiB allocation (one volatile store per 4 KiB) and writes R; the test takes the loaded sample and closes its ends, which releases the descendant to free the load and exit.
  4. Every wait in the handshake is a blocking pipe read. A descendant that dies closes its ends, so the waiting read returns end-of-file instead of blocking. The ready-file poll loop (rss_tree_wait_ready, 10 ms interval) is removed; no handshake step has a sleep, a poll interval or a deadline.

The floor stays at 96 MiB and is now half the load. The root does not grow between the two samples, so a probe that counts only the root misses the floor by the full 96 MiB, while a probe that counts the descendant clears it with 96 MiB to spare (page rounding, allocator bookkeeping, a working-set trim between the two samples).

The two member-fault tests (subprocess_tree_rss_fails_when_a_live_member_cannot_be_measured, subprocess_tree_rss_skips_a_member_that_exited_mid_probe) use the same handshake and keep their assertions; the injected fault still applies to a sample taken while the descendant is alive.

Not changed: production code (only tests/test_main.c and tests/test_subprocess.c change), and rss_tree_finish, which still drives the tree to its terminal state through cbm_subprocess_poll, the subprocess API's only way to advance supervision.

Evidence

  • macOS arm64, ASan build: build/c/test-runner subprocess passes (60 passed, 2 skipped: the two Windows-only Job Object probes). The rewritten test, 20 consecutive runs: 20/20 pass, idle 43 MiB and loaded 259 MiB in every run.
  • Binding: with the macOS probe temporarily limited to the root process (non-root members skipped), the test fails at the floor assertion with idle 33 MiB and loaded 33 MiB. Restoring the probe returns the suite to green.
  • Linux arm64 container (gcc, ASan; scripts/test.sh --suites subprocess in the test-infrastructure test image): 60 passed, 2 skipped. The rewritten test, 20 consecutive runs: 20/20 pass, idle 462 MiB and loaded 654 MiB in every run.
  • Windows: both changed files compile cleanly with -fsyntax-only and the test build's warning flags (-Wall -Wextra -Werror) under llvm-mingw for x86_64 and aarch64 (the image pinned in test-infrastructure/Dockerfile.mingw). The handshake itself first runs on Windows in this PR's CI; the Windows leg should show the test passing with its tree rss: idle=... loaded=... line.

Checklist

  • Every commit is signed off (git commit -s)
  • Tests pass locally: the subprocess suite on macOS and on Linux arm64 (see Evidence); the full suite is left to PR CI
  • Lint: CI lint (cppcheck + clang-format) covers src/, which is unchanged. On the changed test lines, clang-format (Homebrew LLVM) is clean and cppcheck with the lint-cppcheck flags reports nothing. No NOLINT added.
  • New behavior is covered by a test: this change is the test; the binding check above shows it fails when the probe stops at the root

subprocess_tree_rss_measures_contained_descendants compared two
independently spawned process trees, one idle and one whose descendant
touched 128 MiB, and required the loaded total to be at least 96 MiB
above the idle total. Two separate trees need not start from the same
footprint. On the hosted Windows CLANG64 leg they did not: the loaded
tree measured only 78 MiB above the idle one (167 MiB vs 245 MiB), and
the test failed on two unrelated PRs within two days (#2965 on
889170c, #2975 run 43499140720). The diagnosis that the two separate
trees are the cause comes from #2965.

The test now samples ONE tree twice, and every wait in the handshake is
a blocking pipe read instead of a poll of a ready file:

- The test creates two pipes, "up" (tree to test) and "down" (test to
  tree): FIFOs in its private temporary directory on POSIX, named pipes
  named after that directory's unique leaf on Windows. The
  __cbm_rss_tree_root role connects both as its first action and starts
  one __cbm_rss_tree_hold descendant that inherits the connected ends
  (fork on POSIX, an explicit two-handle inherit list on Windows); the
  root then drops its own copies.
- The descendant writes 'I'; the test takes the idle sample and writes
  'L'; the descendant touches every page of a 192 MiB load and writes
  'R'; the test takes the loaded sample and closes its ends, which
  releases the descendant to free the load and exit.
- A descendant that dies closes its ends, so the waiting read returns
  end-of-file instead of blocking. No handshake step has a sleep, a poll
  interval or a deadline.

The floor stays at 96 MiB and is now half the load: the root does not
grow between the two samples, so a probe that counts only the root
misses the floor by 96 MiB, and a probe that counts the descendant
clears it with 96 MiB to spare. The two member-fault tests use the same
handshake and keep their assertions. Production code is unchanged.

Evidence:
- macOS arm64, ASan: subprocess suite 60 passed, 2 skipped (the
  Windows-only Job Object probes); the rewritten test passed 20 of 20
  consecutive runs, idle 43 MiB and loaded 259 MiB in every run.
- Binding: with the macOS probe temporarily limited to the root process,
  the test fails at the floor assertion (idle 33 MiB, loaded 33 MiB);
  restoring the probe returns the suite to green.
- Linux arm64 container, gcc ASan: subprocess suite 60 passed,
  2 skipped; the rewritten test passed 20 of 20 consecutive runs, idle
  462 MiB and loaded 654 MiB in every run.
- Windows: both changed files compile with -fsyntax-only -Wall -Wextra
  -Werror under llvm-mingw for x86_64 and aarch64.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData DeusData mentioned this pull request Nov 14, 2026
4 tasks done

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant