Skip to content

fix(process): kill the gate tree, bound the launch lock, allow one suite - #133

Merged
ManSio merged 5 commits into
mainfrom
fix/process-hardening
Nov 15, 2026
Merged

ManSio merged 5 commits into
mainfrom
fix/process-hardening

Conversation

@ManSio

@ManSio ManSio commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Three defects found while measuring why the suite looked like an 18-minute hang. All three produced real orphaned processes or unbounded waits on this machine.

1. The hook killed only the direct child (.githooks/pre-commit)

On TimeoutExpired it called proc.kill(). The gate is a Python script that spawns its own children: verify_diary -> pytest tests/. The direct child died, the grandchild kept running for 10+ minutes and competed with the next run. Observed twice (PIDs 7732 -> 14156 -> 16264).

Now gates are process-group leaders (CREATE_NEW_PROCESS_GROUP / start_new_session) and the timeout path kills the tree (taskkill /PID <pid> /T /F or killpg), with proc.kill() as a fallback.

2. The launch lock was unbounded (src/core/embedder/onnx_client.py)

ensure_server_running() did with _client_lock: with no timeout. A leaked thread holding it turns the next call into an indefinite wait: in the full run one test was reported at 654s while the same file takes 2.7s alone and that test < 0.01s. It now acquires with _LOCK_TIMEOUT = 150s and returns False (fail closed) instead of hanging.

3. Two suites could run on one checkout (tests/conftest.py)

A session file lock (.pytest_cache/suite.lock) makes the second run exit 3 with an explicit message. A nested pytest spawned by a test inherits MSCB_SUITE_LOCK_INHERITED=1 and skips the check, so test_bootstrap_pipeline still works. The OS releases the lock when a process dies, so a crash cannot leave it stale.

Evidence

tests: 20 passed (test_suite_lock, test_precommit_hook_contract, test_idle_reload)
ruff:  All checks passed

Real processes, not fakes:

parent=14720 child=5544 child_alive_before=True
child_alive_after=False parent_poll=1          # _kill_tree removed the grandchild

second_pytest_exit=3
! Another pytest run already holds .pytest_cache/suite.lock. ...

The new tests are the negative controls: test_timeout_kills_the_process_tree fails if the /T flag or fallback disappears, and test_second_acquire_fails_and_release_frees fails if the lock stops excluding.

Limits

  • Full pytest tests was not run to completion for this commit (1092s under a concurrent benchmark); the commit used the documented MSCB_PRECOMMIT_FAST=1 mode, so gate-zero runs in CI.
  • A benchmark process does not take the suite lock yet; it still competes for CPU. Wiring experiments/bench_2026_11/run.py to the same lock is a reasonable follow-up.

Three defects found while measuring the suite: (1) the pre-commit hook killed only the direct gate child, so a timed-out verify_diary left a full pytest running for 10+ minutes and raced the next run; the gate is now a process-group leader and the timeout path uses taskkill /T or killpg. (2) ensure_server_running waited on a module lock forever; it now gives up after 150s and returns False instead of hanging. (3) two suites could run on one checkout; a session file lock makes the second exit 3 with a clear message, while nested pytest spawned by tests inherits MSCB_SUITE_LOCK_INHERITED and skips. Real-process checks: a grandchild was gone after _kill_tree; a second pytest exited 3 while the lock was held.
@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a3a76168-538e-46f5-8f6d-f739ea7560fe

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@ManSio ManSio closed this Nov 15, 2026
@ManSio
ManSio deleted the fix/process-hardening branch November 15, 2026 07:43
@ManSio
ManSio restored the fix/process-hardening branch November 15, 2026 07:44
@ManSio ManSio reopened this Nov 15, 2026
@ManSio
ManSio merged commit 1d129ca into main Nov 15, 2026
14 checks passed
@ManSio
ManSio deleted the fix/process-hardening branch November 15, 2026 07:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant