Skip to content

Bound full GC passes in tests to one test's allocations - #1847

Draft
DABH wants to merge 3 commits into
mainfrom
flake/gc-pause-deadlock
Draft

Bound full GC passes in tests to one test's allocations#1847
DABH wants to merge 3 commits into
mainfrom
flake/gc-pause-deadlock

Conversation

@DABH

@DABH DABH commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

What was changed

  • tests/conftest.py: a pytest_collection_finish hook runs gc.collect() then gc.freeze() once, after all test modules are imported.
  • tests/contrib/langgraph/test_replay.py: asserts the replay imports nothing into the sandbox after initial workflow load (same guard as the OpenAI Agents replay test).

Why

test_replay tripped the deadlock detector on the 3.10 macOS runner while the first activation was in StateGraph.compile(); the interrupt landed in a weakref callback inside ast.parse, i.e. during a cyclic GC pass. That activation does 3-8 ms of work and imports nothing into the sandbox. The test process heap grows to ~1.7M objects, and a full GC pass over it (160-290 ms locally, several times that on a 3-core runner shared by three xdist workers) runs on whichever thread allocates, so first activations occasionally absorb it inside the 2 s budget. Freezing the import-time heap once removes it from every later full pass. An earlier revision froze after every test; that made each test's survivors permanent (RSS 77 MB to 3.6 GB over 229 serial tests, versus a 490 MB plateau without the hook) and killed the 2-core ubuntu-arm CI runners mid-run on every attempt.

Testing

Serial tests/worker/test_workflow.py with RSS sampled: no hook plateaus at 490 MB, single freeze at 390 MB, per-test freeze climbs to 3.6 GB. Full gc.collect() per test on a 51-test subset: 70 ms mean / 179 ms max without the hook, 37 ms / 122 ms with the single freeze. LangGraph replay and sandbox suites pass on 3.10 and 3.14; poe lint clean.

tests/contrib/langgraph/test_replay.py tripped the workflow deadlock
detector on the 3.10 macOS runner while the first activation was in
StateGraph.compile(); the interrupt landed in a weakref callback inside
ast.parse, i.e. during a cyclic GC pass. That activation does 3-8 ms of
work and imports nothing into the sandbox. The test process heap grows
to ~1.7M objects and a gen2 pass over it (160-290 ms locally, several
times that on a 3-core runner shared by three xdist workers) runs on
whichever thread allocates, so first activations occasionally absorb it
inside the 2 s budget.

Freezing each test's survivors at teardown keeps later passes to one
test's allocations: on tests/contrib (3.10, -n 3) the largest GC pause
inside an activation dropped from 291 ms to 20 ms and wall time from
48 s to 33 s. The replay test also asserts nothing is imported into the
sandbox after initial workflow load, matching the OpenAI Agents replay
test.
@DABH DABH added the skip-changelog PR changes do not require changelog updates label Sep 10, 2026
Freezing after every test made each test's surviving objects permanent:
running tests/worker/test_workflow.py serially, RSS grew from 77 MB to 3.6 GB
over 229 tests versus a 490 MB plateau without the hook, and on CI the 2-core
ubuntu-arm runners died mid-run with "runner has received a shutdown signal"
on every attempt. A single freeze after collection keeps the import-time heap
out of every later full pass without accumulating anything: RSS plateaus at
390 MB and a full gc.collect() per test drops from 70 ms mean / 179 ms max to
37 ms / 122 ms on the same subset.
@DABH

DABH commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Reworked in 764aa2b (see updated description): a single post-collection freeze replaces the per-test freeze, which leaked each test's survivors and exhausted the 2-core ubuntu-arm runners.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog PR changes do not require changelog updates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant