Bound full GC passes in tests to one test's allocations - #1847
Draft
DABH wants to merge 3 commits into
Draft
Conversation
tests/contrib/langgraph/test_replay.py tripped the workflow deadlock detector on the 3.10 macOS runner while the first activation was in StateGraph.compile(); the interrupt landed in a weakref callback inside ast.parse, i.e. during a cyclic GC pass. That activation does 3-8 ms of work and imports nothing into the sandbox. The test process heap grows to ~1.7M objects and a gen2 pass over it (160-290 ms locally, several times that on a 3-core runner shared by three xdist workers) runs on whichever thread allocates, so first activations occasionally absorb it inside the 2 s budget. Freezing each test's survivors at teardown keeps later passes to one test's allocations: on tests/contrib (3.10, -n 3) the largest GC pause inside an activation dropped from 291 ms to 20 ms and wall time from 48 s to 33 s. The replay test also asserts nothing is imported into the sandbox after initial workflow load, matching the OpenAI Agents replay test.
Freezing after every test made each test's surviving objects permanent: running tests/worker/test_workflow.py serially, RSS grew from 77 MB to 3.6 GB over 229 tests versus a 490 MB plateau without the hook, and on CI the 2-core ubuntu-arm runners died mid-run with "runner has received a shutdown signal" on every attempt. A single freeze after collection keeps the import-time heap out of every later full pass without accumulating anything: RSS plateaus at 390 MB and a full gc.collect() per test drops from 70 ms mean / 179 ms max to 37 ms / 122 ms on the same subset.
Contributor
Author
|
Reworked in 764aa2b (see updated description): a single post-collection freeze replaces the per-test freeze, which leaked each test's survivors and exhausted the 2-core ubuntu-arm runners. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
tests/conftest.py: apytest_collection_finishhook runsgc.collect()thengc.freeze()once, after all test modules are imported.tests/contrib/langgraph/test_replay.py: asserts the replay imports nothing into the sandbox after initial workflow load (same guard as the OpenAI Agents replay test).Why
test_replaytripped the deadlock detector on the 3.10 macOS runner while the first activation was inStateGraph.compile(); the interrupt landed in a weakref callback insideast.parse, i.e. during a cyclic GC pass. That activation does 3-8 ms of work and imports nothing into the sandbox. The test process heap grows to ~1.7M objects, and a full GC pass over it (160-290 ms locally, several times that on a 3-core runner shared by three xdist workers) runs on whichever thread allocates, so first activations occasionally absorb it inside the 2 s budget. Freezing the import-time heap once removes it from every later full pass. An earlier revision froze after every test; that made each test's survivors permanent (RSS 77 MB to 3.6 GB over 229 serial tests, versus a 490 MB plateau without the hook) and killed the 2-core ubuntu-arm CI runners mid-run on every attempt.Testing
Serial
tests/worker/test_workflow.pywith RSS sampled: no hook plateaus at 490 MB, single freeze at 390 MB, per-test freeze climbs to 3.6 GB. Fullgc.collect()per test on a 51-test subset: 70 ms mean / 179 ms max without the hook, 37 ms / 122 ms with the single freeze. LangGraph replay and sandbox suites pass on 3.10 and 3.14;poe lintclean.