Skip to content

test_sbc_smoke: sample the tiny fits in-process so 3.14's forkserver default cannot time the fixture out - #612

Merged
jdeast merged 1 commit into
masterfrom
fix/sbc-smoke-314-timeout
Nov 19, 2026
Merged

jdeast merged 1 commit into
masterfrom
fix/sbc-smoke-314-timeout

Conversation

@jdeast

@jdeast jdeast commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Problem

On the ubuntu Python 3.14 legs, all three tests in tests/test_sbc_smoke.py keep erroring at setup with Failed: Timeout (>600.0s) from pytest-timeout. Master runs 43909944076, 44372891494 and PR #611's run 44382820961 all show it, and so do several nightly runs on the unchanged e89acda (11-11, 11-13, 11-14). The failure follows whichever shard holds the file.

Root cause

Python 3.14 changed the default multiprocessing start method on Linux from fork to forkserver. The module fixture calls pm.sample 21 times with cores=2: one prior run plus 20 replicates. pymc passes mp_ctx=None, so each call starts two worker processes with the platform default. A forkserver worker re-imports pymc/exozippy and rebuilds the compiled model on every call.

Evidence:

  • Fixture setup time from the --durations transcripts of the last ~30 master runs:
    • ubuntu 3.12: 112-387 s
    • ubuntu 3.14: 226-606 s, with roughly 2.5-4x the 3.12 time in the same run
    • 3.14 has been within ~100 s of the limit since at least 2026-11-08 (599.31 s on b567b74; 600.33 s timeout on c71244d).
    • The pass or fail outcome on 3.14 is just runner variance around 600 s. It is not a regression in the fixture, and it is not caused by the ci: refresh tests/durations.json from master run 42809842699 #607 durations refresh.
  • Per-replicate fit times printed by the failing 3.14 jobs: 20-34 s each, and 67-88 s for replicate 0.
  • Local reproduction on CPython 3.12, same model, with only mp_ctx forced:
    • fork: 2-4 s per pm.sample call
    • forkserver: 44-49 s per call (5 replicates: 81 s vs 239 s)
  • Dependency versions on the 3.14 legs (pymc 6.3.2, pytensor 3.3.6, numpy 2.4.6, scipy 1.18.1) are the same in passing and failing runs.

Fix

Set cores=1 in the fixture so the two chains run sequentially in the test process. This model samples 2 x 600 iterations in a second or two, so worker processes only ever cost time here. The fixture now runs 123 s locally (3.12, all 21 calls "Sequential sampling"; all three tests pass), and its cost no longer depends on the start-method default.

examples/validation/sbc.py itself is unchanged. The full cluster runs choose their own --cores.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AWjnwpYHm56hSpKBzbPhrG

… 3.14's forkserver

On Python 3.14 the Linux default multiprocessing start method changed from
fork to forkserver.  The SBC smoke fixture called pm.sample 21 times with
cores=2, so every call started two forkserver workers that re-import
pymc/exozippy and rebuild the compiled model: ~45 s per call against ~2-4 s
with fork (measured on CPython 3.12 with mp_ctx forced, same model).  The
fixture ran 226-606 s on the ubuntu 3.14 legs against 112-387 s on 3.12 over
the last ~30 master runs and kept hitting the 600 s pytest-timeout.

The PK model samples 2 x 600 iterations in a second or two, so worker
processes only ever cost time here.  With cores=1 the fixture is 123 s
locally (3.12) and no longer depends on the start-method default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWjnwpYHm56hSpKBzbPhrG
@jdeast
jdeast merged commit d2fa196 into master Nov 19, 2026
20 checks passed
@jdeast
jdeast deleted the fix/sbc-smoke-314-timeout branch November 19, 2026 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant