Run MPI tests under mpi-pytest instead of one mpirun per test case

[pancetta]

Addresses #539.

pySDC ran MPI tests by having pytest subprocess a fresh mpirun per test case, with each test
file re-entering itself through a __main__ argparse block. That is slow, and the useful part of a
failure — the assertion, the rank it happened on — was reduced to a return code.

This moves pySDC/tests to mpi-pytest, which
runs pytest itself under mpiexec and gives each @pytest.mark.parallel(n) test
MPI.COMM_WORLD of a right-sized job. The suite pays one interpreter start per rank count instead
of one per test.

Measured on this CI

	master	this PR
machine-minutes	276.6	187.2
wall clock	27.7 min	20.8 min
mpi4py job (each of 5)	22–24 min	2.2–2.9 min

Per the earlier CI-duration work the mpi4py jobs were what set the pipeline's wall clock; they
are no longer on the critical path, which is now parallelSDC_reloaded and its floor of genuine SDC
numerics. The firedrake job is unchanged at 9.7 min — it only has five MPI tests, so there was
never much launch cost there; porting it was about consistency and dropping pytest-isolate-mpi.

Why this plugin

Measured on test_MPI_sweeper.py (72 cases, 2 ranks, median of 3, one machine):

Model	Configuration	Wall clock
—	status quo: subprocess + mpirun per case	78.0 s
pytest under MPI	mpi-pytest non-forking	2.71 s (29x)
pytest under MPI	pytest_parallel (ONERA), static scheduler	2.30 s (34x)
MPI job per test	pytest-isolate-mpi 0.3	89.6 s (1.15x slower)
MPI job per test	mpi-pytest forking mode	166.2 s (2.1x slower)

Two things worth stating plainly, because #539 bundles "slow" and "bad error messages" as one
problem and they pull in opposite directions:

• Both job-per-test plugins are slower than what we already do. A pytest subsession costs more
  to start than the plain python script.py child the current code spawns. Migrating to
  pytest-isolate-mpi for the diagnostics would have been a ~15% wall-clock tax.
• The cost is Python imports, not mpirun. Locally mpirun -np 2 python -c pass is 0.03 s
  while a full child is 0.58 s. The win comes from amortising the pySDC/numpy/scipy import.

mpi-pytest over pytest_parallel on operational grounds rather than speed: it is on PyPI, it does
not collide with pytest-timeout (our only deadlock guard), its isolate-mode equivalents work on
macOS, it was already in the Firedrake image, and it ships parallel_assert for rank-divergent
assertions that otherwise deadlock.

Coverage

Model B needs no setup — coverage run wraps each rank, so coverage measures the process that
executes the test. Measured on generic_implicit_MPI.py:

	data files	coverage	needs COVERAGE_PROCESS_START?
status quo (subprocess)	13	78%	yes — each test file sets it by hand
mpi-pytest non-forking	2	78%	no
a job-per-test plugin	1	0%	yes, and nothing sets it

That last row is a silent failure: tests pass, the run looks healthy, the number quietly drops.

CI shape

For the mpi4py, petsc, firedrake and RayleighBenard jobs: one mpiexec -n N pytest launch
per rank count in use, plus one ordinary pass. parallel[1] selects tests that are unmarked or
explicitly serial, so nothing runs twice and nothing falls back to forking mode.

run_pass exists because shell: bash -l {0} carries no -e — GitHub adds that only for the
bash shorthand — so a failing pass mid-loop would otherwise be swallowed and the step's status
would come from the last command alone. It propagates failures but tolerates pytest's exit 5 for a
rank count no test in that environment asks for.

The rank counts live in the workflow while @pytest.mark.parallel(n) lives in the test, so a test
asking for a count nobody launches would be deselected by every pass and never run — invisible in a
green pipeline. etc/check_parallel_partition.sh asserts the passes claim exactly as many tests as
the marker selects, and runs in all four jobs before their tests. It needs no MPI (the markers are
attached at collection time) and takes seconds. It fails as intended when a parallel(6) test is
added while the workflow launches 2..5.

Verification that the tests actually run

Collection partition is exact, and so is execution. With a plugin asserting that every
parallel-marked test sees the world size it declares:

n=2  world=2  executed=250  all declaring 2     250 passed, 0 skipped
n=3  world=3  executed=17   all declaring 3      17 passed, 0 skipped
n=4  world=4  executed=82   all declaring 4      82 passed, 0 skipped
n=5  world=5  executed=3    all declaring 5       3 passed, 0 skipped
serial world=1 executed=339 all declaring 1     339 passed, 0 skipped
                                                ---
                              691 executed = 691 collected

No test is skipped anywhere, and none runs on a communicator smaller than it asked for.

The tests also still bite. Deleting a collective in generic_implicit_MPI fails them:

mutation	n=2	n=3
drop the end-point Allreduce	20 failed	1 failed
drop the uend Bcast	39 failed	3 failed

A real bug this surfaced: leaked mpi4py-fft communicators

A PFFT allocates MPI communicators (MPI_Cart_create plus an MPI_Cart_sub per axis) and never
releases them. Creating and dropping IMEX_Laplacian_MPIFFT fails at 1022 instances on MPICH:

MPI_Cart_create(MPI_COMM_WORLD, ndims=3, ...) failed

Pre-existing and not test-specific — any long-running driver that builds many problems hits the same
ceiling. It stayed invisible because every MPI test used to get its own short-lived mpirun.
Sharing one session reaches it: without the fix the serial pass fails 108 tests while each of those
files passes in isolation and master shows zero failures.

Fixed in pySDC/helpers/fft_helper.py: a PFFT subclass whose __del__ calls destroy(). One
definition, since the owners (three problem/transfer classes and SpectralHelper's cache) share no
base. destroy() is idempotent and arrays made with newDistArray stay usable after it, both
checked before tying release to garbage collection.

Not related to the exit-143 MPICH flakiness fixed in #671: that was a silent SIGTERM ~1 s into
collection, before any test body runs and therefore before any PFFT exists, and it was
Azure-region correlated. This fails deep into a run with a loud MPI error stack and no region
dependence.

Three smaller defects found while porting

• test_differentiation_matrix2D never accepted a communicator — it built
  SpectralHelper(comm=None) unconditionally, so test_differentiation_MPI was running N redundant
  serial copies under mpirun, paying the launch cost while testing nothing about the
  distributed path.
• Rectilinear.setupMPI is class-level state, so after the fieldsIO writes every later read
  returned only the local block. The old model got the global read for free because the checks ran
  in the parent process. Now reset explicitly, in a finally.
• The Firedrake container needs the OpenMPI 4 oversubscribe variable. Fix the exit-143 CI flakiness, and stop the duplicate runs #671 set
  PRTE_MCA_rmaps_default_mapping_policy, which is the OpenMPI 5 spelling; that image ships
  OpenMPI 4 and reads OMPI_MCA_rmaps_base_oversubscribe. It only showed up now because the job
  never launched more than two ranks before.

The firedrake job has never measured coverage

Its coverage database carries ~365 file rows and one row of line data on master. The container
mounts the workspace at /repositories, the job sat in ./pySDC, and pySDC is installed editable
from /repositories/pySDC — the same directory under two path strings, so what its modules report
never matched source = ['pySDC'] resolved against the working directory. Running from
/repositories/pySDC makes them agree:

	file rows	line data
master	365	1
this PR	371	60

Now covering firedrake_ensemble_communicator, pySDC_as_gusto_time_discretization,
firedrake_mesh, GenericGusto, HeatFiredrake, TransferFiredrakeMesh and both step_7 tutorials.

The tutorial

C_pySDC_with_PETSc.main() read sys.argv directly, so the only way to drive it was as a script.
It now takes num_procs_space and fname, with argv parsing moved into __main__ — still runnable
exactly as the tutorial text describes, and callable from the tests.

pytest-isolate-mpi removed

Nothing uses it, so the dependency and the 0.3 pin are gone. The pin existed because 0.4 broke
module teardown (#691); 0.3 in turn refuses to load inside an mpiexec session, which made
incremental migration impossible without -p no:pytest_isolate_mpi.

Not ported, deliberately

• test_crash.py — aborts a rank on purpose, which needs isolation.
• test_datatypes.py::test_PyTorch_dtype — runs in the pytorch job, which has no mpi-pytest; that
  is why the file keeps its launcher and __main__.

Developer workflow

Unchanged: plain pytest <file> and pytest <file>::<test> still run the MPI tests, via
mpi-pytest's forking mode. Slower, but you do not need to remember a rank count to run one test.

🤖 Generated with Claude Code

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
pySDC/helpers
fft_helper.py					26
spectral_helper.py					1721-1726
pySDC/implementations/problem_classes
AllenCahn_Temp_MPIFFT.py
generic_MPIFFT_Laplacian.py
pySDC/implementations/transfer_classes
TransferMesh_MPIFFT.py
pySDC/tutorial/step_7
C_pySDC_with_PETSc.py					35, 43
Project Total

This report was generated by python-coverage-comment-action