Split the longest e2e tests into concurrent sub-tests

[Amaury Chamayou (achamayou)]

Summary

The two virtual e2e jobs were the CI critical path. This PR changes no buckets and no workflows — ctest still runs each bucket serially. The tests inside it just stop being serial.

Measured across three consecutive green runs (33527819778, 33531420031, 33537384996):

Job	Before	After (3 runs)
VMSS Virtual A	13.1	13.7, 13.9, 13.7 (untouched)
VMSS Virtual B	15.2	14.6, 14.2, 13.9
VMSS Virtual C	15.5	13.2, 13.5, 14.4
ACI SNP Milan	12.7	12.8, 12.9, 12.9 (untouched)
ACI SNP Genoa	11.1	11.4, 11.0, 11.2 (untouched)

Critical path: 15.5 -> 14.2-14.6 min. Run-to-run variance is around +/-0.6 min per job; the 14.4 for job C above is mostly that, its Build Debug step alone took 3.48 min against a typical 2.9.

Where the time went

A bucket costs the sum of its tests' wall times, and each test costs its longest sub-test. Measuring node process lifetimes from the logs of run 33500039436 showed the time concentrated in a few long serial chains:

Test	Before	After	Why
partitions	282s	157s	Was the only e2e test not using ConcurrentRunner at all
nodes_test	162s	138s	29s serial tail after the main chain, run twice (IPv4 and IPv6)
schema_test	105s	73s	104s of its 105s was a single sub-test

Changes

partitions. One network ran fourteen tests in sequence, then two further networks ran one after the other. Now eight sub-tests, each on its own network, preserving the original relative order within each group.

nodes_test. run_join_old_snapshot (22s) and run_join_no_snapshot_against_original_primary (7s) ran sequentially at the end of reconfiguration.run_all, once for IPv4 and once for IPv6. Each builds its own single-node network and shares no state with the main chain, so they become four sub-tests of their own.

schema_test. Paced almost entirely by download-snapshot: one network running five backup-snapshot-fetch tests in sequence. Each finds the primary and issues its own transactions, so they split into three groups.

Partitioner made safe for concurrent networks. It now owns a private iptables chain named for the pid and an instance counter, instead of a global CCF-TEST chain that every new instance flushed. Its DROP rules already matched only its own nodes' addresses and ports, so co-existing chains do not interfere. iptc.easy operates on a table object that is shared process-wide and committed and refreshed on every call, so interleaved calls from several threads lose updates; all access takes a lock, held across a whole set of rules so a partition appears and disappears in one step. cleanup_iptables.py gains cleanup_all for leftovers from a killed run.

ConcurrentRunner uses a bounded pool. It previously ran sub-tests in fixed batches, joining each batch before starting the next, which made a cap expensive: one long sub-test held back everything queued behind it. A pool starts the next sub-test as soon as any finishes, so the cap costs nothing when a critical path dominates. The default is one per core, a ceiling on new growth rather than a tightening of what already worked. partitions_test passes a lower value, since its groups are the least tolerant of both CPU contention and iptables contention.

Leaked networks reaped. Two governance_test nodes outlived their test by ~8.5 min, logging ~660 lines/min on a join retry loop through the rest of the bucket. test_all_members stops the network it is given and recovers into a new one which it returns, but gov only rebound its local variable: the enclosing context manager still owned the original, so nothing stopped the recovered network. The nodes that leaked were the ones test_no_quote and test_node_data deliberately add without ever trusting, which is why they sat retrying rather than exiting. gov now owns the recovered network and stops it in a finally. partitions_test.run_recovery_elections had the same shape and is fixed the same way, which matters more now those groups run concurrently. Cost: governance_test 53.3s -> 57.7s, partitions unchanged at 157.6s.

This pattern is wider than these two — any test that recovers into a new network and returns it leaks unless its caller takes ownership, and recovery.py does it in several places. Left alone here.

What the intermediate runs taught us

All caught and fixed in this PR rather than guessed at:

• A first attempt capped at half the cores, which made recovery_test slower (166.9s -> 184.4s): it has nineteen sub-tests and ran unbounded before, so the cap throttled a test that was already comfortable. Raised to one per core; recovery_test is back to 164.4s.
• Running the partitions groups four at a time failed with basic timing out waiting for an election 75s after dropping its partition, which is what stale DROP rules look like. Locking per rule was not enough given the shared, auto-refreshing iptables table; the lock now spans a whole rule set, and the groups run two at a time.

Coverage and testing

• Verified by AST that the eight partitions groups make exactly the same test_* calls, in the same order, as the old run(), and that both standalone checks are still registered.
• Verified by AST that run_all keeps its tests minus the two hoisted networks, that nodes_test registers the four hoisted sub-tests alongside its original five, that the three download-snapshot groups make the same five test_* calls in the same order, and that gov and run_recovery_elections still call exactly what they did before.
• Unit-tested the ConcurrentRunner pool against stubbed infrastructure: cap respected and reached, all sub-tests run, thread names still carry the sub-test name (the log format depends on it), failures propagate naming the sub-test, --regex and -N still work, a long sub-test does not stall the queue behind it, and two runners in one process do not share sub-tests.
• Verified the iptables chain-name guard raises ValueError rather than asserting, so it survives python -O.
• All 23 e2e tests still run, and every ctest test passed in all three green runs.

Why this is close to the floor for the current job layout

Job A carries ~12.4 min of work unrelated to e2e buckets, dominated by the Debug build with clang-tidy at 7.9 min. Jobs B and C each pay ~5.35 min of fixed overhead. With the remaining e2e work spread across the three, the best achievable maximum is about 13.8 min.

So further bucket work is worth at most ~0.4 min, and the critical path cannot go much below ~13.9 min without one of:

• Reducing job A. The clang-tidy build is the single largest item in the whole workflow; nothing else in CI comes close.
• A fourth e2e runner. Considered and rejected: it only moves the critical path to ACI SNP Milan at ~12.9 min, so it buys under a minute for a whole extra runner.

Remaining test-side headroom

bucket_b is the constraint, and most of it is four tests each bounded by one long sub-test: recovery_test (164s, ~123s sub-test), nodes_test (138s, ~131s reconfiguration.run_all), e2e_logging_http2 (73s) and schema_test (73s).

Next candidates, deliberately left out:

• e2e_logging.do_main_tests, a ~35-test serial chain pacing the cpp sub-test of both e2e_logging and e2e_logging_http2, so splitting it would help both buckets. But test_historical_query_range and test_historical_receipts read transactions accumulated by earlier tests in the chain, so it needs those dependencies untangled rather than a contiguous cut. Worth ~0.3 min, since job A caps the benefit.
• reconfiguration.run_all. Its node count drifts 3 -> 4 -> 3 -> 2 across the sequence and later tests carry @reqs.at_least_n_nodes constraints that depend on that drift, so it needs per-group node counts rather than a contiguous cut.

[Copilot]

Pull request overview

This PR reduces CI critical-path time by splitting several long-running end-to-end (e2e) tests into smaller sub-tests that can run concurrently within each existing ctest bucket, and by making partitioning infrastructure safe under concurrent execution.

Changes:

• Split schema_test backup snapshot download work into multiple concurrent sub-tests.
• Hoist two reconfiguration join-snapshot checks into their own concurrent sub-tests (including IPv6 variants).
• Replace ConcurrentRunner’s fixed-batch threading with a bounded thread-pool model, and update iptables partitioning to use per-instance chains with locking for thread-safety.

Custom instructions used:

• .github/copilot-instructions.md

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/schema.py	Registers multiple backup-snapshot download sub-tests instead of one serial sub-test.
tests/reconfiguration.py	Removes tail-serial join-snapshot calls from run_all, adds IPv6 wrappers with shared availability check.
tests/partitions_test.py	Splits partitions test into concurrent groups, each with its own partitioned network, and caps concurrency.
tests/nodes.py	Registers join-snapshot checks as separate sub-tests to run concurrently with other nodes tests.
tests/infra/runner.py	Implements bounded concurrency via ThreadPoolExecutor and changes sub-test tracking to support pooling.
tests/infra/partitions.py	Makes iptables partitioning concurrency-safe via per-instance chains and a global lock, plus adds cleanup-all support.
tests/e2e_operations.py	Refactors backup-snapshot download runner to execute grouped test lists, enabling concurrent grouping upstream.
tests/cleanup_iptables.py	Updates cleanup tool to dump/cleanup all leftover per-instance chains.

Suppressed comments (2)

tests/infra/runner.py:294

• FAILURES is a module-level list populated by threading.excepthook. ConcurrentRunner.run() appends FAILURES into its local failures list but never clears FAILURES at the start of a run, so a previous run's background-thread failure can make a later run fail even if all its sub-tests pass.

Clear FAILURES at the start of run() to make repeated runs in one process reliable.

        LOG.configure(**config)

        tests = self.tests

tests/infra/partitions.py:127

• Typo in log message: "[unamed]" should be "[unnamed]".

        LOG.info(f'Dropping rules "{self.name or "[unamed]"}"')

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.