Repository navigation
Skip the Docker containers for suites too small to gain from them #426
Description
Part of #394.
Since wp-cli/.github#313, every Linux Behat job runs its scenarios in 4 parallel workers in Docker containers. Large suites gain a lot from it. Small suites pay more than they get back, because every job has a fixed setup cost of about 70 to 80 seconds, whatever its size:
- 40 to 63 seconds before the first worker starts. Most of this is building the worker image from
utils/docker/Dockerfile(see Use a prebuilt Docker image for the parallel workers #425 for that part). - Another 20 to 30 seconds for starting the containers and database servers, and for removing them again.
In the full-matrix runs of all 29 packages on 2026-11-02, compared with the median of their October runs on main, runner time per Testing run changed as follows:
| Packages | Before (min) | After (min) | Change | |
|---|---|---|---|---|
| Large suites | 9 | 3,762 | 1,893 | −50% |
| Small suites | 20 | 1,734 | 2,214 | +28% |
In the job for PHP 8.4, WP latest and MySQL 8.0:
| Package | Scenarios | Behat step | Longest worker |
|---|---|---|---|
maintenance-mode-command |
3 | 71 s | 10 s |
role-command |
5 | 84 s | 14 s |
site-health-command |
5 | 87 s | 16 s |
super-admin-command |
5 | 91 s | 20 s |
eval-command |
21 | 91 s | 13 s |
entity-command, for comparison |
511 | 457 s | 372 s |
Before the change, maintenance-mode-command used 49 runner-minutes per Testing run; now it uses 97.
Proposal: with WP_CLI_TEST_DOCKER=auto, run the scenarios without Docker when their expected cost is too low for the containers to pay off, and report it like the other fallbacks ("... Running the scenarios without Docker."). The expected cost per scenario that utils/behat-parallel.php already computes for balancing the workers can decide this, so the threshold does not depend on scenario counts alone. Below the threshold, parallel workers on the runner itself (#405) are still available. These add only a few seconds, as long as the test user can create the worker databases.
WP_CLI_TEST_DOCKER=1 keeps forcing containers, so that a small suite can still be run in them deliberately.
Measuring: pick the threshold from the per-job numbers above. Then run the full matrix of a few small packages (for example maintenance-mode-command, role-command, eval-command, cron-command) and a few packages near the threshold (for example cache-command, export-command) against the change, and compare runner time and Behat time per job with the runs of 2026-11-02. If #425 lands first, measure against it, as a cheaper image moves the break-even point.
Activity