Skip to content

Split the pull request tests into the same six shards as the release tests #4144

Closed

Description

@anth-volk

Problem

The pull request workflow's Test job runs the whole make test suite on one runner. On #4140's last run it took 28 minutes (13:58 to 14:26 UTC). The release run already splits the same suite over six runners (#4134), where each shard finishes in 3 to 5 minutes.

The split also runs only on release commits, so a pull request can merge with a change that breaks it. That is what happened in release run 41063091599: a test that starts pytest itself failed only on shard 6 (#4139), and nothing ran the split before the merge.

Proposal

  • pr.yml's test job becomes a matrix of six shards (fail-fast: false). Each shard runs make test with the shard plugin loaded through PYTEST_ADDOPTS, as .github/scripts/run_release_tests.sh does for push.yml's release-tests job, and uploads the tests it collected and ran.
  • A new job downloads every shard's report and runs python -m scripts.release_test_shards check-coverage, which fails unless each collected test ran in exactly one shard. CI complete needs it.
  • The shards' coverage reports go to Codecov in one upload from a job that needs every shard, so Codecov gets one upload for the test suite, as before the split.
  • A test checks that pr.yml and push.yml use the same shards.

The cross-database integration tests took 29 minutes on the same run, so the pull request workflow stays about that long until that job is also faster.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions