Skip to content

Keep the release test split's settings out of pytest subprocesses #4139

Closed

Description

@anth-volk

Problem

Release run 41063091599 (the first with #4134) failed on the release test shard 6 of 6, so no deploy job ran.

run_release_tests.sh turns on the split by setting PYTEST_ADDOPTS="-p tests.release_test_shard_plugin" and the three RELEASE_TEST_SHARD_* variables. A pytest process that a test starts inherits all four, so the shard plugin runs inside it too:

  • tests/unit/test_cloud_run_push_workflow.py::test_live_staging_tests_get_one_pytest_worker_each starts pytest --collect-only on the two live staging test files. The plugin in that process splits just those two files over six shards, keeps none of them on shard 6, and pytest exits with "no tests collected (9 deselected)".
  • The plugin writes the shard's report when it collects, so the inner process also overwrites the outer process's report. On a shard where the inner run keeps its tests, the test passes, but check-release-test-shards would then fail because that shard's report lists the wrong tests.

Which shard the test file lands on depends on the files and test counts on master, so this fails every release run while the test's file lands on shards 3 to 6, and corrupts the report on shards 1 and 2.

Fix

  • The plugin reads its settings once, when pytest configures it, and then removes the RELEASE_TEST_SHARD_* variables and its own -p option from PYTEST_ADDOPTS in the process environment, so pytest processes that tests start run unsplit.
  • write_report refuses to replace a report that already exists, so a second writer fails loudly instead of overwriting a shard's report.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions