Run the full test matrix in the Release workflow before publishing

[laughingman7743]

WHAT

• test.yaml can be called by other workflows (workflow_call).
• release.yaml first runs it as the test job for the tagged commit. The release job (build, PyPI publish, GitHub release) needs: test, so nothing is built or published unless every suite passes on every supported Python version.
• Python versions per trigger, all selected in the changes job from the single PYTHON_VERSIONS list:

Trigger	Python versions
Pull request	newest (unchanged from #863)
Weekly schedule	newest (was: every version)
Manual dispatch	every version, or those listed in the new python-versions input, such as 3.12 or 3.11,3.14
Release (a workflow_call from a tag push)	every version

• The dispatch input is validated against PYTHON_VERSIONS:
  • Whitespace and duplicates are ignored.
  • An unsupported value fails the changes job, so no suite starts.
• Every trigger except pull requests still runs every suite (PyAthena, SQLAlchemy, Spark).
• Permissions:
  • The workflow-level permissions of release.yaml are now {}.
  • The test job grants what the Test workflow needs: contents: read, id-token: write for OIDC, and pull-requests: read for the changes job.
  • release keeps id-token: write and contents: write.
• Documentation only covers released versions:
  • docs-trigger.yaml ("Trigger Docs on Tag" → "Trigger Docs on Release") runs on workflow_run of Release with conclusion == 'success', instead of on every v* tag push.
  • docs.yaml gets a new step, before building, that deletes local v* tags that have no published (non-draft) GitHub release. The step fails if it finds no releases at all.
• docs/testing.md covers:
  • the trigger table,
  • the dispatch input,
  • the release gate,
  • what to do when a release's tests fail.

WHY

Closes #851, part of #834.

• Since Run AWS test suites only on ready pull requests and related changes #837 and Run pull-request AWS tests on the newest Python version only #863, pull requests skip unrelated suites and run on the newest Python only. A release must still pass every suite on every supported version at the released commit. Until now, release.yaml published without checking anything.
• This PR runs the tests inside the Release workflow, option 2 of the issue. It is chosen over a verification script, and it replaces the manual "dispatch on master, then tag" step, so the full matrix still runs once per release.
• Because a release now always runs the full matrix, the weekly schedule drops to the newest version, as the maintainer decided. The weekly run keeps catching service-side changes and regressions in suites that pull requests skipped. A version-specific regression is found at release time at the latest, and cannot ship.
• The dispatch input lets a maintainer test a specific version on a branch without paying for the full matrix.
• The Release workflow creates the GitHub release only after the tests and the PyPI upload succeed. So "has a GitHub release" means "published".
  • Before this PR, the "dispatch, then tag" step put about 20 minutes between the last merge and the tag.
  • Without that step, a Docs run from a master push, which waits in the pages concurrency group behind an earlier ~18-minute build, can check out after the tag is pushed. It would then document a version whose Release run is still testing, or whose tests fail.
  • Filtering by releases closes that path. The trigger change then rebuilds the documentation once the release exists.

Consequences:

• A tag whose tests fail stays in the repository until deleted, but the documentation leaves it out. Measured: all 183 existing vX.Y.Z tags have a GitHub release, so no current version disappears from the documentation.
• The Test workflow's concurrency group is ${{ github.workflow }}-${{ github.ref }}. In a called workflow, the github context belongs to the caller, so a release run uses Release-refs/tags/<tag>. It cancels only a previous run of the same tag, and it can overlap a Test run on master.
• The deployed OIDC role github-actions-oidc-pyathena trusts token.actions.githubusercontent.com:sub StringLike repo:pyathena-dev/PyAthena:* (read with iam get-role), as the template does. GitHub's OIDC token for a reusable workflow carries the caller's claims in sub, with the called workflow only in job_workflow_ref. So a tag-push Release run's repo:pyathena-dev/PyAthena:ref:refs/tags/<tag> matches.
• Tags on the 3.x maintenance branch keep that branch's workflows, because a tag push runs the workflow files from the tagged commit. On 3.x, keep dispatching before tagging, unless this is backported.

TEST

Tested commits: 9b79418 (rebased on master after #863 was merged), and b1abeb5 (docs tag filter).

• just scripts passed (including actionlint with ShellCheck on the run scripts), and so did just docs lint.
• The changes job's selection script was extracted from the workflow file and run locally with bash -eo pipefail:

EVENT_NAME	python-versions input	Output
pull_request	–	["3.14"]
schedule	–	["3.14"]
push (Release call)	–	["3.10","3.11","3.12","3.13","3.14"]
workflow_dispatch	empty	["3.10","3.11","3.12","3.13","3.14"]
workflow_dispatch	3.12	["3.12"]
workflow_dispatch	3.14 , 3.11,3.11	["3.11","3.14"]
workflow_dispatch	3.9	jq error, exit 5
workflow_dispatch	3.12,foo	jq error, exit 5

• The docs tag-filter step was extracted from docs.yaml and run on a local clone with two extra tags (v99.0.0 and vfoo), with GH_TOKEN and GITHUB_REPOSITORY=pyathena-dev/PyAthena. It deleted exactly those two tags and kept all 183 released tags (185 → 183), including v3.36.0.

Not run:

• The Release workflow has not run from a real tag push, so the workflow_call, the actual OIDC exchange from a tag ref, and the docs trigger have not been exercised. The docs tag filter first runs on the next master push after merge. A tag push that passes the tests would publish to PyPI. The first real run is the next release (4.0.0), unless it is tried first with a test tag on a fork.
• The weekly schedule and a dispatch with the new input have not run yet. The next Sunday run shows the schedule change, and a dispatch after merge can show the input.

AWS CI on head b1abeb5 (Test run 36321766073): success, with 3 AWS jobs on Python 3.14 (test, test-sqla, test-sqla-async). All offline checks passed, including the docs build.

🤖 Generated with Claude Code

[laughingman7743]

Self-review round one (implementation behavior): CLEAN

Base 531185619465b809331a9a8cf2083884365de193, head 600b13da7abd522ecda1e90d8fe17ba0482b4818.

Covered:

• scripts/verify_release_tests.py
• scripts/tests/test_verify_release_tests.py
• .github/workflows/release.yaml
• docs/testing.md

Checked:

• Run selection. The status=success filter plus the event check leaves only workflow_dispatch and schedule runs. The run listing is newest first. An incomplete newer run falls through to an older complete one (covered by test_find_full_run_skips_incomplete_runs).
• Re-runs. jobs?filter=latest evaluates only the latest attempt. A job that failed in attempt 1 and passed on re-run counts as success. A run with a re-run still in progress is not success, so it is not listed.
• Job matching. incomplete_jobs compares exact job names. It reports missing jobs, and jobs that are skipped, failed, or still in progress. Non-suite jobs (lint, changes) are not required individually, but a failure in either makes the run's conclusion non-success.
• Failure paths fail closed.
  • An HTTP error raises and exits non-zero.
  • Missing GITHUB_TOKEN exits with code 2.
  • Classifiers with no 3.x version raise ValueError.
  • No qualifying run exits with code 1.
    In every case release does not start, because it needs: verify.
• Commit identity. The tag checkout's git rev-parse HEAD is the commit even for annotated tags, and head_sha needs the full SHA.
• Permissions. Workflow-level {}. verify has read-only actions and contents. release keeps id-token: write (trusted publishing) and contents: write (GitHub release).
• Tests. They assert behavior (the returned run, exit codes, summary text, the requested query parameters) through a patched _paginate. The real API response shape was checked separately by running the script read-only against real runs (see the TEST section).

No findings.

[laughingman7743]

Round-one follow-up on the round-two repair (600b13da7abd522ecda1e90d8fe17ba0482b4818 → 345ad32029f8e5f7b30a20d342c120406edbef99, one commit changing .github/workflows/docs-trigger.yaml and docs/testing.md): CLEAN

• Trigger. workflow_run.workflows: [Release] matches release.yaml's name: Release.
• Condition. if: github.event.workflow_run.conclusion == 'success' skips runs that failed, were cancelled, or were refused by verify. A refused tag makes the Release run conclude failure, because verify exits 1 and release is then skipped.
• Actor. A Release run started by the maintainer's tag push is not a GITHUB_TOKEN-initiated event, so workflow_run fires.
• Where the definition comes from. workflow_run uses the default branch's definition. That is the one merged here, and it still dispatches docs.yaml on master as before.
• Unchanged. Permissions (actions: write) and the dispatch script.
• Checks. actionlint passed within just scripts, and so did just docs lint.

[laughingman7743]

Self-review round two (claims, callers, operations): FINDINGS, repaired

Base 531185619465b809331a9a8cf2083884365de193. Reviewed head 600b13da7abd522ecda1e90d8fe17ba0482b4818; repair head 345ad32029f8e5f7b30a20d342c120406edbef99.

Findings:

1. Operational: the docs build was not gated. .github/workflows/docs-trigger.yaml:10-12 dispatched docs.yaml on every v* tag push. docs/conf.py and docs.yaml build tag versions with sphinx-multiversion. So a tag that the new gate refuses, or one whose PyPI upload fails, would still get a documentation build for an unpublished version.
   • Repair: the trigger now runs on workflow_run of Release with conclusion == 'success', and the workflow is renamed "Trigger Docs on Release".
   • docs.yaml also builds on every push to master, so a refused tag that is left in place would still appear later. docs/testing.md now says to delete a refused tag, run the tests, and push the tag again.
2. Claim: "about 30 minutes". The 2026-09-25 dispatch run 36107643451 took 19 minutes. Corrected to about 20 minutes.
3. Claim: the 3.x note was vague. It was replaced with measured facts. origin/3.x has the same test, test-sqla, and test-sqla-async / run (<version>) job names, and a 3.10–3.14 matrix equal to its classifiers.

Checked and held:

• "v3.36.0 was moved to Update gh-action-pypi-publish to v1.14.2 for metadata 2.5 support #765, which changed only release.yaml": git show --stat 073635f shows 1 file, and 4ef4d3e has no Test run. The gate would have stopped that re-tag, which is stated as a consequence.
• "Pull-request runs can skip suites": test.yaml's changes job gates the suites for pull_request only.
• "No AWS cost": the gate makes GitHub API calls only.
• Docs sentence (every suite × every classifier version, schedule or dispatch): matches find_full_run and expected_jobs.

Validation after the repair: just scripts (actionlint, 124 script tests) and just docs lint passed. Neither trigger has run through a real tag push; this is stated in the PR's TEST section.

[laughingman7743]

Independent review (relayed): Codex CLI 0.157.0, model gpt-6-sol, reasoning effort high, codex exec -s read-only, session 01a0e0df-674a-7c62-9a05-a274baed7450.

• Snapshot: detached worktree at head 345ad32029f8e5f7b30a20d342c120406edbef99, base 531185619465b809331a9a8cf2083884365de193. The prompt contained the literal diff and repository context only (no PR number, description, or prior findings). The review was static: no tests, no network, no edits. The snapshot was unchanged afterwards.

Reviewer result, verbatim:

Surfaces covered: the Release, Test, reusable test suite, Docs, and docs-trigger workflows; the verifier and its offline tests; the relevant documentation and version configuration. This was a static review; I ran no tests or network commands.

Verdict: FINDINGS

• P2 — scripts/verify_release_tests.py:152: filter=latest requests jobs from the latest run attempt. After a maintainer uses Re-run failed jobs, that attempt contains only the rerun jobs. Earlier successful matrix jobs appear missing to incomplete_jobs, so the release is refused even when the Test workflow’s final conclusion is success.

• P2 — .github/workflows/docs-trigger.yaml:12: Gating this trigger does not prevent documentation for a rejected tag from being published. The separate Docs workflow still runs on every master push and builds every matching vX.Y.Z tag. If verification rejects a tag and it remains in the repository when master next changes, that Docs run publishes the unreleased version.

Non-blocking suggestions: None.

Author verification:

1. filter=latest after "Re-run failed jobs" (P2): rejected, disproved by measurement.
   • Run 36252040653 (PR Require SQLAlchemy 2.0 or later for the SQLAlchemy dialect #858) is on attempt 2, with only test-sqla-async / run (3.11) re-run.
   • GET /actions/runs/36252040653/jobs?filter=latest returns all 17 jobs with run_attempt: 2. The jobs that succeeded in attempt 1 are carried into the latest attempt, so they are not reported missing.
2. Docs for a refused tag via docs.yaml on master pushes (P2): known limitation, deferred.
   • It is documented in docs/testing.md (delete a refused tag, run the tests, push the tag again) and in the PR's consequences.
   • Fully preventing it would mean making sphinx-multiversion select only published versions, which is outside Require a full test run on the release commit before publishing #851. The trigger change still stops the immediate build on a refused tag or a failed upload.

[laughingman7743]

Self-review round one (implementation behavior), full pass on the rework: CLEAN

Base 3cb9663de659d7cb8cb6e049a017c43e2b428b3a (master after #863), head 9b7941825291a862c3db9dec6b05e162048f0a0f. This replaces the earlier script-based gate, so every file was reviewed afresh.

Covered:

• .github/workflows/test.yaml
• .github/workflows/release.yaml
• .github/workflows/docs-trigger.yaml
• docs/testing.md

Checked:

• Called-workflow context. GitHub documents that in a reusable workflow "the github context is always associated with the caller workflow". So a Release call has github.event_name == 'push'.
  • changes runs (its if only restricts pull_request).
  • The * branch selects every version, and sqla/spark are true.
  • test-suite.yaml's fork guard passes.
• Inputs. inputs.python-versions is defined only for workflow_dispatch. Under workflow_call and schedule it is empty, and those branches ignore it.
• Dispatch validation, run locally with bash -eo pipefail:
  • Whitespace, empty items, and duplicates are removed (unique sorts, and 3.1x strings sort correctly).
  • An unsupported item makes jq exit 5. Because of -e in the assignment, the step fails before python-versions is written, so no suite starts.
• Permissions. Workflow-level {} in release.yaml. The test job grants contents: read, id-token: write, and pull-requests: read, which covers test.yaml's top level (id-token: write, contents: read) and the changes job (pull-requests: read). Called workflows can only lower permissions, and they do not raise them here. release keeps id-token: write and contents: write.
• Concurrency. The group resolves to Release-refs/tags/<tag> under the caller's context. release.yaml defines no concurrency group of its own, so the caller cannot be cancelled by its callee.
• Side workflows. database-sweep.yaml still follows only Test workflow_run events with event == 'schedule'. A called Test inside Release is not a separate Test run, so the sweep behavior is unchanged.
• Nesting. Release → Test → Test Suite is 3 levels, within the documented limit of 10.

No findings.

[laughingman7743]

Round-one follow-up on the repair (9b7941825291a862c3db9dec6b05e162048f0a0f → b1abeb5c73a927db9506c41c671f3ef9e1522993, one commit: docs.yaml and docs/testing.md): CLEAN

• Release lookup. gh api --paginate applies --jq to each page, so every non-draft tag_name is collected, beyond 100. contents: read, which docs.yaml already has, can read releases.
• Failure handling. Under bash -eo pipefail, an API failure fails the assignment and the step. An empty list fails explicitly. The grep -qxF … || git tag --delete loop cannot fail on a released tag.
• Effect on the build. Deleted tags are local refs/tags, which is what sphinx-multiversion lists (smv_tag_whitelist). docs/conf.py's git describe --tags for the master label then sees only released tags.
• Checks. Verified on a local clone: 185 → 183 tags, only the two fake tags removed. actionlint passed within just scripts, and so did just docs lint.

[laughingman7743]

Self-review round two (claims, callers, operations), full pass: CLEAN after a description fix

Base 3cb9663de659d7cb8cb6e049a017c43e2b428b3a, head 9b7941825291a862c3db9dec6b05e162048f0a0f.

Claims checked:

• "Nothing is built or published unless every suite passes on every version." release needs: test. The called workflow's jobs all run on push with the full list: lint (just lint), and test, test-sqla, and test-sqla-async for 3.10–3.14. A failed or cancelled job fails test, so release is skipped.
• OIDC from a tag ref.
  • GitHub's OIDC documentation for reusable workflows says the token carries the calling workflow's standard claims, and the called workflow only in job_workflow_ref.
  • The deployed role github-actions-oidc-pyathena trusts sub StringLike repo:pyathena-dev/PyAthena:* (read-only iam get-role).
  • The description first said the deployed role was unchecked. It was corrected with this evidence.
• The docs caveat. docs/conf.py:218 has smv_tag_whitelist = r"^v\d+\.\d+\.\d+$", and docs.yaml builds on every push to master. So a refused tag left in place would appear as a version, as docs/testing.md says.
• "Replaces the manual dispatch step; full matrix once per release." It holds for master tags. For 3.x tags, the old workflows apply and dispatching is still needed, which the description states.
• Docs table.
  • The schedule and PRs use the newest version.
  • Dispatch uses the requested versions or all of them.
  • Release uses all of them.
  • This matches the case in test.yaml:98-114.
  • The example gh workflow run ... -f python-versions=3.11,3.14 matches the input name.

Operational:

• A release now takes about 20 minutes longer before the upload, because the test matrix runs first.
• A failing release leaves a tag to delete. The steps are documented.
• Neither is a regression from the old manual flow, which needed the same dispatch first.

Not exercised: a real tag-push Release run (stated in TEST).

[laughingman7743]

Round-two follow-up on the repair (b1abeb5c73a927db9506c41c671f3ef9e1522993): CLEAN

• "A GitHub release means published": in release.yaml, softprops/action-gh-release is the last step of release, after uv build and pypa/gh-action-pypi-publish, and release needs: test.
• "No current version disappears": git tag -l 'v*' against gh release list shows 183 = 183, with no tag missing a release.
• Docs. docs/testing.md now says a failed tag is left out of the documentation because only tags with a GitHub release are listed. The earlier "delete before the next master push" caveat was removed, because it no longer applies.
• Description. It explains why the race matters more after this PR (the dispatch step is gone), and records the measurement.

[laughingman7743]

Independent review (relayed): Codex CLI 0.157.0, model gpt-6-astra, reasoning effort high, codex exec -s read-only, session 01a0e2ea-5b94-7441-bcf1-3ae56c75f382.

• Snapshot: detached worktree at 9b7941825291a862c3db9dec6b05e162048f0a0f, base 3cb9663de659d7cb8cb6e049a017c43e2b428b3a. The prompt contained the literal diff and neutral context only (no PR number, description, or prior findings). The review was static, and the snapshot was unchanged afterwards.

Reviewer result, verbatim:

Covered surfaces: exact diff, reusable-workflow context and tagged checkout, permissions/OIDC, concurrency, version parsing and failure propagation, suite matrices, docs triggering, database sweep, and documentation accuracy. Static review only; no tests or network access.

Verdict: FINDINGS

• P2 — .github/workflows/docs-trigger.yaml:13: The successful-Release gate does not prevent unpublished tags from reaching documentation. docs.yaml:12–13 still triggers on master pushes, fetches all tags, and builds every matching version tag (docs/conf.py:218). Concrete scenario: push master, then push its release tag before the queued Docs checkout runs. Docs can build and deploy that version while Release tests are pending—even if those tests subsequently fail. The instruction at docs/testing.md:177 to delete the tag before the next master push cannot prevent this existing run. Gate version inclusion on successful publication, or otherwise ensure every docs execution path excludes unpublished tags.

Non-blocking suggestions: None.

Author verification: confirmed, and made more likely by this PR.

• docs.yaml builds on every master push, waits in the pages concurrency group (cancel-in-progress: false, builds take about 18 minutes), and checks out all tags with fetch-depth: 0. docs/conf.py:218 whitelists every vX.Y.Z tag.
• The old "dispatch, then tag" step (about 20 minutes) used to separate a merge from its tag. This PR removes that step, which makes the race realistic.

Repair in b1abeb5 (this line):

• Before building, docs.yaml deletes local v* tags that have no published (non-draft) GitHub release.
• The Release workflow creates the release only after the tests and the PyPI upload succeed.
• An empty release list fails the step, so a broken lookup cannot silently drop every version.
• Verified on a local clone: two fake tags were deleted, and all 183 released tags were kept (every existing vX.Y.Z tag has a release).
• docs/testing.md and the description were updated. The "delete the tag before the next master push" caveat is gone.

[laughingman7743]

Independent follow-up (relayed): Codex CLI 0.157.0, model gpt-6-astra, reasoning effort high, codex exec -s read-only, session 01a0e2ed-fab7-7cf0-b44b-4eca9f0a6131.

• Scope: the repair diff 9b7941825291a862c3db9dec6b05e162048f0a0f..b1abeb5c73a927db9506c41c671f3ef9e1522993, plus the prior finding.
• The review was static, and the snapshot was unchanged afterwards.

Reviewer result, verbatim:

Covered: GitHub token permissions, releases API, gh --paginate --jq, draft exclusion, and exact tag matching; shell -e/pipefail behavior, API failures, empty results, and tag deletion; local sphinx-multiversion 0.2.4 ref discovery, version labels, dropdown, and master redirect; release publication ordering and docs/testing.md wording.

Verdict: CLEAN. The repair closes the reported path: tags without published GitHub releases are deleted before version discovery, and sphinx-multiversion does not fetch them again. Retained tag names and labels remain intact; master remains buildable. The documentation accurately describes the failed-release case.