Skip to content

Backport #906: Run SQLAlchemy and Spark tests when shared core modules change #910

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
laughingman7743 merged 1 commit into from
Oct 1, 2026
Merged

Backport #906: Run SQLAlchemy and Spark tests when shared core modules change #910

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 48 additions & 32 deletions .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,10 @@ jobs:
# requests run none. A ready pull request always runs the PyAthena suite; it
# runs the SQLAlchemy tests (the compliance suites and the PyAthena suite's
# SQLAlchemy tests) and the Spark tests only when their code, tests,
# dependencies, or this workflow change. Pull requests and the schedule test
# the newest Python version; a dispatch tests the requested versions or
# every version, and the Release workflow every version.
# dependencies, this workflow, the top-level modules of pyathena and
# pyathena.aio, or the shared test fixtures change. Pull requests and the
# schedule test the newest Python version; a dispatch tests the requested
# versions or every version, and the Release workflow every version.
changes:
if: >-
github.event_name != 'pull_request' ||
Expand All @@ -73,16 +74,53 @@ jobs:
permissions:
pull-requests: read
outputs:
sqla: ${{ steps.filter.outputs.sqla }}
spark: ${{ steps.filter.outputs.spark }}
python-versions: ${{ steps.filter.outputs.python-versions }}
sqla: ${{ github.event_name != 'pull_request' || steps.paths.outputs.sqla == 'true' }}

@laughingman7743 laughingman7743 Oct 1, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round two (backport claims and operations): FINDINGS, repaired in the PR body

  • Scope: claims in the commit message and PR body of ac8b11ee2dd80576f9359efacbe80a3ee344c5d2..056da1af8a856d97746f5119c15b51916e97cc14.
  • Finding (repaired): the PR body said tests/pyathena/tables.py "came with master's fixture work". git log --diff-filter=A shows it was added by 12cfd65, merged in Define the shared test tables in Python and generate their data #874; the body now names Define the shared test tables in Python and generate their data #874. The pattern stays to keep the commit identical to master and matches nothing on 3.x.
  • Claims checked:
    • "applies without conflicts, lines match ci: run SQLAlchemy and Spark tests when shared core modules change #906 exactly": the line-by-line comparison above.
    • "differs from master's only in the 3.x-specific lines": git diff origin/master HEAD -- .github/workflows/test.yaml shows only the header and 3.10.
    • The 3.x import list in WHY: grep over 3.x's pyathena/(aio/)?(sqlalchemy|spark)/. 3.x does not import pyathena.cursor, converter, or aio.cursor there, and the body does not claim it.
    • "CI only, no release-note entry": only .github/workflows/test.yaml changes.
  • Operations: the action's first run on 3.x is this PR's Ready run. That run selects every suite, because test.yaml changes. The AWS cost effect on later 3.x pull requests follows ci: run SQLAlchemy and Spark tests when shared core modules change #906, and 3.x pull requests are few.
  • Evidence limits: the simulation is local, and AWS results come from the Ready run, which has not run yet.

spark: ${{ github.event_name != 'pull_request' || steps.paths.outputs.spark == 'true' }}
python-versions: ${{ steps.versions.outputs.python-versions }}
steps:
- id: filter
- id: paths
if: github.event_name == 'pull_request'
uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3

@laughingman7743 laughingman7743 Oct 1, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round one (backport behavior): CLEAN

@laughingman7743 laughingman7743 Oct 1, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed): CLEAN

  • Reviewer: Codex CLI 0.157.1, model gpt-6-sol, reasoning effort high, --sandbox read-only, --ephemeral, session 01a0f4dd-4ffe-7372-9a8d-3eda092b8d9b. Static review only.
  • Scope: ac8b11ee2dd80576f9359efacbe80a3ee344c5d2..056da1af8a856d97746f5119c15b51916e97cc14 on a detached snapshot, compared with master's merge commit 9b9421d. The package also contained the pinned paths-filter source (ceb8a2b8, v4.0.3), outside the tree. The prompt omitted the PR number, description, and prior findings. The snapshot's HEAD was unchanged afterwards; the only untracked entry was that reference directory.
  • Result:
    • The added and removed lines match master's change; only the line numbers differ.
    • 3.x's Python list still includes 3.10.
    • No 3.x-only dependency outside the intentionally excluded subpackages is missed.
    • The outputs follow the matched paths for ready same-repository PRs. Schedule, dispatch, and the Release workflow_call select both suites. 3.x's test-suite.yaml inputs remain compatible.
    • The action is SHA-pinned with pull-request read permission, and a filter or API failure fails changes.
  • Pre-existing note (deferred): paths-ignore also applies to converted_to_draft. A PR reduced to docs-only changes that is converted to Draft starts no replacement run, so an earlier active run is not cancelled. The behavior predates this commit and is documented in the comment above on: ("paths-ignore applies to both, so a pull request that now changes only docs starts neither").

with:
filters: |
shared: &shared
- .github/workflows/test.yaml
- .github/workflows/test-suite.yaml
- justfile
- pyproject.toml
- uv.lock
# The SQLAlchemy and Spark packages load the top-level modules,
# directly or transitively, and their tests use the shared fixtures.
core: &core
- pyathena/*.py
- pyathena/aio/*.py
- tests/__init__.py
- tests/pyathena/__init__.py
- tests/pyathena/conftest.py
- tests/pyathena/tables.py
- tests/pyathena/util.py
- tests/pyathena/aio/__init__.py
- tests/pyathena/aio/conftest.py
- tests/resources/**
sqla:
- *shared
- *core
- setup.cfg
- pyathena/sqlalchemy/**
- pyathena/aio/sqlalchemy/**
- tests/sqlalchemy/**
- tests/pyathena/sqlalchemy/**
- tests/pyathena/aio/sqlalchemy/**
spark:
- *shared
- *core
- pyathena/spark/**
- pyathena/aio/spark/**
- tests/pyathena/spark/**
- tests/pyathena/aio/spark/**
- id: versions
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
REQUESTED_VERSIONS: ${{ inputs.python-versions }}
# Every supported version, oldest first; keep in sync with the
# pyproject.toml classifiers.
Expand All @@ -106,28 +144,6 @@ jobs:
;;
esac
echo "python-versions=$versions" >> "$GITHUB_OUTPUT"
if [[ "$EVENT_NAME" != "pull_request" ]]; then
{
echo "sqla=true"
echo "spark=true"
} >> "$GITHUB_OUTPUT"
exit 0
fi
files=$(gh api "repos/$REPO/pulls/$PR_NUMBER/files" --paginate --jq '.[].filename')
printf 'Changed files:\n%s\n' "$files"
shared='^(\.github/workflows/test(-suite)?\.yaml|justfile|pyproject\.toml|uv\.lock)$'
sqla="$shared|^pyathena/(aio/)?sqlalchemy/|^tests/sqlalchemy/|^tests/pyathena/(aio/)?sqlalchemy/|^setup\.cfg$"
spark="$shared|^pyathena/(aio/)?spark/|^tests/pyathena/(aio/)?spark/"
if grep -qE "$sqla" <<< "$files"; then
echo "sqla=true" >> "$GITHUB_OUTPUT"
else
echo "sqla=false" >> "$GITHUB_OUTPUT"
fi
if grep -qE "$spark" <<< "$files"; then
echo "spark=true" >> "$GITHUB_OUTPUT"
else
echo "spark=false" >> "$GITHUB_OUTPUT"
fi

# The three suites create their own schemas and tables, so they run in
# parallel; each is still a separate job for "Re-run failed jobs".
Expand Down
Loading