Backport bug fixes and the CI test policy to 3.x for v3.36.1

[laughingman7743]

WHAT

Backport bug fixes from master to the 3.x maintenance branch for v3.36.1, and port the CI test policy and release gate so that this and later 3.x pull requests stop running every suite on every Python version.

Each commit is one change:

1. CI (Skip AWS integration jobs for external fork pull requests #789, Run AWS test suites only on ready pull requests and related changes #837, Run the full test matrix in the Release workflow before publishing #862, Run pull-request AWS tests on the newest Python version only #863 policies)
   • A lint job runs for every pull request.
   • Draft and external-fork pull requests run no AWS suites; a ready pull request runs them on the newest Python version.
   • workflow_dispatch accepts python-versions.
   • The Release workflow runs every suite on every supported Python version for the tag before building and publishing.
   • The parallel suites and the per-change suite selection are not ported.
   • The fixed S3 Tables namespace no longer exists in the test account, so AWS_ATHENA_S3_TABLES_NAMESPACE is dropped and the SQLAlchemy S3 Tables tests skip on 3.x.
2. Find an existing table in to_sql whatever the name's case #800 to_sql finds an existing table whatever the case of its name (clean cherry-pick).
3. Shut down the AsyncSparkCursor executor when session termination fails #830 AsyncSparkCursor.close() shuts down its executor when session termination fails (clean cherry-pick).
4. Stop Spark session readiness polling on failure states #831 Spark session readiness polling stops on TERMINATED, DEGRADED, and FAILED instead of polling forever.
5. Terminate a newly started Spark session when cursor setup fails #832 A Spark session that a cursor starts is terminated when cursor setup fails. The helper is named _terminate_session_by_id, as on master after Replace name-mangled helpers with single-underscore methods #881.
6. SQLAlchemy compliance: audit numeric range and float precision #826 (compiler fix only) CAST to sa.Double or DOUBLE_PRECISION renders DOUBLE instead of REAL.
7. fix: render Hive STRUCT syntax in table column DDL #870 (reimplemented) CREATE TABLE renders STRUCT<name:type, ...> for AthenaStruct at every depth of a column type, including inside MAP and ARRAY.
   • Field names are quoted with the DDL identifier preparer.
   • Unlike master, integers inside MAP and STRUCT stay INTEGER, which Athena's DDL accepts.
   • Direct type compilation, CAST output, and an empty AthenaStruct() are unchanged.

Release notes for v3.36.1:

• CAST to Double / DOUBLE_PRECISION now yields DOUBLE instead of REAL.
• Compiled CREATE TABLE strings contain STRUCT<...> instead of ROW(...) for struct columns.

WHY

These fixes are merged on master (4.x development) and apply to 3.x users without behavior changes beyond the corrected output. #855 promised the STRUCT fix for 3.x.

The 3.x workflows still ran every suite on every Python version for every pull request event, including drafts, and had no release test gate.

TEST

Tested on the pre-reorder head 7c1db8f with Python 3.13.1 and SQLAlchemy 2.0.46. The reorder moved the CI commit to the front without changing any code or test files.

• just lint: passed (ruff check, ruff format --check, mypy).
• Real AWS (the maintainer's test account), 98 passed, 0 failed, 0 skipped:

  uv run --env-file .env pytest -n 1 tests/pyathena/sqlalchemy/test_compiler.py \
    tests/pyathena/spark/test_common.py "tests/pyathena/spark/test_async_cursor.py::TestAsyncSparkCursor" \
    tests/pyathena/pandas/test_util.py tests/pyathena/sqlalchemy/test_base.py \
    -k "test_compiler or test_common or executor or close_retries or test_to_sql or struct or map_types or array_types or complex_nested"
  

  The run includes the new live STRUCT round-trip test (CREATE, INSERT, read-back) and to_sql.
• Regression checks, run offline against the pre-fix code:
  • The new test_floating_point_types fails for Double and DOUBLE_PRECISION (2 failed).
  • The new DDL tests fail with the 3.x compiler (2 failed).
• actionlint 1.7.12: no findings for test.yaml, test-suite.yaml, and release.yaml.
• CI on this Draft PR (GitHub Actions, head 610d6c3): only the lint jobs ran. versions, test, test-sqla, and test-sqla-async were skipped.
• s3tables:GetNamespace for pyathena in the test table bucket returns NotFound, so the fixed namespace is gone. The SQLAlchemy S3 Tables tests (test_base.py:2041, :2082, :2117) are all gated on AWS_ATHENA_S3_TABLES_NAMESPACE.
• markdownlint-cli2 docs/sqlalchemy.md: 0 errors.
• Not run locally: Spark integration tests (cost). The Spark changes are covered by their mocked unit tests.
• Pending: the full suites on the newest Python version when this pull request is marked ready.
• Not exercised before tagging: the Release workflow's test gate. It runs only on a tag push, and master's equivalent (Run the full test matrix in the Release workflow before publishing #862) has not run for a release yet either. If it fails, nothing is published; the failed jobs can be re-run, or the tag can be deleted and recreated after a fix.

🤖 Generated with Claude Code

[laughingman7743]

Self-review round one (behavior and implementation), by the authoring model; not an independent review.

Scope: base 4ef4d3e, head 610d6c3, all 13 changed files.

Result: CLEAN. No actionable findings within this scope.

Checked:

• STRUCT DDL (fix: render Hive STRUCT syntax in table column DDL #870 reimplementation):
  • Only get_column_specification passes a Column as type_expression. CAST passes a TypeClause, so CAST output is unchanged.
  • A TypeDecorator keeps kw when it resolves to its implementation.
  • Field names use AthenaDDLIdentifierPreparer, so date renders as `date` and backticks double.
  • Empty AthenaStruct() and direct type compilation are unchanged.
  • The new DDL tests fail with the 3.x compiler (2 failed).
• CAST Double (SQLAlchemy compliance: audit numeric range and float precision #826): the check requires Double, which exists only in SQLAlchemy 2.0. On older SQLAlchemy versions, CAST renders as before. DOUBLE subclasses Double, and the Double check runs before the Float branch.
• Spark (Shut down the AsyncSparkCursor executor when session termination fails #830 Stop Spark session readiness polling on failure states #831 Terminate a newly started Spark session when cursor setup fails #832):
  • 3.x SparkBaseCursor.close() calls _terminate_session() on every call, so a failed termination is retried.
  • Startup cleanup calls the synchronous _terminate_session_by_id, so AioSparkCursor's coroutine override is never invoked from the constructor.
  • The executor and the S3 client are created before any session starts.
• to_sql (Find an existing table in to_sql whatever the name's case #800): schema defaults to "default" and is never None. Quotes in both literals are doubled.
• CI:
  • On this Draft PR, only lint ran; versions, test, test-sqla, and test-sqla-async were skipped.
  • A tag push calls test.yaml with event_name == 'push', which selects every Python version.
  • The S3 Tables tests (test_base.py:2041, :2082, :2117) are all gated by requires_s3_tables, which needs the namespace, so they skip.

Out of scope (pre-existing on 3.x, not changed here):

• CAST to STRUCT/MAP/ARRAY renders ROW(date STRING, ...) and MAP<...>, which Trino rejects. master fixed this in the ARRAY work (Support native SQLAlchemy ARRAY types and typed round trips #774) and later changes.
• The generic sqlalchemy.types.ARRAY renders ARRAY<STRING>.

[laughingman7743]

Self-review round two (claims, callers, and operations), by the authoring model; not an independent review.

Scope: base 4ef4d3e, head 610d6c3, covering the PR body, the commit messages, the changed docstrings and comments, and docs/sqlalchemy.md.

Result: FINDINGS. There were two gaps in the PR description; both are corrected, and the code is unchanged.

Claims checked:

• The fixed S3 Tables namespace no longer exists. s3tables:GetNamespace(pyathena) returns NotFound. All three S3 Tables tests are gated on the namespace, so they skip.
• Draft and fork PRs run no AWS suites. For Draft, this was observed on this PR's run: only lint ran. For forks, it follows from the conditions on versions and on test-suite.yaml's run job.
• A tag runs every version before publishing. This holds in the code: release needs test, and the called workflow's event_name is push, so it selects the full list. It is not exercised until a tag is pushed, and neither is master's Run the full test matrix in the Release workflow before publishing #862.
• "Scheduled runs only use the default branch's workflow" is GitHub's documented behavior, so the 3.x schedule never ran.
• The reorder changed only the workflows: git diff 7c1db8f 610d6c3 touches only .github/workflows/*, so the local test results still apply.
• STRUCT DDL, Double CAST, and INTEGER being accepted in DDL: covered by the round-one evidence and the Athena probe on fix: render Hive STRUCT syntax in table column DDL #870.

Existing callers:

• 3.x allows sqlalchemy>=1.0.0. The compiler code only uses Column and a hasattr(types, "Double") guard, and the new Double test skips on SQLAlchemy versions below 2.0.
• No other docs describe DDL with ROW(...). docs/usage.md:596 is a SELECT with CAST.

Operations:

• A ready 3.x PR now runs the three suites on Python 3.14 only.
• A tag runs the three suites on five Python versions, with 3.x's retry behavior and without master's rerun-once (Rerun tests once on Athena service-side query failures #811). A transient failure blocks publishing until the failed jobs are re-run.

Corrections: the PR body's TEST section now records the Draft CI observation, the namespace check, and the fact that the release gate has not run yet.

Separate finding, not in this PR: the test table bucket holds 539 namespaces, mostly pyathena_test_*. Per-session namespaces from master (#815) are accumulating. To be reported separately.

[laughingman7743]

Superseded by #893, which cherry-picks the master PRs themselves in master order, including the ARRAY support (#774) that #870 builds on, instead of reimplementing #870 for 3.x.