Derive the one_row_complex expectations of the Python-object cursors

[laughingman7743]

WHAT

• New tests/pyathena/expected.py derives the expected query results from a table defined in tests/pyathena/tables.py. Its module docstring explains how to read it.
  • ExpectedResult(table, casts=...) is the entry point.
    • It builds the query: the table columns first, then CastColumn items such as TIMESTAMP_TZ, TIME_OF_TIMESTAMP, ARRAY_JSON, and MAP_JSON.
    • It returns the expected rows(values), description(), dbapi_types(), and names.
  • PYTHON_VALUES holds the value rules of the cursors that return Python objects: Cursor, S3FSCursor, pandas.util.as_pandas, and SQLAlchemy rows.
    • The rules are keyed by base type, which is the Athena type without its parameters (decimal for DECIMAL(10,1)).
    • Without type hints, these cursors parse Athena's text rendering of arrays, maps, and structs. So an array whose rendering is valid JSON comes back parsed ([1, 2]), and other nested values come back as text ({"a": "1"}).
    • JSON casts come back as parsed JSON, and timestamp with time zone as a UTC datetime.
  • Coverage is limited to the column types of the shared tables.
    • Types nested inside arrays, maps, and structs can be integers and strings.
    • Any other nesting, or a nested scalar type without a rendering rule, raises NotImplementedError. Such a column needs a rule first.
  • Independence: the rules state what the cursors return. They do not call PyAthena's converters.
• tests/pyathena/util.py: new assert_sqlalchemy_type. It checks a reflected column type against the definition, including the varchar length, the decimal precision and scale, and the array item type.
• tests/pyathena/tables.py: one_row_complex is now the named constant ONE_ROW_COMPLEX. Its note states the constraint on nested string values.
• These tests now derive their expectations instead of spelling them out:
  • TestCursor.test_complex
  • TestS3FSCursor.test_complex and TestAsyncS3FSCursor.test_complex
  • pandas/test_util.py::test_as_pandas
  • TestSQLAlchemyAthena.test_reflect_select

Assertion changes:

• The CAST columns now follow the table columns in the select list, instead of sitting next to their source columns. Every value, Python type, description tuple, and DB API type object is still asserted.
• test_as_pandas now includes col_varchar, which it did not select before. It also asserts the DataFrame column names.
• test_reflect_select compares the reflected column names with the definition, instead of len(...) == 16.

WHY

Part of #848 (step 2), for #834. Stacked on #874.

Each of these tests repeated the one_row_complex row, description, and types by hand. Adding a column meant editing every one of them. With this PR, these five tests pick up a new column from its definition. The rules need a change only when the column's type is not covered by the existing rules. The pandas, Arrow, and Polars cursor tests follow in the next PR, and a column added only through the definition follows after that.

TEST

Tested commit: 36988f4. At the maintainer's request, the expectation code was simplified and renamed in d1db7dc, after the independent reviews below. 36988f4 fixes the two findings of the independent review of that refactor.

• just lint passed.
• Offline, the derived expectations for the test_complex selection were compared with the removed hand-written values. The rows are equal, with the same Python type for every value. The description tuples are equal, and so are the DB API type objects.
• Against AWS, from this worktree: pytest -n 2 on the five rewritten tests gave 5 passed on each pushed revision, a9f4db9 through 36988f4. The offline comparison with the removed literals was rerun on 36988f4 and still shows no difference.
• Not run locally: the full suites. When the PR is Ready, AWS CI runs the PyAthena suite and, because tests/pyathena/sqlalchemy/ changed, both SQLAlchemy compliance suites.

🤖 Generated with Claude Code

[laughingman7743]

Self-review round one (implementation behavior): FINDINGS (2 cleanups + 1 comment fix, fixed)

Base 2760077 (#874 head, the stacked base), head a9f4db9. Full diff reviewed (7 files).

Findings, fixed in db7a792:

• _same and _as_list had no Google-style docstrings.
• _json_compatible had one caller. It is now folded into _parsed_json.
• The PYTHON comment listed DictCursor. DictCursor rows are dicts, and no test uses PYTHON with it, so it is removed from the list.

The offline output is unchanged, and the five tests pass again on AWS.

Checked:

• Assertion strength: every removed literal row, description tuple, DB-API type object, and reflected-type assertion has a derived equivalent. Offline, the derived rows, types, descriptions and DB-API objects are equal to the removed literals.
  • test_as_pandas gains col_varchar and a column-name check.
  • test_reflect_select compares names instead of the count.
• Non-circularity: expected.py imports only the DB-API constants and SQLAlchemy type classes the tests already compared against. It imports no converter.
• Cast placement: Selection.rows takes each cast's value from its source column index, and its rule is looked up by the cast's target family. timestamp with time zone, time and json are all covered.
• Failure mode: an unknown family raises KeyError in the test, so a new type fails loudly instead of passing vacuously.

[laughingman7743]

Self-review round two (claims, callers, AWS operations): FINDINGS (description only, fixed)

Base 2760077, head db7a792.

Claims checked:

• "Every value, Python type, description tuple, and DB API type object is still asserted": confirmed by the offline comparison, and test_complex keeps the per-value type() check.
• "Test IDs unchanged": no parametrization changed.
• CI scope: tests/pyathena/sqlalchemy/test_base.py matches the sqla filter in .github/workflows/test.yaml, so both compliance suites run on Ready. The description already says so.
• Callers: ONE_ROW_COMPLEX is still in TABLES at the same position, so the setup order and the DDL are unchanged. Only these five tests import expected.py.

Findings, both corrected in the PR description:

• Overclaim: "cannot inherit a converter bug". A hand-written rule can still copy buggy output, so the sentence now only says that the rules do not call the converters.
• Overclaim: "a new rule only when the type family is new". Element and key types matter too, so it now says a rule changes when the column's type is not covered by the existing rules.

The tested commit is updated to db7a792.

[laughingman7743]

Independent review (relayed Codex result): FINDINGS (3 × P2), finding 1 of 3

Reviewer: Codex CLI 0.157.1 (codex exec -s read-only), model gpt-6-astra, session 01a0e784-02ed-7502-b61d-1a0fa9f8719a. Base 2760077, head db7a792. It ran on a detached snapshot without .env and without PR framing; afterwards the snapshot was unchanged at the head. This was a static review only.

Covered, as reported: all seven files. Removed literals vs derived rows, descriptions, Python/DB-API types, SQLAlchemy reflection checks, cast ordering, fixture generation, and converter independence. "Current fixture assertions remain equivalent; no circular converter calls or vacuous assertions found."

P2 — Nested parameters corrupt outer-column metadata. Adding ARRAY<DECIMAL(10,2)> makes _parameters() extract the element's (10,2) and assign it to the array's description. The array requires precision/scale (0,0), so correct cursor results fail. Restrict parameter extraction to the outer type.

[laughingman7743]

Repair in 9b8cfac: verified. With the old re.search, _parameters("ARRAY<DECIMAL(10,2)>") returned (10, 2). The regex is now anchored to the outer type name, so it returns () for that type and still (10, 1) for DECIMAL(10,1) and (10,) for varchar(10), checked offline. The docstring states this.

[laughingman7743]

Independent review (relayed Codex result), finding 2 of 3

P2 — Array expectations incorrectly preserve typed elements. Adding ARRAY<DATE> containing [date(2017, 1, 2)] produces that same expected list. Without type hints, the cursor parses Athena's rendering into ["2017-01-02"]. The rewritten tests therefore fail despite correct cursor behavior. The array rule needs to account for element representation.

[laughingman7743]

Repair in 9b8cfac: verified against pyathena/converter.py. _to_array tries json.loads first and otherwise splits the rendering into strings. PYTHON["array"] is now _python_array: it parses _athena_text(value, type) as JSON, or else returns each element's text rendering. Offline: ARRAY<date> gives ["2017-01-02"], ARRAY<string> gives ["a", "b"], ARRAY<int> with a null gives [1, None], and col_array still gives [1, 2]. The rule states the expected behavior; it does not call the converter.\n\nLimit: the text rendering of nested timestamps (2017-01-01 00:00:00.000) follows Athena's CSV format for top-level timestamps. No nested timestamp column exists, so it is not verified against Athena.

[laughingman7743]

Independent review (relayed Codex result), finding 3 of 3

P2 — Python stringification does not match map/struct rendering. Adding STRUCT<flag:BOOLEAN> with {"flag": True} produces {"flag": "True"}, while the untyped cursor returns {"flag": "true"}. The map rule has the same problem; nested nulls also become "None" instead of None. These rules need Athena-specific rendering semantics.

[laughingman7743]

Repair in 9b8cfac: verified. _athena_text now renders a boolean as true/false and a null as null, and _python_map/_python_struct turn nested nulls into None, as the converter's null handling does (pyathena/converter.py:455). Offline: STRUCT<flag: boolean> gives {"flag": "true"}, and MAP<string, int> with a null value gives {"a": None}. The derived one_row_complex rows are still equal to the removed literals, with the same value types. The five rewritten tests pass on AWS at 9b8cfac (5 passed).

[laughingman7743]

Self-review of the repair (both perspectives), range db7a792..9b8cfac on the same base 2760077

Round one (behavior): CLEAN

• _athena_text checks None before it dispatches on the family. _python_array catches ValueError, which also covers JSONDecodeError.
• _type_arguments splits only at top-level commas, so nested <...>/(...) arguments stay whole. _struct_fields splits each field at its first colon.
• assert_sqlalchemy_type now unpacks the single array argument. A malformed array type raises instead of passing.
• The derived one_row_complex rows are unchanged: equal to the removed literals, with the same value types. The five tests pass on AWS.

Round two (claims): FINDINGS (description only, fixed)

• The PR description said map and struct values "come back as strings" without mentioning nested rendering. It now explains how nested values are derived from Athena's text rendering, and the tested commit is 9b8cfac.
• Unverified, stated in the reply above: the rendering of nested timestamps. No column uses one, so no current assertion depends on it.

[laughingman7743]

Independent follow-up review (relayed Codex result): FINDINGS (1 × P2)

• Reviewer: Codex CLI 0.157.1, model gpt-6-astra, session 01a0e789-ce64-7462-b947-6d50afe8acd2. The review ran read-only on a detached snapshot at 9b8cfac and covered the range db7a792..9b8cfac. It was a static review only.
• The reviewer reports that all three earlier findings are resolved: outer parameters are isolated, date arrays produce strings, and nested booleans and nulls in maps and structs render correctly. It also reports that the current one_row_complex expectations are unchanged.

P2 — Array fallback incorrectly stringifies structs. For an ARRAY<STRUCT<a:STRING>> column containing [{"a": "x"}], the new helper expects ["{a=x}"]. The untyped converter parses [{a=x}] into [{"a": "x"}] (pyathena/converter.py:309–316). The previous list(value) expectation was correct for this case; adding this column to the shared table now causes false failures in callers of Selection.rows(PYTHON).

[laughingman7743]

Repair in fdad71a: verified against _parse_array_native (pyathena/converter.py:286). Native items are strings or None, except {...} items, which are parsed as structs. Any item containing [, ], = or " makes the converter give up and return the raw string. When an array's rendering is not JSON, _python_array now returns dicts for map and struct elements and returns the rendering itself for nested arrays. To check the rule offline, the converter was run on the rendered text, as a verification step only; the tests do not call it. The rule and the converter agree on 11 shapes: ARRAY<STRUCT<a: string>>, ARRAY<ARRAY<string>>, ARRAY<MAP<int, string>>, ARRAY<date>, ARRAY<string> with a null, ARRAY<int>, ARRAY<boolean>, two maps, and two structs. The derived one_row_complex rows are still equal to the removed literals, and the five tests pass on AWS at fdad71a.

[laughingman7743]

Independent follow-up review 2 (relayed Codex result): FINDINGS (2 × P2)

• Reviewer: Codex CLI 0.157.1, model gpt-6-astra, session 01a0e78d-24c3-75d1-80be-0f2d4c4235dd. The review ran read-only on a detached snapshot at fdad71a, covering the range 9b8cfac..fdad71a. It was a static review only.
• Result: the flat struct case is fixed, and the current one_row_complex expectations are unchanged.

P2 — Nested-array fallback checks only the immediate element type. For ARRAY<STRUCT<a: ARRAY<int>>> containing [{"a": [1]}], the helper expects [{"a": "[1]"}]. However, _to_array checks for [ anywhere inside the rendered array (converter.py:164) and returns the entire string "[{a=[1]}]".

P2 — Nested struct/map values remain incorrectly stringified. For ARRAY<STRUCT<a: STRUCT<b: string>>> containing [{"a": {"b": "x"}}], the helper expects [{"a": "{b=x}"}]. _parse_array_native → _to_struct recursively produces [{"a": {"b": "x"}}] (converter.py:414–418). The map branch has the same mismatch for nested map values.

[laughingman7743]

Repair in cea2dd1: both scenarios verified in pyathena/converter.py.

Fixing one nested shape after another would turn these rules into a copy of the converter's heuristics. Instead, the rules now state the shapes they model and refuse the rest.

• Supported shapes:
  • scalars nested in arrays, maps and structs;
  • arrays of maps or structs whose members are scalars.
• Refused shapes: any array, map or struct nested one level deeper raises NotImplementedError("No expectation rule for nested ...") through _scalar_type. A new column of such a type therefore fails with an explicit request to add a rule, instead of producing an expectation the cursors would not meet.
• Offline check: the supported shapes still agree with the converter's output for the rendered text in 7 cases. ARRAY<ARRAY<string>>, ARRAY<STRUCT<a: ARRAY<int>>>, ARRAY<STRUCT<a: STRUCT<b: string>>> and STRUCT<a: STRUCT<b: int>> raise.
• Current column: the derived one_row_complex rows are still equal to the removed literals, and the five tests pass on AWS at cea2dd1.

[laughingman7743]

Independent follow-up review 3 (relayed Codex result): FINDINGS (3 × P2)

Reviewer: Codex CLI 0.157.1, model gpt-6-astra, session 01a0e791-545a-76c3-943d-80ef4051dab0. The review was read-only, on a detached snapshot at cea2dd1, and covered fdad71a..cea2dd1. It was a static review only. The reviewer confirmed that the current one_row_complex expectations are unchanged.

1. P2 — Unsupported nesting still returns values. ARRAY<ARRAY<int>> with [[1]] returns [[1]] through JSON parsing without checking nesting. Validation also depends on populated, non-null values: ARRAY<STRUCT<a: ARRAY<int>>> with [{"a": None}] returns [{"a": None}], and an empty MAP<string, ARRAY<int>> returns {}. Validate the complete type before processing values.

2. P2 — Supported scalar strings can still produce incorrect expectations. STRUCT<a: string> with {"a": "null"} produces {"a": "null"}. Athena's {a=null} rendering becomes {"a": None} through _parse_named_struct and _convert_value.

3. P2 — Null arrays of maps or structs now fail. A supported ARRAY<MAP<int, int>> column containing SQL NULL raises TypeError when the comprehension iterates None; arrays of structs fail identically. Before this repair, JSON parsing of _athena_text(None, ...) returned None, matching the converter.

[laughingman7743]

Repair in 562f7b5. All three findings were verified.

1. _check_nesting now checks the whole type before any value is read. It runs in _python_array, _python_map and _python_struct, so an unsupported type raises even for a null, empty or JSON-parsable value. Offline, all of these raise NotImplementedError:
   • ARRAY<ARRAY<int>> with [[1]], and with None
   • ARRAY<STRUCT<a: ARRAY<int>>> with [{"a": None}]
   • an empty MAP<string, ARRAY<int>>
   • STRUCT<a: STRUCT<b: int>>
2. A nested string "null" renders exactly like SQL NULL, so this is a constraint on the data rather than on the rules. The ONE_ROW_COMPLEX note in tables.py now says that string values must differ from null, next to the existing separator constraint.
3. A null array, map or struct now returns None before dispatch, as does a null map or struct element of an array. Offline, ARRAY<MAP<int, int>>, ARRAY<STRUCT<a: int>>, MAP and STRUCT nulls give None, and ARRAY<STRUCT<a: int>> with [{"a": 1}, None] matches the converter's [{"a": "1"}, None].

The 7 supported shapes still match the converter's output for the rendered text. The derived one_row_complex rows still equal the removed literals, and the five tests pass on AWS at 562f7b5.

[laughingman7743]

Independent follow-up review 4 (relayed Codex result): FINDINGS (1 × P2)

Reviewer: Codex CLI 0.157.1, model gpt-6-astra, session 01a0e794-a941-7d13-8501-992c7eba5ab3. The review ran read-only on a detached snapshot at 562f7b5 and covered the range cea2dd1..562f7b5. It was a static review only.

The reviewer confirmed:

• Deeper nesting now raises NotImplementedError before any value is inspected, including null, empty, and JSON-parsable values.
• Null containers and null map/struct array elements return None.
• The current one_row_complex expectations are unchanged.

P2 — Documented constraints still permit values the expectation rules misrepresent. For MAP<string, string> with [("k", "x(y)")], both strings satisfy the documented restrictions. The expectation returns {"k": "x(y)"}, but Athena's rendering {k=x(y)} makes _to_map return the original string, because it contains parentheses (pyathena/converter.py:221). Model this fallback or extend the documented data restriction.

[laughingman7743]

Repair in 1928399: verified.

The restriction is no longer a list of forbidden characters that could miss one; the code now enforces it. _element_text accepts a nested string value, including a map key, only if it consists of letters, digits, underscores, and spaces and is not null. For any other string it raises ValueError("Unsupported nested string value: ..."), rather than expecting a value the cursors would return differently. The ONE_ROW_COMPLEX note in tables.py now states the same rule. Top-level strings are not affected, because the cursors do not parse them.

Checked offline:

• x(y) (map value), a(b) (map key), a,b (array element), and null (struct field) are rejected.
• x y_1 is accepted and matches _to_map.
• The derived one_row_complex rows are unchanged.

The five tests pass on AWS at 1928399.

[laughingman7743]

Independent follow-up review 5 (relayed Codex result): FINDINGS (3 × P2)

Reviewer: Codex CLI 0.157.1, model gpt-6-astra, session 01a0e797-e031-7d62-8c4e-80f09909d2d7. Read-only, detached snapshot at 1928399, range 562f7b5..1928399. Static only. The current one_row_complex expectations are unchanged.

1. P2 — Accepted surrounding spaces produce incorrect expectations. MAP<string, string> with [(" key ", " null ")] is accepted and expects {" key ": " null "}, but the converter strips both strings (converter.py:358–368) and yields {"key": None}. Struct values and native-parsed array elements have the same problem.

2. P2 — Accepted empty array strings disappear during conversion. ARRAY<string> with ["", "x"] passes the regex. Its rendering [, x] converts to ["x"], because _parse_array_native skips empty items (converter.py:305–307).

3. P2 — The restriction does not cover the array JSON path. _python_array returns the parsed JSON before calling _element_text, so ["null"] returns [None] instead of raising the promised ValueError, and a string with literal quotes also bypasses the check. The outputs match the converter, but the stated rejection contract is not enforced everywhere.

[laughingman7743]

Repair in 4e205a3. All three findings were verified.

• 1 and 2: A nested string, including a map key, must now match \w+(?: \w+)*: one or more words of letters, digits, and underscores, separated by single spaces. It also must not be null. This rejects empty strings, leading or trailing spaces, and repeated spaces, which the converter skips or strips.
• 3: _python_array now checks every element before it tries the JSON parse. The restriction therefore holds on both paths.
• Offline, rejected: [(" key ", " null ")], ["", "x"], ["null"], ['"x"'], and {"a": "a b"}.
• Offline, accepted and equal to the converter's output for the rendered text: x y_1 (map), ["a b", "c"], ["123", "true"] (JSON path, [123, True] on both sides), {"a": "x"}, and [{"a": "x y"}].
• The ONE_ROW_COMPLEX note in tables.py states the same rule. The derived one_row_complex rows are unchanged, and the five tests pass on AWS at 4e205a3.

[laughingman7743]

Independent follow-up review 6 (relayed Codex result): FINDINGS (2 × P2)

Reviewer: Codex CLI 0.157.1, model gpt-6-astra, session 01a0e79b-0825-7840-af3f-a6bc51685538.

Scope: read-only, on a detached snapshot at 4e205a3, covering 1928399..4e205a3. The review was static only.

Confirmed:

• The string restriction covers every position, including the array JSON path.
• The one_row_complex expectations are unchanged.
• The repair itself introduces no new defect.

Findings:

P2 — Nested binary uses Python's bytes representation. For STRUCT<a: binary> with {"a": b"123"}, the rule expects {"a": "b'123'"}. Athena renders 31 32 33, and the untyped struct converter returns {"a": "31 32 33"}.

P2 — Nested decimals ignore the declared scale. For STRUCT<a: decimal(10,1)> with {"a": Decimal("1")}, the rule expects {"a": "1"}. Athena renders 1.0, and the converter returns {"a": "1.0"}.

Both mismatches predate the reviewed repair, but they concern values the rules still accept.

[laughingman7743]

Repair in e9efc09: both findings verified.

The fallback str(value) in _athena_text is removed, so no type is rendered by accident any more. Each family that can be rendered now has an explicit rule:

• boolean: true/false
• integers, string, varchar, date: str
• timestamp: millisecond ISO format with a space separator
• binary: space-separated hex bytes (31 32 33)
• decimal: formatted with the declared scale (1.0 for decimal(10,1))

Any other type raises NotImplementedError. That includes float and double, which Athena renders with Java formatting (for example 1.0E20).

Offline checks against the converter's output for the rendered text:

• STRUCT<a: binary> gives {"a": "31 32 33"}, as the converter does.
• STRUCT<a: decimal(10,1)> gives {"a": "1.0"}, as the converter does.
• ARRAY<decimal(10,1)> gives [1.0] through the JSON path, as the converter does.
• STRUCT<a: double> raises.
• ARRAY<date> and MAP<int, boolean> still match.

The derived one_row_complex rows are unchanged, and the five tests pass on AWS at e9efc09.

[laughingman7743]

Independent follow-up review 7 (relayed Codex result): FINDINGS (2 × P2)

Reviewer: Codex CLI 0.157.1, model gpt-6-astra, session 01a0e79e-5331-70f3-b539-15de0304acad. The review ran read-only on a detached snapshot at e9efc09 and covered 4e205a3..e9efc09. It was a static review only. The current one_row_complex expectations are unchanged.

P2 — Empty binary array elements produce incorrect expectations. For ARRAY<binary> containing [b"", b"\x12"], Athena's text is [, 12]. The expectation returns ["", "12"], but the converter's native array parser skips the empty item and returns ["12"] (pyathena/converter.py:305–307).

P2 — Unsupported nested types bypass the new rejection for null or empty values. For example, PYTHON["array"]([], "ARRAY<double>") returns [], and PYTHON["struct"]({"a": None}, "STRUCT<a: double>") returns {"a": None} without raising. The rejection is reached only when a non-null scalar is rendered; the declared member types are never validated on their own.

[laughingman7743]

Repair in 756bdd1: both findings verified.

• Member types are now checked by type. _check_nesting requires every nested scalar type to be in _NESTED_SCALAR_FAMILIES: boolean, integers, string, varchar, date, timestamp, binary, and decimal, the same set that _athena_text renders. The check runs before any value is read, so an unsupported type raises even when the value is null or empty. Offline, ARRAY<double> with [] and STRUCT<a: double> with {"a": None} both raise NotImplementedError.
• Empty renderings are rejected. _element_text raises ValueError for any nested value that renders as empty text, such as b"", because the converter skips empty array items. Offline, [b"", b"\x12"] raises.
• Values that should pass still match the converter. Offline, [b"\x12"] (JSON path, [12]), STRUCT<a: binary>, MAP<int, int> and ARRAY<STRUCT<a: int>> agree with the converter's output for the rendered text.
• Current tests are unaffected. The derived one_row_complex rows are unchanged, and the five tests pass on AWS at 756bdd1.

[laughingman7743]

Independent follow-up review 8 (relayed Codex result): FINDINGS (1 × P2)

• Reviewer: Codex CLI 0.157.1, model gpt-6-astra, session 01a0e7a1-2906-7d10-b02a-9fb28d8d14b6.
• Scope: read-only, on a detached snapshot at 756bdd1, covering e9efc09..756bdd1. This was a static review only.
• Result: the reviewer confirmed that both previous findings are resolved, that unsupported nested types raise before any value is inspected, and that the one_row_complex expectations are unchanged.

P2 — Accepted struct dictionaries can omit declared fields. For STRUCT<a: int, b: int> with {"a": 1}, the rule returns {"a": "1"}. The schema-based Arrow construction in tables.py:110–113 fills the missing field with null, so Athena renders {a=1, b=null} and the converter returns {"a": "1", "b": None}. This also affects arrays of such structs. Iterate over the declared fields and supply the missing nulls, or reject incomplete dictionaries.

[laughingman7743]

Repair in c0b2e3c: verified.

Change:

• A new helper, _struct_items, walks a struct type's declared fields in order and takes a missing field as null, matching what pa.Table.from_pylist writes for it.
• It raises ValueError for a field the type does not declare.
• _athena_text and _python_struct now both use it, so arrays of structs are covered as well.

Checked offline against the converter's output for the rendered text:

• {"a": 1} for STRUCT<a: int, b: int> gives {"a": "1", "b": None}, the same as the converter.
• The same value in ARRAY<STRUCT<a: int, b: int>> gives the same result.
• {"b": 2, "a": 1} renders in declared order.
• {"a": 1, "c": 3} is rejected.

The derived one_row_complex rows are unchanged, and the five tests pass on AWS at c0b2e3c.

[laughingman7743]

Independent follow-up review 9 (relayed Codex result): CLEAN

• Reviewer: Codex CLI 0.157.1, model gpt-6-astra, session 01a0e7a7-a446-7f53-a431-969698bc73cc.
• Scope: read-only review of a detached snapshot at c0b2e3c, covering 756bdd1..c0b2e3c. It was a static review only; the snapshot was unchanged afterwards.
• What the reviewer traced:
  • supported scalar nesting, and arrays of scalar maps/structs, including nulls, empty containers, omitted or reordered fields, and rejection of undeclared fields;
  • unsupported nested types, which still raise NotImplementedError, including for null or empty values;
  • the current one_row_complex expectations, which are unchanged.
• Result: "No concrete accepted-value mismatch or defect introduced by the repair found."

[laughingman7743]

Independent review of the refactor (relayed Codex result): FINDINGS (2 × P2), both repaired in 36988f4

Reviewer: Codex CLI 0.157.1, model gpt-6-astra, session 01a0e88a-73e9-79d1-b521-6d185cab0805. It was a read-only static review of a detached snapshot at 7845405 (#876 head), covering c0b2e3c..0b4d138 (this PR's simplification and renaming) and 0b4d138..7845405 (#876 rebuilt). It ran without .env or PR framing.

Covered, as reported: expectation rules, renamed callers, the SQLAlchemy helper relocation, pandas/Arrow/Polars assertions, descriptions, DB-API and Python types, dtypes, schemas, and the fixture round-trip guard. The reviewer found no stale references and no circular converter use, and noted that neither finding affects the current fixture values or ARRAY<string> ["a", "b"].

P2 — Nested nulls silently produce incorrect expectations. For ARRAY<int> [1, None], _athena_text generated "[1, None]" and _python_array expected ["1", "None"]; Athena's [1, null] parses as [1, None]. Null map values and struct fields likewise became "None".

P2 — Struct text depends on dictionary insertion order. {"b": 2, "a": 1} for STRUCT<a: int, b: int> rendered "{b=2, a=1}" instead of "{a=1, b=2}", and the round-trip guard accepts it because dict equality ignores order.

Repair (verified): _athena_text renders a null as null; _member_value turns nested nulls into None for array elements, map values, and struct fields; struct fields render and convert in declared order. The following were checked offline against the converter's output for the rendered text and match: [1, None], ["a", None], MAP<int, int> with a null value, {"b": 2, "a": None}, and ["a", "b"]. The derived one_row_complex rows are unchanged, and the five tests pass on AWS at 36988f4. #876–#878 were rebased onto it: 20 passed and 14/14 equal for #876.

[laughingman7743]

Closing without merging, by the maintainer's decision. Deriving the expected results from the table definitions required tests/pyathena/expected.py to re-model how each cursor returns every type: Athena's text rendering, JSON parsing, and the pandas, Arrow, and Polars types. The module grew large enough to need tests of its own, which is more complexity than the one-place column addition is worth. The hand-written expectations stay. See #848 for the summary. The branch is kept for reference.