Read a Databricks catalog, and pin catalog rows as a shared fixture - #278
Draft
jat255 wants to merge 2 commits into
Draft
Read a Databricks catalog, and pin catalog rows as a shared fixture#278jat255 wants to merge 2 commits into
jat255 wants to merge 2 commits into
Conversation
jat255
marked this pull request as draft
September 4, 2026 06:10
jat255
added
needs-manual-review
jat255
force-pushed
the
jat255/xp65-databricks-reader
branch
from
September 5, 2026 04:58
3babd70 to
dfed967
Compare
|
Preview deployed to Connect ( Deployed from commit 865cb4e. |
|
Preview deployed to Connect ( Deployed from commit dfed967. |
|
Preview deployed to Connect ( Deployed from commit 865cb4e. |
Fourth of the catalog import stack (kata xp65). Ports the catalog-import half of catalog-databricks.R. Unity Catalog is read through system.information_schema, which is a plain query rather than a SHOW, and the information schema itself is excluded so a listing describes the data rather than the catalog. The legacy hive_metastore is not in there at all and needs SHOW TABLES, which is why the listing forks on the catalog name and why selecting hive_metastore requires a schema: there is nothing to enumerate a whole legacy catalog from. DESCRIBE TABLE does not report nullability, so it is read separately from system.information_schema.columns and joined by column name. hive_metastore has no information schema to ask, so its columns keep an unknown nullability rather than a guessed one. DESCRIBE TABLE also appends partition and detail sections after a row whose name starts with `#`; reading past that turns section headings into columns. Metric views are out of scope with the rest of the semantic-model work (kata gcgj). They report as views here, and telling them apart needs the ODBC object type R queries, which only matters once metric views are supported. The comment in the reader says so.
Which rows are relations, what kind each is, which comments count as prose, and where a DESCRIBE reply stops being columns are all user-observable, so they belong in tests/shared/ rather than being asserted in each language. catalog-rows.json carries the rows Snowflake and Databricks actually return and what both implementations must make of them; R's hand-written row assertions are replaced by runners over it. Running the queries stays each language's own business. Only their replies are a contract. One R change beyond the runners: the Snowflake column shaping was inline in snowflake_describe_relation(), which also runs the query, so the contract was not reachable without a connection. It is now snowflake_describe_rows(), called by the same function. Behaviour is unchanged. Databricks already had databricks_columns_from_describe(). Booleans travel as strings, because a bare JSON true reads back as a logical in R and an absent one as NULL, which would make the two runners disagree about a fixture neither had changed. Both runners were confirmed to bite by changing one expected kind and watching each suite fail. Found by roborev job 335.
jat255
force-pushed
the
jat255/xp65-databricks-reader
branch
from
September 6, 2026 02:07
dfed967 to
865cb4e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth of the catalog import stack (kata xp65). Stacked on #277. Ports the catalog-import half of
catalog-databricks.R, and moves both readers' row interpretation into a shared fixture.Unity Catalog is read through
system.information_schema, a plain query rather than aSHOW, with the information schema itself excluded so a listing describes the data rather than the catalog. The legacyhive_metastoreis not in there at all and needsSHOW TABLES, which is why the listing forks on the catalog name and why selectinghive_metastorerequires a schema: there is nothing to enumerate a whole legacy catalog from.DESCRIBE TABLEdoes not report nullability, so it is read separately fromsystem.information_schema.columnsand joined by column name.hive_metastorehas no information schema to ask, so its columns keep an unknown nullability rather than a guessed one.DESCRIBE TABLEalso appends partition and detail sections after a row whose name starts with#; reading past that turns section headings into columns.Metric views are out of scope with the rest of the semantic-model work (kata gcgj). They report as views here, and telling them apart needs the ODBC object type R queries, which only matters once metric views are supported.
Row interpretation is now a shared fixture
Which rows are relations, what kind each is, which comments count as prose, and where a
DESCRIBEreply stops being columns are all user-observable, so they belong intests/shared/rather than being asserted in each language.catalog-rows.jsoncarries the rows both warehouses actually return and what both implementations must make of them. Running the queries stays each language's own business; only their replies are a contract.Booleans travel as strings, because a bare JSON
truereads back as a logical in R and an absent one asNULL, which would let the two runners disagree about a fixture neither had changed.R changes
R/gains one behaviour-preserving refactor and no new behaviour. Snowflake's column shaping was inline insnowflake_describe_relation(), which also runs the query, so the contract it implements could not be reached without a live connection. It is nowsnowflake_describe_rows(), called by that same function with the same rows and returning the same data frame. Databricks already haddatabricks_columns_from_describe()split out, so it needed nothing.In the test suite, two hand-written blocks are replaced by fixture runners: "Snowflake SHOW results retain native relation metadata" and "Databricks information schema retains native relation metadata". Every expectation they held is now a case in
catalog-rows.json, none was dropped, and the Python suite asserts the same cases. The three comparison helpers live in a newhelper-catalog-rows.Rrather than in either test file, because a definition at the top of one test file is not visible to the other when files are run individually, which is how I found the runner failing.Why it is safe:
test-catalog-snowflake.Randtest-catalog-databricks.Rboth pass. The fixture bites rather than passing vacuously, verified by changing one expected kind and watching each suite fail. Nothing outside those two files and the extracted function changed, so no other call site moves.I did not run the full R suite. It hangs for about ten minutes until #268 lands, so I ran the two affected files only, which is the evidence behind the claim above.
pkg-r/tests/testthat/fixtures/shared/is generated byscripts/sync-shared-fixtures.sh; skip it.Verification
677 Python tests, ruff and pyrefly clean. The reader was confirmed to bite by removing the
#metadata cutoff and by removing thehive_metastorefork.