Skip to content

feat(databricks): copy Parquet files in S3 where they lie through a Unity Catalog external location - #341

Merged
kvlonge merged 5 commits into
mainfrom
dbx-external
Nov 18, 2026
Merged

kvlonge merged 5 commits into
mainfrom
dbx-external

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Parquet files of a files source in S3 now load into Databricks straight from the bucket, with no row passing through SQLLoad, when a Unity Catalog external location covers their folder.

  • Discovery and check at plan. plan lists the external locations the service principal can see, once per run, and for a folder holding files to load runs LIST through the covering location. A folder it can't read fails plan with the fix: READ FILES on that location when Databricks refuses access. plan shows route Databricks COPYs directly from s3://… (nothing through the loader).
  • Loading. Each batch is one COPY INTO over its files. A batch is split only when it holds more than 1,000 files (the most FILES takes) or files holding one column as different types, such as INT32 in one file and INT64 in another, which Databricks won't merge in one COPY. Column names are compared case-insensitively, as Spark does. The lineage columns (_sqlload_source_file, _sqlload_source_version, _sqlload_source_row, load id and time) and partition values from the path are filled in the same statement.
  • Atomicity. Every strategy but append copies into a staging or build table, so a batch publishes whole. An append copies straight into the table. When a later COPY in the batch fails, or the rows Databricks reports inserting differ from what the files' footers hold (including a count that doesn't parse), it deletes the rows this load wrote from the batch's files before the load fails.
  • What still goes through SQLLoad. CSV and JSON files, tables with masks, checks or flattened objects, columns a cast wouldn't keep exactly (nanoseconds, times, UUIDs, lists), files on an S3-compatible endpoint (now detected by one check shared with DuckDB), and files in local folders or over SFTP keep the volume route, and plan says why. s3a:// and s3n:// location URLs are treated as s3://.

The docs cover this in the Databricks target page ("Parquet files in S3"), the targets overview, the supported-systems table and the landing page.

Breaking change

Parquet files in S3 that no external location covers now fail plan, naming the folder and the CREATE EXTERNAL LOCATION and GRANT READ FILES to add. They used to load through SQLLoad and the volume; there is no setting to keep that.

Verification

  • Unit tests cover the generated COPY INTO, how files are grouped (the 1,000-file limit, disagreeing types, names differing only in case), external-location matching including s3a/s3n, and the route reasons.
  • Live, against a real Databricks workspace with a Unity Catalog external location on an S3 bucket: append, replace and delete_insert (staging table) loads, including files mixing INT32 and INT64, each compared row for row (EXCEPT ALL both ways, lineage columns included) with the same files loaded through SQLLoad. A second run replaced only the latest partition. A folder under a location the service principal can't read failed plan with the grant.
  • make verify and make test-python (609 passed) pass on the final head.

…ernal location; uncovered S3 Parquet fails plan with the fix
…ly for in-place tables, s3a/s3n URLs, split COPYs by FILES limit and Parquet types, take back a failed direct append, endpoints from the environment
…ils to parse, columns grouped case-insensitively, conditional READ FILES fix, batch atomicity in docs
# Conflicts:
#	website/public/index.html
@kvlonge
kvlonge merged commit ba568dd into main Nov 18, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant