Repository navigation
feat(databricks): copy Parquet files in S3 where they lie through a Unity Catalog external location - #341
Merged
Conversation
…ernal location; uncovered S3 Parquet fails plan with the fix
…ly for in-place tables, s3a/s3n URLs, split COPYs by FILES limit and Parquet types, take back a failed direct append, endpoints from the environment
…ils to parse, columns grouped case-insensitively, conditional READ FILES fix, batch atomicity in docs
# Conflicts: # website/public/index.html
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parquet files of a files source in S3 now load into Databricks straight from the bucket, with no row passing through SQLLoad, when a Unity Catalog external location covers their folder.
planlists the external locations the service principal can see, once per run, and for a folder holding files to load runsLISTthrough the covering location. A folder it can't read failsplanwith the fix:READ FILESon that location when Databricks refuses access.planshowsroute Databricks COPYs directly from s3://… (nothing through the loader).COPY INTOover its files. A batch is split only when it holds more than 1,000 files (the mostFILEStakes) or files holding one column as different types, such as INT32 in one file and INT64 in another, which Databricks won't merge in oneCOPY. Column names are compared case-insensitively, as Spark does. The lineage columns (_sqlload_source_file,_sqlload_source_version,_sqlload_source_row, load id and time) and partition values from the path are filled in the same statement.appendcopies into a staging or build table, so a batch publishes whole. Anappendcopies straight into the table. When a laterCOPYin the batch fails, or the rows Databricks reports inserting differ from what the files' footers hold (including a count that doesn't parse), it deletes the rows this load wrote from the batch's files before the load fails.plansays why.s3a://ands3n://location URLs are treated ass3://.The docs cover this in the Databricks target page ("Parquet files in S3"), the targets overview, the supported-systems table and the landing page.
Breaking change
Parquet files in S3 that no external location covers now fail
plan, naming the folder and theCREATE EXTERNAL LOCATIONandGRANT READ FILESto add. They used to load through SQLLoad and the volume; there is no setting to keep that.Verification
COPY INTO, how files are grouped (the 1,000-file limit, disagreeing types, names differing only in case), external-location matching includings3a/s3n, and the route reasons.append,replaceanddelete_insert(staging table) loads, including files mixing INT32 and INT64, each compared row for row (EXCEPT ALLboth ways, lineage columns included) with the same files loaded through SQLLoad. A second run replaced only the latest partition. A folder under a location the service principal can't read failedplanwith the grant.make verifyandmake test-python(609 passed) pass on the final head.