Repository navigation
feat(bigquery): BigQuery loads Parquet in GCS itself on tables that exclude _sqlload_source_row - #352
Merged
Merged
Conversation
…xclude _sqlload_source_row Parquet files in Google Cloud Storage can load into BigQuery without passing through SQLLoad: BigQuery queries them where they lie, as temporary external tables, into the staging or build table. - A table opts in by listing _sqlload_source_row in exclude_columns: BigQuery doesn't report a row's position in its file, so a table that keeps the column still has SQLLoad read the files, and sqlload plan -v says why and names the change. - Every value loads as SQLLoad's reader would load it, or SQLLoad reads the files: masked columns, a custom endpoint, UUIDs, lists, JSON, native types, decimal(20,0) and column names BigQuery would change keep the reader. - Files with different columns load in one job, one external table each; a batch splits into several jobs past 1,000 files or 900 KiB of SQL. The jobs' inserted rows must match the files' footers, and a row from a file not in the batch fails the query. - BigQuery reads the bucket with its own service account; plan checks that with a dry run over the first file. - BigQuery job starts are retried on transport errors, 429s and 5xx, and a start that already reached BigQuery waits on that job.
Deploying sqlload with
|
| Latest commit: |
c1cb67e
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://837da044.sqlload.pages.dev |
| Branch Preview URL: | https://bigquery-gcs-direct.sqlload.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parquet files in Google Cloud Storage can now load into BigQuery without passing through SQLLoad: BigQuery queries them where they lie, as temporary external tables, into the staging or build table, in one query job per batch, or several for a batch of over 1,000 files or one whose query would pass BigQuery's length limit.
_sqlload_source_row. BigQuery doesn't report a row's position in its file, so it can't fill that column. A table that keeps it, as every files table does by default, still has SQLLoad read the files, andsqlload plan -v <table>says why and names theexclude_columnschange. A table that excludes it gets BigQuery's route.plankeeps SQLLoad's reader for masked columns, agcsconnection with a customendpoint, UUIDs, lists, JSON, native types,decimal(20,0)(an unsigned 64-bit integer, which BigQuery reads as a signed one), and column names BigQuery would change: names other than letters, digits and underscores, names starting with a prefix BigQuery reserves such as_FILE_, and two names differing only in case.UNION ALL. File name, version and partition values come from each row's_FILE_NAME.gcsconnection's key.planchecks that with a dry run over the first file to load, which bills nothing, and fails the table with the fix when BigQuery can't read it.Testing
decimal(20,0), renamed, case-folded and reserved names, custom endpoint) and the query builder (two schema groups, constants by_FILE_NAMEwith an error for any other file, quoting, typed NULLs, casts without parameters) and its splitting of large batches by size and by file count.EXCEPT DISTINCTboth ways: 0 and 0): decimals up to 38 digits, timestamps with and without a time zone, dates from year 1, times, floats and 16-bit integers.INT64's range fails BigQuery's own read, sodecimal(20,0)stays on SQLLoad's reader, which loaded it.make verifyandmake test-pythonpass on the final head.