Skip to content

feat(bigquery): BigQuery loads Parquet in GCS itself on tables that exclude _sqlload_source_row - #352

Merged
kvlonge merged 2 commits into
mainfrom
bigquery-gcs-direct
Nov 19, 2026
Merged

kvlonge merged 2 commits into
mainfrom
bigquery-gcs-direct

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Parquet files in Google Cloud Storage can now load into BigQuery without passing through SQLLoad: BigQuery queries them where they lie, as temporary external tables, into the staging or build table, in one query job per batch, or several for a batch of over 1,000 files or one whose query would pass BigQuery's length limit.

  • Opt in by excluding _sqlload_source_row. BigQuery doesn't report a row's position in its file, so it can't fill that column. A table that keeps it, as every files table does by default, still has SQLLoad read the files, and sqlload plan -v <table> says why and names the exclude_columns change. A table that excludes it gets BigQuery's route.
  • Every value loads as SQLLoad's reader would load it, or SQLLoad reads the files. plan keeps SQLLoad's reader for masked columns, a gcs connection with a custom endpoint, UUIDs, lists, JSON, native types, decimal(20,0) (an unsigned 64-bit integer, which BigQuery reads as a signed one), and column names BigQuery would change: names other than letters, digits and underscores, names starting with a prefix BigQuery reserves such as _FILE_, and two names differing only in case.
  • Files with different columns load in the same job. BigQuery takes one file's columns for a whole external table, so each set of files with the same columns gets its own, joined with UNION ALL. File name, version and partition values come from each row's _FILE_NAME.
  • The row count is checked. The jobs' inserted rows must match the files' footers, or the batch fails and isn't committed. A row from a file BigQuery names differently from the batch's list fails the query rather than loading its file's values as NULL. A batch never falls back to SQLLoad mid-load.
  • BigQuery job starts are retried on a transport failure, a 429 or a 5xx. A job names its own id, so a retried start that already reached BigQuery waits on that job instead of running it twice; this applies to the Kafka exporter's load jobs too.
  • BigQuery reads with its own identity, the BigQuery connection's service account, not the gcs connection's key. plan checks that with a dry run over the first file to load, which bills nothing, and fails the table with the fix when BigQuery can't read it.

Testing

  • Unit tests: the route decision (row number kept, opted in, UUID, list, decimal(20,0), renamed, case-folded and reserved names, custom endpoint) and the query builder (two schema groups, constants by _FILE_NAME with an error for any other file, quoting, typed NULLs, casts without parameters) and its splitting of large batches by size and by file count.
  • Live, against a real bucket and dataset (both deleted afterwards):
    • Parquet in two partition folders with different columns loaded by BigQuery itself and by SQLLoad's reader, and the two tables matched value for value (EXCEPT DISTINCT both ways: 0 and 0): decimals up to 38 digits, timestamps with and without a time zone, dates from year 1, times, floats and 16-bit integers.
    • An unsigned 64-bit value above INT64's range fails BigQuery's own read, so decimal(20,0) stays on SQLLoad's reader, which loaded it.
    • The dry-run probe passes on a readable file and fails on a missing file and an unreachable bucket.
  • make verify and make test-python pass on the final head.

…xclude _sqlload_source_row

Parquet files in Google Cloud Storage can load into BigQuery without passing through SQLLoad: BigQuery queries them where they lie, as temporary external tables, into the staging or build table.

- A table opts in by listing _sqlload_source_row in exclude_columns: BigQuery doesn't report a row's position in its file, so a table that keeps the column still has SQLLoad read the files, and sqlload plan -v says why and names the change.
- Every value loads as SQLLoad's reader would load it, or SQLLoad reads the files: masked columns, a custom endpoint, UUIDs, lists, JSON, native types, decimal(20,0) and column names BigQuery would change keep the reader.
- Files with different columns load in one job, one external table each; a batch splits into several jobs past 1,000 files or 900 KiB of SQL. The jobs' inserted rows must match the files' footers, and a row from a file not in the batch fails the query.
- BigQuery reads the bucket with its own service account; plan checks that with a dry run over the first file.
- BigQuery job starts are retried on transport errors, 429s and 5xx, and a start that already reached BigQuery waits on that job.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Deploying sqlload with  Cloudflare Pages  Cloudflare Pages

Latest commit: c1cb67e
Status: ✅  Deploy successful!
Preview URL: https://837da044.sqlload.pages.dev
Branch Preview URL: https://bigquery-gcs-direct.sqlload.pages.dev

View logs

@kvlonge
kvlonge merged commit 9950812 into main Nov 19, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant