Skip to content

feat(bigquery): BigQuery selects a BigQuery source's rows itself on the same connection - #354

Merged
kvlonge merged 1 commit into
mainfrom
bigquery-same-connection
Nov 19, 2026
Merged

kvlonge merged 1 commit into
mainfrom
bigquery-same-connection

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

A BigQuery source loading into a BigQuery target on the same connection no longer sends its rows through SQLLoad. BigQuery selects each batch's rows with one INSERT ... SELECT query job into the load's staging or build table, which then publishes as any other load does. plan shows:

route   BigQuery INSERT ... SELECT on the source's connection (nothing through the loader)

The select is the one SQLLoad's reader would read. A table is read as of the moment its load starts (FOR SYSTEM_TIME AS OF), as a read session reads it. Views, query tables and tables with where run their query in the same job. Cursor windows, merges and replace build tables work as on the reader route. A merge's row number continues after the earlier batches', in the order the query returns the rows. The reader's parallel streams give no order either. The number is counted within each merge key (ROW_NUMBER() OVER (PARTITION BY <unique_key>)), so BigQuery doesn't number a large batch on one worker; FLOAT64 and JSON keys go through their text form there, as the merge's own ranking does.

SQLLoad still reads the rows itself when it can't promise the same values, and plan -v <table> says why. That covers a table that:

  • has a STRUCT, RANGE, GEOGRAPHY or repeated column, which SQLLoad converts to JSON or text;
  • has a column with no portable type, or one pinned to another type;
  • is in another location than the connection's location (a read session reaches it, a query job doesn't);
  • has a mask, parse, transform or checks.

Other changes:

  • The query-job runner that loads Parquet in GCS is shared with this route.
  • The route checks selects_rows, pulls_unloaded and is_direct are now methods on Route.

Docs: the BigQuery source page ("BigQuery loads the rows itself", cost), the targets overview, the supported-systems table (BigQuery → BigQuery tested live) and the landing page.

Tested:

  • Unit tests for the insert SQL, the plan decision (types, pins, location, masks, parse) and the routing.
  • A live run on a BigQuery project loaded the same tables twice: through the same connection, and through a second connection with the same key (SQLLoad's reader). The tables were an append by cursor, a merge on id with duplicate keys, a merge keyed on a FLOAT64 column with NaNs, a view under replace, a pinned BIGNUMERIC, a STRUCT/ARRAY table and a table in another location. They covered INT64, NUMERIC, FLOAT64 with NaN, BOOL, STRING(n) with non-ASCII text, BYTES, DATE, DATETIME, TIMESTAMP and TIME with microseconds, JSON and NULLs. Over two runs (the second added 300 rows and updated 6 keys), every table matched the reader's: the same column types, and no row in one table that isn't in the other.

…he same connection

A BigQuery source loading into a BigQuery target on the same connection no longer sends its rows through SQLLoad: BigQuery runs one INSERT ... SELECT query job per batch into the load's staging or build table. Tables BigQuery can't select exactly as SQLLoad's reader loads them keep the reader, and plan -v says why.
@kvlonge
kvlonge merged commit 4a43cd5 into main Nov 19, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant