Repository navigation
feat(bigquery): BigQuery selects a BigQuery source's rows itself on the same connection - #354
Merged
Merged
Conversation
…he same connection A BigQuery source loading into a BigQuery target on the same connection no longer sends its rows through SQLLoad: BigQuery runs one INSERT ... SELECT query job per batch into the load's staging or build table. Tables BigQuery can't select exactly as SQLLoad's reader loads them keep the reader, and plan -v says why.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A BigQuery source loading into a BigQuery target on the same
connectionno longer sends its rows through SQLLoad. BigQuery selects each batch's rows with oneINSERT ... SELECTquery job into the load's staging or build table, which then publishes as any other load does.planshows:The select is the one SQLLoad's reader would read. A table is read as of the moment its load starts (
FOR SYSTEM_TIME AS OF), as a read session reads it. Views, query tables and tables withwhererun their query in the same job. Cursor windows, merges andreplacebuild tables work as on the reader route. A merge's row number continues after the earlier batches', in the order the query returns the rows. The reader's parallel streams give no order either. The number is counted within each merge key (ROW_NUMBER() OVER (PARTITION BY <unique_key>)), so BigQuery doesn't number a large batch on one worker;FLOAT64andJSONkeys go through their text form there, as the merge's own ranking does.SQLLoad still reads the rows itself when it can't promise the same values, and
plan -v <table>says why. That covers a table that:STRUCT,RANGE,GEOGRAPHYor repeated column, which SQLLoad converts to JSON or text;location(a read session reaches it, a query job doesn't);mask,parse,transformor checks.Other changes:
selects_rows,pulls_unloadedandis_directare now methods onRoute.Docs: the BigQuery source page ("BigQuery loads the rows itself", cost), the targets overview, the supported-systems table (BigQuery → BigQuery tested live) and the landing page.
Tested:
idwith duplicate keys, a merge keyed on aFLOAT64column with NaNs, a view underreplace, a pinnedBIGNUMERIC, aSTRUCT/ARRAYtable and a table in another location. They coveredINT64,NUMERIC,FLOAT64with NaN,BOOL,STRING(n)with non-ASCII text,BYTES,DATE,DATETIME,TIMESTAMPandTIMEwith microseconds,JSONand NULLs. Over two runs (the second added 300 rows and updated 6 keys), every table matched the reader's: the same column types, and no row in one table that isn't in the other.