-
Notifications
You must be signed in to change notification settings - Fork 1.2k
feat(bigquery): add ArrowQueryResult and ArrowQueryResultImpl for Arrow result streaming #13944
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Open
Changes from all commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
7cc96f6
feat(bigquery): add zero-copy queryArrow API for Arrow VectorSchemaRo…
jinseopkim0 64d1729
refactor(bigquery): remove unused 6-argument constructor in ArrowQuer…
jinseopkim0 7c48bfd
refactor(bigquery): use strongly-typed Arrow Schema and VectorSchemaR…
jinseopkim0 79d9cbd
style(bigquery): remove extraneous empty line in BigQueryImpl
jinseopkim0 359b644
fix(bigquery): resolve Mockito Java 8 JSpecify compatibility and opti…
jinseopkim0 d291158
fix(bigquery): rethrow JVM Error in close and translate stream/storag…
jinseopkim0 1098b0c
feat(bigquery): wait for incomplete fast-path query and stream ByteSt…
jinseopkim0 531eb5e
fix(bigquery): read results via Storage Read API when fast-path query…
jinseopkim0 8765958
fix(bigquery): address review comments on null checks, error handling…
jinseopkim0 4aa070b
fix(bigquery): close BigQueryReadClient, propagate headers, and docum…
jinseopkim0 76b6294
fix(bigquery): remove checked Exception from BigQuery.close signature
jinseopkim0 e23176f
fix(bigquery): narrow lock scope in VectorBatchIterator to allow conc…
jinseopkim0 4906658
fix(bigquery): set bqReadClient = null upon close
jinseopkim0 c687221
fix(bigquery): prefetch and peek Arrow record batches to preserve Ite…
jinseopkim0 e4c72f4
fix(bigquery): provide default no-op implementation for BigQuery.close
jinseopkim0 8d49d2a
fix(bigquery): manage ArrowRecordBatch lifecycle to prevent premature…
jinseopkim0 090f08b
fix(bigquery): ensure exception safe cleanup of ArrowRecordBatch and …
jinseopkim0 8702031
fix(bigquery): mock BigQueryReadClient directly without reflection
jinseopkim0 012c717
fix(bigquery): address review comments on batch loading and project i…
jinseopkim0 6c65fe4
fix(bigquery): protect Arrow batch deserialization under lock and ens…
jinseopkim0 11ac9a5
fix(bigquery): import ByteString and channel types instead of using F…
jinseopkim0 fcf19c0
fix(bigquery): ensure new batch is tracked before closing old batch
jinseopkim0 0e60635
fix(bigquery): hold lock across stream initialization in VectorBatchI…
jinseopkim0 2e556e2
fix(bigquery): rethrow BigQueryException directly in hasNext to avoid…
jinseopkim0 8767fef
fix(bigquery): consolidate lock acquisition in VectorBatchIterator
jinseopkim0 7ff06d1
fix(bigquery): perform blocking gRPC call outside lock during stream …
jinseopkim0 7549809
fix(bigquery): translate IOException to BigQueryException inside getB…
jinseopkim0 7b170df
fix(bigquery): simplify anonymous Callable to lambda in queryArrowWit…
jinseopkim0 6d600ef
fix(bigquery): preserve existing Service contract by removing AutoClo…
jinseopkim0 c857596
refactor(bigquery): isolate Arrow result model and streaming iterator…
jinseopkim0 dc02899
docs(bigquery): add Javadoc comments to non-override methods in Arrow…
jinseopkim0 3065daf
refactor(bigquery): deduplicate loadBatch in ArrowQueryResultImpl
jinseopkim0 3942577
perf(bigquery): ensure zero-copy batch loading via ReadableByteChannel
jinseopkim0 1b283db
fix(bigquery): remove unused gax-grpc and grpc-api dependencies from …
jinseopkim0 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
66 changes: 66 additions & 0 deletions
66
...query/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ArrowQueryResult.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| /* | ||
| * Copyright 2026 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.google.cloud.bigquery; | ||
|
|
||
| import com.google.api.core.BetaApi; | ||
| import org.apache.arrow.vector.VectorSchemaRoot; | ||
| import org.apache.arrow.vector.types.pojo.Schema; | ||
|
|
||
| /** | ||
| * <b>[Beta]</b> A query result container providing zero-copy access to Apache Arrow {@link | ||
| * VectorSchemaRoot} batches. | ||
| * | ||
| * <p><b>Important Usage Warning:</b> The {@link VectorSchemaRoot} returned by the iterator is a | ||
| * single, shared, mutated instance across iterations. Data in the root is only valid during the | ||
| * current iteration step and will be overwritten or cleared on the next call to {@link | ||
| * java.util.Iterator#next()}. Callers needing data across iteration steps must copy the data out of | ||
| * the vectors before advancing the iterator. | ||
| * | ||
| * <p>Implementations manage direct off-heap native memory buffers. Callers must invoke {@link | ||
| * #close()} (idiomatically via a {@code try-with-resources} block) to ensure native allocations and | ||
| * underlying gRPC streaming channels are deterministically released. | ||
| */ | ||
| @BetaApi | ||
| public interface ArrowQueryResult extends AutoCloseable, Iterable<VectorSchemaRoot> { | ||
|
|
||
| /** Returns the Apache Arrow schema of the result vectors. */ | ||
| Schema getArrowSchema(); | ||
|
|
||
| /** | ||
| * Returns the job ID associated with the query execution, or {@code null} if no job was created | ||
| * (e.g. when optional job creation was used). | ||
| */ | ||
| JobId getJobId(); | ||
|
|
||
| /** Returns the query ID associated with the query execution, or {@code null} if unavailable. */ | ||
| String getQueryId(); | ||
|
|
||
| /** | ||
| * Returns the reason a job was created when optional job creation was requested, or {@code null} | ||
| * if no job was created or if the query ran via the fallback path. | ||
| */ | ||
| JobCreationReason getJobCreationReason(); | ||
|
|
||
| /** Returns the total number of rows across all batches if known, or {@code -1} if unknown. */ | ||
| long getTotalRows(); | ||
|
|
||
| /** | ||
| * Releases underlying direct off-heap memory allocations and closes any active stream channels. | ||
| */ | ||
| @Override | ||
| void close(); | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.