Skip to content

feat(gax): add RewindableStreamBuffer for chunk upload recovery - #14423

Draft
whowes wants to merge 1 commit into
whowes/resumable-upload-chunk-retryfrom
whowes/resumable-upload-buffer-window
Draft

whowes wants to merge 1 commit into
whowes/resumable-upload-chunk-retryfrom
whowes/resumable-upload-buffer-window

Conversation

@whowes

@whowes whowes commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Introduces RewindableStreamBuffer to manage a single-chunk buffer for the user-provided InputStream.

Recovery is not implemented in this PR - the buffer is wired up in the chunk coordinator here and the query-command-based recovery process will be implemented in the next.

The length of upload request payloads is now included in the ChunkUploadRequest to allow for more efficient propagation of the content bytes from stream -> buffer -> wire.

@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from ba8840b to 6d89325 Compare September 17, 2026 22:08
gemini-code-assist[bot]

This comment was marked as outdated.

@whowes
whowes added this pull request to stack #14429 September 17, 2026 22:16
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 6d89325 to da1275c Compare September 18, 2026 02:23
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch 2 times, most recently from 589ef8e to f130ccc Compare September 18, 2026 03:21
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from f130ccc to 0e94df8 Compare September 18, 2026 15:05
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 0e94df8 to 00f7932 Compare September 19, 2026 01:36
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 00f7932 to c6c28b1 Compare September 19, 2026 21:12
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from c6c28b1 to 4a4b890 Compare September 19, 2026 22:57
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 4a4b890 to 14f6204 Compare September 19, 2026 23:17
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 14f6204 to 0a389c4 Compare September 20, 2026 00:19
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 0a389c4 to 7d192ff Compare September 20, 2026 05:23
@whowes whowes changed the title feat(gax): add rewindable stream buffer for chunk recovery feat(gax): add RewindableStreamBuffer for chunk upload recovery Sep 20, 2026
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 7d192ff to 33b9862 Compare September 20, 2026 06:51
@whowes
whowes removed this pull request from stack #14429 September 20, 2026 07:20
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 5e5a95e to cac0e5d Compare September 21, 2026 20:26
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from cac0e5d to efa09b6 Compare September 21, 2026 22:16
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from efa09b6 to be3497e Compare September 21, 2026 22:34
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch 2 times, most recently from a431bf0 to 954f899 Compare September 22, 2026 01:19
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 954f899 to a46fb8f Compare September 22, 2026 06:46
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch 2 times, most recently from cfc83ff to 2f7c89c Compare September 22, 2026 07:57
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 2f7c89c to 715822d Compare September 22, 2026 08:52
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch 2 times, most recently from 0925229 to 11ab9b0 Compare September 22, 2026 16:07
@whowes
whowes removed this pull request from stack #14454 September 22, 2026 16:21
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch 2 times, most recently from 3ae0f6e to 97fb372 Compare September 22, 2026 16:45
@whowes
whowes added this pull request to stack #14476 September 22, 2026 17:47
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 97fb372 to 3258c60 Compare September 22, 2026 18:52
Introduce RewindableStreamBuffer managing a single-chunk buffer over an
InputStream, supporting forward compaction and topping up upon recovery
realignment without mark()/reset(). Enforces boundaries by throwing
FailedPreconditionException when a server offset is below the base
offset or beyond the current buffer window.

Use payloadLength in ChunkUploadRequest to avoid allocating temporary
byte arrays for full-sized chunks while reusing a single backing array.
@whowes
whowes force-pushed the whowes/resumable-upload-buffer-window branch from 3258c60 to 074e557 Compare September 22, 2026 19:56
@sonarqubecloud

Copy link
Copy Markdown

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed for 'gapic-generator-java-root'

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

public abstract byte[] getPayload();

/** The number of bytes within {@link #getPayload()} to upload. */
public abstract int getPayloadLength();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have to expose getPayloadLength in the request? I think we can do the slicing in RewindableStreamBuffer and getPayload should always return the byte[] that we can upload directly.

* offset or beyond the current buffer window
* @throws IOException if reading from the stream fails
*/
void realignTo(long committedOffset) throws IOException {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this method used in this PR?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants