Skip to content

Fix oversized Load TsFile piece dispatch - #18627

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/load-tsfile-thrift-frame-slicing
Open

Fix oversized Load TsFile piece dispatch#18627
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/load-tsfile-thrift-frame-slicing

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Load TsFile dispatch could send a serialized LoadTsFilePieceNode larger than dn_thrift_max_frame_size, causing the receiver to reject the Thrift frame.

This change:

  • slices oversized piece RPC payloads below the configured Thrift frame limit;
  • adds slice metadata to TTsFilePieceReq and reassembles slices in order on the DataNode;
  • keeps small requests on the existing wire format;
  • integrates partial-slice cleanup, rollback, and EXECUTE validation.

Verification

  • LoadTsFileDispatcherImplTest: 4 tests passed
  • LoadTsFilePieceNodeAssemblerTest: 4 tests passed
  • DataNode Checkstyle/Spotless passed
  • DataNode compile passed with -Pwith-zh-locale

The sender and receiver DataNodes should be upgraded together because older receivers do not understand the new slice metadata.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant