feat: support to read chain data split by Weixin-Xu · Pull Request #387 · alibaba/paimon-cpp

Weixin-Xu · 2026-06-29T07:54:40Z

Purpose

Linked issue: close #385

Support deserializing, serializing, and reading community-format ChainSplit.

This change adds ChainSplit support for the current Java SplitSerializer wire format, including:

recognizing the SplitSerializer magic, serializer version, and CHAIN_SPLIT type
deserializing ChainSplit fields: logical partition, data files, file bucket path mapping, and file branch mapping
serializing ChainSplit back through SplitSerializer without dropping chain metadata
resolving ChainSplit data file paths through per-file bucket path mapping
failing fast when a non-external ChainSplit file is missing bucket path mapping
preserving existing DataSplit, FallbackDataSplit, and IndexedSplit behavior

Tests

Added ChainSplitTest
- deserialize community-format ChainSplit bytes
- deserialize ChainSplit bytes with multiple data files
- verify per-file bucket path resolution
- verify external file path is preserved
- verify missing bucket path mapping returns an error
- verify ChainSplit serialization preserves bucket and branch mappings
- verify malformed ChainSplit bytes return an error

API and Format

This change does not modify table storage format. It updates split protocol support to handle the community Java ChainSplit format.

The read path consumes fileBucketPathMapping for per-file bucket path resolution. It also deserializes and preserves fileBranchMapping for format compatibility; branch-aware read/schema selection is intentionally deferred to a follow-up change.

Documentation

No user-facing documentation update.

Generative AI tooling

Generated-by: OpenAI Codex

CLAassistant · 2026-06-29T07:54:48Z

All committers have signed the CLA.

zjw1111

Thanks for adding ChainDataSplit support! The overall design looks good. A few minor suggestions below.

lxy-9602 · 2026-06-30T10:52:24Z

+    if (!data_file_path_factory) {
+        PAIMON_ASSIGN_OR_RAISE(data_file_path_factory,
+                               path_factory_->CreateDataFilePathFactory(partition, bucket));
+    }


This design feels a bit too implicit. Why can’t each caller just pass in the correct data_file_path_factory directly?

Also, let’s avoid using default arguments in production code.

Fixed. The callers now construct the appropriate DataFilePathFactory explicitly before delegating:

split-based reads use CreateDataFilePathFactory(data_split), so ChainDataSplit can get the chain-aware factory.

partition/bucket reads create the normal factory with path_factory_->CreateDataFilePathFactory(partition, bucket).

The private helper now requires a DataFilePathFactory argument directly, so there is no nullptr fallback and no default argument.

lxy-9602 · 2026-06-30T10:58:16Z

+                return Status::Invalid(fmt::format("invalid ChainDataSplit byte stream: {}",
+                                                   chain_split.status().ToString()));
+            }
+            return std::static_pointer_cast<Split>(chain_split.value());


Could you please use PAIMON_ASSIGN_OR_RAISE rather than if (!chain_split.ok()).

Fixed. Replaced the manual chain_split.ok() checks with PAIMON_ASSIGN_OR_RAISE in both ChainDataSplit tail parsing paths.

lxy-9602 · 2026-06-30T11:01:23Z

+
+Result<std::shared_ptr<ChainDataSplitImpl>> ReadChainDataSplitTail(
+    const std::shared_ptr<DataSplitImpl>& base_split, DataInputStream* in,
+    const std::shared_ptr<MemoryPool>& pool) {


Could you move the output parameter (in) in to the end of the parameter list?

Fixed. Moved DataInputStream* in to the end of ReadChainDataSplitTail's parameter list and updated both call sites.

lxy-9602 · 2026-07-10T01:56:37Z

+}
+
 Result<std::unique_ptr<ReaderBuilder>> AbstractSplitRead::PrepareReaderBuilder(
    const std::string& format_identifier) const {


I think there is still a semantic gap with the Java ChainSplit read path.

The PR now deserializes and preserves fileBranchMapping, and uses fileBucketPathMapping to resolve each file path. However, Java also consumes fileBranchMapping when reading each file: MergeFileSplitRead#createChainReader builds a ChainReadContext, and ChainKeyValueFileReaderFactory#getDataSchema uses chainReadContext.fileBranchMapping().get(fileMeta.fileName()) to choose the branch-specific SchemaManager before loading fileMeta.schemaId().

In the C++ path, AbstractSplitRead::CreateFieldMappingReader still loads schemas from the current table/current branch only via context_->GetTableSchema() or schema_manager_->ReadSchema(file_meta->schema_id). For a ChainSplit containing files from both snapshot and delta branches, the same schema_id may refer to different schemas across branches, or may not exist in the current branch at all. In that case C++ can read with the wrong schema or fail, while Java reads with the schema from the file’s own branch.

Could we either make ChainSplit reads branch-aware when resolving DataFileMeta::schema_id, using ChainSplitImpl::FileBranchMapping(), or explicitly fail fast for ChainSplits whose files belong to a non-current branch until branch-aware schema selection is supported?

lxy-9602 · 2026-07-11T09:19:32Z

+        std::unordered_map<std::string, std::string> file_bucket_path_mapping);
+    ChainSplitFilePathFactory(std::unordered_map<std::string, std::string> file_bucket_path_mapping,
+                              std::unordered_map<std::string, std::string> file_branch_mapping);
+


If the intended construction path for ChainSplitFilePathFactory in production code is through Create, could we make the other constructors private as well, to avoid external misuse?

lxy-9602 · 2026-07-11T09:27:04Z

+}
+
+std::string ChainSplitFilePathFactory::ToAlignedPath(
+    const std::string& file_name, const std::shared_ptr<DataFileMeta>& aligned) const {


There is already a Java class that serves a similar purpose. Could we keep the naming consistent and use ChainReadDataFilePathFactory, so we can avoid future inconsistencies and make maintenance easier?

lxy-9602 · 2026-07-11T09:31:24Z

+                                                          context_->GetPath(), normalized_branch);
+    auto [inserted_it, _] =
+        branch_schema_managers_.emplace(normalized_branch, std::move(schema_manager));
+    return inserted_it->second.get();


Returning a const SchemaManager* seems a little unusual here. Have you considered using a smart pointer to SchemaManager instead?

lxy-9602 · 2026-07-11T09:36:11Z

+        }
+        PAIMON_ASSIGN_OR_RAISE(schema_manager, GetSchemaManagerForBranch(branch.value()));
+        current_branch = schema_manager == schema_manager_.get();
+    }


current_branch = schema_manager == schema_manager_.get() relies on raw pointer identity, which feels fairly error-prone and easy to misuse. Could we switch to comparing branch names instead?

lxy-9602 · 2026-07-11T09:51:22Z

+          file_branch_mapping_(std::move(file_branch_mapping)) {
+        CopyMetadataFrom(*base_split);
+    }
+


ChainSplitImpl inheriting from DataSplitImpl looks convenient for reusing existing read paths, but it weakens the type semantics compared with Java, where ChainSplit only implements Split. The inherited DataSplit APIs expose fake bucket/bucketPath semantics and may produce wrong behavior if generic DataSplit code calls methods like GetFileList(). Please either introduce a common file-list/read-scope abstraction used by both DataSplitImpl and ChainSplitImpl, or at least override/disable inherited methods whose semantics are invalid for ChainSplit and document that ChainSplit is intentionally accepted by DataSplit read paths.

lxy-9602 · 2026-07-11T09:53:50Z

+
+    const BinaryRow& ReadPartition() const {
+        return read_partition_;
+    }


Is ReadPartition intended to be equivalent to ChainSplit.logicalPartition on the Java side? If they represent the same concept, could we align the function name to keep things consistent?

lxy-9602 · 2026-07-11T10:01:01Z

+    DataInputStream* in, const std::shared_ptr<MemoryPool>& pool) {
+    PAIMON_ASSIGN_OR_RAISE(int32_t size, in->ReadValue<int32_t>());
+    if (size < 0) {
+        return Status::Invalid(fmt::format("invalid data file meta list size: {}", size));


Could you move the output parameter DataInputStream* to the end to keep the code style consistent?

Also, I’m not entirely clear on the rationale for introducing ReadVersion7DataFileMetaList as a separate function. Is there a reason we can’t continue to use DataSplitImpl::GetFileMetaSerializer to get the appropriate versioned serializer here?

lxy-9602 · 2026-07-11T10:09:45Z

+
+    DataFileMetaFirstRowIdLegacySerializer legacy_serializer(pool);
+    DataFileMetaSerializer current_serializer(pool);
+    std::vector<std::shared_ptr<DataFileMeta>> result;


It looks like there are currently two split serialization paths on the Java side: one based on SplitSerializer, and another based on ChainSplit.serialize / DataSplit.serialize. From what I see, this PR implements the SplitSerializer variant, while the existing Flink/Spark java code still relies on ChainSplit.serialize / DataSplit.serialize. That makes me wonder whether this change could lead to compatibility issues.

lxy-9602 · 2026-07-11T10:16:25Z

+    }
+    return data_file_serializer->DeserializeList(in);
+}
+


Thanks a lot for adding support for chain split. Since this change affects the communication protocol, we need to be a bit more careful with the review and merge process. So far, I’ve reviewed the some parts of the PR. Could you please address the comments first? After that, we can continue with the next round of review.

Weixin-Xu changed the title ~~Support chain data split~~ Support to read chain data split Jun 29, 2026

zjw1111 reviewed Jun 30, 2026

View reviewed changes

Comment thread src/paimon/core/operation/abstract_split_read.h

Comment thread src/paimon/core/table/source/chain_split_impl.h

zjw1111 changed the title ~~Support to read chain data split~~ feat: support to read chain data split Jun 30, 2026

lxy-9602 reviewed Jun 30, 2026

View reviewed changes

Weixin-Xu added 2 commits July 9, 2026 14:44

Support chain data split

af12574

Make raw file path factory explicit

8332390

Weixin-Xu force-pushed the support_chain_table_oss branch from 338b67a to 8332390 Compare July 9, 2026 06:57

Weixin-Xu added 2 commits July 9, 2026 15:14

Reorder ChainDataSplit tail parameter

ad8fa33

Use assign-or-raise for chain split parsing

98d0811

Weixin-Xu force-pushed the support_chain_table_oss branch from 4b4e81b to 98d0811 Compare July 9, 2026 07:31

Align chain split parser with community format

2e87f75

lxy-9602 reviewed Jul 10, 2026

View reviewed changes

Resolve chain split schemas by file branch

06eb293

lxy-9602 reviewed Jul 11, 2026

View reviewed changes

Uh oh!

Conversation

Weixin-Xu commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

API and Format

Documentation

Generative AI tooling

Uh oh!

CLAassistant commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zjw1111 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Weixin-Xu commented Jun 29, 2026 •

edited

Loading

CLAassistant commented Jun 29, 2026 •

edited

Loading