fix(jindo): make async reads concurrency-safe#427
Open
zjw1111 wants to merge 2 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the Jindo filesystem InputStream async read path to be concurrency-safe by removing shared per-instance read result state and ensuring async callbacks are completed exactly once, including when the underlying async task fails to start/perform. It also adds opt-in remote OSS-backed Jindo test coverage and clarifies the ReadAsync lifetime contract in the public API.
Changes:
- Refactor
JindoInputStream::ReadAsyncto use per-request state and to propagateperform()failures via the callback. - Remove shared
std::string_viewmember state fromJindoInputStreamand keep sync read results local. - Add
PAIMON_ENABLE_JINDO_TESTSCMake option and gate OSS-backed Jindo tests/integration test parameterization behind it.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/inte/write_and_read_inte_test.cpp | Conditionally enables "jindo" test matrix entries when PAIMON_ENABLE_JINDO_TESTS is set. |
| test/inte/CMakeLists.txt | Adds PAIMON_ENABLE_JINDO_TESTS compile definition to the write/read integration test target when enabled. |
| src/paimon/fs/jindo/jindo_file_system.h | Removes shared std::string_view member from JindoInputStream. |
| src/paimon/fs/jindo/jindo_file_system.cpp | Implements per-async-read state + callback completion-on-perform-failure for concurrency safety. |
| src/paimon/fs/jindo/jindo_file_system_test.cpp | Adds OSS-backed tests for listing across page boundaries and concurrent sync/async reads. |
| src/paimon/common/fs/file_system_test.cpp | Conditionally includes "jindo" in filesystem unit test parameters when PAIMON_ENABLE_JINDO_TESTS is set. |
| src/paimon/CMakeLists.txt | Gates Jindo filesystem test sources/link libs and test compile definitions behind PAIMON_ENABLE_JINDO_TESTS. |
| include/paimon/fs/file_system.h | Clarifies InputStream::ReadAsync caller lifetime requirements in the public API doc. |
| CMakeLists.txt | Introduces PAIMON_ENABLE_JINDO_TESTS option and enforces option dependencies. |
| cmake_modules/DefineOptions.cmake | Exposes PAIMON_ENABLE_JINDO_TESTS via the project’s option definition helpers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: N/A
JindoInputStreamreused one memberstd::string_viewacross synchronous and concurrent asynchronous reads, and ignored failures returned byperform(). This could race between reads or leave a caller's completion callback unresolved.This change:
perform()failsPAIMON_ENABLE_JINDO_TESTSswitch and enables Jindo cases in the existing filesystem and write/read integration test binariesTests
cmake --build build --target paimon-fs-test -j64cmake --build build --target paimon-write-and-read-inte-test -j64FsType/FileSystemTest.TestFactoryCreator/1FileFormatAndFileSystem/WriteAndReadInteTest.TestAppendSimple/1pre-commit run --files ...git diff --checkAPI and Format
Clarifies the existing
InputStream::ReadAsynclifetime contract. No storage format or protocol changes.Documentation
No user-facing documentation changes.
Generative AI tooling
Generated-by: OpenAI Codex (GPT-5)