feat: add read-optimized system table#403
Conversation
gripleaf
force-pushed
the
feat/value_stats_filter
branch
3 times, most recently
from
7aa2715 to
1449d94
Compare
gripleaf
changed the title
gripleaf
force-pushed
the
feat/value_stats_filter
branch
from
1449d94 to
b125cf8
Compare
|
@suxiaogang223 Would you mind helping take a look when you have time, since you’re most familiar with system table. Thanks a lot! |
| Status WriteAndFullCompact(TestHelper* helper, std::unique_ptr<RecordBatch>&& batch, | ||
| int64_t commit_identifier) { | ||
| PAIMON_RETURN_NOT_OK(helper->write_->Write(std::move(batch))); | ||
| PAIMON_RETURN_NOT_OK( |
There was a problem hiding this comment.
Could you place the output parameter at the end of the parameter list?
| .ok()); | ||
| ASSERT_TRUE(result.array->Equals(expected)) << result.array->ToString(); | ||
| } | ||
|
|
There was a problem hiding this comment.
Could you add a test case to verify that system table reads support partial-column projection and that predicate pushdown is effective? For the predicate setup, you can refer to NestedColumnPruningInteTest.NestedStructMapSelectedKeysWithPredicate. If the Parquet/ORC row group or stripe boundaries are configured properly, the test should be able to observe and validate that predicate pushdown is actually taking effect.
| TestHelper::Create(dir->Str(), schema, | ||
| /*partition_keys=*/{}, | ||
| /*primary_keys=*/{}, options, | ||
| /*is_streaming_mode=*/true)); |
There was a problem hiding this comment.
What I’m really trying to test here is predicate behavior on PK tables, because RO tables additionally enable EnableValueFilter, which could affect the result.
| /*expected_commit_messages=*/std::nullopt)); | ||
|
|
||
| ASSERT_OK_AND_ASSIGN(SystemTableReadResult result, | ||
| ReadSystemTable(table_path + "$ro", options, /*streaming_mode=*/true)); |
There was a problem hiding this comment.
please call ReadSystemTable multiple times if you want to test StreamingScan
There was a problem hiding this comment.
@lszskye I think there may be a small misunderstanding here. ReadSystemTable performs a scan first and then reads. What we usually mean by streaming read is doing a single scan and then reading in a while loop from that same scan, rather than calling ReadSystemTable multiple times. For reference, please see TestWithPKWithDvStreamFromSnapshot4 in scan_and_read_inte_test.cpp.
gripleaf
force-pushed
the
feat/value_stats_filter
branch
2 times, most recently
from
8a8f999 to
558f889
Compare
suxiaogang223
left a comment
suxiaogang223
left a comment
There was a problem hiding this comment.
I found two additional issues that are not covered by the existing review threads.
| .WithCacheConfig(context->GetCacheConfig()) | ||
| .WithCache(context->GetCache()) | ||
| .SetReadFieldNames(context->GetReadFieldNames()) | ||
| .SetReadFieldIds(context->GetReadFieldIds()); |
There was a problem hiding this comment.
Could we preserve the projected Arrow schema from ReadContext::SetReadSchema() here? The wrapper currently copies field names and IDs only. For nested-column pruning those vectors are normally empty, so $ro silently falls back to the full table schema and may return a different shape from the requested projection. This is separate from the existing top-level projection test; could we transfer or re-export the read schema and add a nested projection test for $ro?
| For primary-key tables, ``$ro`` only plans data files from the highest LSM | ||
| level, which is the level produced by full compaction. This avoids merging data | ||
| from multiple LSM levels during query planning and reading. The tradeoff is that | ||
| the result may lag behind the latest committed data until a full compaction |
There was a problem hiding this comment.
Could we document that $ro does not necessarily represent one coherent older snapshot? Different buckets may complete full compaction at different times, so the top-level files selected from the latest snapshot can contain bucket states produced at different snapshots. The current “may lag” wording does not make this consistency caveat clear.
gripleaf
force-pushed
the
feat/value_stats_filter
branch
from
558f889 to
5098767
Compare
4 participants
Purpose
Add support for the read-optimized
$rosystem table. For primary-key tables,$roreads only the highest LSM level produced by full compaction and skips merge-on-read across levels. Append-only tables keep the same read behavior as the base table, including streaming reads.This also refactors read-optimized scan configuration into
DataTableBatchScan, uses context builders for$roscan/read contexts, and documents freshness and value-stats filtering behavior.Tests
E2E test:
SystemTableTest.TestReadOptimizedSystemTableRegistrationSystemTableTest.TestReadOptimizedSystemTablePathParsingTableScanTest.TestReadOptimizedPrimaryKeyStreamingScanUnsupportedSystemTableReadInteTest.TestReadOptimizedSystemTableSystemTableReadInteTest.TestReadOptimizedAppendOnlySystemTableWithStreamingScanSystemTableReadInteTest.TestReadOptimizedSystemTableWithBranchSystemTableReadInteTest.TestReadOptimizedSystemTableWithFirstRowMergeEngineAPI and Format
No public API, storage format, or protocol changes.
Documentation
Added user guide documentation for system tables and the read-optimized $ro system table.
Generative AI tooling
Generated-by: OpenAI Codex GPT-5