feat(realtime): improve append table lifecycle and query support

[lxy-9602]

Purpose

Linked issue: #158

This PR completes several follow-up capabilities for append-table real-time reads and writes:

• support batch-level statistics and predicate pushdown in the default Arrow RealtimeStore;
• support count readers and independently schedulable disk splits;
• make CommitWithProgress retries idempotent;
• preserve real-time progress across compaction and require context recreation after rollback;
• clean up expired and orphan real-time offset files;
• extend integration coverage for commit retry, compaction, rollback, schema rename, predicate pushdown, offset cleanup, and memory reclamation.

Tests

API and Format

• Adds the realtime.store.stats-mode option with none and full modes.
• Adds StatisticsMode to RealtimeStoreFactory::Create.
• Clarifies retry and rollback behavior in the public APIs.

Documentation

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

[zjw1111]

Reviewed the realtime lifecycle / query changes. A few questions on the commit idempotency model and on StatisticsMode semantics, two perf points worth confirming, plus one test-coverage gap and three style nits. Inline.

[zjw1111]

+1

[SteNicholas]

FilterCommitted only compares this identifier with the latest identifier for the commit user; it does not establish that this exact (commit_user, commit_identifier) produced a snapshot. If commit A succeeds but its response is lost, and a later commit B from the same user succeeds before A is retried, this branch can classify A as committed merely because B has a higher identifier. When A’s ranges are covered it then returns the latest snapshot (B) at line 959, despite the public API promising the snapshot produced by this commit. Please look up the snapshot for the exact identity and return its ID; if the ranges were committed by another identity, reject the retry.

[lxy-9602]

Thanks for pointing this out. CommitWithProgress intentionally follows the same identifier high-watermark semantics as FilterAndCommit; its return value is the latest snapshot covering the requested offset progress, so it can be passed to RefreshCommittedSnapshot. It is not intended to identify the exact snapshot produced by the retried commit.

I’ve clarified this contract in the API documentation and added a test covering A → B → retry A, where the retry returns B’s latest snapshot ID.

[SteNicholas]

LGTM.