Skip to content

[rust] Cache decoded BLOB indexes across readers - #897

Draft
XiaoHongbo-Hope wants to merge 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/cache-decoded-blob-indexes
Draft

XiaoHongbo-Hope wants to merge 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/cache-decoded-blob-indexes

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

Opening the same immutable .blob file with a new reader currently reloads its footer and index, adding two object-store range reads each time.

This caches the decoded index across readers, following the bounded process-local cache used by PyPaimon in apache/paimon#9547.

Changes

  • Cache up to 16 decoded BLOB indexes in a process-local LRU.
  • Share immutable indexes with Arc; readers and BLOB payloads are not cached.
  • Keep cache locks out of asynchronous I/O.
  • Verify that reopening the same file performs no footer or index reads.

Trade-offs

  • Cold reads are unchanged.
  • Cache hits avoid two range requests per BLOB file.
  • Up to 16 decoded indexes remain in memory.
  • This relies on Paimon data file paths being immutable; there is no API or format change.

Tests

  • cargo test -p paimon --lib blob (139 passed)
  • cargo clippy -p paimon --lib -- -D warnings
  • cargo fmt --all --check
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant