feat(realtime): add primary-key in-memory support - #5

Draft

HaHaJeff wants to merge 95 commits into

mainfrom

jeff/pk-realtime-v1

Commits on Aug 19, 2026

feat(read): support file index and predicate pushdown for data evolution (#215)
lxy-9602
authored
7ff1855
chore(build): reformat arrow.diff patch sections (#217)
lucasfang
authored
ec85415
feat(prefetch): support read-ahead cache for Parquet reads (#209)
lucasfang
authored
223e566

Commits on Aug 20, 2026

feat(parquet): support vector type storage (#198)
ChaomingZhangCN
authored
c7f23c6
fix: optimize shared-shredding read & fix ORC read-size estimation for nested columns (#216)
lszskye
authored
afc8471
fix(parquet): pass ReadHints into VECTOR ParquetFileBatchReader tests (#223)
ChaomingZhangCN
authored
b0e411b
feat(file-index): support writing file indexes (#210)
zjw1111
authored
8a110d4
perf(parquet): reuse leaf column index set across fields in page-filtered reads (#207)
zhf999
authored
dbdce8f
perf(scan): lazily decode manifest bucket entries (#212)
gripleaf
authored
aa60634

Commits on Aug 21, 2026

Commits on Aug 26, 2026

feat(benchmark): add format-level Parquet read/write micro-benchmarks (#232)
SteNicholas
authored
c329090
fix(compaction): persist keys in lookup SST files while first-row merge function (#250)
lszskye
authored
d76c2a6
feat(realtime): add primary-key in-memory writes
HaHaJeff
committed
50d4207
feat(read): merge primary-key realtime memory with snapshots
HaHaJeff
committed
82949b3
test(realtime): cover primary-key realtime lifecycle
HaHaJeff
committed
f8078a5
refactor(realtime): consolidate PK state and validation
HaHaJeff
committed
949356a
fix(read): close PK realtime query readers
HaHaJeff
committed
7f4b60b
fix(realtime): close rejected plugin readers
HaHaJeff
committed
3dee339
fix(read): preserve PK reader metrics after close
HaHaJeff
committed
b36b24f
refactor(realtime): colocate PK realtime option validation
HaHaJeff
committed
20b834e
test(realtime): improve primary key coverage
HaHaJeff
committed
b89a9fc
fix(realtime): prevent sequence reuse and align nested projections
HaHaJeff
committed
b3862d1
fix(realtime): align PK reads across schema changes
HaHaJeff
committed
3acae03
fix(realtime): align PK projections by field ID
HaHaJeff
committed
74d4fee
refactor(mergetree): accept sorted key-value readers
HaHaJeff
committed
2f7d228
feat(realtime): adapt prepared primary-key batches
HaHaJeff
committed
3df2037
refactor(realtime): prepare primary-key batches in framework
HaHaJeff
committed
232d587
refactor(realtime): simplify primary-key write preparation
HaHaJeff
committed
62108e2
test(mergetree): reuse reader failure mock
HaHaJeff
committed
b6bf461
fix(realtime): preserve PK sequence across writer handoff
HaHaJeff
committed
8c4c4c8
refactor(realtime): simplify primary key merge readers
HaHaJeff
committed
aec0a2e
fix(realtime): validate PK reader contracts
HaHaJeff
committed
c248373
refactor(realtime): simplify reader lifecycle cleanup
HaHaJeff
committed
1e1c7d2
fix(realtime): harden primary-key prepared batches
HaHaJeff
committed
352ce21
refactor(realtime): simplify primary-key contracts
HaHaJeff
committed
d818466
fix(realtime): strengthen primary-key recovery coverage
HaHaJeff
committed
ff0c7bd
test(realtime): strengthen failure recovery coverage
HaHaJeff
committed
83ff3cb
refactor(realtime): simplify primary key state and tests
HaHaJeff
committed
5558182
test(realtime): simplify integration test setup
HaHaJeff
committed
e68ac16
fix(realtime): reject PK read-optimized scans
HaHaJeff
committed
f92fe6a
fix(realtime): harden prepared store handling
HaHaJeff
committed
b187e07
refactor(realtime): simplify PK store boundary
HaHaJeff
committed
5a6dcb6
refactor(realtime): simplify PK offset coverage
HaHaJeff
committed
808cc59
refactor(realtime): simplify stores around framework-owned PK offsets
HaHaJeff
committed
7e59253
refactor(realtime): align PK query projection with store
HaHaJeff
committed
7a59f9e
refactor(realtime): address review feedback
HaHaJeff
committed
884270f
fix(realtime): tighten primary key framework boundaries
HaHaJeff
committed
bf00d53
fix(realtime): harden query schema alignment
HaHaJeff
committed
52ee6bd
fix(realtime): validate exact commit reader coverage
HaHaJeff
committed
ace7e1d
fix(realtime): enforce store reader boundaries
HaHaJeff
committed
607cac7
refactor(realtime): align primary key query projection
HaHaJeff
committed
86123e0
refactor(realtime): centralize Arrow array memory retention
HaHaJeff
committed
32844c1
fix(realtime): refine primary key option validation
HaHaJeff
committed
3476342
fix(realtime): validate plugin bitmap bounds
HaHaJeff
committed
961aaaa
fix(realtime): preserve primary keys in projected reads
HaHaJeff
committed
69d9693
fix(read): bound realtime merge fan-in
HaHaJeff
committed
09c53c9
refactor(realtime): simplify primary key maintenance
HaHaJeff
committed
712b1e7
refactor(realtime): simplify primary key schema and reader setup
HaHaJeff
committed
3392f39
refactor(realtime): remove duplicate offset validation
HaHaJeff
committed
dfb3329

Commits on Aug 27, 2026

Commits on Aug 28, 2026

Merge branch 'main' into jeff/pk-realtime-v1
lxy-9602
authored
9142758
fix(realtime): resolve CI failures
HaHaJeff
committed
d4ebe09
test(realtime): remove redundant reader test code
HaHaJeff
committed
49f0ea7
feat(mosaic): support Mosaic file format (#238)
lxy-9602
authored
34d0cb2
test(realtime): streamline realtime test coverage
HaHaJeff
committed
813c644
fix(style): apply clang-format
HaHaJeff
committed
e357418
fix: canonicalize floating-point NaN values (#255)
zjw1111
authored
feb511e
feat(metrics): expose scan and prefetch metrics (#211)
gripleaf
authored
71e415c
fix(executor): support destruction from worker threads (#233)
mrdrivingduck
authored
8644975
docs: preserve comments across refactoring
HaHaJeff
committed
020a14d
test(realtime): streamline primary key reader coverage
HaHaJeff
committed
a77b5f4
test(realtime): remove redundant close coverage
HaHaJeff
committed
a51b207

Commits on Aug 29, 2026

feat(fs): add OSS file system (#205)
mrdrivingduck
authored
19cba8d
feat(core): support input and lookup changelog producers (#248)
lszskye
authored
53f9c86
chore: merge upstream main
HaHaJeff
committed
e6ac99e