Optimize small BitMap with long-backed implementation

[jt2594838]

Summary

• extract the existing byte-array storage into BitMapArrayImpl and add the shared BitMapImpl abstraction
• add BitMapLongImpl for bitmaps up to 64 bits; public constructors keep the array implementation while createDynamicBitMap selects the implementation by length
• optimize markAll, equals, equalsInRange, and hashCode without temporary byte arrays
• update retained-memory estimation and add cross-implementation correctness, serialization, benchmark, and memory coverage

Test

mvn.cmd -P with-java -pl java/tsfile -am test -Dtest=BitMapTest,TabletTest,FloatDecoderTest,BitMapPerformanceTest -Dtsfile.runPerformanceTests=true -Dsurefire.failIfNoSpecifiedTests=false

Test class	Tests	Failures	Errors	Skipped	Time
BitMapTest	15	0	0	0	68.998 s
TabletTest	15	0	0	0	0.268 s
FloatDecoderTest	7	0	0	0	0.251 s
BitMapPerformanceTest	1	0	0	0	1.592 s
Total	38	0	0	0	71.109 s

Coverage includes public-constructor and dynamic-factory selection, the 64-to-65-bit expansion path, point and range operations, cross-implementation equality and hash consistency, exhaustive merge offset/length combinations, Tablet serialization/deserialization and append paths, and Float codec bitmap serialization.

Length-64 performance report

Environment

Item	Value
Test time	2026-07-20 19:23-19:28 (Asia/Hong_Kong)
Base revision	b8825b18, with the BitMap changes applied
OS	Windows 11 amd64, 10.0.26200.0
CPU	Intel64 Family 6 Model 151, 24 logical processors
JDK	Oracle Java 21.0.8, 64-bit HotSpot VM
Maven	Apache Maven 3.9.11
BitMap length	64

Method

• both implementations are exercised through the same public BitMap API
• each operation uses 3 warm-up rounds and 7 measurement rounds, alternating implementation order
• the reported value is the median of the 7 measurement rounds in ns/op
• write positions and states vary during measurement; serialization/copy results are stored in an escaping container to reduce dead-code elimination
• Long/Array < 1 means BitMapLongImpl is faster; Long/Array > 1 means BitMapArrayImpl is faster

Results

Operation	ArrayImpl ns/op	LongImpl ns/op	Long/Array	Result
isMarked	0.542	0.358	0.660x	Long 34.0% faster
isAllMarked/isAllUnmarked	5.173	1.715	0.331x	Long 66.9% faster
mark/unmark	1.300	1.107	0.852x	Long 14.8% faster
markRange/unmarkRange	3.048	2.301	0.755x	Long 24.5% faster
markAll/reset	3.066	0.541	0.176x	Long 82.4% faster
getByteArray	1.497	4.858	3.246x	Long 3.25x slower
getTruncatedByteArray	8.987	14.068	1.565x	Long 56.5% slower
construct from byte[]	8.027	10.218	1.273x	Long 27.3% slower
clone	10.432	4.014	0.385x	Long 61.5% faster
merge	9.943	2.748	0.276x	Long 72.4% faster
append	85.240	61.024	0.716x	Long 28.4% faster
getRegion	114.356	88.900	0.777x	Long 22.3% faster
equals/equalsInRange	5.402	3.045	0.564x	Long 43.6% faster
hashCode	4.412	5.933	1.345x	Long 34.5% slower
toString	151.272	125.010	0.826x	Long 17.4% faster

The optimized equals/equalsInRange paths avoid temporary arrays and are 43.6% faster with the long-backed implementation. The long-backed hashCode also avoids temporary arrays, but its byte-wise calculation remains 34.5% slower than the JVM-optimized Arrays.hashCode path.

Retained memory

RamUsageEstimator includes the outer BitMap, its implementation object, and the byte[] retained by the array implementation, using the current JVM object-header, reference-width, and 8-byte alignment settings.

Scenario	BitMapArrayImpl	BitMapLongImpl	Saving
one length-64 BitMap	72 bytes	40 bytes	32 bytes (44.44%)
1,000,000 BitMaps including reference array	76,000,016 bytes	44,000,016 bytes	32,000,000 bytes (about 30.52 MiB)

Conclusions and limitations

For bitmaps up to 64 bits that primarily perform marking, lookup, merge, clone, or append operations, BitMapLongImpl reduces retained memory and improves most bit operations. Its main cost is byte-array conversion: getByteArray, getTruncatedByteArray, and construction from byte[] are slower because a long-backed bitmap must materialize or decode the array representation.

This is an in-project JUnit microbenchmark rather than JMH. Warm-up, median aggregation, alternating execution order, and escaping result containers reduce common measurement errors, but absolute ns/op values still depend on CPU frequency, JIT compilation, GC, and background load. Relative ratios should be compared on the same machine and JVM.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 288 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.43%. Comparing base (5b4cad2) to head (ae23861).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
.../java/org/apache/tsfile/utils/BitMapArrayImpl.java	0.00%	125 Missing ⚠️
...n/java/org/apache/tsfile/utils/BitMapLongImpl.java	0.00%	96 Missing ⚠️
.../src/main/java/org/apache/tsfile/utils/BitMap.java	0.00%	41 Missing ⚠️
.../main/java/org/apache/tsfile/utils/BitMapImpl.java	0.00%	24 Missing ⚠️
...ava/org/apache/tsfile/utils/RamUsageEstimator.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #871      +/-   ##
===========================================
- Coverage    60.51%   60.43%   -0.09%     
===========================================
  Files          741      744       +3     
  Lines        48722    48963     +241     
  Branches      7728     7771      +43     
===========================================
+ Hits         29486    29589     +103     
- Misses       17828    17965     +137     
- Partials      1408     1409       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hongzhi-gao]

size=64: markAll() makes ArrayImpl's 9th padding byte 0xFF, but LongImpl's
getByteArray() keeps it 0 → cross-impl equals/getByteArray mismatch. Please fix
and add a test.

[jt2594838]

Thanks for catching this. Fixed by preserving the compatibility padding byte in BitMapLongImpl for size 64 and including it in serialization, reset, clone, equality, hash code, and extension. I also added a regression test comparing markAll() across the array-backed and long-backed implementations.