Skip to content

[C++][Feature] Add ALP float/double codec with SIMD - #966

Open
ColinLeeo wants to merge 14 commits into
developfrom
colin/alp-codec
Open

ColinLeeo wants to merge 14 commits into
developfrom
colin/alp-codec

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Add an opt-in ALP encoding for FLOAT and DOUBLE to the C++ implementation,
and expose the new encoding through the Python API.

The existing Gorilla/Chimp codecs are not a good fit for fixed-precision
decimal-like training data. ALP converts such values to integers, applies
frame-of-reference and bit packing, and stores exceptions for values that
cannot be represented exactly. This PR adds:

  • TSEncoding::ALP = 15, supported only for FLOAT and DOUBLE.
  • The same TSEncoding.ALP = 15 value in Python, mapped through the C wrapper
    and Cython layer.
  • A self-contained ALP page/block wire format with ALP and PLAIN fallback.
  • Scalar reference encoder/decoder plus SIMDe AVX2-style kernels.
  • SIMD value conversion, SIMD bit unpacking, and word-based packing.
  • A two-stage exponent/factor search and direct decoding from wrapped page
    buffers.
  • Opt-in integration with the existing EncoderFactory/DecoderFactory.
    The default FLOAT/DOUBLE encoding remains Gorilla.

Java is intentionally untouched. The wire format is language-neutral so a
Java decoder can be added later without changing encoded bytes.

Design

The codec is bit-exact by construction: the encoder decodes each candidate
integer back to the original type and stores an exception when the round-trip
is not bitwise equal. Non-decimal or high-entropy columns fall back to PLAIN;
ALP remains opt-in and should be selected per column.

Benchmark

Codec-level benchmark on Apple Silicon, best of 5, 1M values:

Type Codec Ratio Encode MB/s Decode MB/s
FLOAT decimal ALP 1.773 539.3 2646.9
FLOAT decimal Gorilla 1.457 342.0 1389.3
DOUBLE decimal ALP 5.882 1055.3 6470.7
DOUBLE decimal Gorilla 0.970 276.7 2546.8

The benchmark also includes TS_2DIFF as a decimal-data reference. TS_2DIFF has
a much higher ratio on this workload, but the current C++ TS_2DIFF path is not
bit-exact for all FLOAT/DOUBLE values; ALP is the lossless option.

Testing

  • Full ./mvnw -P with-cpp ... clean verify passed locally with all bundled
    dependencies enabled: 974/974 tests passed, with only the existing
    disabled/skipped tests not run.
  • The CI compatibility build with -Denable.simd=OFF also passed.
  • New C++ tests cover scalar round-trip, SIMD/scalar parity, encoder/decoder
    factory integration, special values, truncated payloads, and end-to-end
    TsFile write/read with ALP.
  • New Python test verifies TSEncoding.ALP and an ALP FLOAT/DOUBLE
    round-trip through the Cython layer.
  • CI passes on Ubuntu, macOS, Windows/MinGW, and Windows/MSVC across the
    supported Python versions; Spotless, cppcheck, RAT, CodeQL, and the
    cross-language fixture validation jobs also pass.

Compatibility

ALP writes a new encoding byte. Older readers that do not know encoding 15
will reject ALP chunks, so the codec is opt-in and the default encodings are
unchanged. Existing files are unaffected.

@codecov-commenter

codecov-commenter commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.21795% with 86 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.22%. Comparing base (9b40e40) to head (1762ad2).

Files with missing lines Patch % Lines
cpp/src/encoding/alp/alp_decoder.h 43.90% 46 Missing ⚠️
cpp/src/encoding/alp/alp_scalar.h 91.63% 22 Missing ⚠️
cpp/src/encoding/alp/alp_encoder.h 75.67% 9 Missing ⚠️
cpp/src/encoding/alp/alp_simde.h 94.69% 6 Missing ⚠️
cpp/src/encoding/alp/alp_format.h 98.57% 1 Missing ⚠️
cpp/src/encoding/decoder.h 50.00% 1 Missing ⚠️
cpp/src/reader/chunk_reader.cc 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #966      +/-   ##
===========================================
+ Coverage    63.96%   64.22%   +0.26%     
===========================================
  Files          758      764       +6     
  Lines        53253    53881     +628     
  Branches      8445     8555     +110     
===========================================
+ Hits         34061    34604     +543     
- Misses       17529    17614      +85     
  Partials      1663     1663              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ColinLeeo ColinLeeo changed the title Colin/alp codec [C++][Feature] Add ALP float/double codec with SIMD Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants