Conversation
ColinLeeo
force-pushed
the
colin/alp-codec
branch
from
September 17, 2026 10:37
8869fe8 to
4e74d03
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #966 +/- ##
===========================================
+ Coverage 63.96% 64.22% +0.26%
===========================================
Files 758 764 +6
Lines 53253 53881 +628
Branches 8445 8555 +110
===========================================
+ Hits 34061 34604 +543
- Misses 17529 17614 +85
Partials 1663 1663 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an opt-in ALP encoding for
FLOATandDOUBLEto the C++ implementation,and expose the new encoding through the Python API.
The existing Gorilla/Chimp codecs are not a good fit for fixed-precision
decimal-like training data. ALP converts such values to integers, applies
frame-of-reference and bit packing, and stores exceptions for values that
cannot be represented exactly. This PR adds:
TSEncoding::ALP = 15, supported only forFLOATandDOUBLE.TSEncoding.ALP = 15value in Python, mapped through the C wrapperand Cython layer.
buffers.
EncoderFactory/DecoderFactory.The default FLOAT/DOUBLE encoding remains Gorilla.
Java is intentionally untouched. The wire format is language-neutral so a
Java decoder can be added later without changing encoded bytes.
Design
cpp/docs/alp-codec-design.mddescribes thepage/block format, exception handling, and the relationship to TS_2DIFF.
cpp/docs/alp-vs-gorilla-benchmark.mdrecords the ALP/Gorilla/TS_2DIFF comparison.
cpp/docs/alp-profiling-analysis.mdrecords the flame-graph analysis and the optimizations that followed it.
The codec is bit-exact by construction: the encoder decodes each candidate
integer back to the original type and stores an exception when the round-trip
is not bitwise equal. Non-decimal or high-entropy columns fall back to PLAIN;
ALP remains opt-in and should be selected per column.
Benchmark
Codec-level benchmark on Apple Silicon, best of 5, 1M values:
The benchmark also includes TS_2DIFF as a decimal-data reference. TS_2DIFF has
a much higher ratio on this workload, but the current C++ TS_2DIFF path is not
bit-exact for all FLOAT/DOUBLE values; ALP is the lossless option.
Testing
./mvnw -P with-cpp ... clean verifypassed locally with all bundleddependencies enabled:
974/974tests passed, with only the existingdisabled/skipped tests not run.
-Denable.simd=OFFalso passed.factory integration, special values, truncated payloads, and end-to-end
TsFile write/read with ALP.
TSEncoding.ALPand an ALP FLOAT/DOUBLEround-trip through the Cython layer.
supported Python versions; Spotless, cppcheck, RAT, CodeQL, and the
cross-language fixture validation jobs also pass.
Compatibility
ALP writes a new encoding byte. Older readers that do not know encoding
15will reject ALP chunks, so the codec is opt-in and the default encodings are
unchanged. Existing files are unaffected.