GH-48701: [C++][Parquet] Add ALPpd encoding - #48345

Open

prtkgaur wants to merge 155 commits into

apache:mainfrom

prtkgaur:gh540-alp-pseudoDecimal-encoding

Commits on Aug 26, 2026

Add alp code

sfc-gh-pgaur
and
dhirhank
committed
ae3d27e
Integrate ALP with arrow
sfc-gh-pgaur
committed
19968d2
Add alp benchmark
sfc-gh-pgaur
committed
2274077
Add datasets for alp benchmarking
sfc-gh-pgaur
committed
06a4027
Update cmake file
sfc-gh-pgaur
committed
3f25cc3
Move hpp files to h
sfc-gh-pgaur
committed
3b0a0d7
Update flow digram and layout digram to use ASCII and not unicode characters
sfc-gh-pgaur
committed
93f291c
Rename cpp files to cc
sfc-gh-pgaur
committed
6931ca5
Update documentation to align with arrow's doxygen style
sfc-gh-pgaur
committed
2a62002
Adapt methods and variable names to arrow style
sfc-gh-pgaur
committed
eaef4d0
Update the tests to adhere to arrow style code
sfc-gh-pgaur
committed
c62111e
Update callers
sfc-gh-pgaur
committed
2832385
Fuse FOR and decode loop
sfc-gh-pgaur
committed
eb2ce54
Reduce memory allocation in the decompress call
sfc-gh-pgaur
committed
66b59e0
Attempt at making decoding faster with SIMD
sfc-gh-pgaur
committed
7db0f18
Revert "Attempt at making decoding faster with SIMD"
sfc-gh-pgaur
committed
eab7cae
Move cpp files to cc
sfc-gh-pgaur
committed
57ec026
Move data file to parquet-testing submodule
sfc-gh-pgaur
committed
af324cb
Update path to the data file
sfc-gh-pgaur
committed
eadc951
Adapt files names to arrow convention
sfc-gh-pgaur
committed
80f9d98
File rename
sfc-gh-pgaur
committed
2629bda
Obtain compressed size and number of elements from page header
sfc-gh-pgaur
committed
595bf65
Fix namespace depth
sfc-gh-pgaur
committed
f2a7e90
Better pack the compression block header
sfc-gh-pgaur
committed
da142b6
Rename class
sfc-gh-pgaur
committed
0f896e8
Rearrage field for vector metadata for better packing
sfc-gh-pgaur
committed
3be05a2
Add spec files
sfc-gh-pgaur
committed
f07c5cc
Add more tests
sfc-gh-pgaur
committed
29a513b
Move number of elements metadata info to uber header
sfc-gh-pgaur
committed
a68121f
Keep version as 1
sfc-gh-pgaur
committed
c500689
Tests pass
sfc-gh-pgaur
committed
77caa8c
Get rid of numElements and derive it for each vector
sfc-gh-pgaur
committed
42b4601
Update the md files
sfc-gh-pgaur
committed
a4f02f2
Fix ubsan error
sfc-gh-pgaur
committed
fa2e820
Use safe copy
sfc-gh-pgaur
committed
0098a01
Add more unit tests
sfc-gh-pgaur
committed
9524032
Templatize AlpEncodedVectorInfo
sfc-gh-pgaur
committed
8b1447b
Fix
sfc-gh-pgaur
committed
15b0344
[SpecReview] Num values in page is at max int32
sfc-gh-pgaur
committed
8e13cc5
[SpecReview] Rename layout to IntegerEncoding
sfc-gh-pgaur
committed
596340b
[SpecReview] Rename enum
sfc-gh-pgaur
committed
63015f5
[SpecReview] Make AlpHeader exact 8 bytes
sfc-gh-pgaur
committed
154ee3a
[SpecReview] Move all vector data towards the start
sfc-gh-pgaur
committed
318d79b
[SpecReview][Metadata] Introduce wrapper to read metadata
sfc-gh-pgaur
committed
a5c3623
[SpecReview] Separate out ALP and FOR metadata
sfc-gh-pgaur
committed
1ad5d78
[SpecReview] Remove unnecessary field
sfc-gh-pgaur
committed
6b2fba0
[FutureFlexibility] Add the FOR specific code behind switch
sfc-gh-pgaur
committed
8629a9f
Add unit test
sfc-gh-pgaur
committed
c592e09
ALP: Implement offset-based layout for O(1) random access
sfc-gh-pgaur
committed
6f159a7
ALP: Update documentation for offset-based layout
sfc-gh-pgaur
committed
69d7e5c
ALP: Add CreateSamplingPreset and EncodeWithPreset APIs
sfc-gh-pgaur
committed
95e020a
ALP: Update all comments and docs for offset-based layout
sfc-gh-pgaur
committed
31d2701
ALP: Add cross-implementation reference blob generator
sfc-gh-pgaur
committed
f845dac
ALP: Add generate_alp_parquet utility for creating ALP test parquet files
sfc-gh-pgaur
committed
0e2fc95
ALP: Remove version field from header (8B → 7B)
sfc-gh-pgaur
committed
afdec0d
Update parquet-testing submodule with ALP test data files
sfc-gh-pgaur
committed
a678c96
ALP: Add C++ correctness tests for ALP-encoded parquet files
sfc-gh-pgaur
committed
f36110f
ALP: Add cross-language tests reading Java-generated ALP parquet files
sfc-gh-pgaur
committed
64607ca
ALP: Add float32 cross-language tests and generator support
sfc-gh-pgaur
committed
e015b6d
Address code review comments
sfc-gh-pgaur
committed
57f4c2f
Address code review comments: endianness assert and decode bounds checks
sfc-gh-pgaur
committed
9fffec8
Address code review comments: renames, API cleanup, and simplifications
sfc-gh-pgaur
committed
35c7fe4
Fix build errors from code review changes
sfc-gh-pgaur
committed
ad0d639
Split EstimateCompressedSize loop into batched passes for vectorization
sfc-gh-pgaur
committed
39dd599
Address review nits: uint64_t→int64_t, spell out ALP, use uint8_t for byte spans
sfc-gh-pgaur
committed
2f78bb0
Move decoder creation outside benchmark loop
sfc-gh-pgaur
committed
4586802
Rename alp_wrapper.h/.cc to alp_codec.h/.cc
sfc-gh-pgaur
committed
38aa1ec
Convert ARROW_CHECK to Status returns on decode/load paths
sfc-gh-pgaur
committed
86ec59c
Add precondition docs to Store methods
sfc-gh-pgaur
committed
6d9cb43
Use signed integer types consistently per Arrow style guide
sfc-gh-pgaur
committed
92dee2c
Move out parameters to last position and remove unused enforce_mode
sfc-gh-pgaur
committed
2469b22
Add ALP encoding to parquet.thrift Encoding enum
sfc-gh-pgaur
committed
76c85fa
Remove ALP spec files; spec belongs in parquet-format repo
sfc-gh-pgaur
committed
7773c8e
Rename decomp/comp params to input/output; remove ARROW_CHECK(false) crash paths
sfc-gh-pgaur
committed
edda0c5
Copy header fields individually instead of bulk memcpy
sfc-gh-pgaur
committed
dc1e9bc
Replace StaticVector with std::vector for flexible vector size
sfc-gh-pgaur
committed
d3ea5a0
Make ALP vector_size configurable on the encode path
sfc-gh-pgaur
committed
1996ea5
Add parquet-level round-trip test for ALP at non-default vector sizes
sfc-gh-pgaur
committed
fab5d30
Address reviewer feedback: use int64_t, reorder out-params last, clean up naming
sfc-gh-pgaur
committed
9cf3d3c
Add sign branching to FastRound for correct rounding of negative values
sfc-gh-pgaur
committed
f96e902
Address reviewer feedback: SafeLoad/SafeStore, struct→class conversion
sfc-gh-pgaur
committed
2c310d8
Convert AlpEncodedVector and AlpEncodedVectorView to private members with getters
sfc-gh-pgaur
committed
275b93b
Remove unused AlpMetadataCache class and redundant GetNumElements()
sfc-gh-pgaur
committed
4b28dc2
Remove private inheritance from AlpCompression and AlpInlines
sfc-gh-pgaur
committed
c805e9a
Replace trailing return type with explicit Result<AlpHeader> in LoadHeader
sfc-gh-pgaur
committed
d62c475
Use SafeLoadAs for all header fields in LoadHeader
sfc-gh-pgaur
committed
df08fb2
Validate header fields in LoadHeader
sfc-gh-pgaur
committed
20f6394
Change AlpCodec public API from byte size to element count
sfc-gh-pgaur
committed
b62c063
Use uint8_t* consistently in AlpCodec API instead of char*
sfc-gh-pgaur
committed
5dc4032
Replace trailing return types with explicit return types
sfc-gh-pgaur
committed
ef1ea11
Cite ALP paper §3.1.2 for combination tie-break rule
sfc-gh-pgaur
committed
620b7b3
Pin AlpMode underlying type to uint8_t for serialization safety
sfc-gh-pgaur
committed
2a8aa77
Address reviewer nits: doc tweaks, sizeof(member_), incremental-decode TODO
sfc-gh-pgaur
committed
12f31e3
Convert AlpSampler uint64_t to int64_t
sfc-gh-pgaur
committed
89460ea
Remove GH-48701 reference from incremental-decode TODO
sfc-gh-pgaur
committed
3777241
Document actual crash/abort behavior in AlpCodec encode-path docs
sfc-gh-pgaur
committed
b96dea5
Use bit_util::CeilDiv for vector-count computations in AlpCodec
sfc-gh-pgaur
committed
6c2a109
Replace memcmp with IsBitwiseEqual helper in alp_test.cc
sfc-gh-pgaur
committed
6e792d2
Type-parameterize alp_test integration tests and fix TruncatedData
sfc-gh-pgaur
committed
81ffb57
Type-parameterize AlpEncodingAdHoc tests in encoding_test.cc
sfc-gh-pgaur
committed
1869d9f
Widen RandomData test range to exercise ALP exception fallback path
sfc-gh-pgaur
committed
9da0605
Remove uncompiled generate_alp_parquet and generate_reference_blobs
sfc-gh-pgaur
committed
81d9cba
Add incremental-encode TODO and snake_case rename in AlpEncoder
sfc-gh-pgaur
committed
60c3d89
Rename EncodeAlp parameter combinations to preset
sfc-gh-pgaur
committed
e20e3d1
Use bit_util::IsPowerOf2 and BytesForBits instead of raw bit manipulation
sfc-gh-pgaur
committed
defe57a
Remove remaining 'non-templated' references in ALP comments
sfc-gh-pgaur
committed
ba08de2
Remove GetBitPackedSize wrapper, inline bit_util::BytesForBits at callers
sfc-gh-pgaur
committed
bb64c79
Code review changes
sfc-gh-pgaur
committed
c3f5b00
Convert AlpCodec public API from ARROW_CHECK aborts to Status returns
sfc-gh-pgaur
committed
a99ae06
Use arrow::csv::TableReader for ALP benchmark dataset loading
sfc-gh-pgaur
committed
504dcea
Reorder AlpEncoded(For)VectorInfo so kStoredSize uses sizeof(member_)
sfc-gh-pgaur
committed
899d2e6
Make ALP num_exceptions and exception positions unsigned
sfc-gh-pgaur
committed
43eb5a7
Enforce the spec's [3, 15] log_vector_size range in ALP
sfc-gh-pgaur
committed
5c6706c
Use array matchers in ALP tests; note PLAIN fallback
sfc-gh-pgaur
committed
9bd4b6f
Document the [3, 15] vector_size range in AlpCodec docs
sfc-gh-pgaur
committed
7b7c1fd
Put EncodeAlp output parameters last
sfc-gh-pgaur
committed
78bde38
Test that ALP rejects malformed page headers
sfc-gh-pgaur
committed
c9d6196
Use std::countr_zero instead of __builtin_ctz in ALP
sfc-gh-pgaur
committed
fc53d86
Apply clang-format 18 to the ALP sources
sfc-gh-pgaur
committed
bb414c9
Validate ALP vector metadata before it steers the decode
sfc-gh-pgaur
committed
8647b0d
Correct inaccurate ALP comments; drop unused UnsafeResize
sfc-gh-pgaur
committed
021e4f9

Commits on Aug 27, 2026

Type-parameterize ALP compression tests; tighten round-trip asserts
sfc-gh-pgaur
committed
fc74a1e

Commits on Aug 31, 2026

Drop the ALP benchmark that needs out-of-tree datasets
sfc-gh-pgaur
committed
b3517dc
Test ALP against the parquet-testing conformance file
sfc-gh-pgaur
committed
a088d72
Put ALP behind an opt-in writer flag
sfc-gh-pgaur
committed
505d122

Commits on Sep 1, 2026

Tighten the ALP tests and document the encoding
sfc-gh-pgaur
committed
1510407
Validate the ALP element count and the offset chain
sfc-gh-pgaur
committed
c678cc7
Take the ALP value count from the page's own header
sfc-gh-pgaur
committed
df3efae
Encode an all-null ALP page as a bare header
sfc-gh-pgaur
committed
7c00dc2
Build the ALP sources once, and under meson too
sfc-gh-pgaur
committed
b768754
Serialize ALP little-endian instead of refusing to build
sfc-gh-pgaur
committed
d8edbe2
Move AlpMode out of the public header
sfc-gh-pgaur
committed
fff642c
Test the ALP element-count and offset-chain validation
sfc-gh-pgaur
committed
d00e0f3
Cover batched ALP decoding, and trim the encoder TODOs
sfc-gh-pgaur
committed
2fe94bb
Decode ALP pages one vector at a time
sfc-gh-pgaur
committed
a444a3d
Move the ALP end-to-end tests to their own file
sfc-gh-pgaur
committed
181b261
Name the ALP headers _internal.h
sfc-gh-pgaur
committed
80529af

Commits on Sep 3, 2026

Commits on Sep 8, 2026

Size the sample buffer without going through a double
sfc-gh-pgaur
committed
1ef3ff6
Say in prose what the target type may be
sfc-gh-pgaur
committed
83d01c9
Export the decode member templates from the shared library
sfc-gh-pgaur
committed
4831e44
Cast the element count the ALP test passes
sfc-gh-pgaur
committed
f048578
Export the internal ALP classes the test links against
sfc-gh-pgaur
committed
04a4872
Widen the shift the ALP test takes a modulus with
sfc-gh-pgaur
committed
db6d15a
Give the ALP roundtrip helper a name of its own
sfc-gh-pgaur
committed
12199dc
Read the stored size through its accessor
sfc-gh-pgaur
committed
d563ad9