GH-48701: [C++][Parquet] Add ALPpd encoding by prtkgaur · Pull Request #48345 · apache/arrow

prtkgaur · 2025-12-05T00:23:45Z

Rationale for this change

ALP significantly improves on the compression ratio and decompression speed over of float/double columns over other encoding/compression techniques.

Spec

Spec
This PR also contains a terse version of the spec in the file cpp/src/arrow/util/alp/ALP_Encoding_Specification_terse.md which can go in the Encodings.md

Parquet Format PR

Dataset PR (parquet-testing)

apache/parquet-testing#100

What changes are included in this PR?

This PR
Introduces ALP (pseudo-decimal) encoding into c++ arrow code.
We also provide benchmarks and dataset to prove the effectiveness of the above algorithm.

Adding above needed us to add following classes.

Alp h/cc : Houses core logic for encoding and decoding.
Sampler h/cc : Houses logic to sample and select parameters for encoding.
AlpWrapper h/cc : Binds together Alp and Sampler classes.

Integration of the above code was done in

Encoder/Decoder cc which exposes wrapper to encode buffer of data.

Are these changes tested?

We have added unit tests to test the code.
Also the benchmarks have been added that cover wide variety of floating point values from low precision to high precision.

Unit tests

alp_test.cc

Benchmark tests

encoding_benchmark.cc and encoding_alp_benchmark.cc

Are there any user-facing changes?

It's a new encoding so the only impact is query performance which we claim will only get better.

DuckDB

We did look at DuckDB's ALP implementation while we were implementing ALP and would like to give that team the desired credit.

GitHub Issue: [ALP][Parquet] Add C++ implementation of ALPpd encoder/decoder #48701

github-actions · 2025-12-05T00:24:08Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

alamb · 2025-12-08T14:07:48Z

Thanks @prtkgaur -- it is super exciting to see this movement.

Unfortunately, I am not familiar with the C/C++ codebase to give this a realistic review.

I started the CI checks on this PR and had some comments about the testing.

prtkgaur · 2025-12-08T23:31:26Z

+    std::string tarball_path = std::string(__FILE__);
+    tarball_path = tarball_path.substr(0, tarball_path.find_last_of("/\\"));
+    tarball_path = tarball_path.substr(0, tarball_path.find_last_of("/\\"));
+    tarball_path += "/arrow/cpp/submodules/parquet-testing/data/floatingpoint_data.tar.gz";


@Reviewer the data sits in the parquet-testing submodule
apache/parquet-testing#100

prtkgaur · 2025-12-08T23:41:27Z


+  // Unsafe resize without initialization - use only when you will immediately
+  // overwrite the memory (e.g., before memcpy). Only safe for POD types.
+  void UnsafeResize(size_t n) {


Using this over resize gave us around 2-3% performance improvement

emkornfield · 2025-12-09T21:32:31Z

Talked offline and wanted to capture notes on high-level changes:

For headers, lets try to reduce duplication with values already in the parquet header.
For remaining items in headers, lets try to be parsimonious with values (i.e. 4 bytes is probably overkill for enums)
Naming convention on files is off (use snake_case).
Given description of ALP, we probably want a top level encoding enum value for the 2 different modes of ALP.

prtkgaur · 2025-12-16T01:24:55Z

Talked offline and wanted to capture notes on high-level changes:

For headers, lets try to reduce duplication with values already in the parquet header.

For remaining items in headers, lets try to be parsimonious with values (i.e. 4 bytes is probably overkill for enums)

Naming convention on files is off (use snake_case).

Given description of ALP, we probably want a top level encoding enum value for the 2 different modes of ALP.

Thanks for the feedback @emkornfield. We have addressed

Reduce duplication of fields between page header and alp header
Other fields have been updated to use 1 byte. Header is now just 8 bytes compared to 40 bytes earlier.
Naming of files has been updated.
We do have the top level enums describing the mode and layout structure.
enum class AlpBitPackLayout { kNormal }; and enum class AlpMode { kAlp };

github-actions · 2026-01-01T20:58:34Z

⚠️ GitHub issue #48701 has been automatically assigned in GitHub to PR creator.

prtkgaur · 2026-01-02T05:56:08Z

+
+    // Slow path: partial read - decode to intermediate buffer
+    // ALP Bit unpacker needs batches of 64
+    if (needs_decode_) {


TODO(prateek) : check with Antoine and other reviewers if there is a way to relax this constraint. Though this has negligible impact on performance.

no action needed

prtkgaur · 2026-01-02T05:59:36Z

umm ideally this submodule shouldn't be attached with this commit.
Should revert the changes to this file.

Pull request for dataset
apache/parquet-testing#100

prtkgaur · 2026-01-02T06:00:11Z

Please check cpp/src/arrow/util/alp/ALP_Encoding_Specification_terse.md for a more terse spec of the encoding.

Also this file will be removed once the spec in parquet format repository is merged.

prtkgaur · 2026-01-02T06:01:08Z

+
+## 2. Data Layout
+
+ALP encoding consists of a page-level header followed by one or more encoded vectors. Each vector contains up to 1024 elements.


Replace 1024 with the constant specified in AlpConstant file.

Each test was checked against the code with its validation removed, so a regression in any of the three shows up as a failure.

The batched test fails if the decoder's remaining count and its scratch index disagree, which is the divergence the old three-variable state allowed.

AlpCodec<T>::VectorReader validates the header and offset chain once, then decodes any single vector on demand, so a batched reader needs scratch for one vector rather than for a whole page.

arrow_reader_writer_test.cc is already long, so the encoding round trips move to arrow/arrow_encoding_test.cc, which later encodings can share.

The names now say what the install rules already do: these headers are not installed and carry no backward-compatibility obligation.

The test hand-duplicated a float block and a double block, which is what the type-parameterized suite is for. Also move the float decode benchmark's output buffer out of the loop and stop timing its SetData, matching the double one.

Incremental decode calls DecodeVector once per vector, and each call built a fresh view and a fresh vector of unpacked integers, so a page's worth of vectors meant a heap allocation per vector (three, when the vector had exceptions). Let the reader hold that scratch and reuse it: ResetDataOnly points an existing view at the next vector's data, and DecompressVectorView now unpacks into a caller-provided span instead of returning a vector. Decoding 32 vectors of doubles allocates 11 times for the first vector and nothing after it, where before it allocated on every one.

The ALP decode benchmarks were the only ones in the file hiding SetData behind PauseTiming, which left their numbers incomparable to their neighbours'.

parquet.thrift is a vendored copy of the format repo's file, and ALP has since been merged there. The comment written here says the same thing in different words, and points at Encodings.md#alp for the full spec, which now lives in AlpEncoding.md.

parquet/encoder.cc and decoder.cc call AlpCodec, whose out-of-line definitions live in libarrow, so the ten symbols they reference have to cross the library boundary. Every other arrow symbol libparquet imports carries ARROW_EXPORT; these did not, which breaks an MSVC shared build and any build that gives arrow hidden default visibility. Also fold the three-deep namespace into one, which is what clang-tidy's modernize-concat-nested-namespaces asks for.

Four of the five const members were copies of AlpConstants values and two of those were only ever read to derive the sampling interval, which is itself a constant. Reference the constants directly and keep the derived one as a static constexpr, which removes the constructor and 40 bytes from every sampler.

wgtmac · 2026-09-07T15:45:33Z

+  const ExactType* data = input_vector.data();
+  const ExactType frame_of_ref = for_info.frame_of_reference();
+
+#pragma GCC unroll AlpConstants::kLoopUnrolls


These currently break Clang and MSVC builds under -Werror. Please make the ALP sources compile cleanly with both Clang and MSVC.

Both pragmas now sit behind defined(__GNUC__) && !defined(__clang__), so Clang and MSVC see nothing to warn about.

I compared the assembly before choosing that over simply deleting them: GCC's output for the two decode loops does change without them, so the guard keeps its codegen rather than trading it away, and the other two compilers fall back to whatever their optimizers work out. A -Wall -Wextra sweep over the ALP sources is clean.

wgtmac · 2026-09-07T15:45:33Z

+  /// can then decode any single vector on its own, so serving a batch of values
+  /// costs work in proportion to the batch rather than to the buffer, and needs
+  /// scratch for one vector rather than for the whole buffer.
+  class VectorReader {


parquet_shared calls these out-of-line methods, but Windows cannot see their symbols.

Please export the VectorReader methods. Then verify that the MinGW shared build links Open, VectorLength, and DecodeVector.

Added ARROW_EXPORT to the nested class as well, which is the idiom the other nested classes in arrow that libparquet calls into follow.

One thing worth recording, because it caught me out: GCC propagates the enclosing class's visibility attribute into a nested class, so the usual local proxy for this defect — compile with -fvisibility=hidden and read the ELF visibility of the symbol — reports DEFAULT for Open, VectorLength and DecodeVector whether or not the nested marker is there. There is no clang or mingw toolchain on the machine I am working on, so I cannot claim the MinGW link is verified; CI is the check for it.

wgtmac · 2026-09-07T15:45:33Z

+      bit-packing; values that cannot be converted losslessly are stored as
+      exceptions. See Encodings.md for the detailed specification.
+   */
+  ALP = 10;


Please run cpp/build-support/update-thrift.sh and commit the updated cpp/src/generated/parquet_types.h and cpp/src/generated/parquet_types.cpp.

Regenerated and committed. I used thrift 0.23.0, which is the version that produced the checked-in files — the vendored copy in the build tree is 0.22.0 and regenerating with it churns the whole file. The diff is the ALP enum value and nothing else.

wgtmac · 2026-09-07T15:45:34Z

+      PARQUET_ASSIGN_OR_THROW(
+          reader_, ::arrow::util::alp::AlpCodec<T>::VectorReader::Open(data, len));
+      total_values_ = reader_.num_elements();
+      if (total_values_ > num_values) {


This compares the ALP non-null count with the page slot count. A corrupt optional page can therefore leave encoded values unread without an error. Please reject the page if the decoder is not exhausted after all definition levels are consumed.

Good catch — an optional page satisfies that count check with room to spare, so it cannot see this.

The decoder now records the page's level count and the levels that carried no value, and rejects the page once every level is accounted for while the payload still has values unread. The accounting is derived from decoder state rather than accumulated per call, which matters because the base DecodeSpaced delegates to Decode: counting in both places would double-count, and deriving it keeps the check idempotent no matter which path a reader takes. A new round-trip test covers it for float and double, and it fails on both when the check is removed.

Clang and MSVC warn on an unknown pragma, which -Werror turns into an error. GCC keeps both; dropping them there changes its codegen.

dllexport does not reach into a nested class, so libparquet cannot resolve the three reader symbols it imports when MSVC links parquet.dll.

Machine output of update-thrift.sh with thrift 0.23.0.

The header count check an optional page passes cannot see this, because such a page legitimately holds fewer values than levels.

wgtmac · 2026-09-08T04:04:55Z

CI is still failing. Could you make them happy?

prtkgaur · 2026-09-08T04:38:51Z

CI is still failing. Could you make them happy?

Ack yes. Let me look into them.

MSVC treats the double-to-size_t narrowing as an error. Integer ceiling division gives the same count.

clang cannot attach a tparam to a member template declared in its class, so the doc build rejects it.

The marker on a class does not reach a member template, so the Windows link could not find the instantiations.

CompressVector takes int32_t, so passing a size_t narrows and clang rejects it with -Wshorten-64-to-32 under -Werror.

Their definitions live in implementation files, so a separate test executable cannot reach them across a shared library boundary without the marker.

The index is 64-bit, so a 32-bit shift operand makes MSVC warn that the shift may have been meant to be 64-bit, and CHECKIN treats that as an error.

The new ALP test file defines a write-then-read helper called DoRoundtrip in an anonymous namespace. arrow_reader_writer_test.cc, which shares its test target, already has a DoRoundtrip whose trailing parameters are defaulted, so a four-argument call matches both signatures exactly. Separate translation units never see each other's helper, so the collision is invisible until the two files are compiled as one, which is what the Windows build does with unity builds enabled. Rename ours after what it does.

AlpEncodedVectorInfo is exported, so for anything linking against the shared library its static constexpr kStoredSize is an import. Reading the value is fine, because a constexpr initializer is folded at compile time, but binding a reference to it asks for an address, and a constexpr member has no out-of-line definition in the library to take the address of. Windows GCC then fails to link the test. The same assertion on AlpEncodedForVectorInfo two lines below is unaffected: that one is a class template, so its static members are instantiated in the translation unit that uses them. Only the non-template class needs the value read by hand, and the test already checks it through GetStoredSize on the next line, which returns by value.

github-actions Bot added Component: Parquet Component: C++ awaiting review Awaiting review labels Dec 5, 2025

prtkgaur force-pushed the gh540-alp-pseudoDecimal-encoding branch 3 times, most recently from 1b78a5c to d563ce0 Compare December 7, 2025 15:46

alamb reviewed Dec 8, 2025

View reviewed changes

Comment thread cpp/src/arrow/util/alp/data/floatingpoint_data.tar.gz Outdated

alamb reviewed Dec 8, 2025

View reviewed changes

Comment thread cpp/src/parquet/types.h

alamb mentioned this pull request Dec 8, 2025

[Parquet] Prototype ALP encoding apache/arrow-rs#8748

Closed

prtkgaur changed the title ~~[Gh540] Add ALPpd encoding to parquet~~ [Gh539] Add ALPpd encoding to parquet Dec 8, 2025

prtkgaur commented Dec 8, 2025

View reviewed changes

prtkgaur changed the title ~~[Gh539] Add ALPpd encoding to parquet~~ [Gh539][Encoding] Add ALPpd encoding to parquet Dec 8, 2025

prtkgaur changed the title ~~[Gh539][Encoding] Add ALPpd encoding to parquet~~ [Gh-539][Encoding] Add ALPpd encoding to parquet Dec 8, 2025

prtkgaur force-pushed the gh540-alp-pseudoDecimal-encoding branch from 0c035b7 to 1cb0852 Compare December 8, 2025 23:48

prtkgaur changed the title ~~[Gh-539][Encoding] Add ALPpd encoding to parquet~~ [Gh-539][ParquetEncoding][c++] Add ALPpd encoding to parquet Dec 9, 2025

prtkgaur force-pushed the gh540-alp-pseudoDecimal-encoding branch from 35f1ad7 to 0908342 Compare December 15, 2025 21:28

github-actions Bot added the Component: Documentation label Dec 31, 2025

prtkgaur changed the title ~~[Gh-539][ParquetEncoding][c++] Add ALPpd encoding to parquet~~ [Gh-48701][ParquetEncoding][c++] Add ALPpd encoding to parquet Dec 31, 2025

kou changed the title ~~[Gh-48701][ParquetEncoding][c++] Add ALPpd encoding to parquet~~ GH-48701: [C++][Parquet] Add ALPpd encoding Jan 1, 2026

prtkgaur commented Jan 2, 2026

View reviewed changes

prtkgaur force-pushed the gh540-alp-pseudoDecimal-encoding branch from 1b08599 to f5f5011 Compare January 12, 2026 16:00

prtkgaur marked this pull request as ready for review January 13, 2026 18:03

prtkgaur requested a review from wgtmac as a code owner January 13, 2026 18:03

alamb mentioned this pull request Jan 14, 2026

GH-533: Adaptive Lossless Floating-Point (ALP) Encoding apache/parquet-format#548

Closed

1 task

sfc-gh-pgaur added 7 commits September 1, 2026 21:07

Test the ALP element-count and offset-chain validation

d00e0f3

Each test was checked against the code with its validation removed, so a regression in any of the three shows up as a failure.

Cover batched ALP decoding, and trim the encoder TODOs

2fe94bb

The batched test fails if the decoder's remaining count and its scratch index disagree, which is the divergence the old three-variable state allowed.

Decode ALP pages one vector at a time

a444a3d

AlpCodec<T>::VectorReader validates the header and offset chain once, then decodes any single vector on demand, so a batched reader needs scratch for one vector rather than for a whole page.

Move the ALP end-to-end tests to their own file

181b261

arrow_reader_writer_test.cc is already long, so the encoding round trips move to arrow/arrow_encoding_test.cc, which later encodings can share.

Name the ALP headers _internal.h

80529af

The names now say what the install rules already do: these headers are not installed and carry no backward-compatibility obligation.

prtkgaur requested a review from wgtmac September 3, 2026 16:47

sfc-gh-pgaur added 4 commits September 3, 2026 22:10

Drop dead ALP code, link the TODOs, unpause the benchmarks

60c899a

The ALP decode benchmarks were the only ones in the file hiding SetData behind PauseTiming, which left their numbers incomparable to their neighbours'.

wgtmac requested changes Sep 7, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Sep 7, 2026

sfc-gh-pgaur added 4 commits September 7, 2026 23:43

Guard the loop pragmas behind the compiler that honours them

1e2d88b

Clang and MSVC warn on an unknown pragma, which -Werror turns into an error. GCC keeps both; dropping them there changes its codegen.

Export the nested VectorReader as well as its enclosing class

8fe553f

dllexport does not reach into a nested class, so libparquet cannot resolve the three reader symbols it imports when MSVC links parquet.dll.

Regenerate the thrift bindings so they carry the ALP enum value

4ee3c1d

Machine output of update-thrift.sh with thrift 0.23.0.

Reject an ALP page whose values outlive its definition levels

b3e581a

The header count check an optional page passes cannot see this, because such a page legitimately holds fewer values than levels.

prtkgaur requested a review from wgtmac September 8, 2026 00:04

sfc-gh-pgaur added 8 commits September 8, 2026 04:58

Size the sample buffer without going through a double

1ef3ff6

MSVC treats the double-to-size_t narrowing as an error. Integer ceiling division gives the same count.

Say in prose what the target type may be

83d01c9

clang cannot attach a tparam to a member template declared in its class, so the doc build rejects it.

Export the decode member templates from the shared library

4831e44

The marker on a class does not reach a member template, so the Windows link could not find the instantiations.

Cast the element count the ALP test passes

f048578

CompressVector takes int32_t, so passing a size_t narrows and clang rejects it with -Wshorten-64-to-32 under -Werror.

Export the internal ALP classes the test links against

04a4872

Their definitions live in implementation files, so a separate test executable cannot reach them across a shared library boundary without the marker.

Widen the shift the ALP test takes a modulus with

db6d15a

The index is 64-bit, so a 32-bit shift operand makes MSVC warn that the shift may have been meant to be 64-bit, and CHECKIN treats that as an error.


		## 2. Data Layout

		ALP encoding consists of a page-level header followed by one or more encoded vectors. Each vector contains up to 1024 elements.

Uh oh!

Conversation

prtkgaur commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Spec

Parquet Format PR

Dataset PR (parquet-testing)

What changes are included in this PR?

Are these changes tested?

Unit tests

Benchmark tests

Are there any user-facing changes?

DuckDB

Uh oh!

github-actions Bot commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

alamb commented Dec 8, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield commented Dec 9, 2025

Uh oh!

prtkgaur commented Dec 16, 2025

Uh oh!

github-actions Bot commented Jan 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wgtmac commented Sep 8, 2026

Uh oh!

prtkgaur commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

prtkgaur commented Dec 5, 2025 •

edited

Loading