[WIP][POC] Add FSST encoding - #3784
Draft
prtkgaur wants to merge 10 commits into
Draft
Conversation
FSST is proposed for Parquet in parquet-format#531, with a C++ implementation on an Arrow branch and nothing in Java. A format change needs a second implementation, so this is the codec core it needs: a symbol table, a trainer, and a compressor and decompressor for the 8-bit code stream. The trainer is a port of FSST's reference implementation rather than a reimplementation, because a writer's ratio has to match what other writers produce for the same input, and the symbol table a reader rebuilds has to be the table the writer chose. Seven constructs did not survive a literal port and are commented where they appear: the symbol descriptor needs 64 bits because the length field alone overflows an int at length 8; the reference hash needs an unsigned shift, which matters because the sampler chains it over full 64-bit values; every byte read needs masking; the hash table becomes parallel long arrays rather than 1024 objects; the pair counter reuses one array across pages with the reference's lazy reset; and the compressor's unconditional eight-byte read needs a padded buffer, since Java cannot read past an array. The file's code space is a permutation of the trainer's. The trainer numbers symbols in an order its own shortcuts depend on, while the file orders them by length so a reader can rebuild the table from a length histogram alone. Rather than renumber and break the shortcuts, the trainer's numbering is kept and translated at the emit site, folded into the compressor's inner loop so there is no scratch buffer and no second pass. Everything here sits behind interfaces that name the concept rather than the codec, because two more codecs are planned against them: FSST_16, which the proposal's own conformance files already contain, and OnPair. Each is a new symbol table, trainer and code stream implementation, not a change to these. Two test classes. One round-trips through serialization on adversarial input: every byte value, values that straddle the compressor's chunk boundary, incompressible bytes, empty values, and a corpus large enough to exercise the sampler. The other compares against the reference implementation directly: for six corpora it asserts that the serialized symbol table and the code stream are byte-identical to what the reference produces. The corpora are generated from a spelled-out generator rather than stored, and each one's digest is asserted before the comparison, so a drifting generator reports itself as a generator problem instead of as a codec regression. Not here yet: the writer and reader that put this on a page, and the symbol table's file-level home, which is thrift-gated.
The page body is the part of this encoding two implementations have to agree on byte for byte, so it is one class with the framing in one place, and it is the same class for every codec built on a symbol table: a code stream plus one end offset per value, behind a fixed nine-byte header. End offsets rather than lengths, because that is what makes a value readable without expanding the ones in front of it, which is the property a codec over a page cannot give and the reason for preferring this over compressing the page whole. The offset section is written through the encoders this module already has rather than by hand, since plain int32 and delta binary packing are both spelled out here already and a second spelling of either would be a second thing to keep compatible. Delta offsets are offered alongside plain ones because on a page of short values the offsets are a large fraction of the payload and almost all of it is the same increment repeated; a test measures that, and it is the argument for making delta the default when the writer lands. The read side validates before it trusts: the header against the page header's own value count, the offset section's length against what the encoding implies, and the offsets against the code section they partition, including the case where they stop short and leave bytes no value can reach. Every later read depends on those bounds, so they are checked once rather than per value. Twelve tests cover the layout byte for byte and each rejection. A symbol table belongs to a column chunk rather than to a page, and a values writer cannot write anything outside its own page, so the writer publishes the trained table through a sink and the reader takes it from a source. Both are one method. That is a test harness today and a page of its own once the format carries one, and neither choice reaches the codec. Alongside them, one class maps a table representation to the code that implements it, so a second representation is a new implementation plus two lines of dispatch rather than a change to anything above.
Adds the values writer and reader that turn buffered binary values into a code stream over a trained symbol table, and back. One writer and one reader serve every symbol table representation. The representation decides how a table is trained, serialized and framed, and all three already sit behind seams, so nothing about them reaches these two classes beyond the type they are asked for. The awkward part of the encoding is that a table belongs to a column chunk while a writer only ever sees a page. The writer trains at the first page, publishes the table through the sink, and keeps it for the rest of the chunk; a later page must not train its own or the reader would decode it wrongly. Values are buffered raw because a trainer reads them in an order of its own choosing and more than once, and because a fallback to another encoding has to replay them. Compressing a page can also make it bigger, which is what the fallback contract is for: whether the codes came out smaller than the values is the only question worth asking, and it is asked once. A page of single bytes with plain offsets is the case that answers no -- and the same page with packed offsets answers yes, which is the argument against writing offsets plain. Encoding.FSST hands out the reader for BINARY and rejects every other type. The reader it hands out has nowhere to get its table from, because the format has nowhere to put one yet; it says so rather than failing later.
Turning the encoding on is a per-column property, off by default, and only BINARY columns honour it. The writer factories give the symbol table the first attempt at a column the dictionary has given up on, and hand back the columns whose codes come out no smaller than the values, so the writer stack is dictionary, then symbol table, then plain or delta byte array. The offsets into a page's code stream default to delta packing rather than four plain bytes a value. On text the encoding halves gets to roughly twenty bytes a value, so plain offsets are about a fifth of the page and delta packing takes them to near one byte; plain offsets are worth having only for a reader that wants them addressable without decoding. A writer built from the format's own settings gets a sink that refuses the table, because the format has nowhere to keep it and the pages would not be readable. It fails at the first page, while the failure can still name the reason. For the same reason converting the encoding to a footer value throws, which is why the two tests that convert every encoding value now skip it.
parquet-column's tests have been migrated off JUnit 4, and an enforcer rule bans the old imports outright: only the four benchmark classes are exempt, because they still need @rule. These four files were written against JUnit 4 and only ever run with the enforcer skipped, so the ban was never checked against them. Converted mechanically, then the comparisons that had become a boolean plus a message were rewritten as the assertion they were describing - isLessThanOrEqualTo, isBetween, isPositive - so a failure prints both sides instead of "expected true".
The page layout in this package is only worth having if it is the same layout everyone else writes. Nothing in the tree proved that: every test so far encoded with this code and decoded with this code, which passes just as well for a private format. The C++ implementation's interop file supplies the missing side. Its pages are lifted out and committed as fixtures, and the reader is required to reproduce what the file's plain reference columns hold - so the fixture does not depend on any FSST decoder being correct, including this one. Five cases: high cardinality values over fourteen pages sharing one table, zero-length values and short pages, the escape code, and a chunk whose offsets are stored plainly rather than packed. Both offset section encodings appear, checked from the page bytes rather than through the reader, so a reader that ignored the mode byte could not hide it. The expected values are kept as digests rather than as payloads: the values themselves would have been fifty kilobytes in a module whose test resources total under three hundred. The header of expected.txt records where the file came from, how the fixture is laid out, and how the digest is computed, so it can be regenerated. The 16-bit columns are skipped. Nothing reads them yet, and the table body they carry is a different shape from the one the format currently describes.
…nery Mirrors the dictionary page seam for seam: a SymbolTablePage alongside DictionaryPage, default methods on PageReader/PageWriter so no existing implementor breaks, and usesSymbolTable()/getSymbolTableBasedValuesReader on Encoding beside the dictionary equivalents. The publish moves off the writer's hot path and onto the chunk-finalize hook (toSymbolTablePageAndClose(), the toDictPageAndClose() analogue), which also fixes a real defect: FallbackValuesWriter.getBytes() used to call the initial writer's getBytes() - which published the table - before deciding whether to fall back, so a column that fell back to PLAIN or dictionary still published a table nobody would read. Publishing now only happens when the initial writer was actually used. SymbolTableSink and SymbolTables.rejectingSink() are retired: the page hook is a strictly better version of the same seam, and nothing depends on the old one since it was never wired to a real writer. ColumnReaderBase reads the symbol table page once per chunk and fails loudly if a column needs one that is not there, rather than guessing.
Values through the column machinery, not just through the values writer/reader pair: a MemPageStore, a real ColumnWriteStoreV1, a real ColumnReadStoreImpl. Covers a high-cardinality column that keeps the encoding and publishes a table, a multi-page chunk sharing one table, nulls and empty strings on an optional column, a column that expands and falls back with no table published, skip() and re-reading a column from the same store, both offset encodings, two row groups each training their own table, and the negative control the plan asked for: strip the symbol table out of the page reader and confirm the column reader refuses to proceed rather than silently decoding as if the encoding were something else.
The existing interop test only proves this implementation can read a packed offset section the C++ writer produced. It does not prove a packed section this implementation writes comes out byte-identical, since delta-binary-packing has framing choices a byte-compatible decoder does not have to make the same way. Decode each fixture page's packed offsets and re-encode them through the same DeltaBinaryPackingValuesWriterForInteger path SymbolTablePayloadWriter uses, then compare against the fixture's own offset bytes. All 26 packed-offset pages across the five interop cases re-encode byte-for-byte.
Compares FSST to DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY and RLE_DICTIONARY, with and without a zstd second pass, on a synthetic corpus with the shared structure (repeated URL templates and vocabulary) that lets each encoding actually exploit redundancy. Reports ratio, encode and decode separately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In the email threads Arnav wanted someone to help with the Java impl of FSST, took a stab at it over the last week.
cc @ArnavBalyan , @emkornfield , @julienledem, @RussellSpitzer, @alamb
Rationale for this change
FSST (Fast Static Symbol Table) is a proposed Parquet encoding for string/binary columns
(parquet-format issue #531). It has a C++ reference implementation on an Arrow branch and
a WIP draft in arrow-rs, but no Java implementation exists — no file, commit, issue, PR,
or branch on the proposal author's fork mentions it as of 2026-09-08. A parquet-format
change needs reference implementations in the ecosystem's major languages before it can
be approved, so the empty Java slot is one of the things keeping the proposal at Draft.
This PR fills that slot.
Two follow-ups are planned and shape the design from the start: FSST_16 (16-bit codes,
already required by the interop conformance fixtures) and OnPair. Both attach as new
implementations behind the seams this PR adds, not as refactors of this PR's code.
What changes are included in this PR?
Scope is parquet-column only, mirroring how PFOR encoding was landed: no new Thrift
page type, no footer fields, no
ParquetFileWriter/ParquetFileReaderplumbing.ParquetMetadataConverter.getEncoding()still throws forEncoding.FSST, so a.parquetfile cannot be written with an FSST column that no released reader can openwhile the format proposal remains unratified. The property defaults to
false.values/symboltable/fsst/), ported from CWI's referenceimplementation (
cwida/fsst@89f49c580c6388acf3b6ed2a49e1bfde6c05e616, MIT — thesame commit Arrow vendored): the trainer's five-round gain loop, the split-plane
Counters, the deterministic sampler, and the scalar 8-bit compressor. AVX-512 and thenative table export/import are dropped; Parquet defines its own table serialization.
values/symboltable/):SymbolTable,SymbolTableTrainer,SampleReservoir,CodeStreamEncoder/Decoder,SymbolTablePayload— shared byFSST now, and by FSST_16/OnPair when they land.
SymbolTableValuesWriter/ReaderimplementingValuesWriter/ValuesReader, reusingFallbackValuesWriter/RequiresFallbackunchanged so a column falls back to dictionary/PLAIN when FSST doesn't shrink the
page, the
usesSymbolTable()/getSymbolTableBasedValuesReader(...)hooks onEncoding, the enablingColumnProperty, and the factory slot (dictionary's fallbackwriter, so low-cardinality columns keep RLE_DICTIONARY and FSST takes the
high-cardinality tail).
SymbolTablePagebesideDictionaryPage,defaultmethods added to
PageReader/PageWriter(so no existing implementor breaks), withthe publish moved onto the writer's chunk-finalize hook rather than the hot
getBytes()path. This is also where a real defect got fixed along the way: a columnthat fell back to PLAIN/dictionary used to still publish a symbol table nobody would
read.
This transport is intentionally the seam the eventual Thrift
SYMBOL_TABLE_PAGEpagetype attaches to — nothing above it should need to change once parquet-format ratifies
the proposal.
Are these changes tested?
Byte-exactness against the C++ format. Symbol table bodies and data page payloads
extracted from the interop file in
apache/parquet-testingPR PARQUET-187: Replace JavaConversions.asJavaList with JavaConversions.seqAsJavaList #121 and committed asparquet-column test fixtures. Java matches C++ byte-for-byte on all five extracted
chunks, including 255-symbol saturation, chunk-boundary crossings, and all-empty
values.
Encode-direction check for the packed offset section: decoding each fixture's
offsets and re-encoding them through the same
DeltaBinaryPackingValuesWriterForIntegerpath the writer uses matches the fixture's own bytes byte-for-byte on all 26
packed-offset pages across the five cases.
End-to-end through real column machinery: values written through a
ColumnWriteStoreV1and read back throughColumnReadStoreImpl/ColumnReader,covering multi-page chunks, nulls/empty strings, values that expand under FSST (must
fall back, with no symbol table page published), both offset encodings, and a
row-group boundary retraining the table. A negative control (strip the symbol table
page from a hand-built page store) fails at
getColumnReader(path)itself, confirmingthe test would catch a missing table rather than silently falling back to PLAIN.
Ratio parity on real corpora. Round-tripped byte-exact on ~10M values across 20
real string columns (TPC-H, ClickBench, and variant-JSON derived corpora, ~305 MB
raw). Compressed/raw ratios ranged 0.14 (repeated short strings) to 0.96 (short,
high-entropy address columns, where FSST buys almost nothing) — 0.36 overall.
Benchmark against DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY and RLE_DICTIONARY,
each with and without a zstd second pass. On a synthetic corpus with realistic
repeated structure (URL templates, repeated vocabulary):
Are there any user-facing changes?
Yes, opt-in only. A new
ParquetPropertiesflag (fsstEnabled) and matchingParquetOutputFormatconf key turn FSST on for BINARY columns, following the sameper-column property pattern used for PFOR. It defaults to
false, so existing writersand files are unaffected.
PageReader/PageWritergain newdefaultmethods, so noexisting implementor of those interfaces needs to change.
Please Note
There is no file-format change: a file written with FSST enabled uses only existing
Parquet page types, and
ParquetMetadataConverterdeliberately throws forEncoding.FSSTin the Thrift-facing path, so this cannot silently produce a file that areleased reader would fail to open through the normal encoding-mapping route.