feat(btrblocks): add VarBinScheme for binary arrays - #9579
Conversation
Canonical binary arrays are VarBinViewArray, which spends a fixed 16 bytes per element on an opaque views buffer. No scheme could compress that buffer, so any binary column that the dictionary scheme declined was written as payload plus 16 B/value, regardless of content. VarBinScheme re-encodes as VarBinArray, replacing the views buffer with an offsets child array that the cascading compressor compresses with the ordinary integer schemes. For fixed-width values the offsets are a constant-stride sequence and collapse to nothing. This mirrors what FSSTScheme already does for strings. Measured at 100k rows (tests/varbin_scheme.rs), compressed nbytes against the same compressor with the scheme excluded: nulls every 7th 2,966,827 -> 1,647,348 (0.56) random 16B (hash) 3,200,000 -> 1,600,000 (0.50) shared prefix 3,200,000 -> 1,600,000 (0.50) random 256B 27,200,000 -> 25,600,000 (0.94) The one golden snapshot that moves also improves: binary_low_cardinality dictionary values go from 96 to 52 bytes as the scheme cascades into the dictionary's values child. Checks: cargo test -p vortex-btrblocks (all pass), cargo test -p vortex-file (144 pass), cargo clippy -p vortex-btrblocks --all-targets --all-features (clean), cargo +nightly fmt --all. Signed-off-by: "Claude" <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GJdgPga43u5rXYwv6R5b19
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | compress_fsst[(500, 64, 8)] |
530.5 µs | 595 µs | -10.85% |
| ⚡ | WallTime | words_gather_scalar_avx2[65536] |
9.4 µs | 8.3 µs | +13.44% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/varbin-scheme (d598e09) with develop (e4b3421)
Footnotes
-
54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
2 participants
Summary
Canonical binary arrays are
VarBinView, which spends a fixed 16 bytes per element on an opaque views buffer that no scheme can compress. This PR adds aVarBinSchemethat re-encodes binary columns as offset-basedVarBin, replacing the views buffer with an offsets child that the cascading compressor can compress with the ordinary integer schemes. For fixed-width values the offsets are a constant-stride sequence and collapse to almost nothing.First of two PRs; the follow-up lets FSST compress binary arrays behind a sampled gate.
Changes
vortex.binary.varbinscheme (vortex-btrblocks/src/schemes/binary/varbin.rs) and register it inALL_SCHEMES.vortex-btrblocks/tests/varbin_scheme.rs: measures the default compressor with and without the scheme at 100k rows and asserts enabling it never grows the output, plus a roundtrip check. Measured compressed sizes: random 16B values 3,200,000 → 1,600,000; random 256B 27,200,000 → 25,600,000; the shared-prefix and nulls cases are unchanged because dictionary/FSST-style schemes win selection there.golden__default__binary_low_cardinalitysnapshot updates because dict values now store asVarBin.Checks run:
cargo test -p vortex-btrblocks,cargo clippy -p vortex-btrblocks --all-targets --all-features,cargo +nightly fmt --all.Generated by Claude Code