Repository navigation
Conversation
Slot 0 of each row in the tag table holds the row's head, not an entry, so the candidate loops tested `matchPos == 0` on every tag hit. Clear that slot's bit group from the match mask once instead. `continue` did not consume an attempt nor store a candidate, so the candidates (and the compressed output) are unchanged.
Row offsets and slot positions were U32, so every tagTable / hashTable access in the insertion loop needed a zero-extension, and on AArch64 the uxtw-extended addressing modes cost an extra uop each. Use size_t, read hashSalt once (BYTE stores into the tag table may alias *ms, which forced a reload per position), and reuse the head byte already loaded for the match mask when inserting the current position. llvm-mca (Neoverse-N1 model) on the generated insertion loop: 33 -> 30 uops per position. Output is unchanged.
In the row matchfinder, 57% of candidate match lengths are 4-7 bytes and ~85% are below 16, so ZSTD_count()'s "first 8 bytes equal?" branch and its loop exit are close to unpredictable (~24% of level 6 cycles are lost to bad speculation on Sapphire Rapids). Add ZSTD_countShort(), which compares the first two words and picks the first differing one with selects; its only branch (>= 16 equal bytes) is taken ~15% of the time. Bit 63 is OR-ed in so the trailing-zero count is defined for a zero word without changing the result for a non-zero one. Only the row matchfinder uses it: applied to ZSTD_count() itself it slowed down the strategies with longer matches (dfast, btlazy2, btopt) by 2-6%. The returned length is identical, so output is unchanged.
Convert 8 sequences per iteration: vld4q_u16 de-interleaves offBase (low/high halves), litLength and mlBase into 16-bit lanes; codes for small lengths are computed by counting the LL_Code / ML_Code thresholds reached, and large ones / offsets from vclzq_u16. All selects are branchless; one narrowed 8-byte store per code table. Only enabled on little-endian 64-bit NEON targets. An SSE2 version was measured slower than the scalar loop on x86-64 (4 lanes, no vector lzcnt), so x86 keeps the scalar loop. Verified equal to the scalar loop for every 16-bit litLength / mlBase and all offBase bit widths.
On x86-64, the out-of-line row search costs callee-saved register spills, the mls/rowLog dispatch and re-deriving constants from ms on every call; perf attributed ~10% of level 6 cycles to that overhead. Inline only the noDict row search there. It was measured neutral on aarch64, so other targets, and the other dict modes, stay outlined to limit code size (+98 KB of .text on x86-64 with gcc 11).
rluvaton
marked this pull request as ready for review
October 13, 2026 17:36
Algunenano
added a commit
to Algunenano/ClickHouse
that referenced
this pull request
Oct 21, 2026
Drop the row matchfinder change, which has a big-endian bug and no gain at `ZSTD(3)`, and add the fix for the legacy v0.5 streaming regression. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 of 4 tasks
This was referenced Nov 4, 2026
zcsizmadia
added a commit
to zcsizmadia/zstd-native
that referenced
this pull request
Nov 5, 2026
* Row finder: mask out the head slot once, candidate count in a register (#61) Part of meta/zstd#4841 (commits 1-2), adapted: - `match_mask` clears slot 0 (the row head) before the rotation, so the candidate loops no longer test `match_pos == 0` on every tag hit (`ZSTD_row_maskOutHead`; here one `and`, since the mask has one bit per slot); - candidate slots are usize (no zero extension in the addressing); - the insertion after the search reuses the head already loaded for the mask (`ZSTD_row_nextIndexFromHead`) instead of reloading the tag row; - the match buffer's count is a local: as a field next to the run-time-indexed array it went through the stack on every push (a load, an increment and a store per candidate in the asm). The other part of 4841 commit 2, reading `hashSalt` once because tag stores may alias `*ms`, has no counterpart: the searcher is a `&mut` (noalias), and the asm shows no reloads of its fields in the loops. EPYC 7543, CPU 8; 3 copies per binary at different path lengths, interleaved in random order, median; geomean over the 12 Silesia files, speed vs main [worst file]: this commit control (main, string change) L5 1.005 [0.989 xml] 1.004 [0.998] L6 1.003 [0.988 xml] 1.003 [0.999] L7 1.009 [0.967 nci] 1.002 [0.997] L9 1.015 [0.994 nci] 1.001 [0.996] L11 1.039 [1.009] 0.997 [0.982] L12 1.042 [1.014] 1.000 [0.994] L15 0.990 [0.969 samba] 1.003 [0.994] source 16 KB frames L5/L9: 1.002 / 0.997 (control 1.003 / 0.999) nci L7 is bimodal by copy (two copies at 152, one at 158 MiB/s; main 157-159), with 3.1% fewer instructions; cycles at fixed work match main's fast mode (3.29G vs 3.30G). L15 (btlazy2) and the 16 KB frames (hash chain) run byte-identical code that only moved: samba L15 has the same instruction count and +1.9% cycles, with the btlazy2 search 32 bytes lower; the next commits' builds read +1.2% at L15 instead. Instructions at fixed work (perf stat): nci L5 -0.9%, nci L7 -3.1%, xml L12 -6.5%, mozilla L9 -5.5%. * Row finder: branchless count of the first 16 bytes (#61) meta/zstd#4841 commit 3: `count_short` (`ZSTD_countShort`) resolves a candidate's first 16 bytes with two 8-byte XORs and selects, and falls back to `count` beyond them, so the "first word equal?" branch of `count`, which mispredicts on the short and unpredictable lengths of the row finder's candidates, is gone. Same result as `count` (new test: every first difference and length around the 8- and 16-byte steps). Used only by the row finder's prefix candidates, like upstream: upstream measured it slower in dfast and the binary trees, whose matches are longer. EPYC 7543, CPU 8; 3 copies per binary, interleaved in random order, median; geomean over the 12 Silesia files vs main [worst file], cumulative with the previous commit (one sweep): this commit L5 1.020 [0.973 nci] L6 1.028 [1.012] L7 1.043 [1.018] L9 1.048 [1.024] L11 1.053 [1.005] L12 1.054 [1.020] L15 1.012 (samba, nci, xml, osdb; btlazy2 code unchanged) source 16 KB frames L5/L9: 0.999 / 0.988 (hash chain, code unchanged) The previous commit alone measured L5 1.005 to L12 1.042 in an earlier sweep. Cycles at fixed work (perf stat, median of 2 copies), main -> previous -> this: xml L12 2.077G -> 2.061G -> 2.040G, mozilla L9 3.643G -> 3.580G -> 3.536G. nci L5 is bimodal by copy (3.02-3.21G here, main 3.09-3.15G), the reason for its low single-file figure. * Lazy: inline the BMI2 noDict row search into the parser (#61) meta/zstd#4841 commit 5, which inlines the noDict row search on x86-64: the out-of-line call costs callee-saved spills and re-derives the search constants per position. Here only the BMI2 build with 16- and 32-entry rows (levels 5-10) is inlined: 64-entry rows keep the generic, non-inlined search, which measured faster than BMI2 there (#53). extDict and dictMatchState stay outlined, like upstream. .text +79 KB. EPYC 7543, CPU 8; 3 copies per binary, interleaved in random order, median; geomean over the 12 Silesia files vs the previous commit, same sweep [worst file]: L5 1.009 [0.981 webster] L9 1.003 [0.991] L6 1.018 [1.008] L11 1.002 [0.998] L7 1.016 [1.006] L12 1.000 [0.991] L15 1.000 (4 files) source 16 KB L5/L9: 0.995 / 1.003 Cycles at fixed work (perf stat, median of 2 copies), previous -> this: nci L5 3.12G -> 3.07G, nci L7 3.33G -> 3.29G, mozilla L9 3.536G -> 3.507G, xml L12 2.040G -> 2.029G. Tried and dropped: usize row slots in the row update loop as well (the rest of 4841 commit 2): one instruction and one stack load fewer per position, but 0.8-1.1% slower at L5-L9 (sao -2.4%), +0.3-0.6% at L11-12; with this commit on top too, mixed (L5 +1.8%, L6-L7 -1%). Not ported: commit 4 (NEON `seqToCodes`), aarch64 only.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Speeds up the row-based lazy matchfinder (
ZSTD_greedy/ZSTD_lazy/ZSTD_lazy2with the row hash, i.e. levels 5-12) by 2-8% on x86-64 and AArch64. Compressed output is byte-identical todevat every level. All changes were guided by profiling (perf, Arm SPE, Intel PEBS / top-down, llvm-mca).One commit per change:
matchPos == 0) on every tag hit; clear its bit group from the mask once instead.uxtwaddressing (an extra uop per access). Also readhashSaltonce (tag-tableBYTEstores may alias*ms) and reuse the head byte already loaded for the mask. llvm-mca (Neoverse-N1): insertion loop 33 → 30 uops per position.ZSTD_count's "first 8 bytes equal?" branch and loop exit are among the worst (57% of candidate lengths are 4-7 bytes, ~85% are < 16). NewZSTD_countShort()resolves the first 16 bytes with selects; it is used only by the row matchfinder: applied toZSTD_count()itself it slowed dfast / btlazy2 / btopt by 2-6% (levels 3, 13, 16), soZSTD_count()is unchanged.ZSTD_seqToCodeswith NEON: 8 sequences per iteration, branchless. NEON only: an SSE2 version measured ~1% slower than scalar on x86-64, so x86 keeps the scalar loop. Verified equal to the scalar loop for every 16-bit litLength/mlBase and every offBase bit width..texton x86-64 (gcc 11; inlining all search functions would be ~2x.textfor ~1% more, so it is not done).Results
One-shot
ZSTD_compressCCtx()onsilesia.tar(211,957,760 bytes), speed in MB/s of input. Median of 5 interleaved runs (binaries run in random order each round), each run = best of repeated compressions over 3 s, pinned to one core of an otherwise idle machine. Same ratio at every level (output is identical).devdevCode-layout noise: x86-64 results move by up to ~1.4% with function/loop alignment alone, so each kept change was also re-measured with
-falign-functions=64 -falign-loops=32(and a second alignment) on both sides; only changes that held at every alignment were kept.Setup
make -C lib libzstd.a(-O3), no-marchNot measured: other compilers (clang), other CPUs, data other than Silesia, streaming / dictionary modes (the extDict / dictMatchState paths share commits 1-3, but their speed was not benchmarked).
Verification
silesia.tarbyte-identical todevat levels 1-19, on both machines.tests/fuzzer -T60sandtests/zstreamtest -T60spass on both machines.-Wall -Wextra -Werrorfor arm64 and x86_64 (clang).Checklist
A gain in one configuration is not a general gain: see the Performance section of CONTRIBUTING.md.
Tools used
Claude Code (AI) was used to profile (perf, Arm SPE, Intel PEBS / top-down, llvm-mca), to write the code changes, and to run the benchmarks and tests. Every change was measured on the machines above, and changes that did not measure as faster were dropped.