Route zstd consumers through the compression-rs autogated router

[guybedford]

This extends the compression-rs autogated router introduced in #7132 to zstd.

As posted, this should not land, due to the 3x slowdown. But updating this PR as Rust zstd improves can allow us to track when this might become viable in future.

The unprefixed ZSTD_* symbol names consumed in the build (the streaming compression/decompression context entry points and the error helpers) are now owned by a compiled routing layer, src/workerd/util/zstd-router.c++, which forwards each call to one of the two implementations linked into the binary, selected by the same process-global atomic set from the compression-rs autogate as the zlib router. This covers node:zlib's zstd modes universally (the only zstd consumer in the build).

No credible pure-Rust zstd encoder exists today (ruzstd's encoder only supports the fastest strategy, with no level selection, streaming flush control, or dictionaries), so only decompression is routed: the gate switches ZSTD_decompressStream and the DCtx entry points to ruzstd, while compression stays on the C implementation on both sides of the gate. The error helpers also stay on the C implementation; the Rust side returns errors in the standard (size_t)-ZSTD_ErrorCode encoding so they interpret results from either implementation.

• The C implementation is rebuilt with its routed entry points renamed to zstd_c_* as @zstd//:zstd_impl, via a single_version_override patch to the BCR zstd module's BUILD (mirroring how build/BUILD.zlib does zlib_impl); @zstd//:zstd becomes the standard headers over the routing layer. One source-line guard is needed in zstd_common.c, which #undefs ZSTD_isError.
• The new src/rust/zstd-rs crate implements the ZSTD_decompressStream contract over ruzstd's FrameDecoder: push-style input buffering with bounded decode steps, frame-boundary and skippable-frame handling, ZSTD_d_windowLogMax enforcement, content checksum verification, and libzstd's hostage-byte protocol at frame end so node:zlib's availIn/availOut bookkeeping and truncation detection behave identically on both sides of the gate.
• The gate is propagated from the existing sync point in autogate.c++.
• A new kj_test verifies the router selects the matching implementation on both sides of the gate (observed via the experimental ZSTD_d_format parameter, which only the C implementation accepts) and that both implementations decode each other's output, including checksummed frames and corruption errors.
• A bench-zstd benchmark measures the node:zlib streaming workload (1MB compressible data, 16KB chunks) against both decoders.

Benchmark results:

Benchmark	Throughput
Zstd::Compress::Native (C, both sides of gate)	472 MiB/s
Zstd::Decompress::Native (C, gate off)	1.38 GiB/s
Zstd::Decompress::ZstdRs (ruzstd, gate on)	485 MiB/s (~2.9x slower)

Compression output is gate-independent (always the C encoder); representative ratios with decode verified byte-exact through both decoders:

Input (1MB)	L1	L3 (default)	L19
repetitive text	163 B	162 B	148 B
JSON records	14.11%	13.79%	11.49%
random	100%	100%	100%
zeros	51 B	50 B	49 B

Test coverage: the new router kj_test, zlib-zstd-nodejs-test across all variants (@all-autogates exercises the ruzstd path end-to-end, including streaming, truncation, and maxOutputLength cases), and the full //src/workerd/... + //src/rust/... suites.

[ask-bonk]

@guybedford Bonk workflow was cancelled.

View workflow run · To retry, trigger Bonk again.

[fhanau]

Zstd::Decompress::Native (C, gate off) | 1.38 GiB/s
Zstd::Decompress::ZstdRs (ruzstd, gate on) | 485 MiB/s (~2.9x slower)

The PR for brotli explicitly asks whether a slowdown is acceptable, but this is not being done here – people often choose zstd because of its high performance, IMHO a 3x slowdown on a compression primitive is a good reason to reconsider replacing a well-maintained C library with Rust. At least, there should be some discussion about this to consider this beforehand.

[guybedford]

The PR for brotli explicitly asks whether a slowdown is acceptable, but this is not being done here

Consider this one posted with the exact same stance as that one, I've added a clarification to the PR description to note that it is not going to land.