A122: zstd message compression

[HosseinNejatiJavaremi]

A122: zstd message compression

gRFC for a first-class, cross-language zstd message codec (grpc-encoding: zstd). Per-message RFC 8878 frames. No dictionaries, skippable frames, or concatenated frames in the gRPC profile.

Blocked grpc/grpc#26460 and grpc/grpc#41813 pending this design.

Summary

• Wire name zstd (IANA / RFC 8878); magic 0xFD2FB528; Dictionary_ID 0
• Per-call grpc-encoding, per-message Compressed-Flag (existing model)
• Encode opt-in; decode+advertise once non-experimental
• C-core GRPC_COMPRESS_ZSTD; Python Compression.Zstd; Java optional artifact; Go encoding/zstd
• C-core env var GRPC_EXPERIMENTAL_ZSTD_COMPRESSION; Java/Go use artifact/import as the flag
• Not in scope: Brotli, Snappy, full-stream compression, C-core BYO compressor API, level remapping

C-core, Java, and Go language owners should review.

[filipsajdak]

Two things that may help move this along: a process note, and some measured data
for the ratio/CPU argument.

The comment period may not have started

The header still reads * Discussion at: (filled after thread exists). Per the
process, the owner sends a notification to
grpc-io and updates the PR with the discussion link, and the >=10-business-day comment
period runs from there. If that thread has not been created yet, that may be why this has
been quiet rather than any objection to the design.

Measured data from a production system

The Rationale currently argues the ratio/CPU tradeoff qualitatively ("gzip is CPU-heavy
relative to the ratio it delivers", "zstd typically dominates ratio at similar CPU"). We had
the same question for our own deployment, so here are numbers from real traffic. These are
protobuf SerializeToString() bytes off a live server, not JSON and not synthetic —
four unary responses from a C++ gRPC service:

payload	bytes	gzip-6	zstd-3	gzip MiB/s	zstd MiB/s
small list response	177	0.92x	0.95x	23	35
single run record	1,039	3.12x	3.05x	81	155
schema/contract response	4,843	3.33x	3.21x	117	336
audit log page	17,216	4.04x	4.20x	121	558

Three observations, including one that cuts against the current wording:

1. On protobuf, ratio is a wash. zstd-3 and gzip-6 are within ~4% of each other, each
   winning two of the four. "zstd typically dominates ratio at similar CPU" may be worth
   softening for the protobuf case — the ratio claim is easy for a reviewer to falsify, and
   the proposal does not need it.
2. The CPU claim holds, and it is the strong one. zstd-3 is 1.9x-4.6x faster at
   equivalent ratio, and the gap widens with message size. That supports the thesis on its
   own; it is the argument worth leading with.
3. The smallest message expands under both codecs (177 B -> 192 B gzip, 186 B zstd).
   Good that §"Negotiation and errors" already says implementations MAY skip compressing a
   message and send Compressed-Flag 0 — worth keeping prominent, since the Rationale's own
   framing ("gRPC compresses many small messages") is exactly the regime where a naive
   always-compress implementation makes payloads bigger.

Caveats, so this is weighed correctly: Python bindings over C zlib and C libzstd rather than
gRPC C-core, so treat the throughput as relative rather than absolute; a small corpus of four
payloads from one application on one machine (Apple Silicon); and we are an interested party,
since we would like zstd on the wire for our own product.

Happy to re-run this on a larger corpus, on aarch64 Linux, or against C-core directly if that
would be useful for the Rationale. Note there is also a closed C-core implementation in
grpc/grpc#41813 (+160/-1, with third_party/zstd, Bazel and CMake wiring already done) that
was closed pending exactly this gRFC.