[comphead]

Which issue does this PR close?

• Closes Spill files are mostly Utf8 offsets, which zstd and lz4 barely compress #26627.

Rationale for this change

Compressed spill files are mostly string offsets, which zstd and lz4 barely compress, see #26627. Spilling is also slower than it needs to be: a batch of the table in #26627 takes about 35 write calls, one per IPC buffer.

This replaces #26698, which needed spill reads to skip validation (#26688).

What changes are included in this PR?

• With lz4_frame or zstd, each top level Utf8, LargeUtf8, Binary and LargeBinary column is spilled as a struct of its value lengths and of its value bytes as one run-end encoded value. Both are valid arrays, so the IPC reader decodes them as usual before the spill reader rebuilds and validates the original arrays. Uncompressed spills keep their layout.
• FileSpillWriter writes through a 256 KB BufWriter.

Benchmarks

Apple M5 Max, interleaved runs against main.

The reproducer from #26627 (datafusion-cli -m 100m, median of 7):

codec	main	this PR
uncompressed	217 ms, 171.7 MB spilled	207 ms, 171.7 MB spilled
lz4_frame	263 ms, 54.0 MB spilled	251 ms, 17.1 MB spilled
zstd	320 ms, 32.0 MB spilled	270 ms, 8.9 MB spilled

sort_tpch queries at SF1 on Utf8 columns (one partition, -m 200m, median of 5), spilled bytes and time against main:

query	uncompressed	lz4_frame	zstd
Q3	-0.4%	92.4 → 78.1 MB, +1.4%	61.4 → 50.5 MB, +0.3%
Q7	-3.6%	306.3 → 230.2 MB, -2.3%	171.3 → 122.6 MB, -5.5%
Q10	-2.7%	398.9 → 309.7 MB, -2.0%	228.2 → 169.2 MB, -3.2%

In the spill_io benchmark, writing and reading spill files is 22-27% faster uncompressed, 4-9% faster with zstd and unchanged to 10% faster with lz4, and the string datasets spill 27-41% smaller.

spill_io results and setup

cargo bench -p datafusion-physical-plan --bench spill_io, mean of two alternating runs per build.

benchmark	main	this PR	spilled, main → this PR
q2/uncompressed	4.33 ms	3.23 ms (-25.3%)	unchanged
q2/zstd	17.40 ms	16.18 ms (-7.0%)	unchanged
q2/lz4_frame	11.75 ms	10.60 ms (-9.8%)	unchanged
q16/uncompressed	10.36 ms	7.85 ms (-24.2%)	unchanged
q16/zstd	41.67 ms	37.81 ms (-9.3%)	4.4 MB → 2.6 MB
q16/lz4_frame	23.52 ms	23.58 ms (+0.2%)	7.8 MB → 5.2 MB
q20/uncompressed	6.13 ms	4.46 ms (-27.3%)	unchanged
q20/zstd	26.94 ms	25.74 ms (-4.4%)	unchanged
q20/lz4_frame	16.91 ms	15.24 ms (-9.9%)	unchanged
wide/uncompressed	25.50 ms	19.91 ms (-21.9%)	unchanged
wide/zstd	104.28 ms	96.08 ms (-7.9%)	11.6 MB → 8.2 MB
wide/lz4_frame	59.34 ms	56.77 ms (-4.3%)	19.9 MB → 14.6 MB

The sort_tpch runs set:

SET datafusion.execution.target_partitions = 1;
SET datafusion.execution.parquet.schema_force_view_types = false;
SET datafusion.execution.spill_compression = 'zstd'; -- or 'lz4_frame', 'uncompressed'

What is the testing strategy for this PR?

• aggregate_memory_spill.slt spills Utf8, LargeUtf8, Binary and LargeBinary group keys with nulls under zstd and lz4_frame and checks their values.
• test_spill_lengths_round_trip covers sliced and empty batches, and test_spill_lengths_compress checks that a zstd spill of similar strings is under a quarter of plain zstd IPC.
• The existing spill_pool tests read files while they are being written, which now relies on the flush after each batch.

Are there any user-facing changes?

Smaller compressed spill files and faster spilling. No configuration or API changes.