Drive VecReduceBackend windows from a resumable merge-walk

[frankmcsherry]

VecReduceBackend presented its windows by walking the trace three times, once per bridge, from a key list it built by scanning every record of every novel batch first. This replaces that with one resumable merge-walk, and fixes two things the old presentation got wrong.

What changes

One identifier namespace for both input runs. MergedWalk walks the novel batches and the prior ones as a single list, so a value carries one identifier whichever run it appears in. Before, the two runs minted independently, so a value retracted by the novel batch could not cancel against its own history: the tactic saw a non-empty input, scheduled a crossing, and called user logic with an input that consolidated to nothing. The module doc described this and asked for "a more attentive identifier selection".

Prior records are advanced to the compaction frontier, and consolidated. ReduceInstance::lower was never read. Advancing alone buys nothing — it rewrites times and removes no records — so the two go together; separately, neither is worth doing. Consolidation also closes a latent double-count: two batches advanced to the same frontier but not yet merged can hold the same (hash, value, time), which the old presentation emitted twice, breaking the sorted-bridge invariant in debug and double-counting in release.

Windows are formed lazily. The active keys are the merge of the novel batch heads with changed, taken as the walk goes, so the whole-trace prescan is gone. The budget is spent in records rather than keys, since records are what the presentations cost, and it is checked at key boundaries so a key is never split across windows.

Collisions are settled when the window is formed. reduce_corrections used to resolve every identifier of every bracket on every wave to learn whether a hash carried more than one real key. The walk sees each value once, so the question is answered there for one comparison per value, and the bracket check becomes a lookup.

Performance: no measurable change

Under the shipped codegen-units = 4 this read as multimoment 21.2ms → 17.2ms, wide 31.3ms → 26.4ms, churn 27.6ms → 25.5ms. Rebuilt with one codegen unit, every ratio fell inside 0.98–1.02 with fully overlapping run ranges — the old code alone got ~19% faster while the new code barely moved. The apparent win was the old arrangement partitioning badly, not the new one doing less.

The in-binary control tactics do not catch this: the cursor and inherent reductions, untouched, moved at most 4.5% across those builds while the proxy moved 19%, because the partitioning damage is confined to the module that was rewritten. Only a one-unit rebuild separates them, so Cargo.toml now pins it; the full workspace release build is unaffected (3m07s against 3m10s), as fat LTO already dominates.

The walk does present 46% fewer records, but that buys no measurable time — which is itself worth knowing before optimising this backend for record volume.

Also here

ReduceTactic::retire and ReduceInstance::lower now state the invariant the advance depends on: every time a tactic evaluates at lies in [lower, upper), so accumulated input and output may be advanced to lower when loaded. It was true and stated nowhere, which left the backend resting on something a reader could not check.

The propagate benchmark reads its shape from the environment like the other programs, so it can be scaled to ask whether an effect grows with the work. That is how the codegen artifact was caught.

Tests

Two new: reduce_collision_multiwindow buckets several real keys into each of two hashes across several windows, exercising the per-window rebuild of the collision set; reduce_collision_inside_iterate does the same under Product times. Suppressing the collision flag fails all five collision tests.

One check is untested and deliberately kept: a hash whose input mentions only key A and whose output only key B collides while each pass alone is consistent, so the input pass leaves a representative for the output pass. Reaching that state needs a correction deferred past a retire's upper bound, which no test here constructs. It costs one comparison per key and what it guards is two real keys' values merging into one reduction.