Streamline the proxy reduce tactic - #827
Closed
frankmcsherry wants to merge 2 commits into
Closed
Conversation
The tactic now works directly from the presented windows: per key bracket it lifts the join partners by the seed meet and re-sorts the bracket to time-major order in place, with suffix meets in shared columns, replacing the per-key EditList/ValueHistory copies. Pending times flatten from BTreeMaps into sorted (key, time) columns consumed by one ascending cursor per retire. Novel times enter the one reached set, matching round_coverage's single novel witness set. Retire scratch pools across retires in the tactic, which names the backend's diff types and so re-parameterizes the tactic over (B1, B2, Bk). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The proxy reduce tactic re-derived each key's working state from scratch: every presented record was cloned into an
EditList, exploded into a per-edit history, and re-sorted — per key, per window; pending times lived inBTreeMap<u64, Vec<T>>s rebuilt per retire; and each key slot carried ~19 heap vectors. This PR reworks the tactic to work directly from the presented windows, with each piece of state pooled at its natural lifetime.next_windowthe harness owns the presentation: per key bracket it lifts the history and output times by the seed meet and re-sorts the bracket to(time, id)order, with suffix meets in shared columns (prepare). The lift is preserved exactly — losing it turns linear sweeps quadratic; see the comment at the call site.EditList/ValueHistoryleave the proxy path entirely, and a key slot shrinks to the six vectors that are genuinely wave-persistent (accumulations,reached,synth,produced). The backend's requirements are unchanged;ReduceWindownow documents that presented records belong to the harness afterward.(key, time)columns: the due/carried split is one scan,changeda dedup of the due keys, due lookup a single ascending cursor per retire, and the next pending set accumulates flat with one sort at retire end. The three BTreeMaps are gone.reachedas they step in, soabsorbtests andclosejoins against the single novel set offormal/Differential/RoundCoverage.lean'sround_coverage, rather than a split acrossreachedand the novel buffer.Scratchheld by the tactic, cleared at its boundary (retire/window/wave/tick) rather than reallocated — profiling showed per-retire setup was ~30% of the tactic's cost on iterative workloads, whose retires are many and small. Naming the backend's diff types re-parameterizes the tactic over(B1, B2, Bk); every construction site isProxyReduceTactic::new(backend)with inference, so no caller changes.ProxyReduceBackend::emitnow documents the guarantee its calls already carried — per tile, disjoint ascending key ranges, each consolidated — which licenses backends to build output incrementally instead of staging rows.history.rsfolds intojoin.rs, and the bench's propagate round count becomes env-configurable (PROP_ROUNDS), which profiling wanted.Measured on
tests/int_proxy_bench(defaults): churn 21.7 → 19.8ms/round (0.47x cursor-same), multimoment 16.2 → 14.6ms (0.90x — the proxy tactic now beats the cursor tactic on identical storage), propagate 2.53 → 2.37ms (1.04x). A scale sweep to the GB range (4M-key churn, 1M-key multimoment, 1M-key string-valued wide) shows the ratios and per-key costs stable across two orders of magnitude, and — measured with interleaved fresh-process repeats — memory parity with the cursor tactic (churn 4M: 3756 vs 3703MB peak RSS).🤖 Generated with Claude Code