Repository navigation
fix(cost): charge KMV builds and merges their measured time (build 110 + 0.62, merge 2.6 + 0.43 per compared hash) - #1375
Merged
Merged
Conversation
zzylol
force-pushed
the
fix/kmv-build-cost-refit
branch
from
November 16, 2026 22:48
c9f874c to
8632711
Compare
zzylol
force-pushed
the
fix/delta-split-over-dependent-readouts
branch
from
November 16, 2026 23:48
50cf197 to
793859e
Compare
zzylol
force-pushed
the
fix/delta-split-over-dependent-readouts
branch
from
November 17, 2026 16:58
793859e to
386ea88
Compare
zzylol
force-pushed
the
fix/kmv-build-cost-refit
branch
from
November 17, 2026 18:43
f9c1da7 to
598f8d4
Compare
zzylol
changed the base branch from
fix/delta-split-over-dependent-readouts
to
main
November 17, 2026 18:43
…0 + 0.62, merge 2.6 + 0.43 per compared hash) The KMV build is refit on 132 executor builds: 110 operations per row and 0.62 per compared hash (was 65 and 0.5). The KMV merge is refit on 45 executor merges: 2.6 per hash and 0.43 per compared hash, with the states of a group sharing the smallest hashes of the column's values. StateInputRows carries the folded column's distinct values for that; a merge of inputs keeps the largest. CostConstant renames KmvOperationsPerComparedHash to KmvUpdateOperationsPerComparedHash and gains KmvMergeOperationsPerHash and KmvMergeOperationsPerComparedHash; merge_terms_per_state charges a KMV merge in those terms. One expected_distinct_values helper replaces the three copies of the formula. The constants describe the executor's pinned asap_sketchlib KMV, whose duplicate check walks every retained hash; they are measured again once the library's check is O(log k).
zzylol
force-pushed
the
fix/kmv-build-cost-refit
branch
from
November 17, 2026 18:45
598f8d4 to
55931e5
Compare
…istinct values; review doc fixes A merge of KMV states built at ingestion (rows per state unknown) took k hashes per state, every one distinct; with the column's distinct values declared it now takes the fewer of k and those values, shared by every state. Docs: distinct_values is the fewer of the declared values and the per-group bound; the merge fit's extra terms bring the error to 0.046 but fit a negative constant; the summation-order example names 0.1 x 7.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1373. Rebased onto main after the δ split merged (its k = 4,323 sizing exposes these charges) and ported onto
NodeCostTerms: each KMV constant is aCostConstant.Summary.
StateInputRowsnow carries the folded column's distinct values for that.Projectorder against main (listed below).Review guide.
crates/cost/src/node/operations_per_unit.rs. It holds the four KMV constants,update_terms_per_rowandmerge_terms_per_state; then theirCostConstantvariants incrates/cost/src/node/cost_terms.rs.StateInputRows::distinct_valuesincrates/cost/src/node/state_bytes/mod.rs, and wherenode_usage_and_cost_termssets it (crates/cost/src/node/usage/mod.rs).crates/cost/src/node/usage/tests.rs:a_kmv_build_charges_its_measured_time_per_row(new);a_kmv_build_costs_per_retained_hash_it_compares(re-measured);a_kmv_merge_charges_its_measured_time_per_state(new);a_kmv_merge_of_states_with_unknown_rows_retains_at_most_the_column_s_values(new);a_kmv_rollup_merge_costs_per_hash_it_inserts(re-measured).crates/integration-tests/tests/rollup_controlled.rs.55931e5bathe build and merge refits with their tests, ported onto main'sNodeCostTerms(the original test-first commits are onbackup/fix/kmv-build-cost-refit-f9c1da7de).471262aa3review fixes: a KMV state of unknown rows retains at most its column's distinct values; doc fixes.Why
The planner should pick the plan that measures faster. In the viewer's distinct rollup (
rollupheavy-t11:COUNT(DISTINCT user_id)by service and by (service, endpoint), 5 min every 60 s, 1,000,000 users), measured whole plans flip with the rows per state:Each plan has to be charged its measured cost for the choice to follow that crossover. The KMV build was the term furthest off. Once it is fixed, 56 grid plans move work into merges and windows, so the merge has to be charged its measured cost too.
About the 12,000-rows-per-state case. It was reported as picking KMV on the base. That report came from a probe that changed only
arrival.rows_per_second; the planner prices a scan fromseries_count×sample_interval, so the probe still planned 600,000,000 rows. Planned at 12,000 rows per state (10,000 series sampled every 25 ms), the base already picks exact. The new testproduction_distinct_rollup_counts_exactly_at_12k_rows_per_statetherefore passes on the base too.What
StateInputRowsgainsdistinct_values: the distinct values of the column the state folds. The build sets it from the column's declared distinct values. The build charge now reads it there instead of computing it a second time. Merging keeps the largest.How
The executor's KMV is
asap_sketchlib::Kmv. Itsupdate_with_hashwalks every retained hash to find a duplicate, unless the state is full and the hash is above its largest. So an update is O(k) while the state fills and for each hash that enters it (about k·ln(n/k) of them once full). A merge updates the group's state with every hash the input retains, so it walks the same way. This is a library defect: a membership check could be O(1) or O(log k), and a merge could be an O(k) sorted merge instead of O(k²). That fix belongs inasap_sketchlibas its own PR, pinned here afterwards, and the KMV constants must be measured again after it. This PR describes the executor as it is today.kmv_compared_hashes_per_rowalready counts that walk: k × the fill fraction while filling, then the walks of the repeats and of the hashes that enter. The build fit keeps it as the size term.The merge passes it the hashes the states share:
Vvalues, a state that foldedDdistinct ones and retainshhashes draws them from theV·h/Dsmallest.Example: 1-minute states of (service, endpoint) at k = 4,323, each of 12,000 rows of 1,000,000 users. Each retains 4,323 hashes of about 11,930 users, drawn from the 362,000 smallest.
Key code interfaces added / modified
The constants in
crates/cost/src/node/operations_per_unit.rsand their names incrates/cost/src/node/cost_terms.rs:StateInputRows(public,crates/cost/src/node/state_bytes/mod.rs) and the crate-private merge charge:pub struct StateInputRows { pub rows_per_state: Option<u64>, pub non_negative: bool, pub value_magnitude_range: Option<ValueMagnitudeRange>, + /// The distinct values the folded column takes in one state: the fewer of the column's declared + /// distinct values and the sources' bound per group; `None` when neither is declared, which + /// takes every row's value as new. Merging inputs keeps the largest. E.g. `Some(1_000_000)`. + pub distinct_values: Option<u64>, } pub(crate) fn merge_terms_per_state( family: StateFamily<'_>, state_bytes: u64, merged_state_bytes: u64, - rows_per_input_state: Option<u64>, + input_rows: Option<StateInputRows>, input_states_per_group: u64, ) -> NodeCostTermsUsage, end to end:
Major algorithm changes
Cost model: the KMV build.
Problem: at 400 states × 60,000 rows of 1,000,000 values at k = 4,323, the build was charged 553 ns per row and measures 712. Every one of the 132 grid builds was under-charged, by 1.17–1.63x.
Before:
65 + 0.5 × comparedper row, fitted on 28 builds of 300,000 rows.After:
110 + 0.62 × comparedper row.How it was fitted: same method as the exact distinct refit.
SummaryAggby (service, endpoint): the build's own CPU (its time minus its input's), median of 3 runs on one pinned core, 10 rows per group per batch.Held-out shapes, not in the grid, each the median of 3 passes:
RMS log error is 0.094 (was 0.322). The first two rows are the shapes in
a_kmv_build_costs_per_retained_hash_it_compares. Their earlier measurements (191 and 7,410) do not reproduce on today's executor (329 and 8,357), so the test now quotes these.Cost model: the KMV merge.
Problem: the merge was charged 65 per hash plus 0.5 per compared hash, taking every merged hash as distinct. Measured, merges cost 0.21–1.10 of that charge on the grid and 0.33–1.58 on held-out rollups. The error grows with the states per group, and it reverses when states share many values.
Before:
retained × (65 + 0.5 × kmv_compared_hashes_per_row(k, retained × states, None))per input state.After:
retained × (2.6 + 0.43 × kmv_compared_hashes_per_row(k, retained × states, Some(shared)))per input state:retainedis the hashes a state keeps, at mostk: the distinct values among its rows.sharedis theV·h/Dsmallest hashes the states draw theirs from (see How).How it was measured:
SummaryMergeover KMV states stored as a materialized view: a 1-minute tumbling-window chain by (service, endpoint) over 5 minutes, read through its time range.ksmallest hashes of their rows'user_ids, hashed as the executor hashes them, inserted in random order.Grid: k = 1,081, 4,323, 26,656 and 106,263 × window states of 1,200, 12,000 and 60,000 rows of 1,000,000 users × 5, 10, 50 and 200 states per group. That is 45 merges; states per group times state size was capped at a few minutes of CPU.
Forms compared by leave-one-setting-out error, where a setting is (k, rows per state, states per group):
Held-out rollups, not in the grid: states merged from 5 windows (6,000 to 300,000 rows each), then rolled up 10 or 100 per group. Each rollup is its node's time less the window merge below it, median of 3 runs. ns per input state, in ms:
This is 12 of the 24. Over all 24, measured ÷ charged is 0.84–1.14 and the RMS log error is 0.091 (was 0.33–1.58 and 0.436).
k = 106,263, 10 × 11,929row is the rollup ina_kmv_rollup_merge_costs_per_hash_it_inserts. It now quotes 299 s for its 1,000 states (median of 3 on an idle core), not the earlier 352 s (a plan's time less its build's).One state per group. A merge whose groups each read one state passes it through. Measured that way, it costs 2–10 µs per state at k = 1,081 and 106,263 (1,000 states). The cost model sends a merge that keeps its input's groups down the pass-through path. A rollup whose estimates tie its input's (10,000 (service, endpoint) states into 10,000 services) still reaches this charge with one state per group. It is then charged as one state filled from empty: 3.9 ms per state at k = 4,323, the most any number of states per group costs. That rule predates this PR and is unchanged (see Limitations).
States of unknown rows. A state built at ingestion has no known rows per state. With the column's distinct values
Vdeclared, it retainsmin(k, V)hashes, the same ones in every state (each folded allV); without them,kdistinct hashes as before. Example: 10 states per group of 500 users at k = 4,323 are charged as 500 shared hashes each, not 4,323 distinct ones (about 9x less).Correctness: the charge stays O(1) per node (the existing 64-slice average). No search or MILP change.
Before this PR
On the base, the viewer rollup at 600,000,000 rows per evaluation is charged 394.0 s per evaluation for a query-time KMV build (331.9 s of it the build), and 589.8 s for exact. The build measures about 1.3x its charge.
After this PR
The planner selects the KMV built in 1-minute tumbling windows at ingestion, then merged over the 5 windows at each evaluation. The three plans are charged:
At the build-only refit (
68a1efa0f) they were charged 453, 491 and 590.Measured end to end at a scaled shape with the same rows per state:
The planner's choice is the measured fastest. Measured, the window plan is 1.26x faster than the query-time build and 1.8–2.0x faster than exact, with either data mapping. Scaled ×10 to the planned 10,000 series, that is about 313 s, 395 s and 571–629 s per evaluation, against 401, 481 and 590 charged.
Evidence
Plan survey, main (
7cfd274b5) against this PR (471262aa3), throughrun_search_overwith defaultPlanSearchSettings, with the default and the per-resource (UsageOverTime) cost inputs. All 422 plan on both:fix(cost): charge sketch states and merges, raw batch visits, grouping keys, distinct counts and KMV at their measured costs, so the measured-best plan is selected #1363's sweep (202 workloads): the grid of 120, showcase 24, held-out 48, 2,000-series 10. All 202 plans are identical node for node, and no cost changes. None of them contains a distinct count.
Distinct-count grid (220 plans):
production_rollup_workloads), plus an ungroupedCOUNT(DISTINCT user_id).rollupheavy-t11and the planner-layering example 2.Project-order changes, and the merge refit's changes below. Typical: the SQL production shape's two distinct counts (by service and by (service, endpoint)) move from two KMV builds (k = 106,263 and 4,323) to one KMV build by (service, endpoint) merged to service; the rollup's query-time build moves to 1-minute tumbling windows merged over the range.The merge refit, against the build refit alone (measured on the earlier base): 29 plans change.
Projectheld for the evaluationProjectstreamed (same operators, same cost)Verification
Run on a 40-core build host, PR
471262aa3against main7cfd274b5:asap-costcargo nextest run -p asap-costasap-integration-tests+asap-devtoolscargo nextest run -p asap-integration-tests -p asap-devtools --no-fail-fastcargo test --release -p asap-devtools --test corpus_snapshots -- --ignoredcargo fmt --all --checkcargo clippy -p asap-cost -p asap-integration-tests --all-targets -- -D warningsjoint_search_equivalence×2,plan_sharing×2,promql_to_post_asap::promql_quantile_of_rate_binds_kll_over_rate_accumulator,materialized_views::materialized_views_equal_the_stored_outputs_main_listed,planner_fuzz×3.New and changed tests:
a_kmv_build_charges_its_measured_time_per_row(new): six measured builds (10 and 1,000 states, 120,000–12,000,000 rows, k = 1,081–106,263) are each charged within 0.8–1.25 of their time. It fails on the base (109 charged vs 176 measured at k = 1,081).a_kmv_build_costs_per_retained_hash_it_compares: now quotes the re-measured 329 and 8,357 ns per row. It fails on the base (222 vs 329).a_kmv_merge_charges_its_measured_time_per_state(new): six measured merges (5 to 200 states per group, k = 1,081–106,263, one of 20,000 users) are each charged within 0.8–1.25 of their time. It fails on the build refit (4,223,195 charged vs 3,115,412 measured for 5 window states at k = 4,323; 88,151 vs 18,109 for 200 states at k = 1,081).a_kmv_rollup_merge_costs_per_hash_it_inserts: now quotes the re-measured 299 s, within 0.8–1.25 (was 352 s, within 0.75–1.33).production_distinct_rollup_counts_exactly_at_12k_rows_per_state(new): at a sample every 25 ms, both queries count exactly. It passes on base and PR (see Why).a_kmv_merge_of_states_with_unknown_rows_retains_at_most_the_column_s_values(new): a merge of states with unknown rows and 500 declared users is charged as states that folded all 500.production_distinct_rollup_builds_kmv_at_60k_rows_per_state(changed): the shape is now set bysample_interval, and a KMV built in tumbling windows also counts as a KMV build.Design scope
StateInputRows::distinct_values(cost, the description of what one summary state folds). It is the folded column's distinct values per state. Any summary whose cost depends on repeated values can read it, not only KMV.CostConstant::{KmvUpdateOperationsPerComparedHash, KmvMergeOperationsPerHash, KmvMergeOperationsPerComparedHash}(cost, the named per-unit constants): one per constant, as for every other.Architectural decisions
Limitations and follow-up
asap_sketchlibKMV, whose duplicate check walks every retained hash. KMV: pin the O(log k) sketchlib fix and recharge KMV builds, merges and memory at its measured cost #1394 pins the O(log k) fix and refits the build and merge; its constants replace these.distinct_valuesis the column's declared distinct values. When a workload declares none, the merge takes every hash as distinct. That charges a merge of states that share many values at about 0.36–0.68 of its time.Human review — do not complete with an agent