Skip to content

fix(cost): charge KMV builds and merges their measured time (build 110 + 0.62, merge 2.6 + 0.43 per compared hash) - #1375

Merged
zzylol merged 2 commits into
mainfrom
fix/kmv-build-cost-refit
Nov 17, 2026
Merged

zzylol merged 2 commits into
mainfrom
fix/kmv-build-cost-refit

Conversation

@zzylol

@zzylol zzylol commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1373. Rebased onto main after the δ split merged (its k = 4,323 sizing exposes these charges) and ported onto NodeCostTerms: each KMV constant is a CostConstant.

Summary.

  • Problem:
    • The KMV build was charged 0.61–0.85 of its measured time. At the viewer rollup's shape (k = 4,323, 60,000 rows per state), it was charged 553 ns per row and measures 712.
    • The KMV merge was charged 0.9x to 4.9x its measured time, and more the more states a group merges. It took every merged hash as distinct, while merged states share hashes.
  • Change:
    • The build is refit on 132 executor builds: 110 per row plus 0.62 per compared hash (was 65 and 0.5).
    • The merge is refit on 45 executor merges: 2.6 per hash plus 0.43 per compared hash (was 65 and 0.5), with the states sharing the smallest hashes of the column's values. StateInputRows now carries the folded column's distinct values for that.
  • Effect:

Review guide.

  • Read first: crates/cost/src/node/operations_per_unit.rs. It holds the four KMV constants, update_terms_per_row and merge_terms_per_state; then their CostConstant variants in crates/cost/src/node/cost_terms.rs.
  • Then StateInputRows::distinct_values in crates/cost/src/node/state_bytes/mod.rs, and where node_usage_and_cost_terms sets it (crates/cost/src/node/usage/mod.rs).
  • Then the tests:
    • crates/cost/src/node/usage/tests.rs:
      • a_kmv_build_charges_its_measured_time_per_row (new);
      • a_kmv_build_costs_per_retained_hash_it_compares (re-measured);
      • a_kmv_merge_charges_its_measured_time_per_state (new);
      • a_kmv_merge_of_states_with_unknown_rows_retains_at_most_the_column_s_values (new);
      • a_kmv_rollup_merge_costs_per_hash_it_inserts (re-measured).
    • crates/integration-tests/tests/rollup_controlled.rs.
  • Commits:
    • 55931e5ba the build and merge refits with their tests, ported onto main's NodeCostTerms (the original test-first commits are on backup/fix/kmv-build-cost-refit-f9c1da7de).
    • 471262aa3 review fixes: a KMV state of unknown rows retains at most its column's distinct values; doc fixes.

Why

The planner should pick the plan that measures faster. In the viewer's distinct rollup (rollupheavy-t11: COUNT(DISTINCT user_id) by service and by (service, endpoint), 5 min every 60 s, 1,000,000 users), measured whole plans flip with the rows per state:

rows per state exact KMV (k = 4,323)
60,000 609.6 s 483.6 s (KMV 1.26x faster)
12,000 127.9 s 178.4 s (exact 1.39x faster)

Each plan has to be charged its measured cost for the choice to follow that crossover. The KMV build was the term furthest off. Once it is fixed, 56 grid plans move work into merges and windows, so the merge has to be charged its measured cost too.

About the 12,000-rows-per-state case. It was reported as picking KMV on the base. That report came from a probe that changed only arrival.rows_per_second; the planner prices a scan from series_count × sample_interval, so the probe still planned 600,000,000 rows. Planned at 12,000 rows per state (10,000 series sampled every 25 ms), the base already picks exact. The new test production_distinct_rollup_counts_exactly_at_12k_rows_per_state therefore passes on the base too.

What

  • The KMV build's per-row and per-compared-hash constants are refit.
  • The KMV merge's per-hash and per-compared-hash constants are refit, and its compared hashes count the hashes its states share.
  • StateInputRows gains distinct_values: the distinct values of the column the state folds. The build sets it from the column's declared distinct values. The build charge now reads it there instead of computing it a second time. Merging keeps the largest.

How

The executor's KMV is asap_sketchlib::Kmv. Its update_with_hash walks every retained hash to find a duplicate, unless the state is full and the hash is above its largest. So an update is O(k) while the state fills and for each hash that enters it (about k·ln(n/k) of them once full). A merge updates the group's state with every hash the input retains, so it walks the same way. This is a library defect: a membership check could be O(1) or O(log k), and a merge could be an O(k) sorted merge instead of O(k²). That fix belongs in asap_sketchlib as its own PR, pinned here afterwards, and the KMV constants must be measured again after it. This PR describes the executor as it is today.

kmv_compared_hashes_per_row already counts that walk: k × the fill fraction while filling, then the walks of the repeats and of the hashes that enter. The build fit keeps it as the size term.

The merge passes it the hashes the states share:

  • A state retains the smallest hashes of the values it folded.
  • So all states of a group draw their hashes from the same smallest hashes of the column's values.
  • Of V values, a state that folded D distinct ones and retains h hashes draws them from the V·h/D smallest.
  • A shared hash is walked until it is found, not added again.

Example: 1-minute states of (service, endpoint) at k = 4,323, each of 12,000 rows of 1,000,000 users. Each retains 4,323 hashes of about 11,930 users, drawn from the 362,000 smallest.

Key code interfaces added / modified

The constants in crates/cost/src/node/operations_per_unit.rs and their names in crates/cost/src/node/cost_terms.rs:

-pub(crate) const KMV_UPDATE_OPERATIONS_PER_ROW: f64 = 65.0;
+pub(crate) const KMV_UPDATE_OPERATIONS_PER_ROW: f64 = 110.0;
-pub(crate) const KMV_OPERATIONS_PER_COMPARED_HASH: f64 = 0.5;
+pub(crate) const KMV_UPDATE_OPERATIONS_PER_COMPARED_HASH: f64 = 0.62;
+/// Per hash a KMV merge reads from an input state, beyond the retained hashes it compares it with.
+pub(crate) const KMV_MERGE_OPERATIONS_PER_HASH: f64 = 2.6;
+/// Per retained hash a KMV merge compares an inserted hash with.
+pub(crate) const KMV_MERGE_OPERATIONS_PER_COMPARED_HASH: f64 = 0.43;

 pub enum CostConstant {
     ...
     KmvUpdateOperationsPerRow,
-    KmvOperationsPerComparedHash,
+    KmvUpdateOperationsPerComparedHash,
+    KmvMergeOperationsPerHash,
+    KmvMergeOperationsPerComparedHash,
     ...
 }

StateInputRows (public, crates/cost/src/node/state_bytes/mod.rs) and the crate-private merge charge:

 pub struct StateInputRows {
     pub rows_per_state: Option<u64>,
     pub non_negative: bool,
     pub value_magnitude_range: Option<ValueMagnitudeRange>,
+    /// The distinct values the folded column takes in one state: the fewer of the column's declared
+    /// distinct values and the sources' bound per group; `None` when neither is declared, which
+    /// takes every row's value as new. Merging inputs keeps the largest. E.g. `Some(1_000_000)`.
+    pub distinct_values: Option<u64>,
 }

 pub(crate) fn merge_terms_per_state(
     family: StateFamily<'_>,
     state_bytes: u64,
     merged_state_bytes: u64,
-    rows_per_input_state: Option<u64>,
+    input_rows: Option<StateInputRows>,
     input_states_per_group: u64,
 ) -> NodeCostTerms

Usage, end to end:

// The viewer's rollup workload: two COUNT(DISTINCT user_id) queries, 5% relative error per
// value, over 10,000 series sampled every 5 ms (600,000,000 rows per evaluation).
let (workload, roots) = fixture();
// Stage 1 offers KMV at k = 4,323 (the δ split). Stage 3 charges each candidate through
// asap_cost:
// - A KMV build's update_terms_per_row charges
//   110 + 0.62 × kmv_compared_hashes_per_row(4_323, Some(60_000), Some(1_000_000)) = 715 ns per row
//   (553 before; 712 measured at 400 states, 757 at 1,000).
// - Its 1-minute window states (12,000 rows each, distinct_values Some(1_000_000)) merge 5 per
//   group at 4,323 × (2.6 + 0.43 × compared) = 3.40 ms per state (3.12 measured; 4.22 charged before).
let run = asap_planner::run_planning(roots, &entries, data, models, PlanSearchSettings::default())?;
// Selected: both queries read one KMV (k = 4,323) by (service, endpoint), built in 1-minute tumbling
// windows at ingestion and merged over the 5 the range covers. Charged 401 s per evaluation; the
// query-time build is charged 481 s and the exact plan 590 s.

Major algorithm changes

Cost model: the KMV build.

  • Problem: at 400 states × 60,000 rows of 1,000,000 values at k = 4,323, the build was charged 553 ns per row and measures 712. Every one of the 132 grid builds was under-charged, by 1.17–1.63x.

  • Before: 65 + 0.5 × compared per row, fitted on 28 builds of 300,000 rows.

  • After: 110 + 0.62 × compared per row.

  • How it was fitted: same method as the exact distinct refit.

    • Measured on the executor's KMV SummaryAgg by (service, endpoint): the build's own CPU (its time minus its input's), median of 3 runs on one pinned core, 10 rows per group per batch.
    • Grid: 10, 100, 400, 1,000, 2,000 and 10,000 states × 1,200, 12,000 and 60,000 rows per state × 10,000, 100,000 and 1,000,000 distinct values × k = 1,081, 4,323, 26,656 and 106,263. That is 132 builds, capped at 24,000,000 rows.
    • The host was shared, and two settings measured up to 1.8x slow in one pass (1,000 × 1,200 and 100 × 12,000). Both were measured three or four times, and the median per point is kept.
    • Weighted least squares on relative error. Forms were compared by leave-one-setting-out error, where a setting is (states, rows per state, distinct values); errors are RMS of the log ratio.
    form leave one setting out worst point leave each k out
    per row + per compared hash (chosen) 0.058 0.25 0.057
    + per inserted hash, or + per inserted hash × log2 k 0.058 0.25
    + per compared hash × cache-miss fraction (10 MB of retained hashes) 0.049 0.22
    constants before this PR 0.280 (whole grid)
    • The heap's insert term adds nothing. The cache term fits a machine-specific cache size for a 0.009 gain, so it is left out.
    • Fitted 108.8 and 0.620, charged 110 and 0.62.
    • Leaving out each states count gives 0.065, each rows-per-state 0.061, each distinct count 0.057.
  • Held-out shapes, not in the grid, each the median of 3 passes:

    states × rows per state, distinct values, k measured ns/row new charge ÷ old charge ÷
    1 × 300,000, 100,000, 4,323 329 305 1.08 222 1.48
    4 × 75,000, 100,000, 26,656 8,357 8,679 0.96 6,975 1.20
    300 × 30,000, 300,000, 8,000 2,499 2,509 1.00 1,999 1.25
    3 × 200,000, 50,000, 2,000 234 183 1.28 134 1.75
    5,000 × 3,000, 2,000,000, 4,323 1,062 1,041 1.02 816 1.30
    30 × 100,000, 3,000,000, 50,000 20,090 18,519 1.08 14,911 1.35
    1,000 × 6,000, 1,000,000, 13,000 1,962 1,963 1.00 1,560 1.26
    200 × 40,000, 1,000,000, 4,323 882 899 0.98 728 1.21
    50 × 20,000, 20,000, 500 133 143 0.93 92 1.45

    RMS log error is 0.094 (was 0.322). The first two rows are the shapes in a_kmv_build_costs_per_retained_hash_it_compares. Their earlier measurements (191 and 7,410) do not reproduce on today's executor (329 and 8,357), so the test now quotes these.

Cost model: the KMV merge.

  • Problem: the merge was charged 65 per hash plus 0.5 per compared hash, taking every merged hash as distinct. Measured, merges cost 0.21–1.10 of that charge on the grid and 0.33–1.58 on held-out rollups. The error grows with the states per group, and it reverses when states share many values.

  • Before: retained × (65 + 0.5 × kmv_compared_hashes_per_row(k, retained × states, None)) per input state.

  • After: retained × (2.6 + 0.43 × kmv_compared_hashes_per_row(k, retained × states, Some(shared))) per input state:

    • retained is the hashes a state keeps, at most k: the distinct values among its rows.
    • shared is the V·h/D smallest hashes the states draw theirs from (see How).
  • How it was measured:

    • The executor's SummaryMerge over KMV states stored as a materialized view: a 1-minute tumbling-window chain by (service, endpoint) over 5 minutes, read through its time range.
    • Two measurements:
      • the window merge: 5 states per group;
      • a rollup that reads the stored windows by service: 5 × (endpoints per service) states per group, 10 to 200.
    • Each is the merge node's own CPU (its time less its input's, which reads the stored states), median of 3 runs, one pinned core, sibling hyperthread idle. Load average before and after each run was under 3.
    • The window states hold the k smallest hashes of their rows' user_ids, hashed as the executor hashes them, inserted in random order.
    • Check against states built by real ingestion runs, at k = 4,323 and 26,656: the window merge measures 3.09 and 79.9 ms per state either way, and the rollup 2.06 against 2.01 and 81.7 against 80.3.
  • Grid: k = 1,081, 4,323, 26,656 and 106,263 × window states of 1,200, 12,000 and 60,000 rows of 1,000,000 users × 5, 10, 50 and 200 states per group. That is 45 merges; states per group times state size was capped at a few minutes of CPU.

  • Forms compared by leave-one-setting-out error, where a setting is (k, rows per state, states per group):

    form leave one setting out worst point leave each k out leave each states-per-group out held-out rollups
    per hash + per compared hash, states share the smallest hashes (chosen) 0.050 0.10 0.122 0.061 0.091
    the same, every hash distinct (the form before this PR) 0.104 0.39 0.106 0.101 0.318
    the same, every hash distinct, each group's first state passed through 0.120 0.38 0.128 0.126 0.329
    chosen + per state, or + per hash × log2 k 0.046 0.10–0.11 0.111–0.134 0.064–0.065 0.080
    constants before this PR 0.524 (whole grid) 0.436
    • The extra terms gain 0.004 and fit a negative per-hash constant, so they are left out.
    • Fitted 2.61 and 0.428, charged 2.6 and 0.43.
  • Held-out rollups, not in the grid: states merged from 5 windows (6,000 to 300,000 rows each), then rolled up 10 or 100 per group. Each rollup is its node's time less the window merge below it, median of 3 runs. ns per input state, in ms:

    k states per group × hashes each (rows each) measured new charge ÷ old charge ÷
    1,081 10 × 1,081 (6,000) 0.150 0.144 1.04 0.234 0.64
    1,081 100 × 1,081 (6,000) 0.034 0.030 1.14 0.101 0.33
    1,081 10 × 1,081 (60,000 of 20,000 users) 0.224 0.238 0.94 0.234 0.96
    1,081 100 × 1,081 (300,000) 0.083 0.077 1.07 0.101 0.82
    4,323 10 × 4,323 (60,000) 2.006 2.261 0.89 2.902 0.69
    4,323 100 × 4,323 (6,000) 0.370 0.441 0.84 0.780 0.47
    4,323 100 × 4,323 (300,000) 1.235 1.202 1.03 0.780 1.58
    4,323 10 × 4,323 (60,000 of 20,000 users) 3.331 3.779 0.88 2.902 1.15
    26,656 10 × 26,656 (60,000) 80.25 85.61 0.94 101.40 0.79
    26,656 100 × 26,656 (60,000) 19.72 19.73 1.00 20.72 0.95
    106,263 10 × 11,929 (12,000) 298.8 281.3 1.06 348.3 0.86
    106,263 10 × 58,237 (60,000) 1,089.4 1,058.1 1.03 1,246.8 0.87

    This is 12 of the 24. Over all 24, measured ÷ charged is 0.84–1.14 and the RMS log error is 0.091 (was 0.33–1.58 and 0.436).

    • The rows of 300,000 per state, and of 20,000 users, are where states share the most hashes. Taking every hash as distinct (refit to this grid) charges them 0.36–0.68 of their time.
    • The k = 106,263, 10 × 11,929 row is the rollup in a_kmv_rollup_merge_costs_per_hash_it_inserts. It now quotes 299 s for its 1,000 states (median of 3 on an idle core), not the earlier 352 s (a plan's time less its build's).
  • One state per group. A merge whose groups each read one state passes it through. Measured that way, it costs 2–10 µs per state at k = 1,081 and 106,263 (1,000 states). The cost model sends a merge that keeps its input's groups down the pass-through path. A rollup whose estimates tie its input's (10,000 (service, endpoint) states into 10,000 services) still reaches this charge with one state per group. It is then charged as one state filled from empty: 3.9 ms per state at k = 4,323, the most any number of states per group costs. That rule predates this PR and is unchanged (see Limitations).

  • States of unknown rows. A state built at ingestion has no known rows per state. With the column's distinct values V declared, it retains min(k, V) hashes, the same ones in every state (each folded all V); without them, k distinct hashes as before. Example: 10 states per group of 500 users at k = 4,323 are charged as 500 shared hashes each, not 4,323 distinct ones (about 9x less).

  • Correctness: the charge stays O(1) per node (the existing 64-slice average). No search or MILP change.

Before this PR

On the base, the viewer rollup at 600,000,000 rows per evaluation is charged 394.0 s per evaluation for a query-time KMV build (331.9 s of it the build), and 589.8 s for exact. The build measures about 1.3x its charge.

After this PR

The planner selects the KMV built in 1-minute tumbling windows at ingestion, then merged over the 5 windows at each evaluation. The three plans are charged:

plan build merges total (s per evaluation)
KMV in 1-minute windows, merged (selected) 191.0 169.8 (windows) + 38.8 (rollup to service) 400.8
query-time KMV build 429.2 38.8 (rollup) 480.5
exact 589.8

At the build-only refit (68a1efa0f) they were charged 453, 491 and 590.

Measured end to end at a scaled shape with the same rows per state:

  • 1,000 series, each sampled every 5 ms: 60,000 rows per (service, endpoint) state per evaluation, 12,000 per 1-minute window.
  • One pinned core, sibling hyperthread idle, load average under 3 before and after.
  • Per evaluation:
    • the window plan is one ingestion run (one new window) plus the query;
    • the other two plans are the query.
  • Two data mappings:
    • one endpoint per service, which the planner assumes (10,000 series, no per-label counts, so 10,000 services);
    • 10 services × 100 endpoints.
plan charged at this shape measured, one endpoint per service measured, 100 endpoints per service
KMV in 1-minute windows, merged (selected) 40.1 s 31.3 s (ingestion 15.8 + query 15.5) 31.7 s (15.6 + 16.1)
query-time KMV build 48.1 s 39.5 s 39.4 s
exact 59.6 s 62.9 s 57.1 s
  • Per node, one endpoint per service:
    • the window build measures 15.8 s against 19.1 s charged;
    • the window merge 15.5 s against 17.0 s;
    • the query-time build 38.8 s against 42.9 s;
    • the rollup to service passes each state through, about 0 s against 3.9 s charged (see Limitations).
  • With 100 endpoints per service, the rollup does merge: 0.5 s in the window plan and 0.8 s in the query-time plan.
  • The exact plan's one-endpoint run ended as another job started on the host (load average 27 after). Its 3 query runs came before that, and its by-(service, endpoint) aggregate matches the clean 100-endpoint run (32.6 s against 32.2 s).

The planner's choice is the measured fastest. Measured, the window plan is 1.26x faster than the query-time build and 1.8–2.0x faster than exact, with either data mapping. Scaled ×10 to the planned 10,000 series, that is about 313 s, 395 s and 571–629 s per evaluation, against 401, 481 and 590 charged.

Evidence

Plan survey, main (7cfd274b5) against this PR (471262aa3), through run_search_over with default PlanSearchSettings, with the default and the per-resource (UsageOverTime) cost inputs. All 422 plan on both:

  • fix(cost): charge sketch states and merges, raw batch visits, grouping keys, distinct counts and KMV at their measured costs, so the measured-best plan is selected #1363's sweep (202 workloads): the grid of 120, showcase 24, held-out 48, 2,000-series 10. All 202 plans are identical node for node, and no cost changes. None of them contains a distinct count.

  • Distinct-count grid (220 plans):

    • Generated from the rollup workload and the SQL production shape (production_rollup_workloads), plus an ungrouped COUNT(DISTINCT user_id).
    • Swept over 100, 1,000 and 10,000 series × sample intervals of 5, 25, 250 and 2,500 ms × 1%, 5% and 10% error, plus rollupheavy-t11 and the planner-layering example 2.
    • Against main, 96 plans change: the build refit's operator or k changes, Project-order changes, and the merge refit's changes below. Typical: the SQL production shape's two distinct counts (by service and by (service, endpoint)) move from two KMV builds (k = 106,263 and 4,323) to one KMV build by (service, endpoint) merged to service; the rollup's query-time build moves to 1-minute tumbling windows merged over the range.
  • The merge refit, against the build refit alone (measured on the earlier base): 29 plans change.

    group build refit only with the merge refit count
    rollup, 250 ms, every series count and error, both cost inputs query-time KMV build, then a rollup merge query-time KMV built in 1-minute windows, then the window merge and the rollup merge 18
    SQL production shape and rollup, 100 series, 2,500 ms KMV at k = 26,656 or 106,263 KMV at k = 4,323 3
    exact plans, 25 ms Project held for the evaluation Project streamed (same operators, same cost) 8
    • The 56 plans the build refit moved into merges or windows all keep them.
    • At 250 ms a state folds 1,200 rows per evaluation. Built in 5 windows of 240 rows, each state's walk is shorter, and merging the 5 window states costs less than the rows' build. The merge measures that cheap (the grid's 1,200-row states cost 0.21–1.47 ms per state).

Verification

Run on a 40-core build host, PR 471262aa3 against main 7cfd274b5:

scope command main PR newly failing
asap-cost cargo nextest run -p asap-cost 109 pass 113 pass none
asap-integration-tests + asap-devtools cargo nextest run -p asap-integration-tests -p asap-devtools --no-fail-fast 339 pass / 9 fail / 8 skipped 340 / 9 / 8 none: the same 9 fail on both
sketch and corpus snapshots cargo test --release -p asap-devtools --test corpus_snapshots -- --ignored 2 pass 2 pass (no snapshot changes) none
fmt cargo fmt --all --check clean
clippy cargo clippy -p asap-cost -p asap-integration-tests --all-targets -- -D warnings clean
  • The 9 failures on both: joint_search_equivalence ×2, plan_sharing ×2, promql_to_post_asap::promql_quantile_of_rate_binds_kll_over_rate_accumulator, materialized_views::materialized_views_equal_the_stored_outputs_main_listed, planner_fuzz ×3.

New and changed tests:

  • a_kmv_build_charges_its_measured_time_per_row (new): six measured builds (10 and 1,000 states, 120,000–12,000,000 rows, k = 1,081–106,263) are each charged within 0.8–1.25 of their time. It fails on the base (109 charged vs 176 measured at k = 1,081).
  • a_kmv_build_costs_per_retained_hash_it_compares: now quotes the re-measured 329 and 8,357 ns per row. It fails on the base (222 vs 329).
  • a_kmv_merge_charges_its_measured_time_per_state (new): six measured merges (5 to 200 states per group, k = 1,081–106,263, one of 20,000 users) are each charged within 0.8–1.25 of their time. It fails on the build refit (4,223,195 charged vs 3,115,412 measured for 5 window states at k = 4,323; 88,151 vs 18,109 for 200 states at k = 1,081).
  • a_kmv_rollup_merge_costs_per_hash_it_inserts: now quotes the re-measured 299 s, within 0.8–1.25 (was 352 s, within 0.75–1.33).
  • production_distinct_rollup_counts_exactly_at_12k_rows_per_state (new): at a sample every 25 ms, both queries count exactly. It passes on base and PR (see Why).
  • a_kmv_merge_of_states_with_unknown_rows_retains_at_most_the_column_s_values (new): a merge of states with unknown rows and 500 declared users is charged as states that folded all 500.
  • production_distinct_rollup_builds_kmv_at_60k_rows_per_state (changed): the shape is now set by sample_interval, and a KMV built in tumbling windows also counts as a KMV build.

Design scope

  • StateInputRows::distinct_values (cost, the description of what one summary state folds). It is the folded column's distinct values per state. Any summary whose cost depends on repeated values can read it, not only KMV.
  • CostConstant::{KmvUpdateOperationsPerComparedHash, KmvMergeOperationsPerHash, KmvMergeOperationsPerComparedHash} (cost, the named per-unit constants): one per constant, as for every other.

Architectural decisions

  • The merge's compared hashes count shared hashes. Taking every hash as distinct fits the grid at 0.104 and the held-out rollups at 0.318. Counting the hashes states share fits them at 0.050 and 0.091.
  • The cache term is left out of the build fit (see the build fit table).
  • No library fix here, as decided. The constants describe today's executor.

Limitations and follow-up

  • These constants describe the executor's pinned asap_sketchlib KMV, whose duplicate check walks every retained hash. KMV: pin the O(log k) sketchlib fix and recharge KMV builds, merges and memory at its measured cost #1394 pins the O(log k) fix and refits the build and merge; its constants replace these.
  • A rollup whose group estimates tie its input's is charged as one state per group filled from empty. A real rollup at k = 4,323 measures 2.0 ms per state at 10 states per group and 0.44 ms at 100, against that 3.9 ms. With no per-label counts, the planner cannot know the states per group. Both KMV plans of the headline rollup carry this charge (38.8 s), and measured with one endpoint per service it costs nothing.
  • distinct_values is the column's declared distinct values. When a workload declares none, the merge takes every hash as distinct. That charges a merge of states that share many values at about 0.36–0.68 of its time.
  • The build grid ran on a shared host. Two settings needed repeated passes; the rest are single passes of median-of-3. The merge grid and the end-to-end runs ran on an idle host.

Human review — do not complete with an agent

  • The MVP boundary is correct.
  • New conceptual layers or public interfaces are necessary.
  • The before/after description matches the intended product behavior.
  • Human reviewer:
  • Decision and rationale:

@zzylol zzylol added bug Something isn't working cost-model Cost formulas, statistics, ranking, and selection labels Nov 16, 2026
@zzylol
zzylol force-pushed the fix/kmv-build-cost-refit branch from c9f874c to 8632711 Compare November 16, 2026 22:48
@zzylol zzylol changed the title fix(cost): charge a KMV build its measured time per row (110 + 0.62 per compared hash) fix(cost): charge KMV builds and merges their measured time (build 110 + 0.62, merge 2.6 + 0.43 per compared hash) Nov 16, 2026
@zzylol
zzylol force-pushed the fix/delta-split-over-dependent-readouts branch from 50cf197 to 793859e Compare November 16, 2026 23:48
@zzylol
zzylol force-pushed the fix/delta-split-over-dependent-readouts branch from 793859e to 386ea88 Compare November 17, 2026 16:58
@zzylol
zzylol force-pushed the fix/kmv-build-cost-refit branch from f9c1da7 to 598f8d4 Compare November 17, 2026 18:43
@zzylol
zzylol changed the base branch from fix/delta-split-over-dependent-readouts to main November 17, 2026 18:43
…0 + 0.62, merge 2.6 + 0.43 per compared hash)

The KMV build is refit on 132 executor builds: 110 operations per row and
0.62 per compared hash (was 65 and 0.5). The KMV merge is refit on 45
executor merges: 2.6 per hash and 0.43 per compared hash, with the states
of a group sharing the smallest hashes of the column's values.
StateInputRows carries the folded column's distinct values for that; a
merge of inputs keeps the largest.

CostConstant renames KmvOperationsPerComparedHash to
KmvUpdateOperationsPerComparedHash and gains KmvMergeOperationsPerHash and
KmvMergeOperationsPerComparedHash; merge_terms_per_state charges a KMV
merge in those terms. One expected_distinct_values helper replaces the
three copies of the formula.

The constants describe the executor's pinned asap_sketchlib KMV, whose
duplicate check walks every retained hash; they are measured again once
the library's check is O(log k).
@zzylol
zzylol force-pushed the fix/kmv-build-cost-refit branch from 598f8d4 to 55931e5 Compare November 17, 2026 18:45
…istinct values; review doc fixes

A merge of KMV states built at ingestion (rows per state unknown) took k
hashes per state, every one distinct; with the column's distinct values
declared it now takes the fewer of k and those values, shared by every
state. Docs: distinct_values is the fewer of the declared values and the
per-group bound; the merge fit's extra terms bring the error to 0.046 but
fit a negative constant; the summation-order example names 0.1 x 7.
@zzylol
zzylol merged commit 5ebca6c into main Nov 17, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cost-model Cost formulas, statistics, ranking, and selection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KMV build is charged 0.61–0.85 of its measured time; the library's O(k) duplicate scan dominates it

1 participant