Repository navigation
fix(accuracy): constraint (K) checks the union bound over the estimates each query depends on - #1405
Merged
Merged
Conversation
zzylol
force-pushed
the
fix/delta-split-over-dependent-readouts
branch
from
November 17, 2026 16:58
793859e to
386ea88
Compare
…er-value query max(q) over 1,000 series at delta 0.01 reads a copy failing with 5e-4 that a per-value query on q at delta 0.5 was given; its union bound is 0.5. Fails on main: (K) adds the selected readout's delta once.
…ut depends on Tests first for the failure-probability rows of (K): a per-value bound P and a whole-output bound W per execution, with the root checking the bound Pass 2's split names (every estimate read, or each covered value). - Stage 3: or, concatenation, many-to-one match, and/unless and count inputs, exact inputs, per-value, whole-result and record-share scopes, an estimate reached on two paths, a top-k readout, a query with several roots, context copies; the reduction regression gets the equal-delta case (a per-value query at 0.01 gives a 0.01 copy). - A brute-force oracle over small shapes and every dataset of two 6-series metrics: every accepted plan's union-bound worst case is within delta, with random looser copies; every shape alone has a plan whenever it has one without delta. - asap-types: AccuracyTarget::failure_probability_bound and Memo::max_estimates_per_evaluation (top-k counts every item it ranks). - Integration: or of two quantiles, group_left onto every series, max reading its own split's level. These use new interfaces and do not compile without the next commit.
…es each query depends on (K) carried one failure probability D per execution, added each selected readout's delta once and checked D <= delta / V at the root. That summed both sides of `or`, divided by V where every output reads one estimate (`q * on() group_left e`), and trusted a readout level given for another query: `max(q)` over 1,000 series could read a per-value query's copy failing with 0.01 per estimate. Each execution now has P (one output value) and W (its whole output), composed by the operator classes Pass 2's split uses, over the inputs whose error reaches the output. A readout has P = delta and W = R delta (a top-k R delta for both). Each query's one row checks W <= delta (every estimate read) or P <= delta / V (each covered value), the bound AccuracyTarget::failure_probability_bound names from the same counts Pass 2 splits by; a query with several roots sums its roots' terms. - asap-types: AccuracyTarget::failure_probability_bound and FailureProbabilityBound replace failure_probability_per_output_value; the per-readout split derives from the bound. Memo::max_estimates_per_evaluation counts every item a top-k ranks. - asap-accuracy: the operator classes and the "error reaches the output" test move to estimate_dependence, shared by Pass 2 and Stage 3; readout_value_counts_of_queries exposes Pass 2's counts. - Stage 3: or and a concatenation take the largest input error, not the sum, as Pass 2's caps assume. - Stage 3: P and W exist only where a query root above checks them; created wherever bounded, RangeHeavy-8users solved about 3x slower. - accuracy-models.md: the (K) rows, which estimates a result depends on, the R assumption, examples.
…hoosing its values `p90 and topk(3, …)`, `p90 unless topk(3, …)` and `topk(3, …) or p90` keep or take a value by which keys the top-k returns, which all of its ranked estimates decide: each output value's failure probability adds the top-k's to the p90's. Fails today: (K) drops the right side of and/unless and takes the larger side of or.
…g a set operator's values - Pass 2 records the readout values (R, fan-in) each query's δ is split over (LogicalASAPMemoCandidates::readout_values_of_query, from ReadoutLevels); Stage 3 checks the bound they name (LogicalASAPMemoCandidates::failure_probability_bound) instead of walking the memo again. readout_value_counts_of_queries is deleted. - An input whose values a top-k chooses (values_chosen_by_estimate) counts its estimates even where beta = 0 (the right of and/unless), and `or` sums its sides' one-value failure probabilities when one side's values are chosen by estimate. One rule for Pass 2's count walk and (K): ErrorMetricsOfMemo::input_estimates_reach_output and each_value_depends_on_one_input. - A readout required at delta = 0 adds 0 whatever its estimate count. - RootLimits compares the bound's kind and limit only (values_covered is not read by any row); the precision error prints the bound in words (FailureProbabilityBound: Display). - Docs: the count walk, R = Memo::max_estimates_per_evaluation and its distinct-item contract, context copies, and the measured cost of scaling failure probabilities by the tightest limit (not adopted). - accuracy_constraints counts each query's outputs once.
zzylol
changed the base branch from
fix/delta-split-over-dependent-readouts
to
main
November 17, 2026 18:14
zzylol
marked this pull request as ready for review
November 17, 2026 18:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Problem. Constraint (K) kept one failure probability
Dper execution. It added each selected readout's δ once and checkedD ≤ δ / n_qat the root. That broke in three ways:max(q)over 1,000 series could read a readout copy given to a per-value query onq. With δ 0.01 per estimate, its real union bound is 1,000 × 0.01 = 10 ([Bug] A reduction reads a readout level given for another query, and Stage 3 accepts it #1395).orgets no sketch plan. On this PR's base,q or radds both sides' δ per value (2e-4 against 1e-4).q * on() group_left edivides δ by 1,000 output values that all read one estimate.Change. Each execution now has:
P, the failure probability of one output value;W, the failure probability of its whole output.Both are built from the same operator classes and counts Pass 2 splits δ by. Each query's one row checks the bound
AccuracyTarget::failure_probability_boundnames:W ≤ δ(every estimate read) orP ≤ δ / V(each covered value).Effect.
max(q)must read the 1e-5 copy.q or rand thegroup_leftmatch read sketches again.Review fixes (head
1070e0a14): Pass 2 records each query's readout counts and Stage 3 reads them (no second walk); a set operator over a top-k counts the top-k's estimates; δ = 0 readouts add 0; the precision error reads as words. Scaling failure probabilities by the tightest limit was measured and not adopted. Details in "Review fixes".Base:
main. #1360 is merged (5e82ecb2b); this PR is rebased onto it (git rebase --onto origin/main 386ea88fb), headbf6288565. Until this merges,mainleavesorand many-to-one matches with no plan at all (no wrong answers).Review guide
main5e82ecb2b(fix(accuracy): split δ over the readout values the covered output values depend on #1360 squash-merged). Rebase conflicts, resolved:accuracy_at_query_roots.rs(main's test(integration): regression guard for each query meeting its own accuracy target over a shared readout group #1384 added aseries_countargument toworkload/selection; this PR'sworkload_with_sourcesandplanning_runtake it too, and its tests passLATENCY_SERIES), andreport.rsimports (main's feat(types, pass1, stage2, stage3, planner, viewer): deployment exclusions: one Stage 2 filter for a target's capabilities and the caller's exclusions #1374ExclusionSourcenext toFailureProbabilityBound).40664b5f9: the main-only regression test for [Bug] A reduction reads a readout level given for another query, and Stage 3 accepts it #1395, cherry-picked fromfix/k-cross-query-regression-test. It fails onmain1760a69f5.2a76cb3f4: all new tests. They use new interfaces, so they compile only with commit 3.ee637c92f: the implementation andaccuracy-models.md.5e93d34c8: review test,k_charges_a_set_operator_for_the_top_k_choosing_its_values. It fails on commit 3 (And 0.001: accepted, should not be).bf6288565: the review fixes (see "Review fixes").AccuracyTarget::failure_probability_boundandFailureProbabilityBound(L402). This is the rule shared by Pass 2 and Stage 3.milp/accuracy.rs:ExecutionError(L39) andAccuracyRules::compose(L422), the composition in f64;ErrorModel::set_checked_failure_probabilities(L787), which decides wherePandWexist;accuracy_constraints(L986), the rows.estimate_dependence.rs, the operator classes moved out ofreadout_capsso both stages read them, withvalues_chosen_by_estimate(L52);ErrorMetricsOfMemo::input_estimates_reach_outputandeach_value_depends_on_one_input(metric_rule/mod.rs, L123, L143), the one rule Pass 2's count walk and (K) both read.LogicalASAPMemoCandidates::readout_values_of_queryandfailure_probability_bound(memo_candidates.rs, L78, L92): the counts Pass 2 split δ by, which (K) checks.Prows oforand concatenation (one row per input, not a sum);Memo::max_estimates_per_evaluationfor top-k.report.rsandlib.rs, and the moved functions.Behavior: before → after
The four cases, each at δ = 0.01 over the whole result unless stated. "Before" is this PR's base, #1360.
quantile_over_time(0.9, latency{side="a"}[5m]) or …{side="b"}…(latency1,000 series,n_q2,000)W = 2,000 × 5e-6 = 0.01: DDSketch planmax_over_time(latency[5m]) * on() group_left quantile_over_time(0.99, reference[5m])(reference1 series)W = 0.01: DDSketch at δ 0.01qper value at δ 0.01, andmax(q)(1,000 series; copies at 0.01 for cost 1 and 1e-5 for cost 3)max(q)reads the 0.01 copy:D = 0.01 ≤ 0.01accepted, true bound 10W10, so the 1e-5 copy is selected for bothcount(distinct user_id)by service, per value at δ 0.05 (10,000 services)k = 4,323at δ 0.05V = 1:P = 0.05, unchanged (rollup_controlledpasses on both)The first two are integration tests (
accuracy_at_query_roots). Both fail on #1360 with no plan selected. The third is the Stage 3 unit test, in two variants: δ 0.5 / 5e-4 fails onmain, δ 0.01 / 0.01 fails on #1360. Through the planner, Pass 2 drops DDSketch's looser level, because DDSketch's error does not depend on δ, soa_max_reads_the_level_its_own_split_givespasses on both and guards the end-to-end path only.Interfaces
crates/types/src/accuracy/target/mod.rs
impl AccuracyTarget { - /// The failure probability each output value must meet: δ over the values the scope covers. - pub fn failure_probability_per_output_value(&self, values_per_evaluation: Option<u64>) -> Option<f64>; + /// Which union bound checks this target's δ, and its limit, from n_q and the query's readout counts; None without a δ or with neither bound known. + pub fn failure_probability_bound(&self, values_per_evaluation: Option<u64>, readout_values: ReadoutValueCounts) -> Option<FailureProbabilityBound>; + /// Pass 2's per-readout level: the bound's limit split over the estimates it counts (R, or the fan-in). pub fn failure_probability_per_readout_value(&self, values_per_evaluation: Option<u64>, readout_values: ReadoutValueCounts) -> Option<f64>; } +/// The union bound a query's failure probability is checked by (Pass 2 splits by it, Stage 3 checks it). +#[derive(Debug, Clone, Copy, PartialEq)] +pub enum FailureProbabilityBound { + /// Summed δ of every estimate any output value depends on ≤ limit (δ). + EveryEstimateRead { limit: f64 }, + /// Summed δ of one output value's estimates ≤ limit (δ / values_covered). + EachCoveredValue { values_covered: u64, limit: f64 }, +} +impl FailureProbabilityBound { + /// Its limit. + pub fn limit(&self) -> f64; +}failure_probability_per_output_valueis removed: dividing δ byn_qis the rule that was wrong foror, matches and reductions.crates/types/src/stages/logical/memo/mod.rs
impl Memo { + /// The estimates one evaluation of a readout group computes: its values, or for a top-k every item it ranks (states × declared distinct items, at most the rows). + pub fn max_estimates_per_evaluation(&self, group: MemoGroupId) -> Option<u64>; }crates/accuracy/src/lib.rs, estimate_dependence.rs, readout_caps/mod.rs, metric_rule/mod.rs
crates/types/src/stages/logical/memo_candidates.rs
pub struct LogicalASAPMemoCandidates { /* memo, edges, failing_logical_edges, pruned, evaluation_grid_of_group, exact_only_groups_of_query unchanged */ + /// Per query: the readout counts (R, fan-in) Pass 2 split its δ over; None without a target. + pub readout_values_of_query: Vec<Option<ReadoutValueCounts>>, } impl LogicalASAPMemoCandidates { + /// The bound query `query`'s δ is checked by: its target's failure_probability_bound over the recorded counts. + pub fn failure_probability_bound(&self, query: usize) -> Option<FailureProbabilityBound>; } +/// In words for error messages: "each of 100 covered values fails with at most 0.0001". +impl std::fmt::Display for FailureProbabilityBound;crates/stage3-plan-selection/src/report.rs
/// Why Stage 3 selected no plan (the type stays). pub enum ErrorSelectingPlan { InvalidSettings(AnalyticalCostError), EntryCount { /* fields unchanged */ }, Cost { /* fields unchanged */ }, NoValidCandidate(Vec<Rejection>), Solver(String), /// A query's target is below the solver's precision after the 1e-10 retry (unsatisfiable request). AccuracyTargetBelowSolverPrecision { query: usize, - failure_probability_per_output_value: Option<f64>, + failure_probability_bound: Option<FailureProbabilityBound>, }, UnsupportedCostModel(String), }Usage. The workload entry is what the user writes. Stage 0 lowers it, Pass 2 splits δ, and Stage 3 checks the bound. Run by
an_or_of_two_quantiles_reads_sketches:On #1360 the same run selects no plan.
Algorithm changes
Constraint (K)'s failure-probability rows
D_c ≥ δ_c + Σ_i H_iover every input.D_v ≤ δ / V.compose,accuracy_constraints):P_c ≥ Σ_{value by value} HP + Σ_{whole} HW, with one row per input fororand a concatenation;W_c ≥ Σ HW.P = δandW = R·δ; a top-k hasR·δfor both.failure_probability_bound:W_v ≤ δorP_v ≤ δ / V;PandWexist only where some root above checks them (set_checked_failure_probabilities).orand a concatenation now take the larger input error inE, not the sum. That is the rule Pass 2's caps assume. With the sum,q or rat relative 0.05 was infeasible even without a δ.P(fail) ≤ max_D Σ_{e ∈ S_q(D)} δ_e, and both rows bound that sum (proof inaccuracy-models.md, "Which estimates a result depends on").Rbounds the estimates per evaluation. Top-k is checked withmax_estimates_per_evaluation; time-bucketed SQL falls back to the rows read, whichmemo_value_boundsexecutes on data.union_bound_oraclechecks every accepted plan, for 21 shapes × 3 scopes plus cross-query pairs with random looser copies, against the union-bound worst case over all 4,096 datasets of two 6-series metrics.Wset back toδ, the oracle reports worst cases of 0.03–0.06 against 0.01.PandWwere created wherever bounded, RangeHeavy-8users solved 3.1× slower with the same objective, so they are now created only where needed.max(A or B): 200 δ for 100 estimates). That costs a stricter δ, never a miss (Constraint (K): count each estimate once per query (Form X) instead of once per path #1400). Each readout is charged its required level's δ, not its build's own (DDSketch's is 0) (Constraint (K): charge the build's own failure probability (DDSketch 0) and one event per state where the guarantee is simultaneous #1401).MILP size and planning time. One build host, Xeon E5-2660 v2 @ 2.20 GHz,
taskset -c 3, releaseasap_planner --run-until stage3, median of 5 (interleaved). Workloads: the 2026-11-15 RollupHeavy and RangeHeavy 8-user workload files.Without the "checked only" rule, the medians of 3 were: RollupHeavy 14.82 s, 936 (K) variables; RangeHeavy 28.99 s, 360 (K) variables.
Decisions and open questions
Decided:
ora summed fan-in. Decided by the user.Rassumption in the doc;PandWexist;Needs reviewer decision:
Wrows sum R·δ terms whereDsummed δ). Dividing every failure probability by the tightest root limit made it worse (+10% and +48% more on the same host), so it is not adopted (recorded inaccuracy-models.md). Both fix(accuracy): split δ over the readout values the covered output values depend on #1360 and this PR are still faster thanmainon RangeHeavy-8users (29.0 s) and this PR is +61% vsmainon the counter variant (13.8 s). Decided 2026-11-17: accepted (optimizer efficiency is deferred); follow-up (K): tighten the failure-probability big-M ceilings (RangeHeavy-8users-counter plans 2x slower) #1413 tightens the failure-probability big-M ceilings.an_or_takes_each_value_from_one_readout, which certifies DDSketch, not (K). It stays.Verification
After the rebase onto
main5e82ecb2b(headbf6288565;cargo nextest run --workspace --no-fail-fast, compared by name):main1845 run / 10 fail, this PR 1871 / 10, the same 10 names (joint_search_equivalence×2,materialized_views,plan_sharing×2,promql_to_post_asap,summary_sharing,planner_fuzz×3).per_query_accuracy_targets(4),accuracy_at_query_roots(7),memo_value_boundsandrollup_controlledpass. Snapshot suites:corpus_plans_match_snapshotspasses;sketch_plans_match_snapshotsdiffers onmainand here inratio-of-quantiles-1h-mvponly, byte-identical between the two (#1360 merged without regenerating it).cargo fmt --checkandcargo clippy --workspace --all-targets -D warningsclean. The measurements below were taken before this rebase (on #1360386ea88fb); the rebase changed only a test helper and an import.Workspace (
cargo nextest run --workspace --no-fail-fast, compared by name):main6b5a728b91721 run / 10 fail; #1360386ea88fb1755 / 10 (before its executor-gaps commit, alsomemo_value_bounds, fixed in #1360); this PR1070e0a141781 / 10. The same 10 fail on all three:joint_search_equivalence×2,materialized_views,plan_sharing×2,promql_to_post_asap,summary_sharing,planner_fuzz×3. Changed crates (asap-accuracy,asap-types, Pass 2, Stage 2, Stage 3): 423 pass; clippy--all-targets -D warningsand fmt clean.Snapshot suites (
--test corpus_snapshots --test devtools --run-ignored all -E 'test(snapshots)'):corpus_plans_match_snapshotspasses on all three;sketch_plans_match_snapshotspasses onmainand differs in one file on #1360 and here (ratio-of-quantiles-1h-mvp), byte-identical between the two, so #1360's own change.Oracles:
union_bound_oraclepasses. The reviewer's extension (random DAGs, not committed: large, review-only) accepts 559 plans with 0 violations; its retry assertion expected scaling and does not apply.Planning time (one idle host, release
asap_planner --run-until stage3,taskset -c 3, trees interleaved, median of 5; showcase, sketch and random-mixed sums of medians of 3 on core 7):MILP size this PR vs #1360: RollupHeavy-8users 8,663 / 21,870 vs 8,627 / 21,810; RangeHeavy-8users 4,297 / 9,680 both.
Plan changes vs #1360 (all 47 workloads above compared by plan; the review fixes change no plan vs this PR before them):
W = R·δ ≤ 0.3, which accepts the readout at δ 3e-5; fix(accuracy): split δ over the readout values the covered output values depend on #1360's per-valueD ≤ δ / n_qrow did not.q5_joint__1h_10hz_60s, random-mixed w07.Plan changes vs main: the above, plus #1360's own: the RangeHeavy ties (same objective to 1e-8) and
sketch ratio-of-quantiles-1h-mvp(same objective, 196 vs 316 variables). RollupHeavy-8users, the other showcase and sketch workloads are unchanged.New tests (28): review round:
k_charges_a_set_operator_for_the_top_k_choosing_its_values(fails onabd1462ad) anda_failure_probability_bound_reads_as_words. Earlier:k_accepts_an_or_whose_readouts_meet_delta_over_its_covered_values:q or rat 1e-4 per value; 2e-4 rejected.k_charges_a_readout_every_series_matches_once: thegroup_leftcase.k_does_not_let_a_reduction_read_a_level_given_for_a_per_value_query: [Bug] A reduction reads a readout level given for another query, and Stage 3 accepts it #1395, in both δ variants.k_adds_nothing_for_an_input_whose_values_are_not_read:and,/ count.k_counts_an_exact_input_as_never_failing: E and W only.k_does_not_divide_a_per_value_failure_probability_by_the_values;k_bounds_a_whole_result_by_its_covered_values_or_every_estimate: withn_qunknown, (X) forces exact reads;k_bounds_a_record_share_scope_by_its_covered_groups;k_charges_an_estimate_both_sides_of_an_or_read_once_per_value;k_charges_an_estimate_two_paths_reach_once_per_path: Form R's loss, pinned (Constraint (K): count each estimate once per query (Form X) instead of once per path #1400).k_charges_a_top_k_readout_for_every_estimate_it_ranks;k_charges_a_concatenation_one_input_per_value;k_sums_the_failure_probabilities_of_a_query_s_roots;contexts_whose_queries_bound_every_estimate_read_at_equal_delta_are_copies.union_bound_oracle::every_accepted_plan_meets_its_failure_probability_on_every_dataset. It computes the union-bound worst case, not the true failure probability. It randomizes copy levels and covers the record-share scope. It also checks that every shape planned alone has a plan whenever it has one without a δ.failure_probability_boundtests, including the property testthe_readout_split_meets_the_bound_it_names, andmax_estimates_per_evaluation_counts_every_item_a_top_k_ranks.accuracy_at_query_roots):an_or_of_two_quantiles_reads_sketches;a_quantile_matched_onto_every_series_reads_a_sketch;a_max_reads_the_level_its_own_split_gives.The first two fail on #1360.
Review fixes
accuracy-models.md. The counter slowdown comes from (K)'sWrows and big-M ceilings (the MILP size is unchanged); no cheap fix found, numbers recorded above.readout_values_of_query; Stage 3 reads it throughLogicalASAPMemoCandidates::failure_probability_bound.readout_value_counts_of_queriesdeleted. Done.values_covereddropped fromRootLimits::bits(kept on the enum: the error text prints it); "Context copies" updated. Done.readout_groups,readout_values, "The assumption it rests on" now namemax_estimates_per_evaluation). Done.accuracy_constraintscounts each query's outputs once. Done.or/and/unless: fixed with a test (values_chosen_by_estimate; one rule for Pass 2's walk and (K)). (b) distinct-item contract documented onmax_estimates_per_evaluationand in the doc. (c) error text in words (Display). (d) δ = 0 adds 0 whatever the count (not observable at a root: a query whose count is unknown has no bound, so no test). Done.Design scope
ReadoutLevels(asap-accuracy, Pass 2's readout step): the step's output, levels and per-query counts. General: every query's δ split.LogicalASAPMemoCandidates::readout_values_of_query(Stage 1 → Stage 3 boundary type): what Pass 2 decided, read by selection.values_chosen_by_estimate(asap-accuracy, estimate dependence): which groups' value sets depend on estimates; general for any operator choosing by estimate.Follow-ups
main. Fixed here; this PR closes it when merged.yassignments againstselection_meets_targets). The oracle and the exact check after each solve cover the rows; the retry tests (a_selection_missing_its_target…) pass unchanged.Closes #1395.