Skip to content

test(integration): regression guard for each query meeting its own accuracy target over a shared readout group - #1384

Merged
zzylol merged 2 commits into
mainfrom
test/per-query-accuracy-targets
Nov 17, 2026
Merged

zzylol merged 2 commits into
mainfrom
test/per-query-accuracy-targets

Conversation

@zzylol

@zzylol zzylol commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Review guide

Behavior: before → after

No planner behavior changes. What the tests pin, for one series of latency sampled every second, queried every 10 s, no raw data retained:

Workload Before #1238 (79d9932) Main (and this PR)
p99 at rank 0.05 + p99 at rank 0.001 one KLL, k = 52 (0.001 missed) one KLL, k = 2,863
l2_over_time at relative 0.1 + 0.05 UnivMon, 624 columns (0.05 missed) UnivMon, 2,367 columns
p99 at rank 0.05 + sort(p99) at rank 0.001 not checked each meets its target

This serves correctness: a plan must meet every query's own target.

Interfaces

None (tests only).

Algorithm changes

None.

Decisions and open questions

None.

Verification

Scope Passed / failed / ignored Newly failing vs base Notes
cargo test -p asap-integration-tests --test integration -- per_query_accuracy_targets:: accuracy_at_query_roots:: 8 / 0 / 0 none on main cb78d07
same, on #1405's head (d8d4e3a) with this PR cherry-picked 8 / 0 / 0 none
mutation: (K)'s root limits disabled (root_limits returns None) on main 2 / 6 / 0 all 4 new tests fail k = 52 and k = 269 against rank 0.001; UnivMon 624 columns against relative 0.05
cargo clippy -p asap-integration-tests --tests, cargo fmt --check clean n/a

New tests:

  • one_query_text_at_two_rank_targets_reads_a_kll_meeting_the_stricter: the p99 at rank 0.05 and 0.001, in both orders. Each query meets its own target, and the plan holds a KLL with k ≥ 2,863.
  • one_query_text_at_two_relative_targets_reads_a_univmon_meeting_the_stricter: l2_over_time(latency[24h]) at relative 0.1 and 0.05, in both orders. The plan holds a UnivMon with at least 2,367 columns.
  • two_query_texts_sharing_a_readout_group_each_meet_their_target: p99 and sort(p99), which root at different groups over one readout group, with the targets swapped between runs. Asserts the roots differ and both read the same readout.
  • every_query_meets_its_own_target_on_the_selected_plan: a property test over 16 fixed-seed sets of 2–3 queries drawn from 9 (query, target) pairs (p99 rank and relative, p50, sort(p99), L2). Every query's own check holds on the selected plan.

The check is independent of the planner. From each query's root it walks to the one SummaryEstimate on the path (no readout, more than one, or a state without θ is an error, so the check cannot pass vacuously), takes the θ of the state that readout reads, and requires asap_accuracy::theoretical_guarantee at the target's δ to be in an allowed metric and within ε.

Probe evidence. The probe is a metamorphic relation: adding a query never lowers the optimal cost, so {q at ε, q at ε/5} must cost at least {q at ε/5} alone. For every query in the sketch-heavy workload set, it plans the pair in both orders, both as one text and with the looser query wrapped in max(...), and compares the cost with the strict query alone. All plans go through the production path:

Commit Pairs checked Violations Errors
79d9932 (before #1238; (K) not built) 330 190 (KLL, UnivMon, KMV, CMS+heap) 0
2f3664d (#1238 merged) 330 0 0

Follow-ups

@zzylol zzylol added accuracy Accuracy requirements, guarantees, and propagation testing Corpus, regression, differential, or benchmark tests labels Nov 16, 2026
…ies share a readout group at different targets

Regression tests through the production stages: one p99 text at rank 0.05
and 0.001 (KLL), one L2 text at relative 0.1 and 0.05 (UnivMon), two texts
sharing the p99 readout group, and a fixed-seed property over query sets
checking each query's readouts against its own target on the selected plan.
@zzylol
zzylol force-pushed the test/per-query-accuracy-targets branch from 2b260fd to f69c7cc Compare November 17, 2026 17:13
…ins the UnivMon and the shared readout, reuses the (K) test's planning helper
@zzylol
zzylol merged commit 542f7c7 into main Nov 17, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Accuracy requirements, guarantees, and propagation testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant