Repository navigation
Open
Description
Design: docs/design_docs/proposals/testing-infrastructure/README.md
(tests T1–T26, decisions D1–D28), added in #1353. This issue tracks the work; the doc holds
the reasoning.
1. Hand-offs to #1207 PRs (decide now, built there)
- PR 6 solver setup follows D18: deterministic solve (one thread, fixed
seed, no time limit, tie-break on canonical ids) for small cases. - PR 9 JSON printer follows D1–D4 and D28: canonical group ids, 12
significant digits, no timings or raw ids, full memo per step, sharing
summary as its own Stage 3 key.
2. Infrastructure (can start now unless noted)
- Golden runner:
libtest-mimic, one case per directory under
tests/goldens/,case.toml,UPDATE_SNAPSHOTS=1, prints the exact
command on failure (D5–D8, D25). Pending cases (D22). -
case.tomlassertions:same_group,different_group,
rule_fires,rule_does_not_fire(D15). - Relations module; runner re-plans every case shuffled and renamed
(D16). Blocked by PR 6 for Stage 3. -
Ruleenum inasap-types; candidates record their rule;
--only-rules,--disable-rule(D9, D13). - Memo invariants always on;
InvariantViolationwith its own exit
code (D12, D14). -
time_series_metricfield on--sql-catalog-table(D11). -
planner_fuzzwrites a case directory per finding (D26); fixed-seed
run on every PR (D16). -
proptestinasap-accuracy(D17). - CI runs all tests in release (D19).
- AGENTS.md: golden review rule covers all goldens (D20).
- T20: every
#[ignore]test runs somewhere or is deleted.
3. Tests
Stage 0 (--run-until stage0)
- T1 lowering golden
- T2 PromQL ≡ SQL ≡ MetricsQL (needs D11)
- T3 equivalence within one language
- T6 group properties
Memo (asap-types::stages::logical::memo)
- T4 schema preservation
- T5 memo consistency, incl. no orphan groups
- T25 schema across stages (Stage 2/3 parts: PRs 5–6)
Stage 1
- T7 Pass 1 candidates
- T8 per-rule positive and negative cases (PR 4; write as pending first)
- T24 rule removal (PR 6)
Stage 2
- T10 physical candidates (PR 5)
Stage 3 (all PR 6)
- T22 sharing summary
- T11 search differential
- T9 missed reuse (
planner_fuzz) - T13 accuracy monotonicity
- T15 order invariance
- T16 separability
- T17 renaming invariance (Stage 0 part can run now)
- T19 plan cost relations
- T26 plan-changing inputs
- T23 corpus snapshots
Accuracy (asap-accuracy)
- T12 propagation (after the memo rewrite of propagation)
- T14 composition algebra
4. Follow-ups
- Migrate corpus snapshots into the golden runner (D27).
- Clean up existing tests per the audit: delete stale, dedupe, migrate.
- Decide which
integration-testsfiles migrate to JSON goldens. - Rule-firing coverage and mutation testing (
cargo-mutants). - Revisit deferring execution oracles (~33 ledger bugs need one).
Metadata
Metadata
Assignees
Labels
No labels
Activity
zzylol commentedon Nov 16, 2026
D2 (12 significant digits) may churn goldens.
While porting the stage printer to the #1207 memo (#1239), reordering code alone, with no change to the cost model or the plan, moved plan costs by up to 3e-11 relative. That is a change in the 11th significant digit, so a 12-digit text comparison would fail on a refactor that changes nothing.
Illustration (not a measured case): a cost of 0.0085 printed to 12 significant digits is
0.00850000000000; the same cost 3e-11 lower prints as0.00849999999974.Two options:
|a − b| ≤ 1e-9 · max(|a|, |b|), and keep 12 digits in the file for reading.Option 1 keeps the files exact and avoids a near-boundary rounding flip that option 2 still allows. #1239 follows D2 as written (12 digits) for now.
milindsrivastava1997 commentedon Nov 16, 2026
Thanks. Let's do both. Option 1 + print 5 significant digits (based on number of significant digits in AWS cloud pricing)
docs(proposals): D2 prints 5 significant digits, compares with a tole…
milindsrivastava1997 commentedon Nov 17, 2026
Execution plan
State. #1207 PRs 4–7 are on main, so most "Blocked by" lines in the doc are stale. Still open: PR 8 (constraint (K); use
--ignore-accuracy-guaranteesuntil then) and PR 9 / #1239 (JSON printer). Stage 3 already solves withthreads = 1and zero gap; D18 still needs the tie-break and a time-limit check.Critical path: #1239 → golden runner → case writing.
Constraints
-j.asap_planner.rsis rewritten in refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239: anything touching it waits.Wave 0 (now, one PR)
case.tomlspec (fields, types, pending, defaults, assertions); refresh "Blocked by" lines. Every later agent builds against it.Wave 1 (parallel, on main, disjoint crates)
devtools/tests):libtest-mimic, defaults, pending cases, D15 assertions; a few Stage 0 cases pending on refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239.asap-types): T4, T5 incl. orphan groups,InvariantViolation; exit-code wiring after refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239.proptest(asap-accuracy).time_series_metriccatalog field (frontend-sql).Wave 2 (after #1239 and the runner)
Sequential (shared code):
Ruleenum,--only-rules,--disable-rule(D9).planner_fuzzwrites case directories (D26).Parallel (each agent adds files in its own directory):
docs/design_docs/concepts, not the rule code, so cases check the spec.Wave 3
Orchestration
Rule): a separate session each, in its own worktree, briefed with doc sections, files, done criteria and test commands.