Skip to content

Testing infrastructure: tracking #1354

Open

Description

@milindsrivastava1997

Design: docs/design_docs/proposals/testing-infrastructure/README.md
(tests T1–T26, decisions D1–D28), added in #1353. This issue tracks the work; the doc holds
the reasoning.

1. Hand-offs to #1207 PRs (decide now, built there)

  • PR 6 solver setup follows D18: deterministic solve (one thread, fixed
    seed, no time limit, tie-break on canonical ids) for small cases.
  • PR 9 JSON printer follows D1–D4 and D28: canonical group ids, 12
    significant digits, no timings or raw ids, full memo per step, sharing
    summary as its own Stage 3 key.

2. Infrastructure (can start now unless noted)

  • Golden runner: libtest-mimic, one case per directory under
    tests/goldens/, case.toml, UPDATE_SNAPSHOTS=1, prints the exact
    command on failure (D5–D8, D25). Pending cases (D22).
  • case.toml assertions: same_group, different_group,
    rule_fires, rule_does_not_fire (D15).
  • Relations module; runner re-plans every case shuffled and renamed
    (D16). Blocked by PR 6 for Stage 3.
  • Rule enum in asap-types; candidates record their rule;
    --only-rules, --disable-rule (D9, D13).
  • Memo invariants always on; InvariantViolation with its own exit
    code (D12, D14).
  • time_series_metric field on --sql-catalog-table (D11).
  • planner_fuzz writes a case directory per finding (D26); fixed-seed
    run on every PR (D16).
  • proptest in asap-accuracy (D17).
  • CI runs all tests in release (D19).
  • AGENTS.md: golden review rule covers all goldens (D20).
  • T20: every #[ignore] test runs somewhere or is deleted.

3. Tests

Stage 0 (--run-until stage0)

  • T1 lowering golden
  • T2 PromQL ≡ SQL ≡ MetricsQL (needs D11)
  • T3 equivalence within one language
  • T6 group properties

Memo (asap-types::stages::logical::memo)

  • T4 schema preservation
  • T5 memo consistency, incl. no orphan groups
  • T25 schema across stages (Stage 2/3 parts: PRs 5–6)

Stage 1

  • T7 Pass 1 candidates
  • T8 per-rule positive and negative cases (PR 4; write as pending first)
  • T24 rule removal (PR 6)

Stage 2

  • T10 physical candidates (PR 5)

Stage 3 (all PR 6)

  • T22 sharing summary
  • T11 search differential
  • T9 missed reuse (planner_fuzz)
  • T13 accuracy monotonicity
  • T15 order invariance
  • T16 separability
  • T17 renaming invariance (Stage 0 part can run now)
  • T19 plan cost relations
  • T26 plan-changing inputs
  • T23 corpus snapshots

Accuracy (asap-accuracy)

  • T12 propagation (after the memo rewrite of propagation)
  • T14 composition algebra

4. Follow-ups

  • Migrate corpus snapshots into the golden runner (D27).
  • Clean up existing tests per the audit: delete stale, dedupe, migrate.
  • Decide which integration-tests files migrate to JSON goldens.
  • Rule-firing coverage and mutation testing (cargo-mutants).
  • Revisit deferring execution oracles (~33 ledger bugs need one).

Activity

  1. zzylol commented on Nov 16, 2026

    @zzylol
    Contributor

    D2 (12 significant digits) may churn goldens.

    While porting the stage printer to the #1207 memo (#1239), reordering code alone, with no change to the cost model or the plan, moved plan costs by up to 3e-11 relative. That is a change in the 11th significant digit, so a 12-digit text comparison would fail on a refactor that changes nothing.

    Illustration (not a measured case): a cost of 0.0085 printed to 12 significant digits is 0.00850000000000; the same cost 3e-11 lower prints as 0.00849999999974.

    Two options:

    1. Compare costs (and other floating-point outputs) with a relative tolerance in the snapshot check, for example |a − b| ≤ 1e-9 · max(|a|, |b|), and keep 12 digits in the file for reading.
    2. Print fewer digits, for example 10, which stays above the observed drift.

    Option 1 keeps the files exact and avoids a near-boundary rounding flip that option 2 still allows. #1239 follows D2 as written (12 digits) for now.

  2. milindsrivastava1997 commented on Nov 16, 2026

    @milindsrivastava1997
    CollaboratorAuthor

    Thanks. Let's do both. Option 1 + print 5 significant digits (based on number of significant digits in AWS cloud pricing)

  3. milindsrivastava1997 commented on Nov 17, 2026

    @milindsrivastava1997
    CollaboratorAuthor

    Execution plan

    State. #1207 PRs 4–7 are on main, so most "Blocked by" lines in the doc are stale. Still open: PR 8 (constraint (K); use --ignore-accuracy-guarantees until then) and PR 9 / #1239 (JSON printer). Stage 3 already solves with threads = 1 and zero gap; D18 still needs the tie-break and a time-limit check.

    Critical path: #1239 → golden runner → case writing.

    Constraints

    Wave 0 (now, one PR)

    • Exact case.toml spec (fields, types, pending, defaults, assertions); refresh "Blocked by" lines. Every later agent builds against it.

    Wave 1 (parallel, on main, disjoint crates)

    Wave 2 (after #1239 and the runner)

    Sequential (shared code):

    • Rule enum, --only-rules, --disable-rule (D9).
    • Shuffle and rename relations in the runner (D16).
    • planner_fuzz writes case directories (D26).

    Parallel (each agent adds files in its own directory):

    • T1/T6 Stage 0 · T7 Pass 1 · T2/T3 cross-language
    • T8, one agent per rule (rollup, topk, chains), briefed from docs/design_docs/concepts, not the rule code, so cases check the spec.

    Wave 3

    • Stage 2/3 cases (T10, T22, T26), D18 tie-break.
    • Corpus snapshots into the runner (D27); CI in release (D19).
    • Existing-test cleanup per the audit (parallel; deletions).

    Orchestration

    • Long implementation streams (runner, invariants, Rule): a separate session each, in its own worktree, briefed with doc sections, files, done criteria and test commands.
    • Short fan-out (case writing, independent review): subagents from one coordinating session.
    • PRs say "Part of Testing infrastructure: tracking #1354".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions