Repository navigation
follow-up: Pre-publish benchmark gate's diffImpact-latency check flakes on unrelated PRs (mirrors #2436's Full build / 1-file rebuild noise) #2587
Description
Summary
tests/benchmarks/regression-guard.test.ts's query-benchmark suite (diffImpact latency, baseline consistently 10.1ms per the recorded dev vs 3.17.0 comparison) has flaked repeatedly across multiple, completely unrelated PRs in a single fixer-batch session on 2026-08-19:
| PR | Change | diffImpact result |
|---|---|---|
| #2581 (issue #2526) | .claude/hooks/guard-git.sh bash hook |
10.1 → 23.2ms (round 1), also flaked separately on No-op rebuild |
| #2582 (issue #2530) | src/domain/analysis/exports.ts query logic |
10.1 → 18/22.0/21.8ms across 3 separate rounds |
| #2585 (issue #2558) | .claude/hooks/normalize-ifs.mjs bash hook |
10.1 → 21.8ms (round 1), also flaked separately on No-op rebuild |
| #2586 (issue #2560) | src/extractors/javascript.ts / crates/codegraph-core/src/extractors/javascript.rs — one new match arm per engine |
10.1 → 21.0ms, then 10.1 → 22.2ms (two consecutive failures on the SAME metric) |
The baseline (10.1) is byte-identical across every single occurrence — strongly suggesting it's a static recorded value from benchmark history (not re-measured each run), while the "dev" measurement is a live, per-CI-run query-latency sample landing consistently in the 18-23ms range regardless of what the PR actually changes (a bash hook script never invoked during buildGraph, TS query-layer code, and now Rust/TS extractor code — none of which share any plausible causal path to diffImpact's query-time cost).
Why this needs its own follow-up
This is the same underlying class of problem #2436/#2550 already documented for the Full build/1-file rebuild metrics (cross-invocation CI-runner variance that an in-process median can't smooth out) — but for a different metric (diffImpact latency, a query-benchmark metric, not a build-benchmark one) that isn't yet covered by any existing mitigation.
Suggested fix
Mirror the existing KNOWN_REGRESSIONS stopgap already used for 3.16.0:Full build / 3.16.0:1-file rebuild (see tests/benchmarks/regression-guard.test.ts) with an equivalent entry for the current version's diffImpact latency query metric, until the deeper cross-invocation noise problem tracked in #2550 is actually resolved.
Related
- Pre-publish benchmark gate fails non-deterministically on unrelated PRs #2436 (original benchmark-gate non-determinism investigation)
- follow-up: benchmark gate noise is cross-invocation (CI runner variance), not within-process — needs job-level replication or dedicated runner (#2436) #2550 (cross-invocation noise needs job-level replication or a dedicated runner — root cause, still open, needs infra I can't provision)
Activity