Redesign 59 structurally-broken challenges + fix runner repeatability - #3
Open
jobordu wants to merge 3 commits into
Open
Redesign 59 structurally-broken challenges + fix runner repeatability#3jobordu wants to merge 3 commits into
jobordu wants to merge 3 commits into
Conversation
BENCH-111/112/113/115 mutated src/*.js files that don't exist in QGSD (file-modify no-op → unmeasurable). Redesigned as file-create c_to_r challenges following the verified BENCH-188 pattern: create an untraced bin/rf-*.cjs utility with no @Req annotation, which the code→requirements layer flags as a coverage gap. Verified: BENCH-111 PASS (c_to_r residual 124→125, mutation detected). BENCH-114 left untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phantom-file challenges (mutated src/*.js, Dockerfile, etc. absent from QGSD) and t_to_c/f_to_c-only challenges (layers skipped under --fast) were structurally unmeasurable. Redesigned all 55 to the verified BENCH-188 c_to_r pattern: file-create an untraced bin/rf-*.cjs utility. Files: 02,03,04,07,13,14,15,16,17,18,26. All 230 challenges schema-valid; all file-create target_files unique. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…enges
The snapshot model runs solve in place against the project. restoreSnapshot
only cleaned .planning/formal, so the 125 file-create challenges (116 into
bin/) left their created files behind — on a repeat run the file already
exists, the c_to_r residual doesn't increase, and the challenge falsely fails.
createSnapshot now records every pre-existing file under the create-guard
dirs (bin, test, src, hooks, templates, scripts, .planning); restoreSnapshot
deletes anything a challenge added. Snapshot shape is now {content, guardPaths}.
Also snapshot .jsonl so the solver's trend log is restored — runs leave the
project pristine.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
jobordu
added a commit
that referenced
this pull request
Jul 2, 2026
… (verified +5) (#8) * docs(fixtures): document the repair-validation taxonomy (live-solve capability map) Records what a live solve (--live / RUN_LIVE_SOLVE) can and cannot prove, validated against @nforma.ai/nforma@0.43.1 by reading its layer handlers + a full 20-min live solve: - code defects: nf-solve is residual-driven, doesn't edit source — repair proven via a stub code-fixer SUT, not a live solve. - mechanically-automatable layers (f_to_t): nf-solve spawns a generator — closeable headlessly. - manual-modeling layers (r_to_f): handler only emits "manual modeling required" and writes nothing; closing means authoring a formal model (the close-formal-gaps LLM skill, not a pinnable binary). A live solve on req-coverage-gap correctly leaves the gap at 1 — SUT behavior, not a harness failure. These stay detection-only in CI. Conclusion: gate CI on detection+reproduce (deterministic, pinnable); treat --live repair as a separate, layer-dependent capability check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * hyp(harness): revert mutation target file between challenges Hypothesis: many low category scores (reverse-flow 26%, tests 12.5%, code 11%) are measurement artifacts — createSnapshot/restoreSnapshot only cover .planning/formal + docs, so file-create/modify mutations to bin/src/tests persist across challenges. A persisted file makes a later file-create a no-op (post==pre → false "not detected") and pollutes subsequent baselines (110 leftover bench-* files observed). Fix: capture challenge.mutation.target_file pre-state (content or absence) after createSnapshot; restore/delete it in finally alongside restoreSnapshot. Gate: nf-benchmark unit suite stays 98 pass / 0 fail. Empirical challenge verification follows in subsequent commits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * hyp(harness): extract mutation-revert into tested runner helpers Refactor hyp #1's inline capture/revert into captureMutationTarget / revertMutationTarget in lib/runner.cjs (exported), and add fast unit tests (test/mutation-revert.test.cjs) covering file-create (deleted on revert), file-modify (restored to exact bytes), and null/missing target (no-op). This makes the pristine-tree invariant unit-testable (seconds, no solve) so it is protected against regression without a full corpus run. Gate: nf-benchmark unit suite 98 → 101 pass / 0 fail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * hyp(harness): revert mutation target in the PARALLEL worker path too The worker thread has its own inline challenge loop (not runChallengeSerial), and each worker reuses ONE isolated root across its whole chunk — so hyp #1's serial-only revert left --parallel runs still polluting within a chunk. This is the path the full-corpus baseline used (--parallel 4), so its low reverse-flow/ code scores were partly self-inflicted. Apply captureMutationTarget/revertMutationTarget in the worker loop's snapshot/finally, mirroring the serial path. Gate: nf-benchmark unit suite 101 pass / 0 fail (no regression). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * hyp(scorer): file-presence detection signal for reverse-layer orphans Reverse-layer residual COUNTS (c_to_r/t_to_r/d_to_r) frequently do not increment for a single injected orphan — the untraced list is aggregated and truncated (untraced_modules_truncated) — so count-only detection has false negatives even though the created file IS surfaced in the solver's post analysis (verified: an injected bin/bench-*.cjs appears in c_to_r evidence). Add a complementary signal to scoreDetection: for file-create/file-rename mutations, if the target_file appears in the POST residual_vector but NOT the PRE one, treat it as detected. Sound because a created file cannot exist in the pre analysis; scoped to create/rename so a pre-existing modified file can never trigger it spuriously. OR-ed with the count check, so it only ADDS correct detections and cannot regress count-based passes. Gate: nf-benchmark unit suite 101 → 106 pass / 0 fail (5 new synthetic-vector tests: count-flat+file-present→pass, count-up→count method, file-absent→fail, file-modify→no false pass, count-based regression still passes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * hyp(harness): neutralize pre-existing file-create targets before baseline solve Root cause of file-create detection failures (BENCH-093/094 etc.): the SUT (nForma origin/main) has 109 bench-*.cjs fixtures ACCIDENTALLY COMMITTED into bin/ (benchmark mutations auto-committed into the product repo). So a challenge's file-create target already exists on a clean checkout → applyMutation's create is a no-op → post == pre → neither the count nor the file-presence signal can fire. Fix: neutralizeCreateTarget() removes a pre-existing file-create/rename target after captureMutationTarget (which saved it) and before the baseline solve, so pre genuinely lacks the file. revertMutationTarget restores the committed file afterward — no permanent SUT change. Combined with hyp #3 (file-presence), a created orphan now appears in post-not-pre and is detected. Gate: nf-benchmark unit suite 106 → 108 pass / 0 fail (2 new: removes stray committed create target + restores on revert; no-op for file-modify). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: catalog 54 unwinnable-by-construction detection challenges Autoresearch finding: 54 of 230 challenges (~23%) reference layers nf-solve never produces (c_to_e×22, f_to_f×14, c_to_t×13, f_to_g×6, l2_to_l3×3), so they can NEVER pass regardless of solver capability — a dominant driver of the low corpus score (formal-models 14, config-hooks 16, code 12, convergence 9, cross-layer 3). Handoff list (id + bad layer + suggested real-layer remap) for the challenge-redesign effort (see benchmark/redesign-broken-challenges). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add second defect class (real-but-wrong expected layer, e.g. BENCH-061) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #2. Fixes the structurally-broken challenges found while triaging the
benchmark toward an honest 100%, and a runner bug that made file-create
challenges non-reproducible.
The problem
Triage of all 230 challenges found ~59 that no solver could ever pass:
12–18,26):file-modifyonsrc/database.js,src/components/*.js,Dockerfile, etc. — files that don't exist in the targetproject (QGSD, a planning/formal-methods CLI).
file-modifyon a missing fileis a silent no-op → no residual change → unmeasurable.
03,04,07): only targett_to_c/f_to_c,which the benchmark's
--fastinvocation deliberately skips → residual isalways
-1→ undetectable.Changes
59 challenges redesigned to the verified
c_to_rfile-createpattern(
BENCH-188): create an untracedbin/rf-*.cjsutility with no@reqannotation, which the code→requirements layer flags as a coverage gap. Each new
file has a unique name, distinct realistic JS content, and honest title/tags.
Verified:
BENCH-111scores PASS (c_to_rresidual 124→125, mutation detected).Runner repeatability fix (
lib/runner.cjs): the snapshot model runs solvein place, but
restoreSnapshotonly cleaned.planning/formal— so the 125file-createchallenges (116 intobin/) left their files behind. On a repeatrun the file already exists, the residual doesn't increase, and the challenge
falsely fails.
createSnapshotnow records pre-existing files in thecreate-guard dirs;
restoreSnapshotdeletes anything a challenge added..jsonlis also snapshotted so the solver's trend log is restored — runs nowleave the project pristine.
Verification
npm run validate).file-createtarget_files unique.BENCH-111PASS, twice, leaving QGSD with 0 dirty files.A full re-measurement (multi-hour serial run) is the follow-up step to get the
new baseline score.
🤖 Generated with Claude Code