Skip to content

Fix stale tier-xhard grader hash and guard locked fixtures - #936

Merged
TheGreatAxios merged 1 commit into
mainfrom
cl-7776-tier-xhard-hash-guard
Sep 12, 2026
Merged

Fix stale tier-xhard grader hash and guard locked fixtures#936
TheGreatAxios merged 1 commit into
mainfrom
cl-7776-tier-xhard-hash-guard

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • The tier-xhard grader pins the fixture's current hash instead of a stale one
  • A hash mismatch reports a broken case when the workdir matches the fixture at setup, and an agent edit otherwise
  • A locked-fixtures guard test fails any case whose TEST_SHA literal diverges from its fixture file

Pre-2026-09-11 tier-xhard results for this case are void: the grader fails every run at setup regardless of agent behavior.

Verification

  • bun test evals/capability/locked-fixtures.test.ts passes, and fails naming tier-xhard when the literal is reverted
  • bun test ./evals ./scripts passes (140 pass, 0 fail)
  • bun run check passes (6980 pass, 0 fail)
  • The grader messages exercise the real verify.sh in a scratch workdir across the pristine, agent-edited, and stale-literal paths

Fixes CL-7776

The tier-xhard grader pinned a hash its fixture no longer has, so every run failed at setup and blamed the agent for an edit it never made.
@linear-code

linear-code Bot commented Sep 12, 2026

Copy link
Copy Markdown

CL-7776

@TheGreatAxios
TheGreatAxios merged commit 8f1413f into main Sep 12, 2026
9 checks passed
@TheGreatAxios
TheGreatAxios deleted the cl-7776-tier-xhard-hash-guard branch September 12, 2026 05:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant