feat(e2e-harness): bucket metadata-hash mismatches as promptChanged in the drift differ - #951
Merged
Conversation
…n the drift differ The aimock recorder stamps fixtures with systemHash/toolsHash fingerprints of the system prompt and tool definitions that produced each recording, but the structural differ ignored them — so a red weekly drift run could not distinguish "our prompt moved underneath the fixture" from model drift. diffFixtures now compares committed vs recorded hashes when BOTH sides carry one and reports mismatches in a new DriftReport.promptChanged category, independent of `changed`. Entries without metadata (pre-stamp fixtures) are never reported. The differ stays advisory: drift.ts exit codes are unchanged; its stderr summary gains a promptChanged count. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
blove
enabled auto-merge (squash)
September 2, 2026 04:30
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
metadata: { systemHash, toolsHash }fingerprints of the system prompt and tool definitions that produced each recording (11 committed entries carrysystemHash, 8toolsHash), but the structural differ ignored them.diffFixturesnow compares committed vs recorded hashes and buckets mismatches into a newDriftReport.promptChangedcategory, separate fromchanged— so a red weekly drift run can say "our prompt moved underneath the fixture" instead of misattributing it to model drift.promptChangedandchangedare independent — a pair can appear in both.drift.tsexit codes are unchanged; its stderr summary line gains apromptChangedcount.Testing
drift-lib.spec.ts: matching hashes, changedsystemHashonly, changedtoolsHashonly, absent metadata on either side (not reported), and structural + prompt drift reported independently.nx test e2e-harnessgreen (16/16).drift.tson the realc-subagents.jsonfixtures — self-vs-self reportspromptChanged=0; a copy with one mutatedsystemHashreportspromptChanged=1(systemHash: 03217c65 -> deadbeef),changed=0, exit 0.🤖 Generated with Claude Code