Repository navigation
feat(fidelity): measure what converting a session into a harness loses - #94
Draft
malihashar wants to merge 2 commits into
Draft
malihashar wants to merge 2 commits into
malihashar wants to merge 2 commits into
Conversation
The two functions that parse or render a harness's native text by HarnessId lived privately in src/wasm.rs, so only the WASM bindings could use them. Move them to a new always-compiled txcript::dispatch module and make wasm.rs call them. No behaviour change. This lets other code (such as a conversion report) render a transcript into any harness's format and read it back without going through WASM. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Add txcript::fidelity. assess(target, source) renders a transcript into a harness's native text, reads it back, and compares the two, so it reports exactly what that codec keeps, changes, loses, or makes up. check_all runs it for every harness that can render. Losses are grouped as content (text, tool calls and results, images), accounting (usage, stop reasons, per-message model, thinking signatures) and metadata (title, model, branch, cwd). Blocks are compared as a flat stream, not message by message, because targets split and merge turns. JSON re-spacing and regenerated ids are not counted as loss. The result is serialisable. The measurement is codec-level and does not include what a store adds, such as artifact files. Tests: unit tests for each category, one golden table of what every harness loses on a rich session (also published in docs/formats), a proptest that no target panics and that Simple reports nothing, and the docs table is checked against the codecs so it cannot drift. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In short
txcript checkis a new command that tells you what converting a session to another agent would lose, before you do it. This is step 2 of 3: the library that does the measuring (txcript::fidelity). It adds no command yet, and nothing visible changes for users.Part 2 of 3 for #92. It builds on #93.
How to review this PR
This branch contains the commit from #93 too, until that PR is merged. Please review only the last commit,
feat(fidelity): measure what converting a session into a harness loses. After #93 merges, this PR will show only its own changes.What this PR adds
A new library module,
txcript::fidelity. It answers one question: if I convert this session into harness X, what will I lose?A
Reportlists each difference as lost, changed or synthesized (made up by the target), grouped by what it is:The types are serializable, and nothing in the module does any I/O or needs a new dependency.
How it measures
dispatchfrom refactor: move the harness text dispatch out of the wasm module #93).So it reports exactly what the real codec keeps, and it cannot drift when a codec changes. Some choices that keep it honest:
The measurement is "codec-level": it does not include what a store adds on top, such as artifact files.
Tests
docs/formats/README.md, and a test fails if the two disagree, so the docs cannot drift. When a codec fix changes a result, the table is updated on purpose.cargo fmtandcargo clippy(0 warnings, with and without default features) are clean.What you should know
check_allreturns oneResultper harness, so one codec failing does not hide the others.What I could not test
Real agent resumes, OpenCode and Cowork writes (I have no
opencodebinary or Cowork account here), and the real WASM build.