Skip to content

docs: describe the verifier's result files and eval.json - #79

Open
reacher-z wants to merge 1 commit into
mainfrom
docs/results-reference
Open

reacher-z wants to merge 1 commit into
mainfrom
docs/results-reference

Conversation

@reacher-z

Copy link
Copy Markdown
Collaborator

Closes #16

New docs/results.md:

  • every file the verifier publishes (reward.txt, eval.json, case-results.json, events.jsonl, results.junit.xml, case-manifest.json, build.log, runtime.log, progress.jsonl, failures/);
  • every eval.json field, including status (VALID_RUN, INVALID_RUN), scorable, reward, score100, rates, tie_break, counts, anticopy, code_redundancy and reason;
  • the tiers and levels (T1, T2 at L1–L3, T3) that classify cases without weighting them.

docs/running.md links it, and docs/README.md lists it.

running.md named reward.txt, eval.json and case-results.json, but not
the other files the verifier publishes (events.jsonl, results.junit.xml,
build and runtime logs, failure screenshots and heatmaps) or what the
eval.json fields mean. results.md lists each file and field, the
VALID_RUN and INVALID_RUN statuses, the tie-break order, and the tiers
and levels that classify cases without weighting them.
@reacher-z reacher-z added documentation Improvements or additions to documentation scoring How cases and rewards are counted labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation scoring How cases and rewards are counted

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document the fields of eval.json and the other verifier outputs

1 participant