Repository navigation
docs(generation-benchmark): describe the generation experiment - #160
Draft
bcleary-nv wants to merge 13 commits into
Draft
bcleary-nv wants to merge 13 commits into
bcleary-nv wants to merge 13 commits into
Conversation
7 tasks
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
mstaats-nvidia
force-pushed
the
feat/generation-benchmark-docs
branch
from
October 30, 2026 21:38
bf65b7a to
5a07a70
Compare
Clarify when scoring can omit executable inputs and document unsigned iteration. Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
mstaats-nvidia
force-pushed
the
feat/generation-benchmark-docs
branch
from
October 31, 2026 10:09
5a07a70 to
2fefc92
Compare
Signed-off-by: mstaats <mstaats@nvidia.com>
mstaats-nvidia
force-pushed
the
feat/generation-benchmark-docs
branch
from
October 31, 2026 11:33
2fefc92 to
d3038a6
Compare
Signed-off-by: mstaats <mstaats@nvidia.com>
…tifacts Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Record the stage flow, adapter contract, and benchmark files so the implementation can follow one design. Signed-off-by: Brian Cleary <bcleary@nvidia.com>
… paths Name the staged directories input/tasks and output/tasks-generated, define the actors once, and say the example generator config is what pins Codex. Signed-off-by: Brian Cleary <bcleary@nvidia.com>
Match the metric benchmark, which uses .yml for its YAML files. Signed-off-by: Brian Cleary <bcleary@nvidia.com>
mstaats-nvidia
force-pushed
the
feat/generation-benchmark-docs
branch
from
October 31, 2026 11:36
d3038a6 to
0ae9311
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
input/tasks/andoutput/tasks-generated/instead of subset notation2 commits on branch
feat/generation-benchmark-docs.Stacked on #96 (
feat/metric-benchmark-coverage).What changed
6 files. Design only. The driver and adapters are in #161.
labs/generation-benchmark/README.mdlabs/generation-benchmark/docs/architecture.mdlabs/generation-benchmark/docs/stages.mdlabs/generation-benchmark/docs/benchmark-format.mddraft_dirlabs/generation-benchmark/docs/tools.mdlabs/generation-benchmark/docs/assets/experiment-flow.svgrequest.jsoncarries the subset score and the staged pathsinput/tasks/,input/coverage/, andoutput/tasks-generated/.response.jsonreturns the new task ids and the suite digest. The reference adapter readsauthored/audit.mdandreport.jsonfrom the coverage work cell. The driver does not.G, A, and C are defined once. The pinned
generator_configis where C is configured. The example config is what pins Codex.Commit summary
Commits
4c2406d0840519Commit
4c2406d-- docs: keep subset notation out of the work-cell paths:generator_configis what pins Codexinput/tasks/andoutput/tasks-generated/Commit
0840519-- docs: describe the generation experiment:draft_dir, and the request/response adapter contractChecklist
AI-assisted draft.
Rebased onto the refreshed #96 after #157 merged; documentation patches unchanged.