Repository navigation
fix(llm): let the task schema disambiguate JSON candidates in model prose - #610
Merged
Merged
Conversation
…rose
Structured output already recovers a JSON object wrapped in prose, but only
when exactly one syntactically valid top-level object is present. A model that
drafts its answer inside a reasoning preamble emits a second valid object, and
even the sentence "return JSON, not {}" is enough, so recovery gave up and a
validation attempt was spent on text that could never parse.
Extraction now returns every balanced candidate and the task's own decoder
chooses among them: exactly one passing the schema is accepted, zero or several
keep the existing error. A candidate that fails DisallowUnknownFields, or
carries the wrong thread or schema version, was never a possible answer, so
this changes which bytes reach the decoder without changing what the decoder
accepts.
monit-reviewer
approved these changes
Oct 9, 2026
monit-reviewer
left a comment
There was a problem hiding this comment.
Automated PR Review
Reviewed commit: 96512a381c81
Profile: open-cli-collective-review - Posting as: monit-reviewer
Summary
| Reviewer | Findings |
|---|---|
| go:implementation-tests | 0 |
Reviewer Coverage
Complete coverage details are retained in the local coverage.json artifact.
go:implementation-tests— complete (broad); scope: 4 files; examples:internal/llm/adapter.go,internal/llm/adapter_test.go,internal/llm/extract.go,internal/llm/extract_test.go; body-inspected: 4 files; examples:internal/llm/adapter.go,internal/llm/adapter_test.go,internal/llm/extract.go,internal/llm/extract_test.go; skipped: none; context (not coverage): 1 file; examples:internal/llm/contracts.go; constraints: Reviewed pinned source and tests only; build, vet, and tests were not executed with the available read-only tools.
0 PR discussion threads considered. 0 summarized; 0 resolved.
Completed in 34s | $0.03 | openai-codex/gpt-6.1-sol | cr 0.10.328
| Field | Value |
|---|---|
| Model | openai-codex/gpt-6.1-sol |
| Reviewers | go:implementation-tests |
| Engine | pi_rpc · openai-codex/gpt-6.1-sol |
| Reviewed by | cr · monit-reviewer |
| Duration | 34s wall · 24s compute |
| Cost | $0.03 |
| Tokens | 12.0k in / 286 out |
Per-workstream usage
orchestrator-selection— openai-codex/gpt-6.1-sol- In: 5.6k
- Out: 133
- Cache read: 0
- Cache create: 0
- Cost: $0.01
- Duration: 5s
go:implementation-tests— openai-codex/gpt-6.1-sol- In: 2.7k
- Out: 103
- Cache read: 22.9k
- Cache create: 0
- Cost: $0.01
- Duration: 16s
orchestrator-rollup— openai-codex/gpt-6.1-sol- In: 3.7k
- Out: 50
- Cache read: 0
- Cache create: 0
- Cost: $0.01
- Duration: 3s
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #608
Problem
Structured-output decoding already recovers a JSON object from surrounding prose, via
extractSingleJSONObject. It requires exactly one syntactically valid top-level object, deliberately, so ambiguous output is never guessed at.A reasoning preamble routinely breaks that. Running the resident extractor against representative inputs:
no braces heredraft: {"decision":"skip"}I must return JSON, not {}.{}is a candidateThe budget is two validation attempts and an exhausted task is blocking, so a model that drafts before answering loses half its retry budget systematically rather than occasionally.
Change
Extraction returns every balanced valid candidate, and the task's own decoder picks among them: exactly one passing the schema is accepted, zero or several keep the existing error. A lone failing candidate still reports its own schema error, since that describes the real violation.
This cannot loosen validation. A candidate failing
DisallowUnknownFields, or carrying the wrongthread_idorschema_version, was never a possible answer — only the bytes reaching the decoder change, never what the decoder accepts. Trailing-data rejection is unaffected: each candidate is a single balanced object, and the raw-text check still runs first.Evidence
Every behavior change mutation-checked by reverting it alone:
recovers_the_schema-valid_object_when_the_preamble_drafts_anotherpassed >= 1rejects_two_schema-valid_objectsi = endadvanceTestExtractJSONObjects/nested_objects_and_arraysjson.Validprose_braces_alongside_one_valid_object,valid_object_nested_in_invalid_prose_bracesgo build ./...,go vet ./internal/...andgo test -count=1 ./...are clean.Note on scope
The originally filed issue claimed no extraction existed; that was wrong and the issue has been corrected. A truncated response remains unrecoverable by extraction and is the retry path's job, not this one's.