FIX: warn instead of raising on reasoning-model token truncation#2166
Open
romanlutz wants to merge 13 commits into
Open
FIX: warn instead of raising on reasoning-model token truncation#2166romanlutz wants to merge 13 commits into
romanlutz wants to merge 13 commits into
Conversation
When a reasoning model hits max_completion_tokens, the API returns finish_reason='length' with empty visible content. Previously this raised EmptyResponseException (misleadingly reported as 'Status Code: 204') and triggered the 10x retry storm. _validate_response now logs a warning explaining that reasoning models consume tokens on hidden reasoning, and returns a graceful empty response (error='empty') instead of raising, so runs continue. Non-length empty responses still raise as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jbolor21
approved these changes
Jul 11, 2026
behnam-o
reviewed
Jul 13, 2026
…se target Mirror the OpenAIChatTarget truncation fix in OpenAIResponseTarget. When the Responses API returns status='incomplete' with reason='max_output_tokens', warn and preserve any completed output (reasoning, partial text) instead of raising a PyritException that halts the run. Falls back to a graceful empty text piece when no visible text was produced, and skips partial tool/function-call sections so an incomplete call cannot re-enter the agentic loop. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00b73356-ae65-4d83-8220-ebad37cc5850
…construction Address review feedback: _validate_response no longer returns a Message. It now only validates (warns and returns None on token-limit truncation, raises on genuinely empty responses so retries still fire). Building the graceful-empty or partial truncated Message moves into _construct_message_from_response_async in both OpenAIChatTarget and OpenAIResponseTarget, and the shared _validate_response return type narrows to None. Behavior is unchanged; responsibilities are cleaner. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00b73356-ae65-4d83-8220-ebad37cc5850
Reconcile the reasoning-model truncation fix with main's extraction of shared Chat Completions response parsing. The chat target keeps its truncation policy (warn on finish_reason='length', graceful empty on no content) layered over the shared validate/build helpers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00b73356-ae65-4d83-8220-ebad37cc5850
behnam-o
approved these changes
Jul 16, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00b73356-ae65-4d83-8220-ebad37cc5850 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00b73356-ae65-4d83-8220-ebad37cc5850
Mirror OpenAIResponseTarget by centralizing the token-limit truncation check in a _is_truncated_response predicate (keyed on finish_reason == 'length'), used by both _validate_response and _construct_message_from_response_async. build_empty_response_for_truncated_completion is now an unconditional builder, with the truncation decision owned by the target. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00b73356-ae65-4d83-8220-ebad37cc5850
Use the OpenAI SDK Response type and direct attribute access in the Responses API truncation helper so type checking can catch field changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00b73356-ae65-4d83-8220-ebad37cc5850
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00b73356-ae65-4d83-8220-ebad37cc5850
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00b73356-ae65-4d83-8220-ebad37cc5850
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Reasoning models spend completion tokens on hidden reasoning before emitting any visible output. When
max_completion_tokensis set too low, the API can returnfinish_reason="length"with empty visible content.OpenAIChatTarget._validate_responsetreated that empty content as a hard failure and raisedEmptyResponseException(surfaced with a misleading "Status Code: 204"), which also triggered the full retry loop even though retrying with the same token budget cannot succeed.Since a low token budget can be a deliberate configuration choice, this changes the behavior to warn rather than raise:
finish_reason="length", log a warning explaining that reasoning models consume tokens on hidden reasoning and thatmax_completion_tokensmay need to be increased.error="empty") so the run continues instead of raising and retrying.EmptyResponseExceptionas before.The final stored result for the empty case is unchanged from today (an
error="empty"piece). The difference is no exception, no retry storm, and a clear warning pointing at the token limit.Tests and Documentation
test_validate_response_success_lengthto assert the truncation warning is emitted when content is present.test_validate_response_length_empty_returns_empty_responseasserting that a truncated-and-empty response returns aMessagewitherror="empty"and raises nothing.tests/unit/prompt_target/target/test_openai_chat_target.pysuite passes (101 tests).No documentation changes needed. JupyText not run (no doc or notebook changes).