Skip to content

fix: derive an output ceiling from the window and answer a refused dial as not_ready - #10

Merged
pengpeng merged 2 commits into
mainfrom
fix/output-budget-and-relaunch-503
Sep 26, 2026
Merged

pengpeng merged 2 commits into
mainfrom
fix/output-budget-and-relaunch-503

Conversation

@pengpeng

Copy link
Copy Markdown
Member

Why

A local chat card reporting max_output_tokens equal to its whole window (131072 of 131072) led clients to reserve the entire context for the reply, which the KV budget refused as kv_budget_exhausted. Separately, each card edit relaunches the engine, and during that gap a request that found nothing listening surfaced as 502 upstream_unreachable, which callers treat as a hard failure.

What changes

  • Output ceiling. In chat mode, when max_output_tokens is unset or not smaller than context_size, it is derived as a quarter of the window, on boot and on PUT /api/model-spec, alongside the existing context_size derivation. A ceiling the author set below the window is kept.
  • Relaunch gap. A refused dial (*net.OpError with Op == "dial") is now 503 not_ready with Retry-After: 5, matching the readiness gate. Nothing reached the engine, so resending is safe. A connection the engine accepted and then dropped is still 502 upstream_unreachable.

Compatibility

  • Cards with an explicit ceiling below the window are unchanged.
  • Clients that retried on 502 now see a 503 with Retry-After, which is the documented retryable shape.
  • The default slot count is not changed here: the single slot observed came from the model application's own card (-np 1).

Verification

  • go vet ./... and go test ./... pass.
  • New and updated tests: TestSyncMaxOutputTokensFromContext, TestProxy_UpstreamErrorJSONEnvelope (now 503 not_ready with Retry-After), and TestProxy_DroppedConnectionStaysBadGateway.
  • CHANGELOG updated under Unreleased.

Made with Cursor

…al as not_ready

Co-authored-by: Cursor <cursoragent@cursor.com>
if syncMaxOutputTokensFromContext(cfg) {
changed = true
slog.Info("model-spec max_output_tokens derived from context_size",
"context_size", cfg.Spec.ContextSize, "max_output_tokens", cfg.Spec.MaxOutputToks)
if syncMaxOutputTokensFromContext(cfg) {
changed = true
slog.Info("model-spec max_output_tokens derived from context_size",
"context_size", cfg.Spec.ContextSize, "max_output_tokens", cfg.Spec.MaxOutputToks)
Co-authored-by: Cursor <cursoragent@cursor.com>
@pengpeng
pengpeng merged commit d185e46 into main Sep 26, 2026
14 checks passed
@pengpeng
pengpeng deleted the fix/output-budget-and-relaunch-503 branch September 26, 2026 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants