Skip to content

fix(tasks): the execution error keeps a reader phrase in any case, so the portal keeps its 504/429 (#4352) - #4358

Merged
vybe merged 3 commits into
devfrom
AndriiPasternak31/issue-4352
Nov 11, 2026
Merged

vybe merged 3 commits into
devfrom
AndriiPasternak31/issue-4352

Conversation

@AndriiPasternak31

@AndriiPasternak31 AndriiPasternak31 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #4352

Journey Impact: none: an unforeseen exception's execution error keeps the status words its readers already relied on; no user journey changes

What

#4340 (#4284) stopped raw exception text from reaching 5xx answers. When an unexpected exception ends a task, the execution error now names only the exception class plus the "reader phrases" that callers match on (timed out → 504, at capacity → 429). It kept a phrase only when it appeared in exact lowercase. The readers that act on those phrases do not all match the same way:

Reader How it matches
sync /task (chat_execution_service._map_task_failure) case-sensitive
public chat (public_chat_service.raise_for_failed_public_result) case-sensitive
inline-auth MCP chat (mcp_auth_service.dispatch_chat) case-sensitive
client portal (client_portal/service.py::portal_chat) lowercases first
scheduler retry delay (scheduler/service.py::_maybe_schedule_retry) lowercases first, on 429 / rate limit / too many requests

So an exception saying Lock Timed Out or Agent At Capacity used to give the portal its 504 / 429, and since #4340 it got a 502. The scheduler's longer retry delay on rate-limit words was lost too, because those words were not passed on.

How

_exception_words (in task_execution_service.py) now finds each phrase the way each kind of reader does, and passes it on in a fixed spelling:

  • the text holds the lowercase phrase → the lowercase phrase (timed out);
  • only the lowercased text holds it → the title-case phrase (Timed Out);
  • otherwise nothing.

For every reader in the table above:

The second check is phrase in text.lower(), the same test the lowercasing readers run, so characters that lower() treats differently (ı, İ, ſ, K) give the same verdict here as there. Neither spelling comes from the exception, so its own letters (tImEd OuT, odd Unicode) never reach the 5xx answer: only the class name and these fixed words (#4284).

The helper has two callers, and both get the change:

  • the generic except Exception in execute_task (Task execution failed (… — details in backend logs))
  • the early-abort network message in _classify_timeout_failure (Task execution aborted after Ns of … allowed (ReadTimeout: Timed Out))

Decision: the scheduler's rate-limit words are added

429, rate limit and too many requests join the phrase list. The scheduler's doubled retry delay (RETRY-001: 2x, capped at 300s) works again for an unforeseen exception that carries them.

429 is passed on only as a number of its own: not the digits inside 14290, 0.429 or 429s. Otherwise a lock held by worker 14290 would read as a rate limit in the error and double the retry delay. This is the one place the error deliberately does not mirror a reader: the scheduler's own "429" in error check is a loose substring (tracked in #4360), and the error should not carry that looseness forward.

This was a close call. Against adding them:

For adding them:

Dropping them later is a one-line revert of the tuple. It also turns these test rows red, which would need updating: two-phrases, 429-sentence-end and the three scheduler rows (429, rate-limit, too-many-requests).

Other readers of this error, outside this fix

Three more places read the execution error's text. All three lost their words in #4340, not in this change, and none is a likely path for an unforeseen exception. This PR does not widen the phrase list for them:

  • Chat panel banner (src/frontend/src/components/ChatPanel.vue:224-228). It lowercases execution.error and shows "Subscription Usage Limit" on usage limit / rate limit / out of usage / out of extra usage. Because rate limit is now passed on, the banner comes back for that word. It does not come back for the other three.
  • Public chat refusal (public_chat_service.py:65,75, _TURN_REFUSED_MARKERS). This gives a 403 on the agent server's refusal sentences. Those sentences are not passed on.
  • Scheduler auth label (scheduler/service.py:1122,1530, _is_auth_failure). It reads the error, but only to label a log line.

Validation

  • tests/unit/test_4284_exception_text_sites.py + tests/unit/test_4284_no_exception_text_in_5xx.py: 221 passed. Before the change, 180 passed on dev.
  • Wider targeted set: the two bug: exception text in HTTPException detail across backend and agent-server routes (py/stack-trace-exposure siblings CodeQL does not flag) #4284 files plus the test files that pin the execution error text or drive these readers (test_2106_timeout_attribution, test_2789_subswitch_retry_budget, test_3340_stream_api_error_detail, test_3603_channel_failure_reply, test_3988_turn_limit_status, test_3993_no_retry_max_turns, test_ent279_secret_scrub, test_2320_portal_failed_turn_visibility, test_3114_pull_route_callers): 433 passed, 1 skipped in a fixed order. Also under shuffle seed 4358 (bug: exception text in HTTPException detail across backend and agent-server routes (py/stack-trace-exposure siblings CodeQL does not flag) #4284 files + 2320 + 3993): 278 passed.
  • The reader table runs one exception through the real execute_task and then through each of the four HTTP readers, including the real portal_chat. For every reader it asserts that the status for the original text equals the status for the class-only error, and that this status is the pinned value. The same check runs through the early-abort network message (httpx.ReadTimeout("… Read Timed Out") → portal 504; 503/502/502 elsewhere, as on the text). The scheduler check drives the real _maybe_schedule_retry with its clock pinned, including worker 14290 → the plain 60s delay.
  • Every answer is checked to carry no planted marker or token, and only the class plus reader phrases in one of their two fixed spellings.
  • A randomized property check (200,000 strings mixing phrases in random case with ı, İ, ſ, K, ß, digits and dots) found 0 cases where a reader's match on the error differed from its match on the text (for 429: from the whole-number rule), and 0 words passed on outside the fixed spellings. It was a local script, not committed.
  • Other shape (empty data): an exception with no text at all (RuntimeError()). The error is the class alone, and all four readers give their generic status (503 / 502 / 502 / 502) for both the text and the error. Passed.
  • ruff on the two files: the same 5 findings before and after, none on changed lines. mypy on task_execution_service.py: identical findings before and after.
  • tests/lint_sys_modules.py: no new violations.

Mutation

With the fix reverted (task_execution_service.py at the branch point on dev, new tests kept), 18 failed, 85 passed: the portal answered 502 where the original text gave 504 (capitalised-timeout) and 429 (capitalised-capacity); the scheduler's delay was 60s where the original text gave 120s (429, rate-limit, too-many-requests); the fixed-spelling row and the _exception_words rows; the early-abort message carried (ReadTimeout) with no Timed Out (all four readers' rows).

Against the previous revision of this PR (spelling copied from the exception, 429 as a substring), 6 red: the fixed-spelling row (tImEd OuT, AT CAPACITY reached the error), overlapping-candidates, the three 429-inside-a-number rows, and the worker 14290 retry-delay row.

Each part of the fix mutated on its own, and each caught:

  • title case swapped for upper case: 17 red (the exact-message rows)
  • the whole-number rule swapped for \b429\b: 1 red (429-in-a-decimal)
  • the whole-number rule swapped for a plain substring: 4 red (the three 429-inside-a-number rows and the retry-delay row)
  • scheduler words dropped: 5 red (two-phrases, 429-sentence-end and the three scheduler rows)

Unverified

  • No live stack: the readers were driven through their real functions with collaborators stubbed. No real portal, MCP or scheduled run hit this path end to end.
  • The three readers outside this fix (the chat panel banner, the public chat refusal, the scheduler auth label) were read in the source, not tested. Their losses date from test: a static guard keeps exception text out of 5xx answers, and the sites it found are fixed (abilityai/trinity#4284) #4340.
  • Local runs used Python 3.14.4 (the test venv). CI's 3.13 is the gate for the full suite.
  • No real exception is known to produce a mixed-case timed out / at capacity or rate-limit words in these paths today. The cases are planted.

… the portal keeps its 504/429 (#4352)

#4340 (#4284) reduced an unforeseen exception's text to its class plus the
reader phrases "timed out" / "at capacity", kept only on an exact lowercase
hit. The client portal lowercases the error before matching, so an exception
saying "Lock Timed Out" or "Agent At Capacity" lost its 504 / 429 there.

_exception_words now finds each phrase case-insensitively and passes it on as
the exception spelled it, preferring the lowercase spelling when the text has
it. The sync /task mapper, the public and MCP chats, the portal and the
scheduler therefore each match the error exactly when they matched the
original text. Nothing else of the text is passed on.

The scheduler's rate-limit words ("429", "rate limit", "too many requests"),
which double the RETRY-001 retry delay, join the reader phrases so that delay
works again for an unforeseen exception carrying them.

Tests: the reader table now drives the sync /task mapper, the public chat, the
inline-auth MCP chat and the real portal_chat; plus the scheduler's retry
delay, the early-abort network message, an exception with no text, and the
overlapping / dotless-i / long-s spellings.
# chats answer 429 on "at capacity" and 504 on "timed out", case and all; the
# client portal does the same after lowercasing. The scheduler lowercases and
# doubles its retry delay on "429", "rate limit" or "too many requests".
_READER_PHRASES = ("timed out", "at capacity", "429", "rate limit", "too many requests")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding the bare digit string "429" to _READER_PHRASES means any number containing 429 (id, port, byte count, hex, elapsed seconds) shows up in the stored execution error as a bare "429", which reads as an HTTP rate limit.

Example: RuntimeError("lock held by worker 14290") becomes Task execution failed (RuntimeError: 429 — details in backend logs). Operators, the UI and the rate-limit bucket in error analysis now see a bare 429 with no context. Consider matching it as a whole token (\b429\b) or keying on error_code instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0df905e. 429 is now passed on only as a number of its own: (?<![\w.])429(?!\w|\.\d). That is a little stricter than \b429\b, which still matches the 429 in 0.429.

RuntimeError("lock held by worker 14290") now gives Task execution failed (RuntimeError — details in backend logs), and a scheduled run's retry keeps the plain 60s delay. These rows pin it: 429-in-a-number, 429-in-a-decimal, 429-with-a-unit, 429-sentence-end, and test_execute_task_error_a_429_inside_a_number_keeps_the_plain_retry_delay.

Against the previous head, 6 tests go red. With \b429\b swapped in, 429-in-a-decimal goes red. With a plain substring, 4 go red.

_READER_PHRASES = ("timed out", "at capacity", "429", "rate limit", "too many requests")


def _phrase_as_said(phrase: str, text: str) -> str | None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design point: keeping each phrase in the case the exception used preserves the readers' disagreement about case rather than removing it. Lock Timed Out still answers 503 on /task and 502 on public/MCP chat but 504 on the portal, and the tests pin that as intended. Setting error_code (TIMEOUT/CAPACITY already exist, and the portal already checks them) in the generic except path, or lowercasing in all readers, would end the per-phrase parity tuning.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed here. #4352 is a regression from #4340: the portal stopped giving an answer it gave on the raw text. This PR puts each reader back to its pre-#4340 answer. The /task 503 vs portal 504 split on Lock Timed Out is older than #4340. It comes from the readers themselves: _map_task_failure checks "timed out" in result.error case-sensitively, while portal_chat lowercases first. The tests pin it as "same status as on the text", not as the end state we want.

Both ways of ending the split change answers the issue did not report as broken:

  • Lowercasing in every reader moves /task, public and MCP chat from 503/502 to 504 on these texts.
  • error_code in the generic except would classify an unknown exception by the words in its text. It also would not reach every reader. _map_task_failure branches on CAPACITY but not TIMEOUT (it still matches "timed out" in the text). The scheduler reads only the stored error string. A row carries a code only through the pull sink's [code] prefix.

A code on the result and on the stored row, which every reader decides on, is what #4360 covers. Its acceptance criteria include a test showing that a wording change moves no reader's status. That is where ending the split belongs.

# Overlapping candidates, each checked with `str.lower` itself: IGNORECASE
# also folds characters `lower()` does not (dotless i, long s).
for found in re.finditer(f"(?=({re.escape(phrase)}))", text, re.IGNORECASE):
if found.group(1).lower() == phrase:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplification: if phrase in text: return phrase / if phrase in text.lower(): return phrase.upper() gives the same verdict for every reader without the overlapping-lookahead IGNORECASE regex plus per-candidate lower() check. It also emits a fixed spelling, which closes the small free-text channel: today the exception's exact letter case (tImEd OuT) and odd lower()-equal Unicode (Kelvin sign) are copied verbatim into the 5xx answer, against #4284's 'only fixed words reach the error' rule.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0df905e, taken as suggested with one change: the fallback spelling is title case (Timed Out, At Capacity, Too Many Requests), not upper case. Both are fixed spellings. Title case reads as the ordinary phrase and matches the HTTP reason phrase.

The lookahead IGNORECASE regex and the per-candidate lower() check are gone. phrase in text.lower() is the same test the lowercasing readers run, so ı, İ, ſ and the Kelvin sign get the same verdict here as there. None of the exception's own letters reach the error.

  • test_execute_task_error_spells_a_phrase_one_fixed_way: Lock tImEd OuT; Agent AT CAPACITY → RuntimeError: Timed Out, At Capacity. This test is red on the previous head.
  • _assert_only_reader_phrases now requires every passed-on word to be exactly one of the two fixed spellings.
  • A randomized check over 200,000 mixed-case and Unicode strings found 0 cases where either reader shape answered differently on the error than on the text.

# inline-auth chats and the client portal answer 429 on "at capacity" and 504
# on "timed out".
_READER_PHRASES = ("timed out", "at capacity")
# The phrases the execution error's readers match. The sync /task mapper

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _READER_PHRASES comment and the _exception_words docstring say every reader answers the status it gave the exception's own text. That isn't true for readers the PR itself lists (the ChatPanel banner's 'usage limit' / 'out of usage' / 'out of extra usage', and the public refusal markers). Worth narrowing the claim so a future reader matching 'usage limit' on execution.error doesn't assume it fires on this path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0df905e.

import src.scheduler.service as scheduler_service
from src.scheduler.models import ExecutionStatus

now = datetime.fromisoformat("2026-11-07T12:00:00") # naive, as the scheduler's clock is

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_retry_delay_seconds permanently appends the repo root to sys.path and imports the scheduler as src.scheduler.service. That leaves a second module identity for the scheduler package for the rest of the session, so under random order later tests that import scheduler.service or patch src.scheduler.failure_classifier can see two module objects. Suggest monkeypatch.syspath_prepend plus importing under the canonical name.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked, and left as is. In tests/unit, src.scheduler.service is the only name the scheduler can be imported under, and it is the name every other unit test that drives it uses. test_1808, test_1823, test_1968 and test_3993 all use the same idempotent if str(_REPO) not in sys.path: sys.path.append(...), at module level. tests/unit/pytest.ini sets the rootdir for these runs, so the repo-root pythonpath = ["src", …] is not applied, and import scheduler.service fails (the docstring of test_1771b records the same thing).

I checked this by running this file together with test_3993, test_1823 and test_1968 in random order. At the end of the run:

  • sys.modules held only src.scheduler.service and src.scheduler.failure_classifier.
  • import scheduler.service raised ModuleNotFoundError: No module named 'scheduler'.

So importing it under the bare name would be what creates a second identity. monkeypatch.syspath_prepend would drop the path at teardown, but the imported module stays in sys.modules either way, so it would not change how many module objects exist.

@dolho

dolho commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Retry-delay side effect (task_execution_service.py ~L1566 / ~L968, outside the diff): now that the scheduler's "429" substring check matters on this error, the platform's own fixed text can trigger the doubled retry delay with no 429 in the exception: aborted after 429s of 3600 seconds allowed (ReadTimeout), timed out after 1429 seconds, or queued turn not claimed in 429s. A plain network failure then gets a rate-limit backoff. Matching 429 as a whole token, or deciding on error_code, avoids it.

…elling, and 429 only as a number of its own (#4352)

The exception's own letters no longer reach the error: a phrase found only
after lowercasing is passed on title-cased, so the 5xx answer carries fixed
words only (#4284). Every status reader still answers what it gave the
exception's own text.

"429" is passed on only as a number of its own. The digits inside 14290,
0.429 or 429s read as a rate limit and doubled a scheduled run's retry delay.
@AndriiPasternak31

Copy link
Copy Markdown
Contributor Author

Retry-delay side effect: … the platform's own fixed text can trigger the doubled retry delay with no 429 in the exception

Real, but older than this PR, and this PR does not change it. Those are the platform's own texts, already present at the branch point (task_execution_service.py on 47e7981):

  • L1548: aborted after {elapsed_s}s
  • L968: queued turn not claimed in {n}s
  • L1540: timed out after {timeout_seconds} seconds

The scheduler's "429" in error_lower dates from #332 (April). So 429s or 1429 seconds doubled the delay before #4340 and before this PR. The words this PR adds come only from the exception's text, and since 0df905e 429 is passed on only as a number of its own.

Fixing these cases means changing the scheduler's own check. The first bullet of #4360 names this exact case (aborted after 429s), and its cheap alternative is to anchor the scheduler's 429 match, so it is tracked there rather than here.

@vybe vybe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge-train: batch validated on train/ (#4454). Head moved only by a dev merge; the PR's own patch is byte-identical to the validated 0df905e.

@vybe
vybe merged commit fb75523 into dev Nov 11, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants