Repository navigation
fix(tasks): the execution error keeps a reader phrase in any case, so the portal keeps its 504/429 (#4352) - #4358
Conversation
… the portal keeps its 504/429 (#4352) #4340 (#4284) reduced an unforeseen exception's text to its class plus the reader phrases "timed out" / "at capacity", kept only on an exact lowercase hit. The client portal lowercases the error before matching, so an exception saying "Lock Timed Out" or "Agent At Capacity" lost its 504 / 429 there. _exception_words now finds each phrase case-insensitively and passes it on as the exception spelled it, preferring the lowercase spelling when the text has it. The sync /task mapper, the public and MCP chats, the portal and the scheduler therefore each match the error exactly when they matched the original text. Nothing else of the text is passed on. The scheduler's rate-limit words ("429", "rate limit", "too many requests"), which double the RETRY-001 retry delay, join the reader phrases so that delay works again for an unforeseen exception carrying them. Tests: the reader table now drives the sync /task mapper, the public chat, the inline-auth MCP chat and the real portal_chat; plus the scheduler's retry delay, the early-abort network message, an exception with no text, and the overlapping / dotless-i / long-s spellings.
| # chats answer 429 on "at capacity" and 504 on "timed out", case and all; the | ||
| # client portal does the same after lowercasing. The scheduler lowercases and | ||
| # doubles its retry delay on "429", "rate limit" or "too many requests". | ||
| _READER_PHRASES = ("timed out", "at capacity", "429", "rate limit", "too many requests") |
There was a problem hiding this comment.
Adding the bare digit string "429" to _READER_PHRASES means any number containing 429 (id, port, byte count, hex, elapsed seconds) shows up in the stored execution error as a bare "429", which reads as an HTTP rate limit.
Example: RuntimeError("lock held by worker 14290") becomes Task execution failed (RuntimeError: 429 — details in backend logs). Operators, the UI and the rate-limit bucket in error analysis now see a bare 429 with no context. Consider matching it as a whole token (\b429\b) or keying on error_code instead.
There was a problem hiding this comment.
Fixed in 0df905e. 429 is now passed on only as a number of its own: (?<![\w.])429(?!\w|\.\d). That is a little stricter than \b429\b, which still matches the 429 in 0.429.
RuntimeError("lock held by worker 14290") now gives Task execution failed (RuntimeError — details in backend logs), and a scheduled run's retry keeps the plain 60s delay. These rows pin it: 429-in-a-number, 429-in-a-decimal, 429-with-a-unit, 429-sentence-end, and test_execute_task_error_a_429_inside_a_number_keeps_the_plain_retry_delay.
Against the previous head, 6 tests go red. With \b429\b swapped in, 429-in-a-decimal goes red. With a plain substring, 4 go red.
| _READER_PHRASES = ("timed out", "at capacity", "429", "rate limit", "too many requests") | ||
|
|
||
|
|
||
| def _phrase_as_said(phrase: str, text: str) -> str | None: |
There was a problem hiding this comment.
Design point: keeping each phrase in the case the exception used preserves the readers' disagreement about case rather than removing it. Lock Timed Out still answers 503 on /task and 502 on public/MCP chat but 504 on the portal, and the tests pin that as intended. Setting error_code (TIMEOUT/CAPACITY already exist, and the portal already checks them) in the generic except path, or lowercasing in all readers, would end the per-phrase parity tuning.
There was a problem hiding this comment.
Not changed here. #4352 is a regression from #4340: the portal stopped giving an answer it gave on the raw text. This PR puts each reader back to its pre-#4340 answer. The /task 503 vs portal 504 split on Lock Timed Out is older than #4340. It comes from the readers themselves: _map_task_failure checks "timed out" in result.error case-sensitively, while portal_chat lowercases first. The tests pin it as "same status as on the text", not as the end state we want.
Both ways of ending the split change answers the issue did not report as broken:
- Lowercasing in every reader moves
/task, public and MCP chat from 503/502 to 504 on these texts. error_codein the genericexceptwould classify an unknown exception by the words in its text. It also would not reach every reader._map_task_failurebranches onCAPACITYbut notTIMEOUT(it still matches"timed out"in the text). The scheduler reads only the stored error string. A row carries a code only through the pull sink's[code]prefix.
A code on the result and on the stored row, which every reader decides on, is what #4360 covers. Its acceptance criteria include a test showing that a wording change moves no reader's status. That is where ending the split belongs.
| # Overlapping candidates, each checked with `str.lower` itself: IGNORECASE | ||
| # also folds characters `lower()` does not (dotless i, long s). | ||
| for found in re.finditer(f"(?=({re.escape(phrase)}))", text, re.IGNORECASE): | ||
| if found.group(1).lower() == phrase: |
There was a problem hiding this comment.
Simplification: if phrase in text: return phrase / if phrase in text.lower(): return phrase.upper() gives the same verdict for every reader without the overlapping-lookahead IGNORECASE regex plus per-candidate lower() check. It also emits a fixed spelling, which closes the small free-text channel: today the exception's exact letter case (tImEd OuT) and odd lower()-equal Unicode (Kelvin sign) are copied verbatim into the 5xx answer, against #4284's 'only fixed words reach the error' rule.
There was a problem hiding this comment.
Fixed in 0df905e, taken as suggested with one change: the fallback spelling is title case (Timed Out, At Capacity, Too Many Requests), not upper case. Both are fixed spellings. Title case reads as the ordinary phrase and matches the HTTP reason phrase.
The lookahead IGNORECASE regex and the per-candidate lower() check are gone. phrase in text.lower() is the same test the lowercasing readers run, so ı, İ, ſ and the Kelvin sign get the same verdict here as there. None of the exception's own letters reach the error.
test_execute_task_error_spells_a_phrase_one_fixed_way:Lock tImEd OuT; Agent AT CAPACITY→RuntimeError: Timed Out, At Capacity. This test is red on the previous head._assert_only_reader_phrasesnow requires every passed-on word to be exactly one of the two fixed spellings.- A randomized check over 200,000 mixed-case and Unicode strings found 0 cases where either reader shape answered differently on the error than on the text.
| # inline-auth chats and the client portal answer 429 on "at capacity" and 504 | ||
| # on "timed out". | ||
| _READER_PHRASES = ("timed out", "at capacity") | ||
| # The phrases the execution error's readers match. The sync /task mapper |
There was a problem hiding this comment.
The _READER_PHRASES comment and the _exception_words docstring say every reader answers the status it gave the exception's own text. That isn't true for readers the PR itself lists (the ChatPanel banner's 'usage limit' / 'out of usage' / 'out of extra usage', and the public refusal markers). Worth narrowing the claim so a future reader matching 'usage limit' on execution.error doesn't assume it fires on this path.
There was a problem hiding this comment.
Fixed in 0df905e.
- The
_READER_PHRASEScomment now speaks of the error's status readers. It says the chat panel banner and the public chat's refusal check also read the error, but on words this path does not pass on (usage limit, the refusal sentences), so they do not fire on it. It points to refactor(tasks): readers decide on execution error text, not a failure code — carry a code so wording changes can't move a status #4360. - The
_exception_wordsdocstring is limited to the readers in_READER_PHRASES, and it notes the exception for429.
| import src.scheduler.service as scheduler_service | ||
| from src.scheduler.models import ExecutionStatus | ||
|
|
||
| now = datetime.fromisoformat("2026-11-07T12:00:00") # naive, as the scheduler's clock is |
There was a problem hiding this comment.
_retry_delay_seconds permanently appends the repo root to sys.path and imports the scheduler as src.scheduler.service. That leaves a second module identity for the scheduler package for the rest of the session, so under random order later tests that import scheduler.service or patch src.scheduler.failure_classifier can see two module objects. Suggest monkeypatch.syspath_prepend plus importing under the canonical name.
There was a problem hiding this comment.
Checked, and left as is. In tests/unit, src.scheduler.service is the only name the scheduler can be imported under, and it is the name every other unit test that drives it uses. test_1808, test_1823, test_1968 and test_3993 all use the same idempotent if str(_REPO) not in sys.path: sys.path.append(...), at module level. tests/unit/pytest.ini sets the rootdir for these runs, so the repo-root pythonpath = ["src", …] is not applied, and import scheduler.service fails (the docstring of test_1771b records the same thing).
I checked this by running this file together with test_3993, test_1823 and test_1968 in random order. At the end of the run:
sys.modulesheld onlysrc.scheduler.serviceandsrc.scheduler.failure_classifier.import scheduler.serviceraisedModuleNotFoundError: No module named 'scheduler'.
So importing it under the bare name would be what creates a second identity. monkeypatch.syspath_prepend would drop the path at teardown, but the imported module stays in sys.modules either way, so it would not change how many module objects exist.
|
Retry-delay side effect ( |
…elling, and 429 only as a number of its own (#4352) The exception's own letters no longer reach the error: a phrase found only after lowercasing is passed on title-cased, so the 5xx answer carries fixed words only (#4284). Every status reader still answers what it gave the exception's own text. "429" is passed on only as a number of its own. The digits inside 14290, 0.429 or 429s read as a rate limit and doubled a scheduled run's retry delay.
Real, but older than this PR, and this PR does not change it. Those are the platform's own texts, already present at the branch point (
The scheduler's Fixing these cases means changing the scheduler's own check. The first bullet of #4360 names this exact case ( |
Fixes #4352
Journey Impact: none: an unforeseen exception's execution error keeps the status words its readers already relied on; no user journey changes
What
#4340 (#4284) stopped raw exception text from reaching 5xx answers. When an unexpected exception ends a task, the execution error now names only the exception class plus the "reader phrases" that callers match on (
timed out→ 504,at capacity→ 429). It kept a phrase only when it appeared in exact lowercase. The readers that act on those phrases do not all match the same way:/task(chat_execution_service._map_task_failure)public_chat_service.raise_for_failed_public_result)mcp_auth_service.dispatch_chat)client_portal/service.py::portal_chat)scheduler/service.py::_maybe_schedule_retry)429/rate limit/too many requestsSo an exception saying
Lock Timed OutorAgent At Capacityused to give the portal its 504 / 429, and since #4340 it got a 502. The scheduler's longer retry delay on rate-limit words was lost too, because those words were not passed on.How
_exception_words(intask_execution_service.py) now finds each phrase the way each kind of reader does, and passes it on in a fixed spelling:timed out);Timed Out);For every reader in the table above:
Lock Timed Outstill gets 503/502 on/taskand the public and MCP chats, as before test: a static guard keeps exception text out of 5xx answers, and the sites it found are fixed (abilityai/trinity#4284) #4340.Lock Timed Out, then read timed outstill gets 504.The second check is
phrase in text.lower(), the same test the lowercasing readers run, so characters thatlower()treats differently (ı,İ,ſ,K) give the same verdict here as there. Neither spelling comes from the exception, so its own letters (tImEd OuT, odd Unicode) never reach the 5xx answer: only the class name and these fixed words (#4284).The helper has two callers, and both get the change:
except Exceptioninexecute_task(Task execution failed (… — details in backend logs))_classify_timeout_failure(Task execution aborted after Ns of … allowed (ReadTimeout: Timed Out))Decision: the scheduler's rate-limit words are added
429,rate limitandtoo many requestsjoin the phrase list. The scheduler's doubled retry delay (RETRY-001: 2x, capped at 300s) works again for an unforeseen exception that carries them.429is passed on only as a number of its own: not the digits inside14290,0.429or429s. Otherwise a lock held by worker 14290 would read as a rate limit in the error and double the retry delay. This is the one place the error deliberately does not mirror a reader: the scheduler's own"429" in errorcheck is a loose substring (tracked in #4360), and the error should not carry that looseness forward.This was a close call. Against adding them:
For adding them:
Dropping them later is a one-line revert of the tuple. It also turns these test rows red, which would need updating:
two-phrases,429-sentence-endand the three scheduler rows (429,rate-limit,too-many-requests).Other readers of this error, outside this fix
Three more places read the execution error's text. All three lost their words in #4340, not in this change, and none is a likely path for an unforeseen exception. This PR does not widen the phrase list for them:
src/frontend/src/components/ChatPanel.vue:224-228). It lowercasesexecution.errorand shows "Subscription Usage Limit" onusage limit/rate limit/out of usage/out of extra usage. Becauserate limitis now passed on, the banner comes back for that word. It does not come back for the other three.public_chat_service.py:65,75,_TURN_REFUSED_MARKERS). This gives a 403 on the agent server's refusal sentences. Those sentences are not passed on.scheduler/service.py:1122,1530,_is_auth_failure). It reads the error, but only to label a log line.Validation
tests/unit/test_4284_exception_text_sites.py+tests/unit/test_4284_no_exception_text_in_5xx.py: 221 passed. Before the change, 180 passed ondev.test_2106_timeout_attribution,test_2789_subswitch_retry_budget,test_3340_stream_api_error_detail,test_3603_channel_failure_reply,test_3988_turn_limit_status,test_3993_no_retry_max_turns,test_ent279_secret_scrub,test_2320_portal_failed_turn_visibility,test_3114_pull_route_callers): 433 passed, 1 skipped in a fixed order. Also under shuffle seed 4358 (bug: exception text in HTTPException detail across backend and agent-server routes (py/stack-trace-exposure siblings CodeQL does not flag) #4284 files + 2320 + 3993): 278 passed.execute_taskand then through each of the four HTTP readers, including the realportal_chat. For every reader it asserts that the status for the original text equals the status for the class-only error, and that this status is the pinned value. The same check runs through the early-abort network message (httpx.ReadTimeout("… Read Timed Out")→ portal 504; 503/502/502 elsewhere, as on the text). The scheduler check drives the real_maybe_schedule_retrywith its clock pinned, includingworker 14290→ the plain 60s delay.ı,İ,ſ,K,ß, digits and dots) found 0 cases where a reader's match on the error differed from its match on the text (for429: from the whole-number rule), and 0 words passed on outside the fixed spellings. It was a local script, not committed.RuntimeError()). The error is the class alone, and all four readers give their generic status (503 / 502 / 502 / 502) for both the text and the error. Passed.ruffon the two files: the same 5 findings before and after, none on changed lines.mypyontask_execution_service.py: identical findings before and after.tests/lint_sys_modules.py: no new violations.Mutation
With the fix reverted (
task_execution_service.pyat the branch point ondev, new tests kept), 18 failed, 85 passed: the portal answered 502 where the original text gave 504 (capitalised-timeout) and 429 (capitalised-capacity); the scheduler's delay was 60s where the original text gave 120s (429,rate-limit,too-many-requests); the fixed-spelling row and the_exception_wordsrows; the early-abort message carried(ReadTimeout)with noTimed Out(all four readers' rows).Against the previous revision of this PR (spelling copied from the exception,
429as a substring), 6 red: the fixed-spelling row (tImEd OuT, AT CAPACITYreached the error),overlapping-candidates, the three429-inside-a-number rows, and theworker 14290retry-delay row.Each part of the fix mutated on its own, and each caught:
\b429\b: 1 red (429-in-a-decimal)429-inside-a-number rows and the retry-delay row)two-phrases,429-sentence-endand the three scheduler rows)Unverified
timed out/at capacityor rate-limit words in these paths today. The cases are planted.