Repository navigation
Fix local inference prompt truncation, tool calls, and memory recovery - #393
Merged
Merged
Conversation
… servers A fixed overall deadline kills a healthy local generation on a slow machine mid-stream, and the timeout then retries the whole prompt against a server still busy with the abandoned request. with_idle_timeout bounds only the silence between reads, which is what a local server needs.
Broccolito
marked this pull request as ready for review
October 12, 2026 06:04
… (F10, F12) Live attempts F10 (lfm2:24b, 32,768-token window) and F12 (command-r7b:7b, 8,192) failed with nothing on screen: Biorouter's instructions and tools alone overflow those windows, the LC-1 check correctly skipped compaction, but the reply loop only yielded an inline notice and broke out, so the turn ended as an ordinary `stop`. The notice was never stored, `biorouter session send` printed `[finished] stop` and exited 0, and the exported chat held only the user's message. Both overflow stops (LC-1 and the spent compaction ladder) now end the turn the way the provider-error stop does: a stored assistant row carrying the sentence and a `context_length` provider failure, which the desktop renders as "Model context limit exceeded" and the CLI as an `[error:...]` line with its exit code. The LC-1 sentence no longer tells the user to raise a window the model may not support. `session send`/`watch`/`attach` also print inline notices as `[notice]` lines; they read only `text` blocks and dropped every notice's `msg`, while the interactive renderer showed them.
…(F11)
Live attempt F11 (deepseek-r1:8b) spent 166 s inventing an SDK because it
never received a tool definition. Ollama lists `tools` among its
capabilities, but neither template /api/show returns for it (the Jinja
`template` nor the Go TEMPLATE in its Modelfile) renders the tool list: the
same 136,952-byte request phi4-mini read as 29,997 prompt tokens reached it
as 19,094. Biorouter trusted the capability, offered the tools and sent the
request.
ModelFacts now records `template_never_shows_tools` when a real template
(not the `{{ .Prompt }}` stub of a renderer-served model, and not a model
with a RENDERER line) mentions neither Go's `.Tools` nor the word `tools`,
in both template sources. Such a model has no tool support: the R19
preflight refuses a tools request before the model loads, with a sentence
that says the template is the reason, and the model list hides it like a
model without the capability. The check fails open for every template that
might show the tools; the shipped llama3.1, mistral-small3.2 and ministral-3
templates and the renderer models are pinned unchanged.
The deepseek-r1:8b templates are saved as fixtures from the live run's
/api/show provenance.
…(F13, F07) Live attempt F13 (GLM 4.7 Flash) had finished the factorial task inside one script: the write, the chmod and both runs returned ok. The script then read `result5.stdout.trim()`, but `shell` hands back its output as a string, so the run failed with Boa's bare "TypeError: cannot convert 'null' or 'undefined' to object". The completed calls were recorded only in `_meta`, which the model never sees, so the error read exactly like one whose calls had failed; the model rewrote the correct file with escaped newlines and destroyed it. - A failed script's error now names the tool calls that had completed successfully before it stopped (names and counts only), and says their effects are in place. - A TypeError now names each variable the script assigned a tool result to and then read a property of that a string lacks, when that tool's result reached this run's script as a string (recorded per run, so the hint is never made about a parsed JSON object or about real string members). Live attempt F07 (ministral-3:8b) embedded a Python file in six scripts as `file_text: """..."""` and got the bare "unterminated string literal" each time. A parse error in a script containing a triple quote now says JavaScript has none and names the two string forms that work.
Live attempt F06 (gpt-oss:20b) contradicted the module's premise that Harmony channels never reach Biorouter as text: a reply whose thinking planned an execute_code call put a malformed Harmony header in its content, after a markdown checklist, and Ollama's parser left it as text. Nothing in the recovery changes: the call sat inside prose, where a quote and an intent cannot be told apart, and its arguments did not fit the tool. The doc now says the exclusion is a choice rather than a premise.
… F05)
In live attempts F04 and F05 the reporter's model, qwen2.5-coder:7b, made
one (recovered) call and then wrote every later reply in Biorouter's own
tool-result frame, `<tool-output untrusted="true" tool="X">...`, wrapped in
`<tool_response>` or `<tool-response>`: an invented "Task updated" for
todo_update, a file's would-be contents for a write, the JavaScript it meant
to run for execute_code. Each was accepted as the turn's answer, because its
body is not a `{name, arguments}` object and `attempted_call` never saw it;
the model was never told that nothing had run.
`forged_tool_result` detects a WHOLE reply that is one such frame (the same
whole-reply rule as `attempted_call`: a frame quoted inside prose is never
asked about), and the native turn spends its one corrective retry saying
that results come only from Biorouter after a real call and asking the
model to make the call, or answer in text. A reply whose frame body repeats
a real tool result or an earlier reply already in the conversation is a
relay and is left alone, so asking can never run a tool twice. Detection
only: nothing in the frame is ever executed. Unit tests use the verbatim
live replies; two replay tests cover the retry and the relay.
…rd_result (F13)
The F13 rerun on the previous commit kept GLM 4.7 Flash's correct
factorial.py intact (the completed-calls sentence did its job), but the
turn still ended without an answer: the model never saw the outputs. It
reported them three ways the sandbox does not support, each answered by
a message that named neither the cause nor record_result:
- `console.log(...)`: "ReferenceError: console is not defined" (four times);
- a top-level `return {...}`: "unexpected token 'return'" (three times);
- no report at all: a bare "Result: undefined" after both runs succeeded,
repeated until the loop guard stopped the turn.
Each now says what to do: there is no console, a module cannot return at
its top level, and a script that made completed tool calls but never
called record_result is told that nothing they returned was passed back.
A script that made no call, or that recorded a value, reads as before.
The second F13 rerun finished the task: GLM 4.7 Flash wrote factorial.py,
ran it for N=5 and N=20 and called record_result({ result5, result20 }),
exactly right. Biorouter then reported 20! as 2.43290200817664e+18. The
shell printed `2432902008176640000`, which is valid JSON, so the script was
handed a double that cannot hold it, and the rounded value is what came
back.
A tool result that is a bare integer beyond 2^53 (decided from its text,
so integers longer than a u64 are caught too) now stays the tool's text and
keeps every digit. Smaller integers, numbers with a fraction or exponent,
and every object or array are parsed as before.
…ts status (F07)
The F07 rerun (ministral-3:8b) moved past its triple quotes and then
called `shell({command: "run", args: [...]})` three times. Each call failed
with "zsh:1: command not found: run", but the script, and through it the
model, was handed only the structured envelope: `{"error":{"kind":
"tool_failure","message":"[shell: command exited with status 127]"}}`.
The output that says why was in the result's content, which a direct call
shows the model and a script never saw. GLM 4.7 Flash's original F13 run
lost "env: python3\nimport: No such file or directory" the same way.
A failed sub-call that carries assistant-visible content now throws that
content; one with nothing but its envelope throws the envelope as before.
The test uses the exact result shape `developer__shell` returns for a
non-zero exit.
…d tool (F07) The F07 rerun (ministral-3:8b) twice called shell(...) in a script that imported only text_editor, and got the bare "ReferenceError: shell is not defined". When the undefined name is a tool of an importable module, the error now names the import that defines it. A name that is no tool gets no invented import.
…o rerun (F07) The F07 rerun (ministral-3:8b) wrote factorial.py successfully and then wrote it twice more, identically, after each "Result: undefined" note. The note said the calls ran but led with what the script did not do. It now leads with the fact that the calls completed successfully and need not be run again, and only then says how to see a value.
Adds a section to the local inference validation record: what each of the 14 raw FAIL attempts traced to after a second reading of the evidence, the commit that changed Biorouter's behavior where one did, and what the reruns showed. F13 (GLM 4.7 Flash factorial) now passes on the final build; F10/F12 fail with the context sentence shown and stored, F11 is refused before the model loads, and F04/F05/F07 remain model failures with Biorouter's corrections and errors now visible to the model. Two items stay with the maintainers: the exit status of session send on an error frame, and a smaller tool set for windows of 32k tokens or less.
…F07)
The 2026-10-12 rerun of live attempt F07 (ministral-3:8b, the factorial
task) found a variant of F13 that the F13 fixes do not reach. The script
wrote factorial.py, ran it for N=5 and N=20 (both shell calls returned
their output, 9 and 26 bytes), and recorded
record_result({ outputN5: result5.output, outputN20: result20.output })
`shell` hands a script its output as one string, so `.output` is
undefined. Nothing is called on it, so nothing throws and the F13 hint
(which explains a TypeError) never runs; record_result drops undefined
values, and the model was shown `Result: {}` ten times with nothing
saying why, until the task hit its 1,800-second limit.
A script that runs to its end now carries `string_result_member_hints`
out with its result, and the text the model sees says the calls
completed, that the named properties were undefined because the tool
returned a string, and how to read the string itself. A script that
records the string, or whose tool returned a JSON object, is shown only
its result, as before.
Tests: the verbatim ministral script through the engine, and a recorded
property of todo_write's string result through the real handler. Both
fail when the added text is removed; the 128 code_execution_extension
tests pass.
…mal cases Every failed attempt that is not purely environmental was rerun at least twice with the PR's own live harness (one Ollama model at a time on an owned one-slot Ollama, the same task prompts and fixtures), and all 29 formal cases were rerun on the final build. - F10, F11, F12, F14: the expected clear failure or refusal, twice each. - F13 (GLM 4.7 Flash): PASS three times out of three. - F06 (gpt-oss:20b): flaky, two of three PASS. - F02-F05, F07-F09: model failures, unchanged; the first ministral run exposed the Code Execution defect fixed in acbefe5. - F01 (the Gemma survey): PASS twice, 39,096 prompt tokens. - Formal: 35 invocations, all 29 cases pass; the five failed verdicts are harness observation races in which the product test itself passed. Evidence: ~/biorouter-runs/local-inference-resume-2026-10-11-delivery-rerun/.
…explain only what record_result lost Review of acbefe5 and 4010c2f (2026-10-12), each case reproduced against run_js_script before this change: - A tool's JSON scalar output was parsed, so `python3 factorial.py 5` reached the script as the NUMBER 120 while 20! (above 2^53) stayed text. Two live models were shown `out5: 120` beside `out20: "2432902008176640000\n"`, `.trim()` threw on one and worked on the other, and the hint "`shell` returned a string, use `result5.trim()`" was false for exactly the outputs that parsed (its suggested call threw). With both outputs numeric the script got a bare `Result: {}`. Now only a JSON object or array is parsed; every other result is its text, so one tool's results are one kind and every digit survives. fits_js_number is gone with the special case it served. The module note says so. - The "holds a string" hint keyed on a per-run, per-TOOL set of tools that ever returned a string. A tool that returned a JSON object on one call and text on the next made `info.name` (a real property of the object) be reported undefined. RESULT_KINDS now records both kinds, and the hint is made only about a tool whose every result in the run was text. - A variable assigned again after its declaration (`lines = lines.split("\n")`) had its array methods reported as a string's missing ones. Such a variable is skipped. - On the success path the note was emitted from the source alone, so a correct script with a complete result was told its values were undefined. record_result now records where the value it was given held undefined (read off own data properties, as Boa's to_json reads them, so no getter runs), and the note is made only when that list is non-empty, naming the lost keys, with the string hint when it applies. Tests for each probe fail without the change.
…(F10, F11, F14) The PR #393 rerun recorded F11 (deepseek-r1:8b) and F14 (gemma3:4b) as "refused before the model loaded". The isolated Ollama's own serve.log says otherwise: about a second after each refusal it began loading the refused model's blob for a POST /api/chat that the harness later cut off as 499. The harness itself sends no /api/chat, so the request was Biorouter's: the chat-title call. maybe_rename_session runs after every reply loop, the title request carries no tools, so it passed the R19 no-tools refusal and loaded the model the turn had just refused. lfm2:24b's only logged request in its overflow-refused F10 turn was the same call. That load is also the "leftover runner" the F11/F14 pair cleanup checks failed on. The reply loop now records when a turn ends on a typed TurnAbortCode::ProviderFailure, and maybe_rename_session then titles the chat from the first user message (maybe_update_name_without_model) instead of calling the provider again. Only the placeholder is replaced that way, so a title an earlier turn's model wrote is kept, and the next turn that reaches the model names the chat as before. Tests: a refusing provider and the overflowing provider are each called no further after their turn; both fail without the change.
…s no item (F07) In the F07 rerun (round 3, ministral-3:8b) the turn's first six calls were todo_write with `- [ ] Create...` and `- [ ] Create...`. The parser required exactly `- [ ] `, so each parsed 0 items, each was answered "Todo list set: 0 item(s)" as a success (silently replacing the checklist with nothing), and the loop guard fired before the model found the spacing. Whitespace after the bullet and inside and after the brackets is now ignored (`- [ ] a`, `- [ x ] b`, `*[~]c`). A write whose non-blank content still names no item is refused with the expected format and its first line, and with_state persists nothing on an error, so the existing checklist stays. Blank content still clears the list.
…0, F12) The rerun of 8e18644's evidence showed that Ollama answered lfm2:24b's and command-r7b:7b's overflowed turns with a 400 to the turn's own request; the title call was a second request to an already loaded model, not the only one. Comments only.
…fected reruns The review of the 2026-10-12 rerun found four Biorouter defects or wrong claims, fixed in 8f90576 (code execution: scalar output as text, a false undefined note), 8e18644 (the chat-title call after a provider failure) and 65be50c (todo checklist spacing). This records them and corrects the claims in place: F11 and F14 were not "refused before loading" end to end, because the title call loaded the refused model, and that load, not the harness's `ollama stop`, is what their cleanup checks found; F07 round 3 began with five checklist writes answered as successes with 0 items. The affected attempts were rerun on a fresh build of 65be50c (evidence in ~/biorouter-runs/pr393-review-fix-reruns-2026-10-12): F10, F11, F12 and F14 fail as designed with no title request, and F11/F14 now pass their cleanup checks; F13 and F06 pass with numeric output handed over as text; F07 still fails on the model's triple quotes.
Calls do not all return strings: a JSON object or array is parsed, and any other result, a number a command printed included, is the tool's text (8f90576). Also say what record_result does with an undefined value.
Broccolito
added a commit
that referenced
this pull request
Oct 12, 2026
Broccolito
added a commit
that referenced
this pull request
Oct 12, 2026
… R-34)
The Ollama panel, the Llama Server memory warnings and the merged
memory module now follow the redesign rules. Checked in Chromium
with stubbed data in light and dark; no model was run.
- Every memory word comes from localModelsCopy and is written in GB.
- Ollama: the daemon's GiB paragraph about memory becomes the Llama
rows' warning word, with an InfoTip in GB. The status line no
longer repeats the model tag. A public server's status line starts
with "Public", in place of a separate caption. The remote server
note becomes one instruction with an InfoTip, after the picker.
The privacy note text stays unchanged.
- Llama: the Windows VRAM note is now the Memory row's InfoTip. The
Expected speed row drops the catalog's GiB memory clause, here and
in the Settings info dialog. The confirmation strings move into
llamaCopy.
- Switch models picker: the fit and state words match the Settings
rows ("Fits this Mac", "Installed"). It no longer says
"Recommended here" or "VRAM unknown".
- Local models: the delete and pull messages move into copy.ts. The
Ollama delete confirm names the model by its display name.
Broccolito
added a commit
that referenced
this pull request
Oct 12, 2026
… the probe address (R-34) With OLLAMA_HOST=0.0.0.0 the public note read "treats the Ollama server at http://127.0.0.1:11434 as public, not private: it counts only localhost, 127.0.0.1 and ::1 as this computer", which contradicts itself, and the status line said "Public · Ollama running" with no host. The card handed every sentence the renderer's probe address, which turns 0.0.0.0 and localhost into 127.0.0.1 for the CSP. #393's card did the same. - Every status line and the public note name the server the user configured (base_url without its trailing slash). Probes still go to the renderer address. - A public server is always named in the status line. Only the default server on this computer stays unnamed. - The note's words, role and test id are unchanged. Privacy UI: on the PR's human-review list. Tests: two failing-then-passing cases (0.0.0.0, running and not running) and one that the default server is not named.
Broccolito
added a commit
that referenced
this pull request
Oct 12, 2026
… defaults, in GB The page cited v1.92.1 for its defaults but named Qwen3.8 27B and qwen3.8:27b as plain defaults, which 1.92.1 does not pick. The Llama Server and Ollama defaults, the earlier catalog and the context window sizes now say 'after v1.92.1' / 'in v1.92.1 and earlier', as the Versa API Azure row already did, and memory is written in GB as the app writes it. landing/scripts/provider-defaults.test.mjs reads each provider's default from the published release's tag and from the tree, and fails when the page states a changed default without the qualifier.
This was referenced Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ollama uses native
/api/chatwith explicit context andtruncate:false; overflow/short usage becomes an error. Whole-call text recovery requires advertised names and checked arguments. Private sessions reject cloud alternates/remote aliases before either native or remembered fallback inference.Managed Llama Server uses checksum-verified b11429, ggml-org's official-weight Qwen GGUF conversion and Google's official Gemma GGUF, a 64k automatic floor, one-request queue and launch authentication.
Automatic memory adaptation happens after the first reply begins and cannot prevent first-load failure. Explicit
OLLAMA_NUM_CTXand distinguishable per-call overrides retain precedence; equal-default numeric limits lack provenance. Recovery checks a bounded schema subset; different-model replacement remains unexercised.Type of Change
AI Assistance
Testing
Validation concluded 2026-10-12 UTC (2026-10-11 PDT); overlapping counts retain source/platform scope.
98fd881just check-everythingPASS; workspace macOS 9,328 / Ubuntu 9,319 / Windows 8,868; Vitest 10,408/19 skips.Historical H1 Windows lock failure and retry remain preserved, cause unproved. Final PR checks pass on H2; H1 supplies no H2 credit.
Gemma G: two strict passes, LOAD quality FAIL, disk-interrupted QUERY, four unrun. R6: two of six attempted, both FAIL—LOAD omits fictional/clinical/schema/workflow facts despite correct persisted stubs; QUERY times out with gated deletion intent, no execution proof. Separate LABS PASS; DEBUG supplemental PASS retains wrapper failure/missing exports; VIZ static PASS has browser rendering UNVERIFIED. Seeded CURATE times out with no final/KB operation; metadata changes remain qualified.
Historical managed Qwen: raw 7/8, strict 4/8, recorded scope 8/8. LOAD invents intrahepatic enrollment; QUERY omits module inspection; CURATE adds invalid YAML/unsupported randomization and no final; VIZ invents mg/dL despite correct Chrome controls. Earlier d06 Qwen scope failures, Gemma strict 4/8 and thinking-off Qwen 3/5 remain failures.
Formal coverage: 29 unique model/test cases PASS across 30 attempts, original undersized survey FAIL retained. One workflow PASS. Full-agent llama3.1/reporter Qwen2.5-Coder tasks each FAIL twice; GPT-OSS factorial FAIL. Reporter TODO recovery and single-shell-call probe PASS establish recovery, not script/read reliability; notifications qualify raw empty-message counts. Mistral Small factorial PASS retains five recovered errors; Ministral/Granite/Phi/DeepSeek/GLM factorial FAIL, LFM/Command R context-refusal FAIL. GLM's original numeric bytes are unretained; final overwrite is invalid. Gemma3's planned no-tools refusal is observed; raw task, original serving census/cleanup FAIL remain, with separate fresh closure PASS. Cleanup, native replacement, port/sampling/priority gaps remain scoped; 39 startup witnesses are unverified; existing session-observer diagnostic/rc0 ambiguity remains a separate follow-up, without emission-loss proof. No benchmark or OS confinement claim.
Native Qwen GGUF factorial/urine PASS; urine's first illustrative quotation reverses source order. All 45 sequences closed: 47 functional attempts, 33 PASS/14 raw FAIL; negative observation adds no whole-case PASS. Two lifecycle controls PASS separately. DB maintenance: 30 logical/integrity PASS, 62 before files; first partial attempt preserved, only offline copy recovered once. Archive: 10,627 files/3.35 GB, all saved bytes and 20 executables verified, source bytes unchanged;
/Users/wgu/biorouter-runs/local-inference-resume-2026-10-11/manifest-sha256.jsonSHA6a147d3e…84eb13e. Owned debug cache detached; image retained.See the validation report for receipts and limitations.
Related Issues
None linked.
Screenshots/Demos (for UX changes)
Chart evidence linked; Electron unverified.
Follow-up: the 14 failed live attempts (2026-10-12)
A second reading of the archived evidence traced four of the 14 raw FAIL attempts to Biorouter, and the reruns found five more defects in the same Code Execution reporting path. Fixed here, each with a test that fails without it (mutation-checked):
context_lengthfailure instead of an unexplainedstop;session send/watch/attachprint inline notices (f28ca994).5b879a69)..stdouton a string result,console, a top-levelreturnand a script that returns nothing each point atrecord_result; a tool's large-integer output keeps every digit (359f0323,d281bd6b,4010c2f9). The task now passes (rerun on the final build: 120 and 2432902008176640000).b672d90b). F07 (ministral-3:8b): triple quotes, a failed call's output, and unimported tools are now explained to the script (359f0323,0e38ef20,21fd6649,35954e31). F06: module doc corrected (49f519e9). These remain model failures.No change for F01 (fixed earlier), F02, F03, F08, F09, F14. Still open for maintainers:
session sendexits 0 on an error frame, and the ~29k-32k token fixed prompt excludes 32k-window models. Details and per-round evidence: the "Follow-up on the 14 failed attempts" section of the validation report; evidence in~/biorouter-runs/pr393-failed-attempt-reruns-2026-10-12/.Live rerun of the failed attempts and the formal cases (2026-10-12)
Every failed attempt that is not purely environmental was rerun at least twice with the PR's own harness (owned one-slot Ollama on 11445, one model at a time, same prompts and fixtures), and all 29 formal cases were rerun on the final build:
result5.outputoffshell's string result was shownResult: {}ten times with no explanation. Fixed inacbefe5a(tests mutation-checked; CI green on that head).1 passed, each followed by a clean retry.Details: the "Rerun of the failed attempts and the formal cases" section of the validation report; evidence in
~/biorouter-runs/local-inference-resume-2026-10-11-delivery-rerun/.