Skip to content

perf(desktop): batch thinking deltas like text deltas while streaming - #258

Merged
GoDiao merged 3 commits into
GoDiao:mainfrom
iruzen-dono:fix/batch-thinking-deltas
Nov 18, 2026
Merged

GoDiao merged 3 commits into
GoDiao:mainfrom
iruzen-dono:fix/batch-thinking-deltas

Conversation

@iruzen-dono

@iruzen-dono iruzen-dono commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #155

What changed

Thinking deltas buffer into the same 50 ms window as text deltas, so one window produces one store update instead of one per event. Ten deltas in a window measure 10 store updates on main and 1 here — measured, not estimated: the batching test fails on a stashed tree with expected 10 to be +0.

The three delta buffers (text, tool input, thinking) are one createDeltaBuffer(onFlush) helper — append, consume, clear, flush, schedule. Each call site keeps only its own flush body, and content_delta no longer reads and deletes its maps by hand.

Review round 2

Must fix 1 — the stale session after the flush. content_start and message_complete now flush first and read get().sessions[sessionId] afterwards, with the null check on that fresh read. The three sequences you named are tests:

  • content_start text, content_delta 'abc', 60 ms, thinking 'hmm', content_start text, content_delta 'def', message_complete → [abc, thinking, def]
  • the same opening with content_start tool_use → [abc, thinking, tool_use]
  • the same opening with message_complete directly → [abc, thinking]

Each fails against the pre-flush read, with the shapes you predicted ([abc, thinking, abcdef], [abc, thinking, abc, tool_use], [abc]).

Should fix 2 — the content_start test hid its own bug. The vi.advanceTimersByTime(60) before content_start is gone, so content_start arrives while the thinking is still buffered and the thinking message can only exist because of the flush at the top of that handler. The test now asserts messages is empty before the call and holds the thinking block after it. Removing the flush fails it.

Should fix 3 — one test per flush and clear site. message_complete, error, tool_use_complete, stopGeneration, disconnectSession, clearMessages, session_cleared and the applyMirror idle path each have a case, plus the local send path. disconnectSession and the two clears are asserted the durable way: the session is recreated (or the messages cleared) and 60 ms are advanced, so a surviving timer would write into the fresh state.

Every one of the 11 new cases was checked the other way round, by deleting the line it covers and re-running it alone:

case line removed result
does not restore text a thinking flush already wrote when content_start repeats it fresh read in content_start fails
does not duplicate text a thinking flush wrote when content_start opens a tool block fresh read in content_start fails
flushes thinking at content_start so the text keeps the streaming state flush at the top of content_start fails
keeps a flushed thinking block when message_complete follows it fresh read in message_complete fails
flushes buffered thinking before a message from this client lands flush in sendMessage fails
flushes buffered thinking before an error message flush in error fails
flushes buffered thinking before a tool use completes flush in tool_use_complete fails
flushes buffered thinking when the user stops generation flush in stopGeneration fails
drops buffered thinking when the session disconnects flush in disconnectSession fails
drops buffered thinking when the messages are cleared clear in clearMessages fails
drops buffered thinking when the server clears the session clear in session_cleared fails
folds buffered text and thinking when a snapshot reports the turn idle merge in the applyMirror idle path fails

Should fix 4 — one buffer helper. createDeltaBuffer owns the pending map and the flush timer, with append, consume, clear, flush and schedule; flush is consume-then-callback, so a manual flush and a timer tick share one path. textDeltaBuffer, toolInputDeltaBuffer and thinkingDeltaBuffer are three instances of it, and the text timer callback is gone: schedule registered the 50 ms window, flush reads the buffer.

Note — the report line is updated. docs/features/performance/optimization-report.md:70 now says content_delta and thinking are throttled, status and tool_result are not.

Behaviour difference from main: a message sent from the desktop composer while thinking is buffered now flushes that thinking first, which is what the remote user_message path already did. Without it the 50 ms timer could land the thinking block after the user bubble.

Checks, with the numbers

  • cd desktop && bunx vitest run src/stores/chatStore.test.ts → 115 pass (104 before, 11 new)
  • cd desktop && bun run lint → exit 0 (tsc, tsc electron, oxlint), no output
  • cd desktop && bunx vitest run → 3111 pass, 7 fail, 4 skipped (3122). Five of the seven fail by name on the stashed tree: pages.test.tsx ×2, LocalSlashCommandPanel, MessageList nested tool calls, UpdateChecker. The other two are wall-clock timeouts on a loaded machine (KanbanPage 5.3 s, WorkspacePanel 20.3 s); both files pass when run alone on this tree.
  • bun test (server + CLI) → 2724 pass, 15 fail, 1 error. The same 15 fail by name on the stashed tree (16 there, one flaky extra): the ripgrep fixture cases, the artifact path and symlink cases, H5AccessService, the two session-link forks, a 5 s DiagnosticsService timeout, and the Windows tar error. None of them imports the store.

Files

  • desktop/src/stores/chatStore.ts — +164/−128
  • desktop/src/stores/chatStore.test.ts — +345/−0
  • docs/features/performance/optimization-report.md — 1 line

No new comment, JSDoc or inline note in any file, flushPendingThinking, createDeltaBuffer and mergeThinkingIntoMessages included. No CHANGELOG entry. No visible UI change: MessageList, buildRenderModel and toolResultMap are untouched, this only stops the store from notifying them per thinking event.

Copilot AI balanced review requested due to automatic review settings November 16, 2026 18:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@GoDiao GoDiao left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up and for going through the #160 review point by point. The batching works: ten deltas in one window are one store update now. The user_message flush is in and tested, the added lines have no comments, the mirror idle path does text before thinking, and timers are cleared on stop, disconnect and clear. A few things before this can merge.

Must fix

1. content_start and message_complete read a stale session after flushing thinking, which duplicates text or drops the thinking.

Both handlers read const session = get().sessions[sessionId] and then call flushPendingThinking(sessionId) (chatStore.ts:1298-1300 and 1516-1518). The flush moves pending text into a message and sets streamingText: '', but the code after it keeps using session.streamingText and session.messages from before the flush.

The server sends no content_start for a thinking block (src/server/session/translate.ts:325-329), so text followed by thinking is a real sequence, for example with interleaved thinking. With content_start text, content_delta 'abc', 60 ms so the text timer fires, then thinking 'hmm', and within 50 ms:

  • content_start text, content_delta 'def', message_complete: the messages end as [abc, thinking, abcdef]. abc is written as a message by the flush and then put back into streamingText from the stale snapshot. On main they end as [abc, thinking, def].
  • content_start tool_use: [abc, thinking, abc, tool_use], against [abc, thinking, tool_use] on main.
  • message_complete directly: [abc]. appendAssistantTextMessage(session.messages, …) runs on the stale messages and overwrites the flushed thinking. On main it's [abc, thinking].

Flushing before reading session (and keeping the null check on that fresh read) fixes both handlers. Please add these three sequences as tests.

Should fix

2. The content_start test passes without the content_start flush. "flushes thinking at content_start so the text keeps the streaming state" (chatStore.test.ts:2606) calls vi.advanceTimersByTime(60) after thinking (line 2616), so the timer flushes the thinking before content_start runs. With the flushPendingThinking call at the top of content_start removed, all 104 tests still pass, and the #160 point 1 bug is back. Dropping that advance, so content_start arrives while the thinking is still buffered, makes the test catch it.

3. Most of the new flush and clear sites have no test. Removing any one of these still leaves 104/104 passing: the flush in message_complete, error, tool_use_complete, stopGeneration and disconnectSession; clearPendingThinking in clearMessages and session_cleared; and the thinking merge in the applyMirror idle path. Only the user_message flush and the batching itself are covered. Please add a test for each, for example "thinking, then session_cleared, then 60 ms leaves no thinking message" for the clear paths. The tests for point 1 will cover message_complete and content_start.

4. This makes three copies of the same delta buffer. Text (pendingDeltaBySession, flushTimerBySession, consumePendingDelta, appendPendingDelta, clearPendingDelta) and tool input (the ...ToolInputDelta set) already repeat each other line for line, and so does the "start a 50 ms timer if none is running" block in content_delta. The thinking buffer is a third copy. Please extract one buffer helper (append, consume, clear, and schedule a flush with a callback) and move text, tool input and thinking onto it, so each call site keeps only its own flush body. The text timer callback, which reads and deletes the maps by hand, becomes a plain consume.

Note

  • docs/features/performance/optimization-report.md:70 still says thinking has no throttle. It's a snapshot from before the change, so updating it is optional.

No CHANGELOG entry needed; we'll add it after merging.

Thinking deltas updated the store on every event, so each one created a new
messages array, rebuilt the render model in MessageList and re-rendered every
tool group through a new toolResultMap. Fast thinking output did that dozens of
times per second.

They now buffer into the same 50ms window as text deltas
(pendingThinkingBySession plus a flush timer), so one window produces one store
update. Ten thinking deltas measure 10 store updates on main and 1 here.

Ordering is explicit rather than incidental:

- pending thinking flushes at the top of content_start for every block type, so
  a reply that thinks before it writes text keeps chatState 'streaming' and
  activeThinkingId null for the rest of the answer instead of falling back to
  'thinking' at the first text delta;
- it flushes before a tool result, turn end, an error, stopGeneration and
  disconnectSession, and in the user_message case after the duplicate check so a
  message from another client cannot land before the thinking;
- clearMessages and a cleared session drop it, and the mirror's idle path folds
  pending text before pending thinking, matching flushPendingThinking.

Tests: three new cases in chatStore.test.ts. One window with ten deltas produces
one update; the real sequence thinking -> content_start text -> content_delta
keeps the streaming state and yields thinking before text at turn end; a remote
user_message lands after the thinking. One existing test advanced its timers for
the buffered timing.

bunx vitest run on the desktop suite and bun run lint (tsc, tsc electron, oxlint)
both pass.
content_start and message_complete captured the session before
flushPendingThinking moved pending text into a message and cleared
streamingText, so the handler then rebuilt that text from the stale
snapshot: [abc, thinking, abcdef] instead of [abc, thinking, def], and
message_complete appended to the pre-flush messages array, overwriting
the thinking block it had just written. Both handlers now flush first and
read the session afterwards.

The three delta buffers were three copies of the same append, consume,
clear and schedule code. They now share one createDeltaBuffer helper, so
each call site keeps only its own flush body and content_delta no longer
manages its maps by hand.

A local send flushes pending thinking the way the remote user_message path
already did, so a thinking block cannot land after the user bubble.

Tests: 11 new cases. The content_start case no longer lets the timer flush
the thinking before content_start runs, and the stale-read sequence, the
flush in message_complete, error, tool_use_complete, stopGeneration,
disconnectSession and a local send, the clears in clearMessages and
session_cleared, and the mirror idle merge each have a case that fails
when the line it covers is removed.
@iruzen-dono
iruzen-dono force-pushed the fix/batch-thinking-deltas branch from 957c4f1 to 3280d0e Compare November 17, 2026 20:11
@iruzen-dono

Copy link
Copy Markdown
Contributor Author

All four points, in 3280d0e.

Must fix 1 — the stale session after the flush. content_start and message_complete now call flushPendingThinking first and read get().sessions[sessionId] afterwards, with the null check on that fresh read. Your three sequences are tests: text + thinking + content_start text + more text + message_complete ends [abc, thinking, def]; content_start tool_use ends [abc, thinking, tool_use]; message_complete on its own ends [abc, thinking]. Restoring the pre-flush read fails all three with the shapes you predicted — [abc, thinking, abcdef], [abc, thinking, abc, tool_use], [abc].

Should fix 2 — the content_start test covered its own bug. The advanceTimersByTime(60) before content_start is gone, so content_start arrives with the thinking still buffered; the test asserts messages is empty before the call and holds the thinking block after it, which only the flush at the top of the handler can produce. Removing that call fails the test.

Should fix 3 — one case per flush and clear site. message_complete, error, tool_use_complete, stopGeneration, disconnectSession, clearMessages, session_cleared and the applyMirror idle path are covered, plus the local send path. For disconnectSession and the two clears the assertion is the durable one: the session is recreated (or the messages cleared), then 60 ms are advanced, so a surviving timer would write into the fresh state. I also ran each new case against the deletion of the line it covers — all 11 fail, listed in the PR body.

Should fix 4 — one buffer helper. createDeltaBuffer(onFlush) owns the pending map and the timer: append, consume, clear, flush (consume-then-callback, so a manual flush and a timer tick take one path) and schedule. textDeltaBuffer, toolInputDeltaBuffer and thinkingDeltaBuffer are three instances, and the four lines that read and deleted two maps by hand in the text timer are gone.

Note — the report line is updated. docs/features/performance/optimization-report.md:70 now says content_delta and thinking are throttled, status and tool_result are not.

One behaviour difference the refactor surfaced: a message sent from the desktop composer while thinking is buffered now flushes that thinking first, the way the remote user_message path already did — otherwise the 50 ms timer could land the thinking block after the user bubble. Tested.

Numbers: chatStore.test.ts 115 pass (104 before, 11 new); bun run lint exit 0; the full desktop suite 3111 pass / 7 fail, five of them the known Windows set (pages.test.tsx ×2, LocalSlashCommandPanel, MessageList nested tool calls, UpdateChecker) that fail identically on a stashed tree, the other two wall-clock timeouts (KanbanPage, WorkspacePanel) that pass alone; bun test 2724 pass / 15 fail, the same 15 by name on the stashed tree, all in ripgrep, artifact-path, H5AccessService and session-link fixtures.

The branch is rebased onto main (86747fe), so the diff is against the current base: chatStore.ts +164/−128, chatStore.test.ts +345/−0, one doc line. No CHANGELOG entry, no new comment anywhere in the added code.

@GoDiao GoDiao left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @iruzen-dono, all four points are in. The stale-session reads are gone, the content_start test now fails without the flush, every flush and clear site has a case that fails when its line is removed, and the three buffers share createDeltaBuffer. We compared each text and tool-input site against main and they behave the same.

I pushed one small commit on top so it could merge today: flushPendingThinking only forwarded to thinkingDeltaBuffer.flush, so the call sites now use the buffer directly and the wrapper is gone. Lint and the 115 chatStore tests pass, and CI is green.

We'll add the CHANGELOG line on main after merging.

@GoDiao
GoDiao merged commit bd423a7 into GoDiao:main Nov 18, 2026
4 checks passed
GoDiao added a commit that referenced this pull request Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch thinking deltas like text deltas while streaming

3 participants