Repository navigation
perf(desktop): batch thinking deltas like text deltas while streaming - #258
Conversation
GoDiao
left a comment
There was a problem hiding this comment.
Thanks for picking this up and for going through the #160 review point by point. The batching works: ten deltas in one window are one store update now. The user_message flush is in and tested, the added lines have no comments, the mirror idle path does text before thinking, and timers are cleared on stop, disconnect and clear. A few things before this can merge.
Must fix
1. content_start and message_complete read a stale session after flushing thinking, which duplicates text or drops the thinking.
Both handlers read const session = get().sessions[sessionId] and then call flushPendingThinking(sessionId) (chatStore.ts:1298-1300 and 1516-1518). The flush moves pending text into a message and sets streamingText: '', but the code after it keeps using session.streamingText and session.messages from before the flush.
The server sends no content_start for a thinking block (src/server/session/translate.ts:325-329), so text followed by thinking is a real sequence, for example with interleaved thinking. With content_start text, content_delta 'abc', 60 ms so the text timer fires, then thinking 'hmm', and within 50 ms:
content_starttext,content_delta 'def',message_complete: the messages end as[abc, thinking, abcdef].abcis written as a message by the flush and then put back intostreamingTextfrom the stale snapshot. Onmainthey end as[abc, thinking, def].content_starttool_use:[abc, thinking, abc, tool_use], against[abc, thinking, tool_use]onmain.message_completedirectly:[abc].appendAssistantTextMessage(session.messages, …)runs on the stale messages and overwrites the flushed thinking. Onmainit's[abc, thinking].
Flushing before reading session (and keeping the null check on that fresh read) fixes both handlers. Please add these three sequences as tests.
Should fix
2. The content_start test passes without the content_start flush. "flushes thinking at content_start so the text keeps the streaming state" (chatStore.test.ts:2606) calls vi.advanceTimersByTime(60) after thinking (line 2616), so the timer flushes the thinking before content_start runs. With the flushPendingThinking call at the top of content_start removed, all 104 tests still pass, and the #160 point 1 bug is back. Dropping that advance, so content_start arrives while the thinking is still buffered, makes the test catch it.
3. Most of the new flush and clear sites have no test. Removing any one of these still leaves 104/104 passing: the flush in message_complete, error, tool_use_complete, stopGeneration and disconnectSession; clearPendingThinking in clearMessages and session_cleared; and the thinking merge in the applyMirror idle path. Only the user_message flush and the batching itself are covered. Please add a test for each, for example "thinking, then session_cleared, then 60 ms leaves no thinking message" for the clear paths. The tests for point 1 will cover message_complete and content_start.
4. This makes three copies of the same delta buffer. Text (pendingDeltaBySession, flushTimerBySession, consumePendingDelta, appendPendingDelta, clearPendingDelta) and tool input (the ...ToolInputDelta set) already repeat each other line for line, and so does the "start a 50 ms timer if none is running" block in content_delta. The thinking buffer is a third copy. Please extract one buffer helper (append, consume, clear, and schedule a flush with a callback) and move text, tool input and thinking onto it, so each call site keeps only its own flush body. The text timer callback, which reads and deletes the maps by hand, becomes a plain consume.
Note
docs/features/performance/optimization-report.md:70still says thinking has no throttle. It's a snapshot from before the change, so updating it is optional.
No CHANGELOG entry needed; we'll add it after merging.
Thinking deltas updated the store on every event, so each one created a new messages array, rebuilt the render model in MessageList and re-rendered every tool group through a new toolResultMap. Fast thinking output did that dozens of times per second. They now buffer into the same 50ms window as text deltas (pendingThinkingBySession plus a flush timer), so one window produces one store update. Ten thinking deltas measure 10 store updates on main and 1 here. Ordering is explicit rather than incidental: - pending thinking flushes at the top of content_start for every block type, so a reply that thinks before it writes text keeps chatState 'streaming' and activeThinkingId null for the rest of the answer instead of falling back to 'thinking' at the first text delta; - it flushes before a tool result, turn end, an error, stopGeneration and disconnectSession, and in the user_message case after the duplicate check so a message from another client cannot land before the thinking; - clearMessages and a cleared session drop it, and the mirror's idle path folds pending text before pending thinking, matching flushPendingThinking. Tests: three new cases in chatStore.test.ts. One window with ten deltas produces one update; the real sequence thinking -> content_start text -> content_delta keeps the streaming state and yields thinking before text at turn end; a remote user_message lands after the thinking. One existing test advanced its timers for the buffered timing. bunx vitest run on the desktop suite and bun run lint (tsc, tsc electron, oxlint) both pass.
content_start and message_complete captured the session before flushPendingThinking moved pending text into a message and cleared streamingText, so the handler then rebuilt that text from the stale snapshot: [abc, thinking, abcdef] instead of [abc, thinking, def], and message_complete appended to the pre-flush messages array, overwriting the thinking block it had just written. Both handlers now flush first and read the session afterwards. The three delta buffers were three copies of the same append, consume, clear and schedule code. They now share one createDeltaBuffer helper, so each call site keeps only its own flush body and content_delta no longer manages its maps by hand. A local send flushes pending thinking the way the remote user_message path already did, so a thinking block cannot land after the user bubble. Tests: 11 new cases. The content_start case no longer lets the timer flush the thinking before content_start runs, and the stale-read sequence, the flush in message_complete, error, tool_use_complete, stopGeneration, disconnectSession and a local send, the clears in clearMessages and session_cleared, and the mirror idle merge each have a case that fails when the line it covers is removed.
957c4f1 to
3280d0e
Compare
|
All four points, in Must fix 1 — the stale session after the flush. Should fix 2 — the Should fix 3 — one case per flush and clear site. Should fix 4 — one buffer helper. Note — the report line is updated. One behaviour difference the refactor surfaced: a message sent from the desktop composer while thinking is buffered now flushes that thinking first, the way the remote Numbers: The branch is rebased onto |
GoDiao
left a comment
There was a problem hiding this comment.
Thanks @iruzen-dono, all four points are in. The stale-session reads are gone, the content_start test now fails without the flush, every flush and clear site has a case that fails when its line is removed, and the three buffers share createDeltaBuffer. We compared each text and tool-input site against main and they behave the same.
I pushed one small commit on top so it could merge today: flushPendingThinking only forwarded to thinkingDeltaBuffer.flush, so the call sites now use the buffer directly and the wrapper is gone. Lint and the 115 chatStore tests pass, and CI is green.
We'll add the CHANGELOG line on main after merging.
Closes #155
What changed
Thinking deltas buffer into the same 50 ms window as text deltas, so one window produces one store update instead of one per event. Ten deltas in a window measure 10 store updates on
mainand 1 here — measured, not estimated: the batching test fails on a stashed tree withexpected 10 to be +0.The three delta buffers (text, tool input, thinking) are one
createDeltaBuffer(onFlush)helper —append,consume,clear,flush,schedule. Each call site keeps only its own flush body, andcontent_deltano longer reads and deletes its maps by hand.Review round 2
Must fix 1 — the stale session after the flush.
content_startandmessage_completenow flush first and readget().sessions[sessionId]afterwards, with the null check on that fresh read. The three sequences you named are tests:content_starttext,content_delta 'abc', 60 ms,thinking 'hmm',content_starttext,content_delta 'def',message_complete→[abc, thinking, def]content_starttool_use→[abc, thinking, tool_use]message_completedirectly →[abc, thinking]Each fails against the pre-flush read, with the shapes you predicted (
[abc, thinking, abcdef],[abc, thinking, abc, tool_use],[abc]).Should fix 2 — the
content_starttest hid its own bug. Thevi.advanceTimersByTime(60)beforecontent_startis gone, socontent_startarrives while the thinking is still buffered and the thinking message can only exist because of the flush at the top of that handler. The test now assertsmessagesis empty before the call and holds the thinking block after it. Removing the flush fails it.Should fix 3 — one test per flush and clear site.
message_complete,error,tool_use_complete,stopGeneration,disconnectSession,clearMessages,session_clearedand theapplyMirroridle path each have a case, plus the local send path.disconnectSessionand the two clears are asserted the durable way: the session is recreated (or the messages cleared) and 60 ms are advanced, so a surviving timer would write into the fresh state.Every one of the 11 new cases was checked the other way round, by deleting the line it covers and re-running it alone:
does not restore text a thinking flush already wrote when content_start repeats itcontent_startdoes not duplicate text a thinking flush wrote when content_start opens a tool blockcontent_startflushes thinking at content_start so the text keeps the streaming statecontent_startkeeps a flushed thinking block when message_complete follows itmessage_completeflushes buffered thinking before a message from this client landssendMessageflushes buffered thinking before an error messageerrorflushes buffered thinking before a tool use completestool_use_completeflushes buffered thinking when the user stops generationstopGenerationdrops buffered thinking when the session disconnectsdisconnectSessiondrops buffered thinking when the messages are clearedclearMessagesdrops buffered thinking when the server clears the sessionsession_clearedfolds buffered text and thinking when a snapshot reports the turn idleapplyMirroridle pathShould fix 4 — one buffer helper.
createDeltaBufferowns the pending map and the flush timer, withappend,consume,clear,flushandschedule;flushis consume-then-callback, so a manual flush and a timer tick share one path.textDeltaBuffer,toolInputDeltaBufferandthinkingDeltaBufferare three instances of it, and the text timer callback is gone:scheduleregistered the 50 ms window,flushreads the buffer.Note — the report line is updated.
docs/features/performance/optimization-report.md:70now sayscontent_deltaandthinkingare throttled,statusandtool_resultare not.Behaviour difference from
main: a message sent from the desktop composer while thinking is buffered now flushes that thinking first, which is what the remoteuser_messagepath already did. Without it the 50 ms timer could land the thinking block after the user bubble.Checks, with the numbers
cd desktop && bunx vitest run src/stores/chatStore.test.ts→ 115 pass (104 before, 11 new)cd desktop && bun run lint→ exit 0 (tsc, tsc electron, oxlint), no outputcd desktop && bunx vitest run→ 3111 pass, 7 fail, 4 skipped (3122). Five of the seven fail by name on the stashed tree:pages.test.tsx×2,LocalSlashCommandPanel,MessageListnested tool calls,UpdateChecker. The other two are wall-clock timeouts on a loaded machine (KanbanPage5.3 s,WorkspacePanel20.3 s); both files pass when run alone on this tree.bun test(server + CLI) → 2724 pass, 15 fail, 1 error. The same 15 fail by name on the stashed tree (16 there, one flaky extra): the ripgrep fixture cases, the artifact path and symlink cases,H5AccessService, the two session-link forks, a 5 sDiagnosticsServicetimeout, and the Windowstarerror. None of them imports the store.Files
desktop/src/stores/chatStore.ts— +164/−128desktop/src/stores/chatStore.test.ts— +345/−0docs/features/performance/optimization-report.md— 1 lineNo new comment, JSDoc or inline note in any file,
flushPendingThinking,createDeltaBufferandmergeThinkingIntoMessagesincluded. No CHANGELOG entry. No visible UI change:MessageList,buildRenderModelandtoolResultMapare untouched, this only stops the store from notifying them per thinking event.