Keep a persistent append handle for `events.jsonl` instead of open/append/close per event (Windows Defender rescan / CPU cost)

[tom-un]

Summary

Each session's event log at ~/.copilot/session-state/<id>/events.jsonl is appended one event at a time via fsPromises.appendFile(path, line, { mode }) (guarded by a per-file runExclusive mutex). When appendFile is called with a path (rather than an open FileHandle), Node opens the file with flag 'a', writes, and closes it on every call — i.e. a CreateFile → WriteFile → CloseHandle cycle per emitted event.

On Windows, that Create+Close per event is what triggers Windows Defender (and other real-time AV) to rescan the file — the close, not the write, is the scan trigger. During an active, tool-heavy turn events are emitted tens–hundreds of times per second, so Defender rescans events.jsonl (which grows monotonically through the turn) that many times per second, producing a sustained CPU/energy regression on the user's machine.

We're seeing this downstream in a desktop app (Microsoft Scout) that bundles the Copilot CLI; events.jsonl is the largest remaining source of AV rescans after we cut our own state-file write churn. Tracked internally as ADO 62904486, and possibly the same root cause as #3907 ("CPU usage at max.").

Where

• Append: runExclusive(lockKey, async () => { await mkdir(dir, { recursive: true }); await appendFile(eventsPath, line, { mode }); }) — one open/append/close per event.
• Compaction: the truncate path does a single writeFile(eventsPath, fullContent, { mode }) (full rewrite). That's occasional and fine.

Ask

Hold a persistent append FileHandle open per session and reuse it:

• Open once per session with fs.open(eventsPath, 'a') (or a cached createWriteStream), then append via fileHandle.appendFile(line) / fileHandle.write(line) for each event, and close() once at session end (or on the truncate/compaction rewrite, then reopen).
• Because the handle stays open, the file identity is preserved and there is no per-event Create/Close, so AV real-time protection incrementally scans only the appended bytes instead of rescanning the whole file on every event.
• The existing runExclusive mutex already serializes writers, so a single long-lived handle is compatible — the only added lifecycle concern is closing/reopening around the truncate rewrite and on session teardown.

If a persistent handle is undesirable, a smaller mitigation is to batch events and flush on a short interval / at turn boundaries, which reduces (but doesn't eliminate) the open/append/close frequency.

Impact

• Removes the dominant per-event AV rescan trigger on Windows; the cost currently scales with turn length × event volume, so long tool-heavy turns — when the machine is already busy — are the worst case.
• Append-only semantics and on-disk format are unchanged; events.jsonl remains the documented hook/event stream (cf. Feature Request: Formalize events.jsonl as an official hook/integration API #3551).

Environment

• OS: Windows 10/11, Microsoft Defender real-time protection enabled.
• Copilot CLI (@github/copilot) bundled inside an Electron desktop app.
• Path: %USERPROFILE%\.copilot\session-state\<session-id>\events.jsonl.
• Observed mechanism: fsPromises.appendFile(path, …) per event (verified in the bundled app.js).

Downstream reference

gim-home/m#4426 — same class of problem (Defender rescans from high-frequency small-file writes) for the desktop app's own .scout state files, mitigated there by coalescing writes; events.jsonl is the CLI-owned remainder.