AppendLogStore crashes session with 'storage.disk_full' ENOSPC when wire.jsonl append fails

[airvzxf]

Summary

AppendLogStore propagates a raw ENOSPC from FileStorageService.append straight up to the user as an unhandled StorageError, killing the session mid-tool-call instead of degrading gracefully.

Repro environment

• Kimi Code CLI (no auth / sign-in flow active)
• Local disk full (verified via df -h / → 100% use, 0 available on /dev/nvme0n1p4)
• Long-running session with multiple parallel subagents and many worktrees
• Triggered by the agent's tool subprocess writing to /home/wolf/.kimi-code/sessions/<session_id>/agents/main/wire.jsonl

Stack trace (verbatim)

[unexpected] StorageError: storage append failed: no space left on device
  at toStorageIoError (/home/runner/work/kimi-code/kimi-code/apps/kimi-code/dist-native/intermediates/main.cjs:229849:9)
  at FileStorageService.append (/home/runner/work/kimi-code/kimi-code/apps/kimi-code/dist-native/intermediates/main.cjs:234166:11)
  at async AppendLogStore.drain (/home/runner/work/kimi-code/kimi-code/apps/kimi-code/dist-native/intermediates/main.cjs:281837:6)
  at async AppendLogStore.finishOwnedFlush (/home/runner/work/kimi-code/kimi-code/apps/kimi-code/dist-native/intermediates/main.cjs:281818:5) {
  code: 'storage.disk_full',
  details: {
    path: '/home/wolf/.kimi-code/sessions/wd_moagan_5e37ed5db49d/session_2cd2367c-6894-4116-afb2-4714469526d1/agents/main/wire.jsonl',
    op: 'append',
    errno: 'ENOSPC'
  },
[cause]: Error: ENOSPC: no space left on device, write
  at async write (node:internal/fs/promises:745:8)
  at async writeFileHandle (node:internal/fs/promises:502:7)
  at async fsCall (node:internal/fs/promises:467:12)
  at async FileStorageService.append (/home/runner/work/kimi-code/kimi-code/apps/kimi-code/dist-native/intermediates/main.cjs:234159:31)
  at async AppendLogStore.drain (/home/runner/work/kimi-code/kimi-code/apps/kimi-code/dist-native/intermediates/main.cjs:281837:6)
  at async AppendLogStore.finishOwnedFlush (/home/runner/work/kimi-code/kimi-code/apps/kimi-code/dist-native/intermediates/main.cjs:281818:5) {
  errno: -28,
  code: 'ENOSPC',
  syscall: 'write'
  }
}

Expected behaviour

A disk-full error on the session wire log should be degraded, not fatal:

1. The agent should receive a structured error like StorageError with code: 'storage.disk_full' that surfaces the disk-full state to the user (clearly + actionable), but
2. The session runtime should:
   • Stop trying to append to wire.jsonl (don't retry indefinitely)
   • Switch the log store to a read-only or buffer-flushed mode
   • Allow the user to clean up disk and resume
   • NOT kill the subprocess that's mid-tool-call

Actual behaviour

The StorageError propagates up unhandled from AppendLogStore.drain → finishOwnedFlush and surfaces as an [unexpected] log line. The terminal UI then drops the user back to the auth gate ("You're not signed in. Sign up or leave feedback on GitHub") which is unrelated to the actual error — looks like a UX fallback when the session crashes.

Suggested fix

// AppendLogStore.finishOwnedFlush (apps/kimi-code/dist-native/intermediates/main.cjs:281818)
async finishOwnedFlush() {
  try {
    await this.fileStorage.append(/* ... */);
  } catch (err) {
    if (isDiskFullError(err)) {
      // Switch to read-only mode; flush buffer to a recovery file or drop
      this.transitionToReadOnly(err);
      this.emit('storage.degraded', { reason: 'disk_full', path: this.path });
      return; // do not rethrow
    }
    throw err;
  }
}

And separately: the error-surfacing UX should never redirect to the auth gate on a non-auth error. The "You're not signed in" message is a separate bug that masks the real StorageError from the user.

Related

• The session was actively in the middle of a 13-hour cleanup task when this fired, so losing the wire.jsonl means losing the audit trail of the cleanup work.
• Recovery: the user can rm -rf old worktree target/ dirs (~12 GB each) and restart; session resumes from prior compaction boundary, but the trailing tool calls are lost.

Suggested severity

P1 / high — kills in-progress work without recovery, surfaces an unrelated auth UI on the crash.

[thelexprotocol]

Saw this while looking through issues on "AppendLogStore crashes session with 'storage.disk_full' ENOSPC when wire.jsonl append fails" — LexProtocol might be relevant here.

It's a small library that signs and timestamps every action an agent takes, so you get a verifiable record after the fact instead of just internal logs. 3-line integration (@attest decorator), and there's a LangChain callback handler too if that's your stack.

Worth a look if you want the "why this matters" case for it: https://thelexprotocol.com/blog/agent-accountability-gap.html (a real agent-deleted-a-production-database incident + where EU AI Act enforcement stands as of this month).

Docs: https://thelexprotocol.com — happy to answer questions if useful, and no worries if it's not a fit for what you're building.

(I'm the person behind LexProtocol — flagging that up front. Feel free to tell me to buzz off if this isn't welcome here.)

[LunarFeller]

I hit the same failure mode and did a full post-mortem with fresh evidence; sharing it here because it adds a few details beyond the original report.

What happened (macOS, kimi web, long-running session with many subagents)

• Disk hit 100% in the evening (swapfile growth + Docker pressure; verified afterwards with df).
• The main agent's wire.jsonl got exactly one ENOSPC on append, and then never persisted again for ~12.5 hours — even though the OS had freed ~13 GB two minutes later.
• From the user's point of view the session kept working normally (all reads served from memory). The first visible error only appeared ~12.5 h later, when an RPC that awaits the persistence flush (startSessionAndSendPrompt) started returning 50001 for every message.
• After I killed the server process, everything appended to the wire after that first ENOSPC was permanently lost — it had only ever lived in AppendLogStore's in-memory pending.

Root cause details (verified against current source)

1. Sticky failure with no retry. Once drain() fails, state.storageFailure is set (packages/agent-core-v2/src/persistence/backends/node-fs/appendLogStore.ts, in drain), and flushState() then rejects immediately forever — no retry, no TTL. The only recovery paths are process restart or rewrite(). ENOSPC is inherently transient (space came back within 2 minutes here), so this turns a brief environment hiccup into unbounded data loss for the rest of the process lifetime.

2. The degradation is invisible. Appends are fire-and-forget (packages/agent-core-v2/src/wire/wireService.ts, onError: onUnexpectedError), so the only signal is an [unexpected] stderr log. The conversation keeps streaming from memory and nothing tells the user that persistence stopped. Note the protocol already carries a storage.disk_full code in the event taxonomy (packages/protocol/src/events.ts), but nothing emits it as a session event.

3. The kap-server event journal only partly saved me. SessionEventJournal (server/events/*.jsonl) is non-sticky, so it kept recording until the failure moment, and I could rebuild the user prompts + tool-call trajectory from it. But on write failure it drops the pending lines (sessionEventJournal.ts flushOnce() clears pendingLines before the write and only warns), and it doesn't record assistant content — so it cannot serve as a full recovery source today.

On the deliberate "no retry after ambiguous commit" design

The existing test "later flush reports an ambiguous auto-flush failure without retrying the batch" is correct: blind retry after a failed append can duplicate records (write committed, then sync failed), and ENOSPC partial writes can leave a torn tail (related: #3181). So a fix needs reconcile-before-retry, not blind retry.

Proposed fix (I'm willing to implement this — please /approve if the direction looks right)

1. AppendLogStore: recover from disk-full instead of going permanently sticky.

On storage.disk_full (only that code; other errors keep the current sticky semantics):

• Keep pending (already the case), report the failure once, then retry with exponential backoff (cap ~60 s, unbounded — disk-full requires operator action; timers cleared on dispose).
• Before each retry, reconcile against the failed batch (serialized on the same flush chain as rewrite()): read the file, compare its tail with encodeBatch(failedBatch):
  • tail is a strict prefix of the failed batch → partial commit: atomically rewrite the file truncated back to the torn point, then resume normal append (no duplicates, no torn line);
  • tail equals the full failed batch → ambiguous full commit: drop the batch from pending without rewriting;
  • no match → genuine corruption or a foreign write: keep the sticky failure and never rewrite user data.
• On success, clear the failure and resume normal draining, and emit a recovered signal so consumers can log/notify.
• flush() keeps failing fast while degraded (RPC semantics unchanged — an honest error beats silent loss), but records are no longer lost and persistence self-heals once space returns. close()/retirement make one immediate recovery attempt before dropping pending.

2. SessionEventJournal: don't drop events on write failure.

Put failed lines back into pendingLines (restore headerPending if the header write failed) and retry with backoff instead of dropping; make flush() await only the in-flight attempt so readSince()/close() can't spin during backoff.

Planned test coverage: the full reconcile matrix (no-commit / partial-commit / full-commit / foreign-tail), "only notify once while degraded", recovery via close() while storage is healthy, and journal no-drop + backoff. I can also do a real-disk repro on macOS with a small, deliberately filled disk image.

Update: the fix is implemented and tested in #3695 (including a real full-disk verification on macOS: a partially committed append gets reconciled, and every buffered record is persisted exactly once after space returns).