Skip to content

fix(components): fail closed on a full or dead repo IndexedDB - #438

Open
lodystage[bot] wants to merge 5 commits into
mainfrom
fix/storage-crisis-recovery
Open

fix(components): fail closed on a full or dead repo IndexedDB#438
lodystage[bot] wants to merge 5 commits into
mainfrom
fix/storage-crisis-recovery

Conversation

@lodystage

@lodystage lodystage Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Related issue

Refs #417

Phase 1 only (classify + fail-closed breaker + blocking recovery screen). The Issue stays open for Phase 2, the filesystem-only Manage storage panel.

Problem / pressure

When a user's disk filled, the renderer's CRDT replica (lody-loro-repo-db-<workspaceId>) hit QuotaExceededError, and Chromium then tore down the backing store while keeping the IDBDatabase object alive. Every later db.transaction() threw InvalidStateError: ... The database connection is closing.

Two things made this worse than a failed operation:

  • It broke on the READ path. IndexedDBStorageAdaptor.loadDoc opens its transaction readwrite because it may consolidate queued updates. So opening a brand-new session room — which writes nothing and returns undefined — still took a readwrite transaction and threw. Session creation failed before any write was attempted, so a guard in the composer would have covered one caller out of many; archive, send, and workspace-catalog writes hit the same dead connection right after.
  • It did not heal. Freeing disk space does not reopen a dying connection, and neither does location.reload() — the store is bound to the renderer PROCESS. So the toast kept firing after the user had already fixed the underlying problem, each retry pasting the raw DOMException into a fresh one.

Summary

Classify the failure under the repo and latch a one-way breaker, then explain it once instead of repeating it.

LoroRepo.create({ storageAdapter: createCrisisAwareStorageAdapter(new IndexedDBStorageAdaptor(…)) })
                                  │
                                  ├── healthy      → delegate
                                  ├── classified   → enterStorageCrisis(), throw StorageCrisisError
                                  └── latched      → throw StorageCrisisError, never touch IndexedDB
                                                     (close() alone stays delegated)
  • lib/storage-crisis.ts — classifies quota / unavailable by walking the cause chain, and latches the FIRST failure for the page lifetime so the recovery screen keeps naming the original cause. Anything unclassified keeps its existing behavior.
  • providers/crisis-aware-storage-adapter.ts — wraps the adaptor passed to LoroRepo.create. Re-throws as StorageCrisisError, so raw IDBDatabase text cannot reach a toast even on the first failure. Only forwards the optional StorageAdapter methods the inner adaptor actually implements.
  • components/storage-crisis-dialog.tsx — mounted in __root.tsx above RuntimeProvider, because loadMeta runs before anything renders. Dismisses the stale toasts, states that a restart is required, keeps the engine text behind a toggle.
  • app.restartApp() / app.quitApp()app.relaunch() + app.quit(). A renderer reload is not a restart here.
  • New --z-storage-crisis: 110, above --z-toast.

Classification sits in Lody rather than in patches/loro-repo.patch: StorageAdapter is a public interface and every method is async, so a bare synchronous db.transaction() throw already surfaces as a rejection the wrapper sees. Patching the published dist/ (two builds) would need re-applying on every version bump for no user-visible gain. Upstream work was opened against loro-dev/loro-repo for the two things a wrapper cannot fix: a stable code on storage errors, and loadDoc not requiring readwrite.

Rationale is routed per .agents/README.md#where-content-goes — a compressed rule in packages/components/AGENTS.md, the explanation in .agents/docs/components-storage-crisis.md, the decision and rejected alternatives in a proposed bug-fix note (EN + ZH).

Before / after

Before After
Every session-create click raises 会话创建失败 with Failed to execute 'transaction' on 'IDBDatabase': The database connection is closing. One blocking screen: local storage is full, free space, then restart. Engine text behind "Technical details".
Toasts keep stacking after the user frees disk space Breaker latches once; further repo calls never reach IndexedDB
Reads could answer undefined from a dead store Reads reject — undefined reads as "no such document" and invites a write over durable history
Recovery offered is a reload, which cannot work restartApp() relaunches the process; browser shells fall back to reload

Test plan

Run on the merged branch (main brought 12 commits, including #431):

  • vitest run in @lody/components: 438 files / 3216 tests pass, 27 new across storage-crisis, crisis-aware-storage-adapter, and storage-crisis-dialog. Covers classification by name and by message, cause-chain walking, a cyclic chain terminating, fail-closed reads, optional methods not being advertised when absent, close() still delegating during a crisis, and the dialog's restart/quit/reload branches.
  • @lody/electron: typecheck clean, 91 tests pass.
  • pnpm test:scripts: 27 pass. tsgo --noEmit clean for @lody/components.
  • pnpm lint (0 errors), check-i18n (en + zh_CN complete), check:public-boundary, check:platform-boundaries, check:code-collab-imports.
  • pnpm run docs check: no errors. packages/components/AGENTS.md is 7905 bytes — under the 8192 gate, but see the gap below.

Not done: the disk-full condition is reproduced from fixtures, not a real exhausted volume; no manual end-to-end run on a full disk. Phase 2 (Manage storage panel) and Phase 3 (upstream PR) are out of scope.

Environment note for anyone re-running locally: with NODE_ENV=production exported in your shell, React 19's production build omits act and every .tsx test fails with act is not a function — including ones untouched here. Use NODE_ENV=test.

Context handoff

Instructions for reviewing agents

  • Review focus: providers/crisis-aware-storage-adapter.ts (does the wrapper preserve StorageAdapter semantics loro-repo relies on?) and lib/storage-crisis.ts classifyStorageFailure (are the match rules narrow enough not to latch on a transient error?).
  • Decisions to challenge: reads failing closed rather than returning undefined; the latch being one-way with no retry; close() being the one method still delegated during a crisis; wrapping in Lody instead of patching loro-repo.
  • Plausible failures / evidence gaps: the message-regex fallback is a heuristic — DOMException prose is not stable API across engines or locales, so a differently worded message is treated as unclassified (fails safe: previous behavior, no wrong latch). Toasts raised after the dialog mounts are not dismissed; they render under the overlay at a lower z-index and carry the controlled message, not engine text. packages/components/AGENTS.md sits 287 bytes under the gate — already past the 7000-byte warning on main at 7489, and I did not restructure other contributors' entries to buy room.

Authoring context

  • User goal / directives: fix Storage Crisis Mode: disk-full / IndexedDB failure UX (会话创建失败 toast) #417 (Storage Crisis Mode); then follow the document-maintenance conventions that landed on main while this was in progress.
  • Constraints / non-goals: Phase 1 only. No Manage-storage panel, no upstream loro-repo change in this PR, no patches/loro-repo.patch edit. Keep the sibling stream-cursor store's fail-open behavior untouched.
  • Risk-bearing decisions: the breaker is one-way for the page lifetime and gates all repo persistence, so a false positive would block writing until restart — which is why classification is narrow and anything unclassified keeps its old path. Reads reject rather than answering empty, chosen specifically to avoid silent overwrite of durable history.
  • Destructive or irreversible behavior: none added. restartApp()/quitApp() end the process; the existing before-quit handler still drains the embedded CLI. The screen deliberately offers no cache-clear button — deleteDatabase() blocks while the runtime holds a connection and is the wrong tool for a full disk.
  • Deliberately not done or tested: no real full-disk end-to-end run; no auto-reopen on InvalidStateError (on a full disk the reopen fails too, and a self-healing storage layer makes failing closed harder); no in-memory repo fallback.
  • Unknowns / confidence: high on the mechanism and its tests; moderate on classification coverage across engines, which is exactly what the upstream code work would settle.

zxch3n and others added 3 commits September 6, 2026 07:08
When the disk or the origin's storage quota ran out, IndexedDB rejected
writes with QuotaExceededError and then killed the connection, so every
later db.transaction() threw InvalidStateError. Session creation broke on
the READ path — openPersistedDoc for a new room already runs a readwrite
transaction — and each retry pasted the raw Chromium DOMException into a
fresh toast. Freeing disk space did not help: the dying connection only
goes away with the process.

Classify the failure and latch one one-way breaker under the repo, then
explain it once in a blocking recovery screen.

- lib/storage-crisis.ts classifies (quota / unavailable) by walking the
  cause chain, and latches the first failure for the page lifetime.
- providers/crisis-aware-storage-adapter.ts wraps the adaptor passed to
  LoroRepo.create. Once latched, every method rejects with
  StorageCrisisError without touching IndexedDB — reads included, since
  an undefined read would look like "no such document" and invite a write
  that overwrites durable history. close() stays delegated: it opens no
  transaction and runtime dispose needs it.
- StorageCrisisDialog (mounted above RuntimeProvider, since loadMeta runs
  before anything renders) dismisses the stale toasts, states that a
  restart is required, and keeps the raw engine text behind a toggle.
- app.restartApp() / app.quitApp() relaunch the process. A renderer
  reload is not enough; Chromium binds the dying backing store to it.

Classification sits in Lody rather than in a loro-repo patch because
StorageAdapter is a public interface and every method is async, so a bare
synchronous throw surfaces as a rejection the wrapper already sees.

Verified: 3208 component tests (27 new), 79 Electron tests, typecheck,
lint, i18n, and the public/platform/code-collab boundary guards.

Refs #417

Model: claude-opus-5[1m]

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merging main brought the document maintenance process (#431), which trims
every AGENTS.md under an 8192-byte gate. The storage-crisis entry landed
as a 17-line block and pushed packages/components/AGENTS.md to 8898, the
only error `pnpm run docs check` reported.

Route it per .agents/README.md#where-content-goes rather than trimming
words: the binding constraint stays as one compressed bullet in the
nearest AGENTS.md, the cross-module explanation moves to
.agents/docs/components-storage-crisis.md, and the decision with its
rejected alternatives becomes a proposed bug-fix note in both languages.

The note records what a future maintainer could plausibly get wrong: an
in-memory repo fallback copied from the resilient cursor store, reads
answering undefined instead of rejecting, auto-reopen on
InvalidStateError, and patching loro-repo's dist. It also records that
loro-repo does not lose data when a save fails — persistDocUpdate rolls
its version pointer back and MetaPersister advances only after save
resolves — and names the message-regex fallback as the known limit that
upstream work would remove.

`pnpm run docs check`: no errors. packages/components/AGENTS.md is 7905
bytes, 287 under the gate; it was already past the 7000-byte warning on
main at 7489.

Model: claude-opus-5[1m]

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
zxch3n and others added 2 commits September 6, 2026 07:28
The upstream session verified all three findings against loro-repo source
and landed loro-dev/loro-repo#129 (a RepoStorageError carrying
code: 'quota' | 'unavailable' | 'unknown', plus loadDoc no longer needing
readwrite) and issue #130 for the missing retry trigger. Name both
instead of "upstream work was opened".

Two corrections to the note's own claims:

- The verification counts predated the main merge. They are now 3216
  tests across 438 files and 91 Electron tests, attributed to the merged
  branch at 90c6eaf, and the limits section admits that toasts raised
  after the screen mounts are not dismissed.
- The brief this repo sent upstream asked for loadDoc's transaction to be
  split in two. That was wrong: the readwrite is deliberate and commented
  upstream, because splitting the read from delete(docId) drops updates
  appended in between. The note records the actual fix, which generalizes
  the existing compare-then-write helper.

Also record why `code` is deliberately not read yet: the contract is open
for review upstream and could be renamed, and a field that never matches
would be dead code behind working heuristics. Verified that #129's rename
to RepoStorageError is safe here — nothing in Lody compares an error name
to 'Error', and classifyStorageFailure walks `cause` regardless.

Adds the Lody PR link the note rules ask for, and states that the issue
stays open for Phase 2.

`pnpm run docs check`: no errors.

Model: claude-opus-5[1m]

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The note said the failed-save behavior is "a UX and correctness-of-failure
problem, not a durability bug". True but imprecise: nothing re-triggers a
flush after a failure, so the re-queued document waits for the next doc
event and the meta Flock for its next subscription callback. No data is
lost, but the last change before a transient failure stays unpersisted
until the user edits again. Filed as loro-dev/loro-repo#130.

Also strengthen the rejected auto-reopen alternative with evidence found
afterwards: ensureDb caches the promise from a FAILED open and never
clears it, unlike the close() and versionchange paths, so the adaptor is
already dead for the page lifetime once opening fails
(loro-dev/loro-repo#131). An app-level reopen would have been built on a
layer that cannot reopen itself.

Both are upstream and out of scope here; they are recorded because one
corrects a claim this note makes and the other supports a decision it
defends.

`pnpm run docs check`: no errors.

Model: claude-opus-5[1m]

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Storage Crisis Mode: disk-full / IndexedDB failure UX (会话创建失败 toast)

1 participant