State WAL replacement by cody-littley · Pull Request #3701 · sei-protocol/sei-chain

cody-littley · 2026-07-06T14:57:38Z

Describe your changes and provide context

This WAL is intended to replace all existing WALs in both SS and SC. Instead of having multiple WALs for multiple different services, we can maintain just a single WAL which is used universally across all parts of the storage stack.

This PR does not integrate the changes, it just creates a WAL utility. Integration work will happen in future PRs. We've got a pre-existing WAL, but that implementation has some serious issues. Namely, the WAL library we currently use has very inefficient garbage collection (it rewrites garbage collected files, as opposed to dropping files as a whole when data in file is all stale).

Testing performed to validate your change

unit tests

github-actions · 2026-07-06T14:59:25Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Jul 15, 2026, 7:58 PM

codecov · 2026-07-06T14:59:49Z

Codecov Report

❌ Patch coverage is 77.16966% with 292 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.06%. Comparing base (c6ab0be) to head (ed97f93).
⚠️ Report is 22 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/seiwal/seiwal_file.go	64.68%	52 Missing and 43 partials ⚠️
sei-db/seiwal/seiwal_impl.go	76.24%	58 Missing and 33 partials ⚠️
sei-db/seiwal/seiwal_serializing.go	85.63%	22 Missing and 5 partials ⚠️
sei-db/state_db/statewal/state_wal_impl.go	81.30%	12 Missing and 11 partials ⚠️
sei-db/seiwal/seiwal_offline.go	71.83%	10 Missing and 10 partials ⚠️
sei-db/seiwal/seiwal_iterator.go	88.18%	9 Missing and 6 partials ⚠️
sei-db/seiwal/seiwal_lock.go	50.00%	5 Missing and 4 partials ⚠️
...ei-db/state_db/statewal/state_wal_serialization.go	79.48%	4 Missing and 4 partials ⚠️
sei-db/seiwal/metrics.go	77.77%	1 Missing and 1 partial ⚠️
sei-db/seiwal/seiwal.go	71.42%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3701      +/-   ##
==========================================
- Coverage   59.85%   59.06%   -0.79%     
==========================================
  Files        2278     2203      -75     
  Lines      189140   180599    -8541     
==========================================
- Hits       113202   106665    -6537     
+ Misses      65861    64541    -1320     
+ Partials    10077     9393     -684

Flag	Coverage Δ
sei-chain-pr	`76.12% <76.12%> (?)`
sei-db	`70.41% <ø> (ø)`
sei-db-state-db	`?`
sei-db-state-db-pr	`83.33% <83.33%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/seiwal/seiwal_config.go	`100.00% <100.00%> (ø)`
sei-db/state_db/statewal/state_wal_config.go	`100.00% <100.00%> (ø)`
sei-db/seiwal/metrics.go	`77.77% <77.77%> (ø)`
sei-db/seiwal/seiwal.go	`71.42% <71.42%> (ø)`
...ei-db/state_db/statewal/state_wal_serialization.go	`79.48% <79.48%> (ø)`
sei-db/seiwal/seiwal_lock.go	`50.00% <50.00%> (ø)`
sei-db/seiwal/seiwal_iterator.go	`88.18% <88.18%> (ø)`
sei-db/seiwal/seiwal_offline.go	`71.83% <71.83%> (ø)`
sei-db/state_db/statewal/state_wal_impl.go	`81.30% <81.30%> (ø)`
sei-db/seiwal/seiwal_serializing.go	`85.63% <85.63%> (ø)`
... and 2 more

... and 87 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cursor · 2026-07-06T17:36:43Z

PR Summary

Medium Risk
New durability-critical storage code with complex crash/rollback semantics; risk is moderated because this PR only lands the library and tests, not production integration.

Overview
Introduces a new seiwal write-ahead log library meant to replace fragmented WAL usage across the storage stack (integration deferred to follow-up PRs). It provides a generic, monotonic-index WAL[T] with async append/flush, file rotation and whole-file pruning (instead of rewriting GC’d segments), crash recovery (orphan sealing, torn-tail tolerance, rollback remnants), and iterators that read hard-linked point-in-time snapshots so pruning does not block replay.

The on-disk format uses sealed/mutable files, CRC-framed records, a directory wal.lock, and offline helpers GetRange, PruneAfter, and VerifyIntegrity. NewGenericWAL adds background serialization with OpenTelemetry metrics (seiwal_* counters/gauges).

Adds statewal, a block-scoped StateWAL over []*proto.NamedChangeSet (ordered Write / SignalEndOfBlock, flush only completed blocks) built on seiwal. .gitignore now ignores sei-db/seiwal/walsim/bin/.

^{Reviewed by Cursor Bugbot for commit ed97f93. Bugbot is set up for automated code reviews on this repo. Configure here.}

blindchaser · 2026-07-07T17:08:20Z

+	if err := w.enforceWriteOrdering(blockNumber); err != nil {
+		return fmt.Errorf("write rejected: %w", err)
+	}
+	if err := w.sendToSerializer(dataToBeSerialized{entry: NewEntry(blockNumber, cs)}); err != nil {


Write keeps caller-owned changesets and serializes them later. Because Write returns after enqueueing, callers can mutate or reuse cs before serializerLoop runs, causing the WAL to persist changed data or race with the caller

This pattern is intentional for performance, but not well documented.

I updated all godocs to say that it is unsafe to modify slices passed into methods or slices received from methods.

blindchaser · 2026-07-10T20:27:15Z

+		index:     index,
+		serialize: func() ([]byte, error) { return s.serialize(data) },
+	}
+	if err := s.submit(req); err != nil {


+1. two implementation notes if we add the gate here: (1) the gate needs to be seeded from inner.Bounds() at construction (same as newWAL does for lastAppendIndex), otherwise the first append after a reopen goes unchecked; (2) TestGenericWALAppendOrdering currently asserts the async-brick behavior as expected, so its assertions need to flip to a synchronous rejection.

blindchaser · 2026-07-14T14:25:48Z

+// from an interrupted rollback and seals any unsealed file left behind by a prior session. After it returns,
+// every record lives in a sealed file whose name matches its content, with no orphans remaining. Shared by
+// the WAL constructor and the offline GetRange/PruneAfter utilities, so all three run the same sanity pass.
+func recoverDirectory(path string) error {


a second wal instance or an offline utility opens the same path while one is live may corrupts the log

can we add a FileLock to the dir?

added file locks

blindchaser · 2026-07-14T15:10:56Z

+			resp := w.startIterator(m.startIndex)
+			m.reply <- resp
+			if resp.err != nil {
+				w.fail(resp.err)


Iterator construction failure bricks the entire write path, and hard-link support becomes a hard filesystem requirement

should we distinguish the different failure cases? like for MkdirAll/os.Link error, just return error instead of bricking the entire write path

Iterator failures are no longer fatal for failure modes that do not lead to corrupted state.

blindchaser · 2026-07-14T15:15:09Z

+}
+
+// Write accumulates a set of changes for the given block number in memory.
+func (w *stateWALImpl) Write(blockNumber uint64, cs []*proto.NamedChangeSet) error {


the nil changeset check is performed asynchronously in state_wal_serialization.go. As a result, Write() and potentially SignalEndOfBlock() can return successfully even when the input contains a nil changeset. The serialization goroutine discovers the error later and bricks the WAL, so the error surfaces at an unrelated subsequent operation.

can we validate nil entries synchronously in Write() ? so the invalid input is rejected at the call site without bricking the WAL.

change made

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ed97f93. Configure here.}

seidroid

A well-documented, thoroughly tested new WAL utility package (seiwal) and a statewal wrapper; the code is not yet integrated and is carefully engineered for crash durability and recovery. The only code finding is a low-severity uint64 overflow in the contiguous-index validation.

Findings: 0 blocking | 5 non-blocking | 1 posted inline

Blockers

None at the file/PR level.

Non-blocking

Cursor's review (cursor-review.md) is empty — that second-opinion pass produced no output.
REVIEW_GUIDELINES.md (from the base branch) is empty, so no repo-specific review standards were applied.
seiwal_iterator.go:290 acknowledges (in a comment) that locating the start index is a linear scan over a fully-loaded file; fine for now, but worth revisiting if startup replay from a deep offset becomes a hotspot, as the comment notes.
Overall this is high-quality, self-contained utility code with strong test coverage; the main integration/correctness risk will land in the follow-up PRs that wire it into SS/SC.
1 suggestion(s)/nit(s) flagged inline on specific lines.

seidroid · 2026-07-15T20:02:48Z

+					"append rejected: index %d is not greater than last appended index %d",
+					index, w.lastAppendIndex)
+			}
+		} else if index != w.lastAppendIndex+1 {


[nit] uint64 overflow edge case in the contiguous check: when lastAppendIndex == math.MaxUint64, lastAppendIndex+1 wraps to 0, so an Append(0, ...) is incorrectly accepted as "contiguous", producing a record with an inverted index relative to the file's existing max. The writer-side backstop at appendRecord (line 489, m.index != w.lastWrittenIndex+1) has the identical wrap. Reaching it only requires the baseline (first) append to use a near-max index rather than 2^64 total records. Realistically block numbers won't approach 2^64, so severity is low, but a cheap explicit guard (reject once the max index is reached) in both the caller-side and writer-side checks would harden it. (Also raised by Codex.)

claude

Beyond the inline findings, I also checked two other candidate issues and ruled them out: walImpl.Close()'s per-call closeErr local means a second Close() call after a failed first one would return nil — but Close() is only ever expected to be called once, so this isn't reachable in practice. And walImpl.Append advancing lastAppendIndex/hasAppended before sendToWriter succeeds is fine because sendToWriter only fails once the WAL is already closed or bricked, at which point no retry could succeed anyway.

Extended reasoning...

This run's two inline findings (the serializing/StateWAL layers still bricking on non-fatal iterator snapshot errors that walImpl itself now tolerates, and a narrow EOF-vs-shutdown race in the iterator) are the substantive output of this pass. I additionally reviewed the two candidate issues surfaced by finder agents and confirmed they are not real bugs, as noted in the message.

blindchaser · 2026-07-16T19:13:21Z

 sei-db/ledger_db/block/blocksim/bin/
-sei-db/db_engine/litt/bin/
+sei-db/db_engine/litt/bin/
+sei-db/seiwal/walsim/bin/


it seems there is no such folder

blindchaser · 2026-07-16T19:16:12Z

+	w.hasAppended = true
+
+	record := frameRecord(index, data)
+	if err := w.sendToWriter(dataToBeWritten{record: record, index: index}); err != nil {


what happens if the w.sendToWriter() fails but lastAppendIndex and hasAppended already advances. should we move w.lastAppendIndex = index / w.hasAppended = true before w.sendToWriter?

Cody Littley added 8 commits July 2, 2026 14:48

flatKV WAL implementation

33a6cb0

iterator improvements

855cbe9

bugfixes

6a1efd7

bugfixes

d86fdc5

bugfixes

c3207da

fix recovery bug

cab8132

fix bugs

12b943f

bugfix

6ea0aab

cody-littley self-assigned this Jul 6, 2026

cody-littley added the non-app-hash-breaking label Jul 6, 2026

rename

54dd10c

cody-littley marked this pull request as ready for review July 6, 2026 17:35

cody-littley requested a review from blindchaser July 6, 2026 17:35

This comment was marked as low quality.

Sign in to view

cursor Bot reviewed Jul 6, 2026

View reviewed changes

Comment thread sei-db/state_db/statewal/state_wal_impl.go

claude Bot reviewed Jul 6, 2026

View reviewed changes

Comment thread sei-db/state_db/statewal/state_wal_entry.go Outdated

Cody Littley added 2 commits July 7, 2026 10:30

made suggested changes

3fbc79a

move statewal to requested location

101e586

This comment was marked as low quality.

Sign in to view

claude Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread sei-db/state_db/statewal/state_wal_impl.go Outdated

blindchaser reviewed Jul 7, 2026

View reviewed changes

Cody Littley added 6 commits July 8, 2026 08:11

split into abstract utility

691a217

Merge branch 'main' into cjl/flatkv-wal

a201ba8

create async serializing utility

9e4df1b

iterate and improve

47b5ac0

document thread safety better

6ebc75a

bugfixes

6191630

Cody Littley added 4 commits July 13, 2026 13:09

static functions

9c696be

better iterators

3a0d826

Added note about possible perf hotspot

efb976e

don't scan DB at startup time

fe3cede

This comment was marked as low quality.

Sign in to view

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread sei-db/seiwal/seiwal_serializing.go

Comment thread sei-db/seiwal/seiwal_file.go

claude Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread sei-db/seiwal/seiwal_file.go

Comment thread sei-db/seiwal/seiwal_iterator.go

Comment thread sei-db/state_db/statewal/state_wal.go Outdated

made suggested changes

59562f2

This comment was marked as low quality.

Sign in to view

made suggested fix

52142bc

This comment was marked as low quality.

Sign in to view

blindchaser reviewed Jul 14, 2026

View reviewed changes

make suggested changes to iterator

086b4e2

cursor Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread sei-db/seiwal/seiwal_impl.go

This comment was marked as low quality.

Sign in to view

claude Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread sei-db/seiwal/seiwal_offline.go Outdated

Comment thread sei-db/seiwal/seiwal_impl.go

Comment thread sei-db/seiwal/seiwal_impl.go

Comment thread sei-db/seiwal/seiwal_impl.go

Comment thread sei-db/state_db/statewal/state_wal_impl.go

Cody Littley and others added 6 commits July 15, 2026 12:11

added file locks

e62672f

made suggested changes

12ccd55

made suggested changes

06f64a5

bugfix

011ae8b

bugfixes

054f452

ignore sei-db/seiwal/walsim/bin build artifacts

ed97f93

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cursor Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread sei-db/seiwal/seiwal_serializing.go

seidroid Bot approved these changes Jul 15, 2026

View reviewed changes

claude Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread sei-db/seiwal/seiwal_serializing.go

Comment thread sei-db/seiwal/seiwal_iterator.go

blindchaser reviewed Jul 16, 2026

View reviewed changes

blindchaser approved these changes Jul 16, 2026

View reviewed changes

yzang2019 approved these changes Jul 16, 2026

View reviewed changes

Uh oh!

Conversation

cody-littley commented Jul 6, 2026

Describe your changes and provide context

Testing performed to validate your change

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cursor Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

This comment was marked as low quality.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as low quality.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as low quality.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as low quality.

Uh oh!

This comment was marked as low quality.

Uh oh!

This comment was marked as low quality.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as low quality.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

seidroid Bot left a comment

Choose a reason for hiding this comment

Blockers

Non-blocking

Uh oh!

seidroid Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 •

edited

Loading

codecov Bot commented Jul 6, 2026 •

edited

Loading

cursor Bot commented Jul 6, 2026 •

edited

Loading