test(integration): state-sync fixture — conventional 4-validator topology, witness-freshness gate

[bdchatham]

Summary

The state-sync fixture was the harness's only single-validator chain, and that topology is the root of the divergence found during the live validation run: a solo validator is gossip-free (it self-votes to quorum), producing at the unthrottled ~15 blocks/s execution rate, while a follower replays consensus at the gossip-paced ~2-3 blocks/s ceiling (100ms block-part pacing + 100ms vote pacing + 50ms commit timeout + per-height WAL fsync). The witness follower diverged from head immediately after its caught-up gate — silently, since catching_up is a one-way latch and this fork has no consensus→blocksync switchback — and the state-sync bootstrap's light client failed its snapshot-height cross-check with "witnesses don't have the blocks."

The fix is the conventional topology, not a custom pacing knob:

• SEI_VALIDATORS defaults to 4 in both suites — the harness convention (release/chaos/bench hardcode 4). Consensus among four paces production to the same gossip cadence followers replay at, so the witness tracks head structurally.
• rpcNodes returns to 1 — the validator set clears blocksync's >1-peer caught-up floor natively; the interim sibling follower is no longer needed.
• New witness-freshness gate: before the bootstrap, assert the witness follower sits within one snapshot interval of the validator head via RPC height reads. catching_up cannot certify at-head, so a future change that makes production outpace replay fails here by name instead of wedging silently.
• The follower keeps snapshot_interval=0 (witness role only; chunks come from the validators over p2p).

A closing detail from review: the old comment's "2-validator genesis blocksync-bootstrap deadlock" — the reason the fixture went single-validator — was the same pool.IsCaughtUp peer floor all along (two validators = one peer each). Four validators clears it, as the sibling suites prove nightly.

Review

xreview ledger: bdchatham-designs/designs/seinode-task/xreview/workflow-complete-at-release.md Rounds 12-13 — blinded sei-network-specialist (assigned dissenter) + idiomatic-reviewer, both RATIFY. The dissenter's outpacing hypothesis (parallel validator gossip vs serial follower pull) was refuted from source: a consensus-mode follower receives parts and votes from all four validators in parallel, one gossip hop behind the quorum. Non-blocking notes on record: the freshness gate is a single-shot read (consistent with the file's point-read idiom), and its 50-block bound is a regime separator relying on co-pacing persistence.

Mechanism citations: sei-tendermint/internal/consensus/reactor.go (gossip pacing, WaitSync latch), internal/blocksync/pool.go:211-214 (peer floor), config/config.go:1269 (100ms gossip sleep), internal/consensus/state.go (commit timeout, WAL fsync).

A gated validation run on harbor from the rebuilt harness image follows the merge.

🤖 Generated with Claude Code

[cursor]

PR Summary

Low Risk
Changes are limited to integration test defaults, fixture topology, and a pre-bootstrap assertion; no production controller or runtime behavior is modified.

Overview
Aligns the state-sync integration fixture with the rest of the harness by defaulting SEI_VALIDATORS to 4 in TestWorkflowStateSync and TestGigaStoreMigration, and provisioning one plain RPC follower instead of two—validators alone satisfy blocksync’s multi-peer caught-up requirement.

Before bootstrapping the state-sync follower, bringUpStateSyncFollower now compares validator-0 and rpc follower-0 via LatestHeight and fails if the follower lags the validator by more than one storage.snapshot_interval, so a witness that drifted behind head is caught before light-client snapshot checks fail opaquely.

Comments in harness_test.go, workflow_test.go, and giga_migration_test.go are updated to document the 4-validator convention, plural validators as snapshot producers, and the new gate; the harness package doc lists TestGigaStoreMigration in the -run set.

^{Reviewed by Cursor Bugbot for commit 1ef5181. Bugbot is set up for automated code reviews on this repo. Configure here.}