test(integration): state-sync fixture — peer-floor fix, followers stop producing snapshots

[bdchatham]

Summary

The state-sync suites (TestWorkflowStateSync, TestGigaStoreMigration) stalled unconditionally in their shared fixture: sei-tendermint's blocksync caught-up check has a hard >1-peer floor (pool.IsCaughtUp returns false for a single-peer node before any height math, internal/blocksync/pool.go:211-214, present since 2023), and the fixture's first rpc follower had exactly one peer — the lone validator — at its WaitCaughtUp gate. catching_up could never flip; both suites burned their 60m scenario deadline inside provision. The fixture never had a green execution: the state-sync round-trip was proven manually on a multi-peer chain, the single-validator topology landed later on review + compile gates only, and no nightly cron runs these suites.

Two changes, both required:

• provision brings rpc nodes up in two phases — create every node (createRPCNode: CreateNode + WaitReady), then gate each (gateRPCNode: WaitCaughtUp + WaitEVMServing). Gating node i before node i+1 exists strands a follower whose only peer is the validator.
• The fixture provisions two followers, and followers produce no snapshots (rpcConfig zeroes the interval). The sibling exists to be the second blocksync peer. A light-client witness serves RPC trust points; snapshot chunks come from the producing validator over p2p — and a producing follower at this chain's block rate requested snapshots faster than one takes to write, keeping a snapshot operation perpetually in progress (ErrConflict every interval).

Blast radius on the other suites sharing provision is nil: release/chaos/bench are 4-validator topologies (floor already cleared), genesis/upgrade have zero rpc nodes.

Root cause

Root-caused live on harbor with blinded chain + infra lenses (/root-cause). Environment falsified by the assigned dissenter's own tests: workloads on disjoint nodes, no CPU limits, provisioned 10k-IOPS gp3, no chaos/netpol/pressure, and the co-tenant started 13 min after the stall began and completed while it persisted. Peer floor confirmed from source and live signals (n_peers=1; apply ≈14.5 blk/s vs produce ≈14.7 — a follower tracking seconds behind a bar it structurally cannot be credited with reaching).

Review

xreview ledger: bdchatham-designs/designs/seinode-task/xreview/workflow-complete-at-release.md Rounds 10-11. First fix attempt (rpcNodes 1→2 alone) was caught broken by three independent blinded reviewers — provision's serial gate meant the second peer arrived too late — and repaired with the phase split. Round 11 unanimous RATIFY (sei-network-specialist as assigned dissenter, systems-engineer, idiomatic-reviewer).

Two named empirical residuals, settled by the first live run (this fixture has never run green, so the run is the proof):

1. With the floor cleared, IsCaughtUp still needs the follower to touch maxPeerHeight-1 against ~15 blk/s production. Snapshot-drag removal is the material lever; maxPeerHeight is a periodically-reported (slightly stale) bar. Moderate-to-good confidence; a timeout here means the apply-rate flake persists and needs a different lever.
2. The phase split assumes WaitReady's serve-probes clear pre-catch-up (they gate on liveness, not catching_up — probe.go:46-48,83-85); a seid image that errors eth_blockNumber until synced would re-import the stall into the create loop.

A validation run on harbor from the rebuilt harness image follows the merge.

🤖 Generated with Claude Code

[cursor]

PR Summary

Low Risk
Changes are limited to integration test harness and fixture comments; production controller/SDK behavior is unchanged.

Overview
Fixes integration stalls where RPC followers never cleared catching_up during provision, because sei-tendermint’s blocksync IsCaughtUp stays false when a node has only one peer.

provision now uses two phases: create every RPC node (createRPCNode: running only), then gate all of them (gateRPCNode: caught-up + EVM). The old serial “create then fully gate” path could leave the first follower with only the validator as a peer until a second follower existed.

The shared state-sync fixture (bringUpStateSyncFollower) provisions two plain RPC followers (the extra one is a blocksync peer) and sets storage.snapshot_interval: "0" on followers so only the genesis validator produces snapshots. Comments are updated to match: witnesses supply RPC trust points; snapshot chunks come from the validator over p2p.

^{Reviewed by Cursor Bugbot for commit d3bb76d. Bugbot is set up for automated code reviews on this repo. Configure here.}