test(integration): TestNightlyGigaMixedRelease — one exclusive conformance run

[bdchatham]

Problem

The nightly TestNightlyGigaMixedRelease suite failed (nightly-harness-suite-manual-validate). Root cause: the test launched two release-test conformance Jobs concurrently against one shared chain (one per RPC follower). The release-test suite is written for a single exclusive tenant — its stateful sequences (EVM filters, send-then-wait) assume one consistent mempool + filter-store view — so the two runs cross-talked on global block state and inclusion timing:

• The failing gas case asserts on the EIP-1559 base fee, which is per-block and driven by every tx in the block, not per-account — so the sibling run's traffic perturbed it (off-by-one).
• The erc721 failed-tx case reads a receipt after a fixed 1s wait; under the doubled load the receipt wasn't mined in time → null-receipt cascade.

Separate admin accounts isolate per-account state but cannot isolate global block state, so the failures were nondeterministic test artifacts, not a chain or giga-store regression. A scoped reproduction confirmed this: re-running the original design failed with a completely different set of cases, and the same conformance image passes single-run in TestNightlyRelease.

Fix

Redesign to the suite's true invariant — a mixed giga/v2 follower set is invisible to consensus — which needs only one traffic source:

1. Provision one 4-validator, snapshot-producing chain + two RPC followers; migrate one follower to giga (evm-ss-split, pebbledb).
2. Run the conformance harness once, against the v2 (full-history, shipped-default) follower, with exclusive chain access.
3. Assert that through the load the chain kept advancing and both followers stayed at the validator head.

Per-node SS read+serve parity is already covered by TestNightlyGigaStoreMigration; this suite's distinct job is the mixed-fleet-under-load consensus invariant.

Notable

• Head-parity, not catching_up. The invariant is asserted by comparing each follower's LatestHeight to the validator head (within one snapshot interval), plus a head-advancement check. catching_up is a one-way latch that already flips pre-load, so it would false-green a post-load halt or a giga follower silently falling behind — the one failure mode this suite exists to catch.
• Shared runReleaseTest helper extracted (dedups the Job-launch orchestration; TestNightlyRelease now uses it, behavior unchanged); concurrency-only waitJobErr deleted.
• Root test now reads as five named sequential steps; ctx envelope 90m → 110m for the now-sequential critical path.

Testing

• gofmt clean; go vet -tags integration ./... and go build -tags integration ./... clean.
• Two independent review rounds (Kubernetes/harness correctness + Go idiom): both fixes confirmed sound, no blocker.

Note: fixing the live nightly also requires building a new integration-harness image from this branch and pointing the nightly-harness-suite CronJob at it — a build/deploy step separate from this code change.

🤖 Generated with Claude Code

[cursor]

PR Summary

Low Risk
Integration test-only refactor with no production controller or runtime behavior changes; nightly coverage intent is clarified and should be more stable.

Overview
Fixes flaky TestNightlyGigaMixedRelease by dropping two concurrent release-test Jobs on one chain. The harness assumes a single mempool/filter view; parallel runs interfered on global block state (e.g. EIP-1559 base fee) and inclusion timing, so failures were test artifacts, not giga regressions.

The suite now runs one conformance Job against the v2 control follower, then checks the real invariant: validator head advanced under load and both giga and v2 followers stay within snapshot-interval lag of that head (not catching_up, which can false-pass after a halt). Giga migration is factored into migrateFollowerToGiga, assertGigaFollowerServing, and assertFollowerAtHead; one admin account replaces two.

runReleaseTest in release_test.go centralizes REST probe, Secret, Job launch, and log tail; TestNightlyRelease calls it unchanged aside from wiring. Concurrent waitJobErr and inline k8s Job code are removed. Mixed-release timeout is 110m (was 120m for overlapping runs).

^{Reviewed by Cursor Bugbot for commit 45e33dd. Bugbot is set up for automated code reviews on this repo. Configure here.}