Skip to content

test(integration): TestNightly<Suite> convention + no-match guard for the consolidated nightly#490

Merged
bdchatham merged 3 commits into
mainfrom
test/nightly-cron-consolidation
Jul 15, 2026
Merged

test(integration): TestNightly<Suite> convention + no-match guard for the consolidated nightly#490
bdchatham merged 3 commits into
mainfrom
test/nightly-cron-consolidation

Conversation

@bdchatham

Copy link
Copy Markdown
Collaborator

Summary

Prepares the integration harness for the platform-side consolidation of the 5 per-suite nightly CronJobs into one. Test-only; no controller/CRD/SDK change.

  1. Per-suite image env vars — running every suite in one process means one environment, so the three suites that can't use the shared SEID_IMAGE default read their own: SEID_IMAGE_MOCK (benchmark), SEID_IMAGE_CHAOS (chaos), SEID_UPGRADE_FROM_IMAGE/SEID_UPGRADE_TO_IMAGE (upgrade's pinned pre/post pair). Release/workflow/giga/giga-mixed keep the shared SEID_IMAGE.
  2. TestNightly<Suite> naming convention — all 7 nightly suites renamed so -test.run '^TestNightly' selects the whole set by convention; a new suite joins by taking the prefix, no allowlist to maintain. TestGenesisCeremonyProducesBlocks stays outside the prefix (onboarding check, run standalone).
  3. No-match-fails-loud guardgo test exits 0 when -test.run matches nothing; against a live cluster that means the harness binary and the pattern are out of sync (an image built before a rename) — a false-healthy nightly that ran nothing. TestMain now fails the run when zero suites started against a cluster. Counts executed suites at requireCluster (the universal entry point), so no per-suite wiring.
  4. Reused-code refactorwitnessNamespace + assertWitnessFresh extracted to harness_test.go; the namespace-derivation and witness-freshness gate were duplicated verbatim between the state-sync and giga-mixed suites (a must-change-together correctness coupling).

Companion / sequencing

The platform PR (sei-protocol/platform#feat/nightly-cron-consolidation) collapses the 5 CronJobs into one nightly-harness-suite running -test.run '^TestNightly'. Merge order matters: this PR must land first so its ECR push-build produces a harness image carrying the TestNightly* names; the platform CronJob then pins that image atomically. The no-match guard is the backstop if that ordering ever slips.

Deliberately not included (recorded in the review ledger): per-suite panic recovery — the completion SLI (count of --- PASS/FAIL: TestNightly == 7, an observability follow-up) is the robust layer for detecting a dropped suite, not scattered recovers.

Review

xreview ledger: bdchatham-designs/designs/seinode-task/xreview/workflow-complete-at-release.md rounds 20-22. Four-lens cross-repo review (kubernetes-specialist, platform-engineer, sre-engineer as assigned dissenter, idiomatic-reviewer) + a systems-engineer raise-the-bar pass + a verify round on the guard/refactor (systems + idiom, both RATIFY). Gates: gofmt, go vet -tags integration, go test -c -tags integration, -test.run '^TestNightly' selection all clean.

🤖 Generated with Claude Code

bdchatham and others added 2 commits July 14, 2026 17:19
Running every nightly suite in one process (one CronJob, one -test.run
regex over the whole set) means one os-level environment: the three
suites whose required image can't be the shared SEID_IMAGE default now
read their own env var name instead of colliding on it.

- TestBenchmark: SEID_IMAGE -> SEID_IMAGE_MOCK (mock_balances flavor)
- TestChaosSuite: SEID_IMAGE -> SEID_IMAGE_CHAOS
  (mock_chain_validation-mock_balances flavor)
- TestChainUpgrade: SEID_IMAGE/SEID_UPGRADE_IMAGE ->
  SEID_UPGRADE_FROM_IMAGE/SEID_UPGRADE_TO_IMAGE (a pinned historical
  version pair, never "latest")

release_test.go, workflow_test.go, giga_migration_test.go, and
giga_mixed_release_test.go are unchanged: all four already want the same
vanilla nightly image and keep reading the shared SEID_IMAGE.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…htly run

Renames the seven suites the nightly schedule runs (TestBenchmark,
TestChaosSuite, TestChainUpgrade, TestRelease, TestWorkflowStateSync,
TestGigaStoreMigration, TestGigaMixedRelease) to TestNightly<Suite>, so
`-test.run '^TestNightly'` selects the whole nightly set by convention —
a new suite joins the schedule by taking the TestNightly prefix, with no
separate allowlist to maintain in this repo or the platform CronJob.
TestGenesisCeremonyProducesBlocks keeps its name: it's an onboarding
check, run standalone, never nightly.

Builds on the prior commit's per-suite image env vars (the actual
one-process, one-environment constraint the combined run has to satisfy;
this commit is purely the selection mechanism on top of that).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cursor

cursor Bot commented Jul 15, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Test-only integration harness and Dockerfile comments; no controller, SDK, or production runtime changes. Operational risk is mainly merge ordering with the platform CronJob consolidation.

Overview
Prepares the integration harness for one nightly CronJob running -test.run '^TestNightly' instead of per-suite jobs. The Dockerfile and package docs now describe that single-binary, whole-set invocation.

Naming: Seven scheduled suites are renamed to TestNightly<Suite> so the regex selects them by convention; TestGenesisCeremonyProducesBlocks stays outside the prefix for standalone onboarding runs.

One process, multiple images: Benchmark, chaos, and upgrade suites read dedicated env vars (SEID_IMAGE_MOCK, SEID_IMAGE_CHAOS, SEID_UPGRADE_FROM_IMAGE / SEID_UPGRADE_TO_IMAGE) because a combined run shares one environment; release/workflow/giga suites keep SEID_IMAGE.

Safety: TestMain fails exit 0 when SEI_NODE_CLUSTER is set but requireCluster never incremented suitesStarted—catching a -test.run pattern that matches nothing after a rename or image skew.

Refactor: witnessNamespace and assertWitnessFresh move to harness_test.go and replace duplicated logic in state-sync and giga-mixed suites.

Reviewed by Cursor Bugbot for commit 8d68711. Bugbot is set up for automated code reviews on this repo. Configure here.

Two bar-raisers from the systems + reuse review of the consolidation:

- TestMain no-match guard: go test exits 0 when -test.run matches nothing,
  which against a live cluster (SEI_NODE_CLUSTER set) means the harness
  binary and the pattern are out of sync (an image built before a suite
  rename) — a false-healthy nightly that ran nothing. TestMain now fails
  the run when zero suites started against a cluster. Counts executed
  suites (incremented in requireCluster, the universal entry point) rather
  than enumerating registered tests, so it needs no per-suite wiring and
  upholds the TestNightly-prefix-is-the-only-join-step invariant. Bounds
  the zero-match case only; positive "all 7 ran" proof is the completion
  SLI on the metrics layer.

- Extract witnessNamespace + assertWitnessFresh into harness_test.go: the
  aggregate-RPC-host namespace derivation and the witness-freshness gate
  were copied verbatim between the state-sync and giga-mixed suites. Both
  now call the shared helpers; giga-mixed's witness section is reordered to
  match (derive ns, build the set, assert freshness) for one reading order.

Panic-isolation (per-suite recover) was considered and declined: it is
per-suite wiring a new suite can silently forget, and the completion SLI
detects a dropped suite regardless of cause (panic, OOM, timeout) — the
robust layer for that concern is the metric, not scattered recovers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bdchatham bdchatham force-pushed the test/nightly-cron-consolidation branch from dd0c279 to 8d68711 Compare July 15, 2026 01:21
@bdchatham bdchatham merged commit 6ebc089 into main Jul 15, 2026
5 checks passed
@bdchatham bdchatham deleted the test/nightly-cron-consolidation branch July 15, 2026 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant