Skip to content

fix(test): make UBI fast-seed rung test time-independent (anchor to now)#624

Merged
SoundMindsAI merged 1 commit into
mainfrom
fix/ubi-fast-seed-es-flake-resilience
Jul 12, 2026
Merged

fix(test): make UBI fast-seed rung test time-independent (anchor to now)#624
SoundMindsAI merged 1 commit into
mainfrom
fix/ubi-fast-seed-es-flake-resilience

Conversation

@SoundMindsAI

@SoundMindsAI SoundMindsAI commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Summary

test_synthetic_ubi_seed_round_trip_hits_rung_3 has been red on main since ~2026-06-28 — a deterministic time-bomb, not an ES flake (my initial diagnosis on PR #623 was wrong; see below).

The test hardcoded seed_anchor_iso="2026-05-29", but classify_rung counts events only within a trailing [now - 30d, now] window (DEFAULT_READINESS_WINDOW_DAYS = 30). Once "now" drifts >30 days past the anchor, every seeded event falls outside the window → count 0 → rung collapses to rung_1, for everyone, regardless of ES health. It surfaced on the 2026-07-11 security PR simply because that was the first CI run past the bomb.

Fix

Anchor the synthetic events to now - 1 day so they always land inside classify's window — matching the real demo paths (demo_seeding.py:1448, seed_meaningful_demos.py:2469), which already anchor at now. This is the only call site that runs the real now-windowed classify_rung against a hardcoded past anchor (the unit tests anchor-relative internally, so they're unaffected).

Diff is two lines of substance: a datetime import and the relative anchor.

How the real cause was found

The first attempt on this branch added a searchable-_count verify + classify retry, on the theory that an ES "name resolution" flake was under-counting the read. CI then reported "verified 640 searchable in ES … got rung_1" — proving the events were present and searchable, so the flake theory was wrong and the count was a window problem, not a presence problem. That scaffolding was reverted; this PR is just the anchor fix.

Test plan

  • ruff check + mypy + py_compile clean
  • CI backend (contract + integration) — now expected to pass (events land inside the classify window)

Once green and stable, backend (contract + integration) becomes a candidate for a required status check under the new main branch protection.

🤖 Generated with Claude Code

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves the resilience of the integration test test_demo_seeding_ubi_fast.py against transient Elasticsearch connectivity issues on CI runners by verifying that seeded events are searchable before asserting the rung and adding a retry loop for the classification step. The review feedback suggests enhancing error handling in these new additions: first, by catching TypeError when parsing the Elasticsearch response to handle unexpected JSON structures gracefully, and second, by wrapping the classify_rung call in a try...except block within the retry loop so that transient network exceptions do not prematurely fail the test.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread backend/tests/integration/test_demo_seeding_ubi_fast.py Outdated
Comment thread backend/tests/integration/test_demo_seeding_ubi_fast.py Outdated
test_synthetic_ubi_seed_round_trip_hits_rung_3 has been red on main since
~2026-06-28: it hardcoded seed_anchor_iso='2026-05-29', but classify_rung
counts events only within a trailing [now - 30d, now] window. ~30 days
after the anchor date every seeded event falls outside the window, the
count drops to 0, and the rung collapses to rung_1 — deterministically,
for everyone, regardless of ES health. (It surfaced on the 2026-07-11
security PR simply as the first CI run past the time-bomb.)

Fix: anchor the synthetic events to (now - 1 day) so they always land
inside classify's window — matching the real demo paths (demo_seeding.py,
seed_meaningful_demos.py), which already anchor at now. This is the only
call site that runs the real now-windowed classify_rung against a
hardcoded past anchor.

Unblocks 'backend (contract + integration)' as a future required status
check under the new main branch protection.

Signed-off-by: SoundMindsAI <eric.starr@soundminds.ai>
@SoundMindsAI SoundMindsAI force-pushed the fix/ubi-fast-seed-es-flake-resilience branch from 38e0b5f to 9811bb6 Compare July 12, 2026 00:57
@SoundMindsAI SoundMindsAI changed the title test(demo-ubi): make fast-seed rung test resilient to ES flakes fix(test): make UBI fast-seed rung test time-independent (anchor to now) Jul 12, 2026
@SoundMindsAI

Copy link
Copy Markdown
Owner Author

Review adjudication (Gemini Code Assist)

Both inline comments target the first revision of this branch (a searchable-_count verify + classify_rung retry loop). That approach was reverted after CI proved the failure was a time-window bug, not an ES flake — this PR is now just the relative-anchor fix, so both comments are obsolete (GitHub marks them outdated). Resolving.

# Location Verdict
1 retry loop Obsolete — code reverted
2 ES unexpected-response handling Obsolete — code reverted

@SoundMindsAI SoundMindsAI merged commit da91906 into main Jul 12, 2026
21 checks passed
@SoundMindsAI SoundMindsAI deleted the fix/ubi-fast-seed-es-flake-resilience branch July 12, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant