Skip to content

Flaky CI: watcher lifecycle integration test and Postgres refresh negative control #1955

Open

Description

@phernandez

Two tests failed CI on PRs that don't touch their code during the 1.0 merge run (2026-10-29), and passed on re-run.

test-int/mcp/test_new_project_watch_lifecycle.py::test_new_project_copied_files_and_live_changes_are_searchable

tests/services/test_search_refresh_concurrency.py::test_negative_control_relation_rows_collide_without_the_tolerance

Fix direction

  • Lifecycle test: find the wait that can outlast its deadline under load; isolate _initial_index_tasks per test.
  • Negative control: force the overlap deterministically (e.g. an explicit barrier between the two refreshes) instead of relying on timing, or drop it if the positive tests cover the tolerance.

🤖 Generated with Claude Code

Activity

  1. claude commented on Oct 29, 2026

    @claude
    Contributor

    Claude finished @phernandez's task in 14s —— View job


    Triage

    • Type: Bug (flaky tests, CI reliability)
    • Component: Testing (watcher/indexing lifecycle and Postgres search refresh). Not Cloud-specific.
    • Priority: Medium. Nothing breaks for users, but the flakes fail unrelated PRs and cost a re-run each time.
    • Complexity: Medium. The two tests need separate fixes.

    Duplicates: None found. #940 covers a similar class (timing-sensitive tests) but different tests, and it is closed.

    Notes

    • Lifecycle test (test-int/mcp/test_new_project_watch_lifecycle.py): the issue already names a likely cause. The module-global _initial_index_tasks is shared across tests, and a fake _CapturedTask from tests/services/test_initialization.py can be gathered in this test's teardown. That would explain the TypeError and may also explain the teardown hang. The asyncio.timeout failures seen on SQLite shard 2/2 could be a separate load-sensitive wait, so fix both: isolate the task set per test and review the wait deadlines.
    • Negative control (tests/services/test_search_refresh_concurrency.py): it asserts that a race happens, so it depends on timing. The issue's suggestion of an explicit barrier or event between the two refreshes is the right fix. If the positive tests already cover the tolerance, dropping the control is also fine.

    No reproduction steps are needed. The CI run links are in the issue. I've stopped at triage and made no code changes.

  2. phernandez commented on Oct 29, 2026

    @phernandez
    MemberAuthor

    Two more failures on 2026-10-29, now with a third symptom: bm_search raising sqlite3.OperationalError: vtable constructor failed: search_index inside the watcher lifecycle test.

    A virtual-table constructor failure on an FTS5 table points at the search_index being created or dropped concurrently while another connection opens it. The watcher's new-project catch-up may be (re)creating search tables while the test searches.

  3. phernandez commented on Oct 29, 2026

    @phernandez
    MemberAuthor

    Another occurrence on main at 1e8e64c (run 40970750756, SQLite integration shard 2/2): test_new_project_copied_files_and_live_changes_are_searchable timed out at line 83, the narwhal loop. The first loop (copied file indexed after attach) passed, so the project was attached and only the later live edit was missed. Same symptom as described above; the next main run (fb90bbb) passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions