Skip to content

fix(db): keep lease releases durable and deflake PostgreSQL lane tests - #8676

Merged
matt2e merged 1 commit into
mainfrom
flakey-tests-pg-lease
Oct 29, 2026
Merged

matt2e merged 1 commit into
mainfrom
flakey-tests-pg-lease

Conversation

@matt2e

@matt2e matt2e commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When the user-deletion worker dropped a lease heartbeat, its pooled connection could be left inside an open transaction. The worker's final lease release then reported success but never committed, and a rerun saw the request as "leased by another worker" for up to 60 s. The heartbeat is now one autocommit UPDATE.

The PR also fixes six PostgreSQL-lane test flakes from the 21–28 Oct 2026 CI audit. It moves one relay test off the 100 ms-acquire test helper before that test starts flaking the same way.

This is one of seven PRs split out of the flakey-tests branch (siblings below). It touches only buzz-db, buzz-deletion and buzz-relay. Every relay file except connection.rs and boot_lifecycle.rs changes only for the helper rename in the last row.

Related issue

None found. Nothing on main (eda26b9) or in an open PR covers these. G, a single hermit pgschema 504 that needs a workflow change, is out of scope.

Flaky tests fixed

Test Failing runs Cause Change
buzz-deletion user::worker::postgres_tests::dummy_executor_failure_retries_same_community_stage (:47) and next_step_advances_one_handler_at_a_time_through_completion (:131) 40439041234 (main), 40399635380#3, 40413732817#1 Product bug. run_stage_with_heartbeat drops the renewal future when the stage handler wins select!. The heartbeat opened a transaction. sqlx 0.9 bumps transaction_depth only after BEGIN returns, so a begin() dropped mid-BEGIN queues no ROLLBACK, and the connection goes back to the pool inside an open transaction. The worker's autocommit release then ran inside it and never committed. EntityDeletionStore::heartbeat is one autocommit UPDATE, guarded by lock_owned's ownership/status/expiry predicate. A superseded lease is never renewed. A new regression test, dropped_heartbeat_leaves_no_open_transaction_for_the_release, drops the real heartbeat mid-BEGIN through a pausing proxy.
buzz-db session_statement_timeouts_bound_writer_and_reader_sessions (runtime/tests.rs:3138, PoolTimedOut) 40128463987, 40314950864#1, 40374286094 (main), 40394821009#1, 40413732817#1 Each config's lazy reader pool dialled cold inside the 150 ms READER_ACQUIRE_TIMEOUT. Db::new never runs the boot ping. Warm the reader the way the relay boots it (read_pool_boot_ping_for_tests + await_idle_floor).
buzz-relay startup_owns_the_dependency_sampler (boot_lifecycle.rs) Torn scrape: 39865872891 (main), 39624074244#2, 40385799750#1, 40413293801#1. Early exit: 39704588623 metrics-exporter-prometheus 0.18.3 snapshots counters, then gauges. The scrape that first shows the completion gauge can therefore predate the checks_total counters. Assert on a second scrape, taken after the gauge is seen. On an early relay exit, the helper now panics with the exit status, stdout and stderr. The early exit itself was not reproduced.
buzz-db sample_writer_waits_for_shared_floor_lock_before_sampling_time (replica_fence.rs:1086) and replica_floor_probe_waits_for_shared_writer_and_records_after_release (runtime/tests.rs:3452) 39601763602 (main), 40391445284#1, 39825846222, 40301667418 Both scan cluster-wide pg_stat_activity but ran outside the serialized postgres-cluster-global group. Connection churn from other tests masked activity (MaskedActivity { masked: 1 }). Rename both with a cluster_global_ prefix. nextest show-config test-groups --profile postgres-ci lists both in the group.
buzz-db stalled_reader_sessions_release_their_slots_for_new_connections (reader_pool_postgres_tests.rs:354) 40399635380#2 The floor warms one connection, so the second warm-up acquire dials cold inside 150 ms. Retry only that warm-up acquire on PoolTimedOut, for up to 5 s (#8472's precedent). The stall under test keeps the production budget.
buzz-db readiness_check_cancellation_balances_waiter_and_inflight_connection (runtime/tests.rs:391, Elapsed) 39785370353 The test aborted pg_sleep(5) after a fixed 25 ms. Recovery lands at server start + 5 s, but the bound is abort + 5 s, a ~22 ms margin locally. Poll pg_stat_activity until the query is active, then abort. Use pg_sleep(1) within the unchanged 5 s bound.
buzz-relay upgrade_would_deny_line_is_written_at_auth_while_open (connection.rs) None yet (preventive, added by #8634) It used the 100 ms-acquire test_state_with_database_url against the real test database, the same shape as the PoolTimedOut flakes above. Use test_state_with_database_url_and_pool_options with sqlx's default timeout. The 100 ms helper is renamed test_state_with_unreachable_database_url; all its remaining callers use closed ports.

Local before/after counts are from the original flakey-tests commit (1f0451915), on macOS with OrbStack Postgres:

  • Regression test: 3/3 FAIL before, 3/3 PASS after.
  • Worker stress: 3 failures in 288 iterations before; 2,100 passing runs after.
  • Statement timeouts: 44/50 FAIL before, 50/50 PASS after.
  • Sampler: 2/200 FAIL before, 200/200 PASS after.
  • MaskedActivity under pgbench -C: 2/50 iterations FAIL before. After, the lane passed 3× (613/613) but was not re-run under pgbench.
  • Stalled reader: 10/30 FAIL before, 30/30 PASS after.
  • Forced late-Execute cancellation: 10/10 FAIL before, 10/10 PASS after.

Verification

This branch was pushed without local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. Hosted CI is the check. It was pushed with git push --no-verify, because the pre-push hook runs clippy and unit tests.

  • Every file on this branch is byte-identical to its final version on flakey-tests (67185df6a), where the counts above were measured.
  • Not re-run for this PR: clippy, nextest and the Postgres lanes.
  • Checked for this PR: git diff --check, rustfmt --check on the changed files, and a conflict-free merge with all six siblings onto main.

Readiness

This PR has a product fix (the buzz-db lease heartbeat). Under AGENTS.md "Before opening a PR", it needs an agent review and a human check against a local relay before buzz-review-completed is added. Suggested check:

  1. With local Postgres and a relay, submit and approve a user deletion with buzz-admin user-deletions submit / approve.
  2. Run buzz-admin user-deletions run until it completes, then run it again at once.
  3. Working looks like this: both runs complete, neither reports the request as leased by another worker, and buzz-admin user-deletions inspect <id> shows no leftover lease. The original race is rare and is not expected to reproduce by hand; this check confirms releases still commit.

Merge order

Independent of the other six. It merges cleanly onto main, and with every sibling, in any order.

Sibling PRs

Split from flakey-tests (one PR per area): #8677 (relay E2E helper + Windows shell), #8678 (mobile), #8679 (Tauri apt cache + Windows Rust cap), #8680 (desktop product races), #8681 (desktop E2E state gates), #8682 (desktop E2E composer + row actions).

🤖 Generated with Claude Code

Fixes the PostgreSQL-lane flakes from the 21-28 Oct 2026 CI audit:
families B, C, D, E, F1, F2 and the §A residual. G (one hermit pgschema
504, a CI workflow change) is out of scope. Nothing on origin/main
(eda26b9) or in an open or merged PR covers any of these.

Local runs: macOS, OrbStack Postgres, cargo-nextest 0.9.136, Rust
1.98.1. Baselines ran from a nextest archive built before any fix
(HEAD plus the B regression test). "Load" means one `yes` per core (16).

B. PRODUCT FIX: a dropped lease heartbeat stranded the release.
  Tests: buzz-deletion user::worker::postgres_tests
    dummy_executor_failure_retries_same_community_stage (:47) and
    next_step_advances_one_handler_at_a_time_through_completion (:131)
  Runs: 40439041234 (main), 40399635380#3, 40413732817#1
  Cause: run_stage_with_heartbeat drops the renewal future whenever the
    stage handler wins its select!. EntityDeletionStore::heartbeat opened
    a transaction. sqlx 0.9 PgTransactionManager::begin bumps
    transaction_depth only after BEGIN returns, so a begin() dropped
    mid-BEGIN queues no ROLLBACK. The release ping drains the BEGIN, and
    the connection goes back to the pool inside an open transaction. The
    worker's final autocommit `release` then ran inside it and reported
    success, but nothing committed. user-deletions run shares this path:
    the release is lost at exit, and reruns see "leased by another
    worker" until the lease expires (up to 60 s).
  Change: heartbeat is one autocommit UPDATE guarded by lock_owned's
    ownership/status/expiry predicate. The only callers are worker.rs and
    transition_postgres_tests.rs; the community loop's heartbeat is a
    different store. The worker comment now states why heartbeat must not
    open a transaction.
  Locking guarantee, as the leases.rs doc now states it: a superseded
    lease is never renewed; an expired lease can be renewed only while
    nobody has reclaimed it. Postgres re-checks the WHERE after a lock
    wait only when the holder committed an update of the row, so a holder
    that only took lock_owned's FOR UPDATE (save_stage_success writes
    entity_deletion_stage_history, never the parent row) or rolled back
    leaves the pre-wait expiry check in force. That is safe: claim takes
    the row with FOR UPDATE SKIP LOCKED while the waiting UPDATE holds the
    tuple lock, so no one can reclaim during the wait, and a reclaim that
    commits first bumps the generation, which the predicate re-checks.
    Only claim acts on expiry.
  Regression test: store::entity_deletion::postgres_tests::
    dropped_heartbeat_leaves_no_open_transaction_for_the_release. It
    drops the production heartbeat with its BEGIN held by PausingProxy
    (1-connection pool, test_before_acquire(false)), then releases and
    asserts on a direct connection.
  Before -> after: regression test 3/3 FAIL (lease_owner
    Some("worker") after release() returned true) -> 3/3 PASS. Worker
    stress under load, before: 2/138 iterations FAIL at load average
    ~200, then 1/150 at ~25, all at :131 (3/288). After, at ~25:
    300/300 iterations (2100 runs) PASS.
  Out of scope: the shutdown/renewal-failure branches can still drop a
    handler's begin() mid-BEGIN. The general fix belongs in sqlx.

C. session_statement_timeouts_bound_writer_and_reader_sessions
  (buzz-db runtime/tests.rs:3138 "reader timeout: PoolTimedOut")
  Runs: 40128463987, 40314950864#1, 40374286094 (main), 40394821009#1,
    40413732817#1
  Cause: each config's lazy reader pool dialed cold inside the 150 ms
    READER_ACQUIRE_TIMEOUT. Db::new never runs the boot ping.
  Change: warm the reader the way the relay boots it
    (read_pool_boot_ping_for_tests + await_idle_floor). await_idle_floor
    moved from reader_pool_postgres_tests.rs to runtime/tests.rs.
  Before -> after, 30% of sessions delayed 250 ms by proxy, 50
    iterations: 44/50 FAIL at :3138 -> 50/50 PASS.

D. startup_owns_the_dependency_sampler (buzz-relay boot_lifecycle.rs)
  Runs: torn scrape 39865872891 (main), 39624074244#2, 40385799750#1,
    40413293801#1; early exit 39704588623
  Cause: metrics-exporter-prometheus 0.18.3 snapshots counters, then
    gauges, with no single snapshot. The scrape that first shows the
    completion gauge can predate the checks_total counters registered
    just before it.
  Change: assert on a second scrape taken after the gauge is seen. The
    counters were written before the gauge and never idle out. Fix the
    doc comment that claimed no window. When the relay exits early,
    wait_for_scraped_metric now panics with its exit status, stdout and
    stderr.
  Before -> after, 200 iterations under load: 2/200 FAIL at :857 (the
    scrape had the gauge and histograms but no checks_total) -> 200/200
    PASS.
  Not reproduced: the early exit (:173). Only diagnostics were added.

E. sample_writer_waits_for_shared_floor_lock_before_sampling_time
  (replica_fence.rs:1086) and
  replica_floor_probe_waits_for_shared_writer_and_records_after_release
  (runtime/tests.rs:3452), both "MaskedActivity { masked: 1 }"
  Runs: 39601763602 (main), 40391445284#1; 39825846222, 40301667418
  Cause: both scan cluster-wide pg_stat_activity but were outside the
    serialized postgres-cluster-global group, so other lane tests'
    connection churn masked activity.
  Change: rename both with a cluster_global_ prefix (each migrates its
    own scratch DB, so not migration_schema_). `cargo nextest show-config
    test-groups --profile postgres-ci` lists both in
    postgres-cluster-global.
  Before, 50 iterations with pgbench -C -c 8 on the cluster: 2/50
    iterations FAIL (3 MaskedActivity panics). After: the buzz-db lane
    passes 3x (613/613). Not re-run under pgbench; the group serializes
    lane tests only.

F1. stalled_reader_sessions_release_their_slots_for_new_connections
  (reader_pool_postgres_tests.rs:354 "second warm connection:
  PoolTimedOut")
  Run: 40399635380#2
  Cause: the floor warms only one connection, so the second warm-up
    acquire dials cold inside the 150 ms budget.
  Change: retry only that warm-up acquire on PoolTimedOut within 5 s
    (#8472's precedent). The stall under test keeps the production
    budget.
  Before -> after, session-delay proxy, 30 iterations: 10/30 FAIL at
    :354 -> 30/30 PASS.

F2. readiness_check_cancellation_balances_waiter_and_inflight_connection
  (runtime/tests.rs:391 "cancelled in-flight query must return or
  replace its connection: Elapsed(())")
  Run: 39785370353
  Cause: the test aborted pg_sleep(5) after a fixed 25 ms sleep. sqlx's
    release ping waits for the query, so recovery lands at server start
    + 5 s, against a bound of abort + 5 s (~22 ms margin locally).
  Change: poll pg_stat_activity until the query is active, then abort.
    Use pg_sleep(1), so recovery takes about 1 s inside the unchanged 5 s
    bound.
  Before -> after, a proxy delaying only the Bind/Execute chunk after
    pg_sleep's Parse by 40 ms: 10/10 FAIL at :391 -> 10/10 PASS.

§A residual. upgrade_would_deny_line_is_written_at_auth_while_open
  (buzz-relay connection.rs) used the 100 ms-acquire
  test_state_with_database_url against the real test database. No
  failure observed yet; not reproduced. It now uses
  test_state_with_database_url_and_pool_options with sqlx's default
  timeout. The 100 ms helper is renamed
  test_state_with_unreachable_database_url; all of its remaining callers
  use closed ports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Matt Toohey <contact@matttoohey.com>
@github-actions

Copy link
Copy Markdown

🔐 Codex Security Review

Note: This is an automated, security-focused review generated by Codex.
Use it as a supplement to human review; false positives are possible.

Scope

  • Exact PR diff: eda26b9423328a06960bb0d5fd4c1af54600c419...dae720ed982f0360fba95e3dc5038eb32f626edd
  • Model: gpt-6.1-sol

💡 Click "edited" above to see earlier reviews for this PR.


Review Summary

Overall Risk: NONE

No concrete security, correctness, or reliability findings in the authorized PR range. Lease renewal retains owner and generation checks that prevent superseded workers from renewing or releasing a successor’s lease.

Findings

No concrete security, correctness, or reliability findings were identified.

Notes

  • Review used read-only source and history inspection. Builds and tests were not executed.

Generated by Codex Security Review |
Requested by: @matt2e |
Workflow run

@jedwards27 jedwards27 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE

Reviewed: eda26b9423328a06960bb0d5fd4c1af54600c419..dae720ed982f0360fba95e3dc5038eb32f626edd (exact head dae720ed982f0360fba95e3dc5038eb32f626edd)

Risk: high — this touches database lease ownership/recovery for user deletion plus PostgreSQL/relay CI behavior. No schema or migration files changed.

Behavior/contracts traced:

  • EntityDeletionStore::heartbeat now uses one owner/generation/status/expiry-fenced autocommit UPDATE, avoiding the cancellable explicit transaction that could leave a pooled session open and make the following release appear successful without committing (crates/buzz-db/src/store/entity_deletion/leases.rs:111).
  • The worker still keeps heartbeat cancellation local to run_stage_with_heartbeat; the store-side autocommit contract is the boundary that makes dropping the renewal future safe (crates/buzz-deletion/src/user/worker.rs:121).
  • The new PostgreSQL regression exercises the real dropped-heartbeat path through a pausing proxy, then proves release commits and no session remains idle in transaction (crates/buzz-db/src/store/entity_deletion_postgres_tests.rs:452).
  • The test-only changes are scoped to the stated flakes: reader warm-floor setup, readiness cancellation timing, cluster-global PostgreSQL serialization, unreachable-database helper naming, and relay boot diagnostics.

Findings: no blocking findings.

Author action: none for code. Keep the PR-body readiness checklist: do not add buzz-review-completed until the repo checklist's human local-relay check has been performed.

Verification owner: CI for exact-head automated gates; author/human operator for the local relay readiness checklist.

Validation:

  • gh api user --jq .login → jedwards27; live PR author is matt2e.
  • /Users/judeedwards/.buzz/scripts/buzz-pr-review-preflight 8676 at dae720ed982f0360fba95e3dc5038eb32f626edd: PASS; base/head matched, no unresolved review threads, and exact-head CI checks showed required PostgreSQL/relay/backend/desktop/Rust/security/DCO gates successful or intentionally skipped by path selection.
  • . ./bin/activate-hermit && git diff --check eda26b9423328a06960bb0d5fd4c1af54600c419...HEAD && scripts/test-postgres-test-discovery.sh: PASS at dae720ed982f0360fba95e3dc5038eb32f626edd.
  • . ./bin/activate-hermit && cargo test -p buzz-deletion user::worker::tests --lib: PASS at dae720ed982f0360fba95e3dc5038eb32f626edd.

Manual/native evidence: no desktop/manual native journey run; this is a database/relay change. I did not run the full local PostgreSQL lane or local relay user-deletion workflow; exact-head CI supplies the automated PostgreSQL/relay gates, and the human local-relay readiness check remains external to this code review.

Residual risk: the original race is timing-sensitive and local infrastructure was not re-run here. The exact-head PostgreSQL regression and hosted CI are the merge evidence; the repo checklist's human local-relay smoke remains required before claiming buzz-review-completed.

matt2e added a commit that referenced this pull request Oct 29, 2026
… by keyboard (#8682)

## Summary

Test-side fixes for the two largest desktop E2E flake clusters in the
21–28 Oct 2026 CI audit:

- **Composer `fill()` races the app's deferred focus-to-end** (§1). Text
was appended to the old draft, or a clear did nothing.
- **Clicks on the hover-only message action bar while the timeline
re-pins** (§3/§8/§18). Reply and more-actions clicks timed out or were
intercepted.

Two shared helpers replace the per-spec workarounds, and §9 rides along
because its spec now imports them. The PR changes test code only. It is
one of seven PRs split out of the `flakey-tests` branch (siblings
below).

### Related issue

None found. #8559, #8622 and #8613 touch these specs or helpers, but had
no coverage for these clusters. Draft #8661 edits
`empty-edit-delete.spec.ts`'s `submitEmptyEdit` and `beforeEach`, the
same lines as this PR.

### New helpers

- `tests/helpers/composer.ts`:
- `clearComposer` waits for the app's focus, then repeats select-all +
Delete through the editor keymap until the editor is empty (250 ms per
try, 5 s overall).
- `replaceComposerText` clears, then fills. On an empty document, a late
collapse to the end is a no-op.
- `tests/helpers/channelHeadReady.ts`:
  - `GENERAL_CHANNEL_ID`.
- `openGeneral` clicks `#general`, then calls
`waitForMockChannelHeadReady`.
- `pressRowAction(button)` does focus + Enter. The action bar also shows
on focus-within, and more-actions is a Radix trigger that opens on
Enter.

### Flaky tests fixed

| Test | Failing runs | Cause | Change |
|---|---|---|---|
| §1 `inbox-edit.spec.ts:525` | 39641311877, 39821769576, 40333705511,
40348069929, 40388246283 | **Append:** entering edit mode, or closing
the non-member mention dialog, focuses the end of the draft in deferred
steps: RAF, a nested RAF, then ProseMirror's 20 ms selection resync.
`fill()` selects and inserts in separate protocol steps, so a step that
lands in between collapses the selection and the text is appended.
**Clear does nothing:** `fill("")` presses Delete before ProseMirror
handles `selectionchange`, so the editor blocks it. | `clearComposer` /
`replaceComposerText` |
| §1 `inbox-edit.spec.ts:560` | 39538411386, 39821769576 | Same as
above. | Same |
| §1 `mention-recipients.spec.ts:591` (both variants) | 39799090750,
40374286094, 39641311877 | Same | Same |
| §1 `mention-recipients.spec.ts:691` (both variants) | 40348069929,
40556019361, 39601763602 | Same | Same |
| §1 `messaging.spec.ts:3799` | 39827234118, 40445481983 | Same, plus a
bare `#general` click and a `force: true` Reply | Same, plus
`openGeneral` and `pressRowAction` |
| §1 `messaging.spec.ts:4022` | 40389392495 | Same | Same |
| §1 `messaging.spec.ts:4289` | 39822883395, 40557808800, 40566841699 |
Same | Same |
| §1 `messaging.spec.ts:4368` | 40574058918 | Same | Same |
| §1 `remote-owned-mentions.spec.ts:463` (both variants) | 39618692657,
40374286094, 40382726903, 40389392495, 40582387515 | Same, after the
Invite dialog closes | Same |
| §3 `messaging.spec.ts` :3531, :3647, :4122, :4208, :4450 (root and
reply) and :4633 (keyboard and button) | :3531 40583830861, 40595483966.
:3647 40566841699, 40597484657. :4122 39641311877, 40368315739,
40368363357. :4208 39525092486, 39796569342, 40367495044. :4450
39796569342, 40348069929, 40373404267. :4633 39530486096, 40348069929,
40368315739, 40582387515, 40368363357 | The action bar is
`pointer-events-none` until its row is hovered. Opening `#general`
refreshes the head after the live subscription lands, and the timeline
re-pins, so the row moves out from under the pointer. `force: true` only
hid this. | `openGeneral`, then `pressRowAction` on Reply or
more-actions. Six more messaging tests that used hover + `click({ force:
true })` get the same change, and each first waits for its last row to
contain the sent root. |
| §18 `thread-follow.spec.ts:350` | 39529963092, 40348069929,
40583830861 (all 3), 40595483966; PR 39541263507 (all 3) | Same; the
composer overlay intercepts the more-actions click. | Same |
| §8 `thread-head-stale-edit.spec.ts:55` | 39525092486, 39618692657,
39796569342, 39822883395, 40011714848, 40281448810, 40368363357,
40389392495, 40445481983, 40574058918; PR 40031299575 (all 3) | Same
(used `force: true`) | Same |
| `file-attachment.spec.ts:165` | 1–2 CI failures (audit long tail) | A
more-actions click on a moving timeline | Same |
| `empty-edit-delete.spec.ts` (`openMoreActionsMenu`, `submitEmptyEdit`,
the non-empty edit, `beforeEach`) | Covered under §11 in the
desktop-races sibling | The same hover and `fill` races | The head-ready
wait before emitting, `pressRowAction`, `clearComposer` and
`replaceComposerText`. The autocomplete table rows keep their
back-to-back keys, which is what they test. |
| §9 `file-attachment.spec.ts:419` "upload progress floats above the
dock and lifts Jump to latest" | 10 of 50 main jobs, including
40583830861, 40597484657, 40608383847 | A programmatic `scrollTop` keeps
the bottom intent armed. The upload card resizes the content, the
timeline re-pins, and the pill unmounts under `boundingBox()`. The fixed
250 ms wait read the card mid-slide. | `scrollTimelineUp()` leaves the
bottom with a real wheel, and one `expect.poll` checks all three layout
relations. |

Local before/after counts are from the original `flakey-tests` commits
(e286a6ae0, 816c7c35e, c66f411f3, 67185df6a), running `main`'s copy and
the changed copy side by side under forcings:

- **§1 at 4× CPU throttle:** 130/130 after; `main`'s copies ranged from
5/10 to 10/10.
- **§1 with `selectionchange` delayed, a stretched resync, or a gap
inside `fill()`:** `main`'s copies 0/5 to 4/5 and 0/20; after 65/65 and
20/20.
- **§3 at 4× throttle:** `main` had 3 Reply-click timeouts at :4208 and
3 at :4633 (keyboard); after 120/120.
- **`file-attachment:165`:** 0/3 before, 3/3 after.
- **§9 unforced:** 4/5 before, 5/5 after.

These were measured with the helper's first version, which waited two
frames plus the 20 ms resync. The final state-based helper ran unforced
only: 157/157 and then 196/196 over the touched specs. Its retry was not
checked under a forced late collapse. :3531, :3647, :4122, :4450 root,
:4633 button, `thread-head-stale-edit:55` and the six force-click tests
were not force-reproduced; they get the fix that matches their CI
signatures and reproduced siblings.

Dropped:

- `mention-recipients.spec.ts:933` (40608383847) fails on
`flavors.handled`, which is unrelated.
- `persistent-agent-audience.spec.ts:828` (`@Mor@Morg`) did not
reproduce in 79 forced runs.

### Verification

This branch was pushed without local re-verification, by the user's
choice: local test runs were overloading the machine and failing for
that reason. **Hosted CI is the check.** It was pushed with `git push
--no-verify`, because the pre-push hook runs typecheck and the unit
suites. #8613's flake-check lane will also repeat the changed specs.

- Every file except `empty-edit-delete.spec.ts` is byte-identical to its
final version on `flakey-tests` (67185df6a). That file carries only this
PR's hunks; see "Merge order".
- Checked for this PR: `git diff --check`, biome on the partial spec (no
changes, the same as on `main`), and a conflict-free merge with all six
siblings onto `main`.

### Readiness

Test-only change. AGENTS.md's readiness steps (agent review, then human
confirmation) are still outstanding, so `buzz-review-completed` is not
set.

### Merge order

No required order. `desktop/tests/e2e/empty-edit-delete.spec.ts` is also
edited by **flakey-tests-desktop-races**, which adds the autocomplete
table after line 158. This PR changes imports, helpers, `beforeEach` and
line 143. The hunks don't touch, so the two merge cleanly in either
order, as checked with `git merge-tree`. Draft #8661 edits the same
`submitEmptyEdit`/`beforeEach` lines; whichever lands second will need a
rebase.

### Sibling PRs

Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease +
lane), #8677 (relay E2E helper + Windows shell), #8678 (mobile), #8679
(Tauri apt cache + Windows Rust cap), #8680 (desktop product races),
#8681 (desktop E2E state gates).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
matt2e added a commit that referenced this pull request Oct 29, 2026
## Summary

Test-side fixes for ten desktop E2E flake clusters and six long-tail
rows from the 21–28 Oct 2026 CI audit. Each spec now waits for the state
it assumes (a rendered row, a mounted listener, a finished animation, a
held boot read), not a fixed delay or a one-shot probe. Two mock-bridge
seams support this.

The PR changes test code and mock-bridge seams only, apart from one
product constant extracted with its value unchanged. It is one of seven
PRs split out of the `flakey-tests` branch (siblings below).

### Related issue

None found.

- #8622, #8244 and #8386 touch some of these spec files, but not the
tests fixed here.
- §7 mode A (`navigation:472`) is left to open PR #6852, which already
sets the width and measures in one polled evaluate.
- Draft #8279 queues cold-launch nostr-bind links in the product. Its
mock returns null, so the §12 listener seam is still needed.

### Mock-bridge seams

- `__BUZZ_E2E_EVENT_LISTENER_COUNT__(event)` counts live **app**
listeners per Tauri event. It wraps the mocked
`plugin:event|listen`/`unlisten`. The spec-facing
`__TAURI_INTERNALS__.listen` passes a marker target that the wrapper
skips, so a spec's own subscription can't satisfy the wait. Tauri
2.11.1's mock ignores targets, so delivery is unchanged.
- `deferFirstChannelsRead` arms the existing channels-read hold at boot.
`__BUZZ_E2E_RELEASE_CHANNELS_READ__` releases it.
- `CREATE_PROJECT_REFOCUS_DELAY_MS` moves to `createProjectFormFocus.ts`
with its value unchanged (50 ms). Both `CreateProjectFormContent` and
`project-commit-detail.spec` import it.

### Flaky tests fixed

| Test | Failing runs | Cause | Change |
|---|---|---|---|
| §2 `overscroll-boundary.spec.ts:34` (and its sibling) | 39601763602,
40367495044, 40583830861, 40597484657, 40373404267 | The
`message-timeline` test id first lands on the loading skeleton, which
doesn't overflow, so the scroll lock rightly cancels the wheel. | Wait
for a rendered row, and content taller than the viewport, before
dispatching wheels. |
| §4 `project-conversation-load-failure.spec.ts:73` | 40367495044 (all 3
attempts), 40608383847, 40583830861; PRs 39541263507, 40509460168 | The
seeded root and reply were dated `now`. The panel's live REQ (`since =
floor(now)`) still replayed the reply, so the error card never appeared.
| Date both events a minute in the past, and correct the comment. |
| §6 `workflows.spec.ts` `createWorkflow` cluster (:222, :365, :297,
:525, :471, :406, :614, :997) | 14 of 50 main jobs; :222 ×4, e.g.
40353619362 | `AnimatePresence mode="wait"` keeps the trigger body
mounted through its exit, so "Message text" could fill the trigger's
field (:222 and the callers). For :297, `isVisible()` read "closed" one
frame before the dialog opened its own channel list. | Wait for the step
body's "Step details" control before filling. Create dialogs wait for
the list; only the duplicate dialog clicks it. |
| §12 `nostr-bind.spec.ts` (:156, :198, :228, :247, :260, :335, :417,
:447) | 8 main jobs, each test once | The spec emitted
`deep-link-nostr-bind` before the app subscribed, and the mock drops
events that have no listener. | Wait for
`__BUZZ_E2E_EVENT_LISTENER_COUNT__` to report the app listener before
emitting. |
| §15 `profile.spec.ts:1128` (integration) | 7 of 50 main jobs | The 320
ms motion-sampling window started before the click. | Start the window
when `activeTab` reads "runtime". |
| §17 `agent-availability.spec.ts:602`, both shutdown-failure variants
(integration) | 6 jobs (4 disconnected-online, 2 failed-online) | The
"shutdown refused" toast sits over the Delete row and pauses while the
pointer is on it, so the retry click is intercepted. | Assert the toast,
then retry with focus + Enter. The toaster has no close button. |
| §19 `virtualization.spec.ts:247` "08 — cascading older pages…" | 4 of
50 main jobs; PR 40605736093 (all 3 attempts) | The 120 ms and 400 ms
windows were timed from before the wheel bursts, and CDP can deliver
wheels late. | Count the wheels the page receives. Each window opens at
the first wheel and closes only after all have landed, then waits for
still frames. **A second, product-side mode remains** (see below). |
| §20 `message-copy-link.spec.ts:160` | 4 of 50 main jobs (5 failed
attempts), e.g. 40595483966 | The huddle row reused `"d".repeat(64)`,
which is a seeded ordinary `#general` message. | Use ids that no seed
uses, and emit only after `waitForMockChannelHeadReady`. |
| §21 `navigation.spec.ts:687` | 4 of 50 main jobs | The pending chip's
`boundingBox()` was read once, and can be null while the row mounts. |
Poll until the chip has a width. |
| §23 `sidebar-snapshot.spec.ts:358`, `:388`, `:438` | PR 40427822003
(all 3 attempts); never on main | The 500 ms loading-state budget
counted app boot, so a slow runner finished the channels read first. |
Hold the first channels read with `deferFirstChannelsRead`, assert the
loading state, then release it. |
| `project-commit-detail.spec.ts:259` | 1–2 CI failures (audit long
tail) | A 50 ms refocus timer can land mid-fill and type the description
into the name field. | Once the name field is focused, wait out a page
timer of `CREATE_PROJECT_REFOCUS_DELAY_MS`; equal-delay timers fire in
order. |
| `project-pr-review.spec.ts:3520` | 1–2 CI failures | The same timer
pattern in `CreateProjectWorkItemDialog`. | Wait for `toBeFocused()` on
the title, which only that timer can satisfy. |
| `thread-load-failure.spec.ts:79` | 1–2 CI failures |
`openWelcomeThread` branched on a one-shot `count()` and fell back to a
hover-only Reply click. | Wait for the summary row, then click it. |
| `virtualization.spec.ts:201` | 1–2 CI failures | The drop point sat 2
px above the target's bottom edge, after two fixed frames. | Drop at the
centre, after dnd-kit's live region names the target. |
| `channel-composer-overflow.spec.ts:217` | 1–2 CI failures |
Same-second emits sort by id, and the clearance was checked once. |
`createdAt: base + i` and `expect.poll`. |
| `scroll-history.spec.ts:1289` | 1–2 CI failures | A fixed 250 ms wait
before idle sampling. | Poll for coverage first. |

Local before/after counts are from the original `flakey-tests` commits
(e286a6ae0, 816c7c35e, 67185df6a). Each used a forcing that made
`main`'s copy fail with CI's signature:

| Test | Before | After |
|---|---|---|
| §2 | 3/15 | 15/15 |
| §4 | 0/10 | 10/10 |
| §6 :222 | 0/3 | 3/3 |
| §6 :297 | 0/3 | 3/3 |
| §12 | 0/3 | 3/3 |
| §15 | 0/3 | 3/3 |
| §17 | 1/4 | 4/4 |
| §19 late-wheel modes | 0/3 | 3/3 and 2/3 |
| §23 | 2/6 | 6/6 |
| project-commit-detail | 0/3 | 3/3 |
| project-pr-review | 0/3 | 3/3 |

§20, §21, `thread-load-failure:79`, `virtualization:201`,
`channel-composer-overflow:217` and `scroll-history:1289` were not
reproduced; they get the fix that matches their CI signature.

Not fixed (follow-up): `virtualization:247` also has a product-side
mode. In one frame after a prepend, `scrollHeight` can drop 458–639 px
while `scrollTop` moves only 50–114 px, so the anchor row jumps 382–457
px. That is lost anchor compensation, and this test can still fail on
it.

### Verification

This branch was pushed without local re-verification, by the user's
choice: local test runs were overloading the machine and failing for
that reason. **Hosted CI is the check.** It was pushed with `git push
--no-verify`, because the pre-push hook runs typecheck and the unit
suites. #8613's flake-check lane will also repeat the changed specs.

- Every file except `virtualization.spec.ts` is byte-identical to its
final version on `flakey-tests` (67185df6a). That file carries only this
PR's hunks; see "Merge order".
- Checked for this PR: `git diff --check`, biome on the partial spec (no
changes, the same as on `main`), and a conflict-free merge with all six
siblings onto `main`.

### Readiness

Test and mock-bridge change; the one product edit is a constant
extraction with its value unchanged. AGENTS.md's readiness steps (agent
review, then human confirmation) are still outstanding, so
`buzz-review-completed` is not set.

### Merge order

No required order. `desktop/tests/e2e/virtualization.spec.ts` is also
edited by **flakey-tests-desktop-races**, which adds one test after line
929; this PR's hunks are all above line 551. They don't touch, so the
two merge cleanly in either order, as checked with `git merge-tree`.

### Sibling PRs

Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease +
lane), #8677 (relay E2E helper + Windows shell), #8678 (mobile), #8679
(Tauri apt cache + Windows Rust cap), #8680 (desktop product races),
#8682 (desktop E2E composer + row actions).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Jitter <2b2e6415e748c35180847a37aadae06a11e30dddc76b784e7fd9354c4eb42e7a@buzz.block.builderlab.xyz>
matt2e added a commit that referenced this pull request Oct 29, 2026
…cap (#8679)

## Summary

Fixes the two CI workflow defects that turned `main` red most often in
the 21–28 Oct 2026 audit:

- **Desktop Tauri:** the lane now caches its apt `.deb`s. It no longer
re-downloads 62 MB from a slow Ubuntu mirror on every run.
- **Windows Rust:** the lane drops a duplicate `cargo check` and gets a
60 min cap, so a cold run can save its cache.

It also updates #8613's `scripts/flake-check.test.mjs`, which breaks
once the Tauri package list moves into an env var. There are no product
code or test-behaviour changes.

This is one of seven PRs split out of the `flakey-tests` branch
(siblings below).

### Related issue

None found. #8613 touched both workflows after the audit, but not these
steps. No open PR changes how the Tauri packages are fetched, the
Windows timeout, or its Check (workspace) step:

- #8422, #8321 and #8373 only add packages to the list, and would now
add them to `TAURI_APT_PACKAGES`.
- #8414 adds two mesh-llm Tauri steps to the Windows job. If it lands,
they must fit the cold budget below.

### CI failures fixed

| Lane / test | Failing runs | Cause | Change |
|---|---|---|---|
| Desktop Tauri, "Install Tauri dependencies (Linux)"
(`_ci-desktop.yml`) | 62 jobs in 51 run attempts, 22–27 Oct. On main:
39821769576, 39822883395, 39827234118, 40333705511, 40342468627,
40348069929, 40353619362, 40368315739, 40373404267, 40374519539,
40556742929 | All 3 `ci-apt-retry.sh` attempts hit their 420 s limit
while `azure.archive.ubuntu.com` served the 62 MB at 20–65 KB/s. The
25.5 MB `libwebkit2gtk-4.1-0` alone takes a whole attempt, and each
attempt loses its partial download. | The package list moves to a
job-level `TAURI_APT_PACKAGES`. A new "Resolve Tauri apt archives" step
builds the cache key from `apt-get install --print-uris -o
Acquire::ForceHash=SHA256` (name, size and SHA256 per file).
`actions/cache/restore` uses that key. The install step copies only the
restored files whose SHA256 matches the signed index, because apt 2.8.3
reuses any same-size cached `.deb` without hashing it. A best-effort
rebuild keeps exactly the listed files. `actions/cache/save` runs only
on core-lane pushes after a successful rebuild; PR runs never save. |
| Windows Rust (`_ci-rust.yml`) | All on main: 39796569342 (job
119885795086), 39821769576 (119975358772), 39827234118 (119994286695),
39865872891 (120128351939), 40402556064 (121852634685) | Every test
passed. Cold steps take 41.3–43.5 min, so the 45 min cap cancelled the
`Swatinem/rust-cache` post-run save 1.6–3.7 min in. No cache was
written, and the next push was cold again. | Delete "Check (workspace)".
The clippy step before it covers the same `--workspace --all-targets`
set with `-D warnings`, which saves 2.0–2.8 min. Raise `timeout-minutes`
from 45 to 60, with the measured breakdown in a comment: 39–41 min of
steps, a ~4.5 min save and ~15 min of margin. |
| `scripts/flake-check.test.mjs` "repeat invocations mirror the lanes
they stand in for" | None in CI yet: it fails only with this change
applied | #8613 added this test after `flakey-tests` branched. It checks
that the advisory `flake-tauri` lane installs the same packages as the
Desktop Tauri core lane, by regex-matching the inline `apt-get install
-y --no-install-recommends \` list in `_ci-desktop.yml`. That list now
lives in `TAURI_APT_PACKAGES`, so the regex matched nothing and the test
failed with "Desktop Tauri apt packages". | Read `TAURI_APT_PACKAGES`
from `_ci-desktop.yml` and compare package names (12 on each side) with
`_ci-flake.yml`'s list. Renaming one package in `_ci-flake.yml` still
fails the test. Giving the `flake-tauri` lane the same read-only `.deb`
restore is a follow-up. |

Local evidence is from the original `flakey-tests` commit (9aa768a3b):

- The apt steps ran verbatim in Ubuntu 24.04 containers with apt 2.8.3,
with the cache restore and the save step's `if:` emulated.
- On a miss, the install fetches everything ("Need to get 254 MB"). On
an exact hit with the network down, it fetches nothing ("Need to get 0
B/254 MB").
- On a fallback with one `.deb` missing and one corrupted at the same
size, the hash check rejects the corrupt file, and apt re-fetches only
the corrupt and missing files.
- The key was identical in all four simulated jobs.
- The Windows numbers come from the jobs API for the five runs above.

On this branch, the split session ran the `scripts/` node tests:
`ci-selection`, `ci-required-gates`, `ci-apt-retry` and `flake-check` on
macOS gave 84 pass, 0 fail and 8 skipped (Linux-only). Before the
`flake-check.test.mjs` fix, that test failed 1 case. The same session
ran actionlint 1.7.12 with shellcheck, which was clean, and the
rust-cache contract scripts, which passed.

### Verification

This branch was pushed without further local re-verification, by the
user's choice: local test runs were overloading the machine and failing
for that reason. **Hosted CI is the check.** It was pushed with `git
push --no-verify` over SSH. The pre-push hook runs tests, and the HTTPS
token lacks the `workflow` scope.

Only hosted CI can show:

- The runner's real package list (about 62 MB) and the same key in all
three Tauri lanes.
- The `actions/cache` round trip. The first PR run misses, because
`main` has no key yet. After merge, the first push to `main` saves it.
- A cold Windows run saving its cache within the new cap. The next
toolchain bump or cache eviction will show it.

Remaining risk: after Ubuntu updates a large package, runs fetch it from
the mirror until a push to `main` saves a new key.

### Readiness

CI-only change. AGENTS.md's readiness steps (agent review, then human
confirmation) are still outstanding, so `buzz-review-completed` is not
set.

### Merge order

Independent of the other six. It merges cleanly onto `main`, and with
every sibling, in any order.

### Sibling PRs

Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease +
lane), #8677 (relay E2E helper + Windows shell), #8678 (mobile), #8680
(desktop product races), #8681 (desktop E2E state gates), #8682 (desktop
E2E composer + row actions).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
matt2e added a commit that referenced this pull request Oct 29, 2026
…#8677)

## Summary

Fixes two CI flakes from the 21–28 Oct 2026 audit, both in test code
with no product change:

- The relay E2E helper `collect_until_eose` now drops repeat copies of
an event id.
- Two Windows shell tests drop a 5 s cap that a cold Git Bash start can
exceed.

This is one of seven PRs split out of the `flakey-tests` branch
(siblings below).

### Related issue

None found for these two. The buzz-terminal `passwd_shell()` `getpwuid`
race
(`env_fence_tests::resolve_shell_falls_through_to_passwd_not_the_default`,
run 38590095289) is left to open PR #7687, which already replaces
`getpwuid`. #8636 and #7976 edit `buzz-test-client/src/lib.rs`, and
#7607 and #7915 edit `buzz-dev-mcp/src/shell.rs`, but none of them
changes `collect_until_eose` or these timeouts.

### Flaky tests fixed

| Test | Failing runs | Cause | Change |
|---|---|---|---|
| Relay E2E
`e2e_project::test_project_tombstone_deletes_coordinate_and_spares_members`
(e2e_project.rs:338, "left: 2 right: 1") | 40408678412#1, job
121879630394 (PR #8620, schema-only) | The EVENT handler sends OK, then
fans out in a spawned task. `handle_req` registers the live subscription
before its history query, and `req.rs`'s `seen_ids` dedups history rows
only. If fan-out runs after the REQ is registered, the event arrives
once live and once from history, both before EOSE, and
`collect_until_eose` kept every frame. | `collect_until_eose` keeps the
first copy of each event id, as production clients do. Its doc comment
says a test of the relay's own dedup must read frames with `recv_event`.
A new unit test, `collect_until_eose_keeps_one_copy_of_each_event_id`,
uses a fake relay that sends the duplicate. |
| Windows `buzz-dev-mcp` `shell::tests::basic_echo` (shell.rs:1026, exit
124) | 39675698081#1, job 119485363399 (green on re-run) | 124 is the
shell tool's own timeout exit code. `basic_echo` capped the shell at 5
s, and it is one of the first tests in the process to start Git Bash,
which can take longer than 5 s on a cold runner. | `timeout_ms: None`
(the production 120 s default) in `basic_echo` and in
`workdir_is_honored`, which has the same exposure. `timeout_fires` still
covers timeouts. |

Pre-check: no e2e test relies on the relay's own history dedup. Every
exact-count assertion uses a single filter. The NIP-33 "only newest
remains" assertions would see a failed replacement as two different ids,
and this dedup keeps both.

Local evidence is from the original `flakey-tests` commit (fc4518d39):

- The new unit test: FAIL before, PASS after.
- 13 of the 14 e2e files that call the helper, plus `e2e_thread_roots`,
against an isolated local relay. All pass, except `e2e_relay`
`test_unarchive_emits_member_added_notification`. That test fails by
itself 3/5 with the original helper too, and CI's Relay E2E lane doesn't
select it.
- The Windows failure can't be reproduced on macOS. The Windows Rust
lane is the check.

Follow-up (out of scope): make the relay deliver each event id at most
once per subscription, keeping the live/history overlap.

### Verification

This branch was pushed without local re-verification, by the user's
choice: local test runs were overloading the machine and failing for
that reason. **Hosted CI is the check.** It was pushed with `git push
--no-verify`, because the pre-push hook runs clippy and unit tests.

- Both files are byte-identical to their final versions on
`flakey-tests` (67185df6a).
- Checked for this PR: `git diff --check`, `rustfmt --check`, and a
conflict-free merge with all six siblings onto `main`.

### Readiness

Test-only change. AGENTS.md's readiness steps (agent review, then human
confirmation) are still outstanding, so `buzz-review-completed` is not
set.

### Merge order

Independent of the other six. It merges cleanly onto `main`, and with
every sibling, in any order.

### Sibling PRs

Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease +
lane), #8678 (mobile), #8679 (Tauri apt cache + Windows Rust cap), #8680
(desktop product races), #8681 (desktop E2E state gates), #8682 (desktop
E2E composer + row actions).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@matt2e
matt2e merged commit ef4130d into main Oct 29, 2026
97 checks passed
@matt2e
matt2e deleted the flakey-tests-pg-lease branch October 29, 2026 23:00
loganj added a commit that referenced this pull request Oct 29, 2026
… helper

main does not compile its relay lib tests: #8676 renamed
test_state_with_database_url to test_state_with_unreachable_database_url,
and #8670 added two new calls to the old name in state.rs postgres_tests.
These tests need a real database, so they now use
test_state_with_database_url_and_pool_options with default pool options.

Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>
Signed-off-by: Logan Johnson <loganj@squareup.com>
matt2e added a commit that referenced this pull request Oct 30, 2026
…#8678)

## Summary

Fixes three mobile flakes from the 21–28 Oct 2026 CI runs, in test code
only:

- In two iOS `RunnerTests` classes, late bridge callbacks no longer fail
whichever test is running when they fire.
- The waits on a busy test-host main thread now cap at 10 s.
- Two Flutter avatar tests poll for the ready frame instead of sleeping
a fixed 50 ms.

This is one of seven PRs split out of the `flakey-tests` branch
(siblings below).

### Related issue

None found.

### Flaky tests fixed

| Test | Failing runs | Cause | Change |
|---|---|---|---|
| `BuzzPushSnapshotEnrichmentTests` (RunnerTests) | 40362671989 (job
121712747802, main), 39681273328#1 (job 119504560518), 40658588384 (job
122715950541, PR #8640) | The 1 s waits ran out while the test host's
Flutter app held the main thread, and the test returned and deleted its
temp directory. The bridge callback's assertions then ran later and were
blamed on whichever test was running (`ProfileSetupRecoveryTests`,
`IdentityFilePluginTests`). | Callbacks only record the value and
fulfill. Every assertion, including the `isRestricted` checks, runs
after the wait, so a late callback writes into a dead local. The seven
waits cap at 10 s (`callbackTimeout`). |
| `ProfileSetupRecoveryTests`
`testStartedEditorWithoutReadyRestoresProfileAndAllowsRetry` |
40585805357 (job 122477274956, PR #8568) | Engine release and editor
dismissal must both happen within 2 s, on the main thread the host app
keeps busy. | Both predicate waits cap at 10 s. |
| Flutter `channel_detail_page_test.dart` "profile sheet shows the
poster before autoplay and restores it on tap" and
`settings_profile_header_test.dart` "shows the poster until the animated
avatar is ready" | 40368315739 (job 121732711851, main) | After the
image response is released, both tests slept a fixed 50 ms of real time
before `pumpAndSettle`. A slower real decode left no frame scheduled, so
the ready key was missing. | A new `WidgetHelpers.pumpUntilFound` pumps
in 10 ms real-time steps until the ready key appears, capped at about 5
s. |

Local evidence is from the original `flakey-tests` commit (53aaec97c),
Xcode 27.1 on the iOS 26.2 simulator:

- Enrichment, with `Self.complete` delayed 1.5 s and 4 s: 6 failures
before, 107/107 after. With a 12 s delay, every failure lands in its
owning test. A mutated error code is still caught.
- Profile setup, with the engine release held 4 s: 2 failures before,
4/4 after. A leaked engine still fails.
- Avatar tests: under load, 94/96 before and 96/96 after. With a 200 ms
image delay, 0/5 before and 5/5 after. A widget that never signals ready
still fails.

Not included:

- The planned `AppDelegate` / `Runner.xcscheme` change that keeps
Flutter out of the RunnerTests host. It did not work locally: restored
scene sessions and the `UIMainStoryboardFile` fallback, and a normal
launch came up black afterwards.
- `IdentityFilePluginTests`
`testSyntheticDocumentsReadInPlaceAndRejectInvalidOrOversize`
(40441255307#1). It is left unchanged and stays open.

### Verification

This branch was pushed without local re-verification, by the user's
choice: local test runs were overloading the machine and failing for
that reason. **Hosted CI is the check.** It was pushed with `git push
--no-verify`, because the pre-push hook runs the Flutter tests.

- Every file is byte-identical to its final version on `flakey-tests`
(67185df6a).
- Checked for this PR: `git diff --check`, and a conflict-free merge
with all six siblings onto `main`.
- #8613's flake-check lane will repeat the two changed Dart test files.

### Readiness

Test-only change. AGENTS.md's readiness steps (agent review, then human
confirmation) are still outstanding, so `buzz-review-completed` is not
set.

### Merge order

Independent of the other six. It merges cleanly onto `main`, and with
every sibling, in any order.

### Sibling PRs

Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease +
lane), #8677 (relay E2E helper + Windows shell), #8679 (Tauri apt cache
+ Windows Rust cap), #8680 (desktop product races), #8681 (desktop E2E
state gates), #8682 (desktop E2E composer + row actions).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Jitter <2b2e6415e748c35180847a37aadae06a11e30dddc76b784e7fd9354c4eb42e7a@buzz.block.builderlab.xyz>
shu-hui-cash added a commit that referenced this pull request Oct 30, 2026
…nity-storage-calculation

* origin/main:
  fix(mobile): resolve iOS pushes on a fresh connection and retry a stale one (#8722)
  ci(loadgen): publish buzz-loadgen as ghcr.io/block/buzz-loadgen (#8717)
  test(relay): fix main's build after the test-state helper rename (#8718)
  fix(db): keep lease releases durable and deflake PostgreSQL lane tests (#8676)
  test: dedupe collect_until_eose by event id and uncap two shell tests (#8677)
  ci: cache the Tauri apt .debs and fit a cold Windows Rust run in its cap (#8679)
  test(desktop): gate flaky E2E specs on the state they assume (#8681)
  test(desktop): settle composer edits and press hover-only row actions by keyboard (#8682)

Signed-off-by: Shu Hui <shu@squareup.com>
matt2e added a commit that referenced this pull request Oct 30, 2026
## Summary

Fixes the four desktop product races behind recurring E2E flakes in the
21–28 Oct 2026 CI audit. Each fix has a regression test:

- In the channel browser, Enter acted on a stale deferred list.
- A stale autocomplete swallowed the Enter that submits an empty edit.
- Same-second messages swapped order on screen.
- The unread pill read "Jump to latest" instead of "N new messages".

Two E2E specs that failed for the same ordering reason also get
test-side fixes.

This is one of seven PRs split out of the `flakey-tests` branch
(siblings below).

### Related issue

None found for these. Draft #8559 does not fix the voice-note flake.
Draft #8661 (Refs #6733) closes the `#channel` list synchronously when
an edit is cleared. §11 below also covers that case, plus the `@mention`
and `:emoji` lists, so the two PRs overlap in `useChannelLinks.ts` and
`empty-edit-delete.spec.ts`.

### Flaky tests fixed

| Test | Failing runs | Cause | Change |
|---|---|---|---|
| §5 `channel-browser.spec.ts:402` "Enter with no matches jumps to
create" | 15 of 50 main runs, e.g. 40597484657, 40582387515,
40566841699, 40367495044 | **Product.** The list renders from
`useDeferredValue(query)`. An Enter that landed before the deferred
render read the old list, so it opened a stale match or skipped the
create row. | While the deferred query lags, Enter filters the live
query with the same `selectBrowserChannels` the list renders from. |
| §11 `empty-edit-delete.spec.ts:88`, `:111`, `:132` | :88 in 3 jobs,
:111 in 3, :132 in 2. :111 failed all 3 attempts on main 39822883395 and
39799090750 | **Product.** Clearing the draft went through the 120 ms
query debounce. The `#channel` list stayed open, so Enter re-inserted
the suggestion instead of submitting the empty edit. |
`useChannelLinks`, `useEmojiAutocomplete` and `useMentions` (typed `@`
only) close as soon as the trigger is gone. Typing a query still
debounces. `ChannelAutocomplete` gains
`data-testid="channel-autocomplete"`. |
| §10 `video-attachment.spec.ts:1469` "right-click menus expose distinct
selectors…" | 10 of 50 main runs, e.g. 40528171327 | **Product.** The
live cache merge (`sortMessages`, ascending id within a second) and the
channel-window projection (descending id) disagreed, so two same-second
rows swapped on screen as each writer landed. | A new
`shared/lib/eventOrder.ts` holds `compareRelayOrder` and its exact
reverse, `compareChronologicalOrder`. `sortMessages` and the window and
reconciliation sorts all use the window's order, so older-page paging is
unchanged. `sortEvents` keeps ascending id. The spec gives its three
messages distinct seconds. |
| §16 `voice-note.spec.ts:906` "starting a duplicate voice-note player
pauses the other instance" | 6 of 50 main runs. All 3 attempts on PRs
39622488737 and 40309414135 | Same-second messages plus index-based card
lookup. | Distinct seconds, and each card is found through its own
message row. |
| §13 `stream.spec.ts:477` "keeps scroll position when new messages
arrive above the fold" (integration) | 7 of 50 main runs. All 3 attempts
on PR 39546556833 | **Product.** Frozen rows came back reordered (7 of
12 instrumented runs). The selector ended older history at the
snapshot's first id, so moved rows were both prepended and kept in the
frozen tail. The duplicates filled the row count, `pendingCount` fell to
0, and the pill read "Jump to latest". Deleting a frozen row also
dropped the whole freeze. | `useBufferedTimelineMessages` keys rows by
`renderKey`, which is stable across an optimistic send and its ack
(`frozenRowKeys`). It drops only deleted frozen rows, and ends older
history at the first frozen row of any key. `stream.spec` is unchanged.
|

New regression tests:

- `channel-browser.spec`: two forced Enter variants.
- `empty-edit-delete.spec`: a `#`/`@`/`:` autocomplete table.
- `autocompleteTriggerClose.jsdom-test`.
- `projectChannelWindow.test`: the cache merge and the window agree in
both arrival orders.
- `channel-dense-second-reach.spec`: a RAF sampler sees one order.
- `useBufferedTimelineMessages.test`: deletion, reorder and ack rows.
- `virtualization.spec`: deleting a row while reading keeps the next
arrival counted.
- A `useThreadReplies.test` row: the thread fetch and a live append
agree on same-second order.

Local before/after counts are from the original `flakey-tests` commits
(238c08c7c, c66f411f3):

| Test | Before | After |
|---|---|---|
| channel-browser | 0/20 | 20/20 |
| autocomplete table | 0/30 | 30/30 |
| jsdom | 0/2 | 2/2 |
| dense-second sampler | 0/10 | 10/10 |
| buffered-timeline unit | 4/7 | 7/7 |
| virtualization deletion | 0/10 | 10/10 |
| `stream:477` against an isolated relay | 25/30, all failures "Jump to
latest" | 30/30, then 20/20 and 10/10 |

### Verification

This branch was pushed without local re-verification, by the user's
choice: local test runs were overloading the machine and failing for
that reason. **Hosted CI is the check.** It was pushed with `git push
--no-verify`, because the pre-push hook runs typecheck and the unit
suites. #8613's flake-check lane will also repeat the changed specs and
unit tests.

- Every file except `empty-edit-delete.spec.ts` and
`virtualization.spec.ts` is byte-identical to its final version on
`flakey-tests` (67185df6a). Those two carry only this PR's hunks; see
"Merge order".
- Checked for this PR: `git diff --check`, biome on both partial specs
(no changes, the same as on `main`), and a conflict-free merge with all
six siblings onto `main`.

### Readiness

These are **product fixes in the desktop app**. Under AGENTS.md "Before
opening a PR", this needs an agent review, an agent run of the affected
flows, and a **human in-app check** before `buzz-review-completed` is
added. Suggested check, in `just dev` or a local build against any
community:

1. **Channel browser:** open "Browse channels", type a name that matches
nothing and press Enter straight away. The create-channel flow opens
with that name. Then type an existing channel's exact name and press
Enter straight away. That channel opens, not an earlier match.
2. **Empty-edit delete:** edit one of your own messages and type `#gen`
so the channel list opens. Select all, delete, and press Enter at once.
The empty edit is submitted (the delete-by-clearing flow) and no
suggestion is re-inserted. Repeat with `@` (mention list) and `:sm`
(emoji list).
3. **Same-second order:** send two short messages in quick succession,
within one second. They appear in send order and never swap while they
land.
4. **Unread pill:** in a busy channel, scroll up past the fold, and have
a second client post one message. The pill reads "1 new message". While
still scrolled up, delete one of your visible messages, then have the
second client post again. The pill still counts the new message.

### Merge order

No required order, but two files are shared:

- `desktop/tests/e2e/empty-edit-delete.spec.ts` is also edited by
**flakey-tests-desktop-e2e-actions**. This PR adds the autocomplete
table after line 158. The sibling changes imports, helpers, `beforeEach`
and line 143.
- `desktop/tests/e2e/virtualization.spec.ts` is also edited by
**flakey-tests-desktop-e2e-gates**. This PR adds one test after line
929. The sibling's hunks are all above line 551.

The hunks don't touch, so each pair merges cleanly in either order, as
checked with `git merge-tree`. Draft #8661 overlaps this PR in
`useChannelLinks.ts` and `empty-edit-delete.spec.ts`; whichever lands
second will need a rebase.

### Sibling PRs

Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease +
lane), #8677 (relay E2E helper + Windows shell), #8678 (mobile), #8679
(Tauri apt cache + Windows Rust cap), #8681 (desktop E2E state gates),
#8682 (desktop E2E composer + row actions).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Jitter <2b2e6415e748c35180847a37aadae06a11e30dddc76b784e7fd9354c4eb42e7a@buzz.block.builderlab.xyz>
ravarora2 added a commit that referenced this pull request Oct 30, 2026
…vent-guards

* origin/main: (29 commits)
  Restyle community invitations (#8728)
  feat(relay): add incremental artifact reads (#8523)
  fix(mobile): show agent DM typing and balance reply bubble padding (#8731)
  perf(relay): route author- and #p-scoped global subscriptions by selector (#8666)
  fix(desktop): fix the product races behind four E2E flakes (#8680)
  test(scripts): stop file-size fixtures leaking git repos and fsmonitor daemons (#8727)
  test(mobile): keep late callbacks and fixed sleeps from failing tests (#8678)
  fix(mobile): resolve iOS pushes on a fresh connection and retry a stale one (#8722)
  ci(loadgen): publish buzz-loadgen as ghcr.io/block/buzz-loadgen (#8717)
  test(relay): fix main's build after the test-state helper rename (#8718)
  fix(db): keep lease releases durable and deflake PostgreSQL lane tests (#8676)
  test: dedupe collect_until_eose by event id and uncap two shell tests (#8677)
  ci: cache the Tauri apt .debs and fit a cold Windows Rust run in its cap (#8679)
  test(desktop): gate flaky E2E specs on the state they assume (#8681)
  test(desktop): settle composer edits and press hover-only row actions by keyboard (#8682)
  test(desktop): measure inline video controls against their player (#8683)
  relay: open the search pool lazily so a dead reader can't block boot (#8662)
  fix(mobile): address #8650 second-pass review (#8691)
  Revert "fix(mobile): push every message in your own Solo chats (#8615)" (#8690)
  fix: don't notify workflow owners through their attribution tag (#8694)
  ...

Signed-off-by: Ravneet Arora <rarora@squareup.com>

This branch was previously deployed

1 inactive deployment
codex-review — dae720ed Deployed Oct 29, 2026 by matt2e via Run Codex Security Review #10459
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codex-security-review-current The posted Codex security review matches its recorded range.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants