Repository navigation
fix(db): keep lease releases durable and deflake PostgreSQL lane tests - #8676
Conversation
Fixes the PostgreSQL-lane flakes from the 21-28 Oct 2026 CI audit: families B, C, D, E, F1, F2 and the §A residual. G (one hermit pgschema 504, a CI workflow change) is out of scope. Nothing on origin/main (eda26b9) or in an open or merged PR covers any of these. Local runs: macOS, OrbStack Postgres, cargo-nextest 0.9.136, Rust 1.98.1. Baselines ran from a nextest archive built before any fix (HEAD plus the B regression test). "Load" means one `yes` per core (16). B. PRODUCT FIX: a dropped lease heartbeat stranded the release. Tests: buzz-deletion user::worker::postgres_tests dummy_executor_failure_retries_same_community_stage (:47) and next_step_advances_one_handler_at_a_time_through_completion (:131) Runs: 40439041234 (main), 40399635380#3, 40413732817#1 Cause: run_stage_with_heartbeat drops the renewal future whenever the stage handler wins its select!. EntityDeletionStore::heartbeat opened a transaction. sqlx 0.9 PgTransactionManager::begin bumps transaction_depth only after BEGIN returns, so a begin() dropped mid-BEGIN queues no ROLLBACK. The release ping drains the BEGIN, and the connection goes back to the pool inside an open transaction. The worker's final autocommit `release` then ran inside it and reported success, but nothing committed. user-deletions run shares this path: the release is lost at exit, and reruns see "leased by another worker" until the lease expires (up to 60 s). Change: heartbeat is one autocommit UPDATE guarded by lock_owned's ownership/status/expiry predicate. The only callers are worker.rs and transition_postgres_tests.rs; the community loop's heartbeat is a different store. The worker comment now states why heartbeat must not open a transaction. Locking guarantee, as the leases.rs doc now states it: a superseded lease is never renewed; an expired lease can be renewed only while nobody has reclaimed it. Postgres re-checks the WHERE after a lock wait only when the holder committed an update of the row, so a holder that only took lock_owned's FOR UPDATE (save_stage_success writes entity_deletion_stage_history, never the parent row) or rolled back leaves the pre-wait expiry check in force. That is safe: claim takes the row with FOR UPDATE SKIP LOCKED while the waiting UPDATE holds the tuple lock, so no one can reclaim during the wait, and a reclaim that commits first bumps the generation, which the predicate re-checks. Only claim acts on expiry. Regression test: store::entity_deletion::postgres_tests:: dropped_heartbeat_leaves_no_open_transaction_for_the_release. It drops the production heartbeat with its BEGIN held by PausingProxy (1-connection pool, test_before_acquire(false)), then releases and asserts on a direct connection. Before -> after: regression test 3/3 FAIL (lease_owner Some("worker") after release() returned true) -> 3/3 PASS. Worker stress under load, before: 2/138 iterations FAIL at load average ~200, then 1/150 at ~25, all at :131 (3/288). After, at ~25: 300/300 iterations (2100 runs) PASS. Out of scope: the shutdown/renewal-failure branches can still drop a handler's begin() mid-BEGIN. The general fix belongs in sqlx. C. session_statement_timeouts_bound_writer_and_reader_sessions (buzz-db runtime/tests.rs:3138 "reader timeout: PoolTimedOut") Runs: 40128463987, 40314950864#1, 40374286094 (main), 40394821009#1, 40413732817#1 Cause: each config's lazy reader pool dialed cold inside the 150 ms READER_ACQUIRE_TIMEOUT. Db::new never runs the boot ping. Change: warm the reader the way the relay boots it (read_pool_boot_ping_for_tests + await_idle_floor). await_idle_floor moved from reader_pool_postgres_tests.rs to runtime/tests.rs. Before -> after, 30% of sessions delayed 250 ms by proxy, 50 iterations: 44/50 FAIL at :3138 -> 50/50 PASS. D. startup_owns_the_dependency_sampler (buzz-relay boot_lifecycle.rs) Runs: torn scrape 39865872891 (main), 39624074244#2, 40385799750#1, 40413293801#1; early exit 39704588623 Cause: metrics-exporter-prometheus 0.18.3 snapshots counters, then gauges, with no single snapshot. The scrape that first shows the completion gauge can predate the checks_total counters registered just before it. Change: assert on a second scrape taken after the gauge is seen. The counters were written before the gauge and never idle out. Fix the doc comment that claimed no window. When the relay exits early, wait_for_scraped_metric now panics with its exit status, stdout and stderr. Before -> after, 200 iterations under load: 2/200 FAIL at :857 (the scrape had the gauge and histograms but no checks_total) -> 200/200 PASS. Not reproduced: the early exit (:173). Only diagnostics were added. E. sample_writer_waits_for_shared_floor_lock_before_sampling_time (replica_fence.rs:1086) and replica_floor_probe_waits_for_shared_writer_and_records_after_release (runtime/tests.rs:3452), both "MaskedActivity { masked: 1 }" Runs: 39601763602 (main), 40391445284#1; 39825846222, 40301667418 Cause: both scan cluster-wide pg_stat_activity but were outside the serialized postgres-cluster-global group, so other lane tests' connection churn masked activity. Change: rename both with a cluster_global_ prefix (each migrates its own scratch DB, so not migration_schema_). `cargo nextest show-config test-groups --profile postgres-ci` lists both in postgres-cluster-global. Before, 50 iterations with pgbench -C -c 8 on the cluster: 2/50 iterations FAIL (3 MaskedActivity panics). After: the buzz-db lane passes 3x (613/613). Not re-run under pgbench; the group serializes lane tests only. F1. stalled_reader_sessions_release_their_slots_for_new_connections (reader_pool_postgres_tests.rs:354 "second warm connection: PoolTimedOut") Run: 40399635380#2 Cause: the floor warms only one connection, so the second warm-up acquire dials cold inside the 150 ms budget. Change: retry only that warm-up acquire on PoolTimedOut within 5 s (#8472's precedent). The stall under test keeps the production budget. Before -> after, session-delay proxy, 30 iterations: 10/30 FAIL at :354 -> 30/30 PASS. F2. readiness_check_cancellation_balances_waiter_and_inflight_connection (runtime/tests.rs:391 "cancelled in-flight query must return or replace its connection: Elapsed(())") Run: 39785370353 Cause: the test aborted pg_sleep(5) after a fixed 25 ms sleep. sqlx's release ping waits for the query, so recovery lands at server start + 5 s, against a bound of abort + 5 s (~22 ms margin locally). Change: poll pg_stat_activity until the query is active, then abort. Use pg_sleep(1), so recovery takes about 1 s inside the unchanged 5 s bound. Before -> after, a proxy delaying only the Bind/Execute chunk after pg_sleep's Parse by 40 ms: 10/10 FAIL at :391 -> 10/10 PASS. §A residual. upgrade_would_deny_line_is_written_at_auth_while_open (buzz-relay connection.rs) used the 100 ms-acquire test_state_with_database_url against the real test database. No failure observed yet; not reproduced. It now uses test_state_with_database_url_and_pool_options with sqlx's default timeout. The 100 ms helper is renamed test_state_with_unreachable_database_url; all of its remaining callers use closed ports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Toohey <contact@matttoohey.com>
🔐 Codex Security Review
Review SummaryOverall Risk: NONE
FindingsNo concrete security, correctness, or reliability findings were identified. Notes
Generated by Codex Security Review | |
jedwards27
left a comment
There was a problem hiding this comment.
Verdict: APPROVE
Reviewed: eda26b9423328a06960bb0d5fd4c1af54600c419..dae720ed982f0360fba95e3dc5038eb32f626edd (exact head dae720ed982f0360fba95e3dc5038eb32f626edd)
Risk: high — this touches database lease ownership/recovery for user deletion plus PostgreSQL/relay CI behavior. No schema or migration files changed.
Behavior/contracts traced:
EntityDeletionStore::heartbeatnow uses one owner/generation/status/expiry-fenced autocommitUPDATE, avoiding the cancellable explicit transaction that could leave a pooled session open and make the followingreleaseappear successful without committing (crates/buzz-db/src/store/entity_deletion/leases.rs:111).- The worker still keeps heartbeat cancellation local to
run_stage_with_heartbeat; the store-side autocommit contract is the boundary that makes dropping the renewal future safe (crates/buzz-deletion/src/user/worker.rs:121). - The new PostgreSQL regression exercises the real dropped-heartbeat path through a pausing proxy, then proves release commits and no session remains
idle in transaction(crates/buzz-db/src/store/entity_deletion_postgres_tests.rs:452). - The test-only changes are scoped to the stated flakes: reader warm-floor setup, readiness cancellation timing, cluster-global PostgreSQL serialization, unreachable-database helper naming, and relay boot diagnostics.
Findings: no blocking findings.
Author action: none for code. Keep the PR-body readiness checklist: do not add buzz-review-completed until the repo checklist's human local-relay check has been performed.
Verification owner: CI for exact-head automated gates; author/human operator for the local relay readiness checklist.
Validation:
gh api user --jq .login→jedwards27; live PR author ismatt2e./Users/judeedwards/.buzz/scripts/buzz-pr-review-preflight 8676atdae720ed982f0360fba95e3dc5038eb32f626edd: PASS; base/head matched, no unresolved review threads, and exact-head CI checks showed required PostgreSQL/relay/backend/desktop/Rust/security/DCO gates successful or intentionally skipped by path selection.. ./bin/activate-hermit && git diff --check eda26b9423328a06960bb0d5fd4c1af54600c419...HEAD && scripts/test-postgres-test-discovery.sh: PASS atdae720ed982f0360fba95e3dc5038eb32f626edd.. ./bin/activate-hermit && cargo test -p buzz-deletion user::worker::tests --lib: PASS atdae720ed982f0360fba95e3dc5038eb32f626edd.
Manual/native evidence: no desktop/manual native journey run; this is a database/relay change. I did not run the full local PostgreSQL lane or local relay user-deletion workflow; exact-head CI supplies the automated PostgreSQL/relay gates, and the human local-relay readiness check remains external to this code review.
Residual risk: the original race is timing-sensitive and local infrastructure was not re-run here. The exact-head PostgreSQL regression and hosted CI are the merge evidence; the repo checklist's human local-relay smoke remains required before claiming buzz-review-completed.
… by keyboard (#8682) ## Summary Test-side fixes for the two largest desktop E2E flake clusters in the 21–28 Oct 2026 CI audit: - **Composer `fill()` races the app's deferred focus-to-end** (§1). Text was appended to the old draft, or a clear did nothing. - **Clicks on the hover-only message action bar while the timeline re-pins** (§3/§8/§18). Reply and more-actions clicks timed out or were intercepted. Two shared helpers replace the per-spec workarounds, and §9 rides along because its spec now imports them. The PR changes test code only. It is one of seven PRs split out of the `flakey-tests` branch (siblings below). ### Related issue None found. #8559, #8622 and #8613 touch these specs or helpers, but had no coverage for these clusters. Draft #8661 edits `empty-edit-delete.spec.ts`'s `submitEmptyEdit` and `beforeEach`, the same lines as this PR. ### New helpers - `tests/helpers/composer.ts`: - `clearComposer` waits for the app's focus, then repeats select-all + Delete through the editor keymap until the editor is empty (250 ms per try, 5 s overall). - `replaceComposerText` clears, then fills. On an empty document, a late collapse to the end is a no-op. - `tests/helpers/channelHeadReady.ts`: - `GENERAL_CHANNEL_ID`. - `openGeneral` clicks `#general`, then calls `waitForMockChannelHeadReady`. - `pressRowAction(button)` does focus + Enter. The action bar also shows on focus-within, and more-actions is a Radix trigger that opens on Enter. ### Flaky tests fixed | Test | Failing runs | Cause | Change | |---|---|---|---| | §1 `inbox-edit.spec.ts:525` | 39641311877, 39821769576, 40333705511, 40348069929, 40388246283 | **Append:** entering edit mode, or closing the non-member mention dialog, focuses the end of the draft in deferred steps: RAF, a nested RAF, then ProseMirror's 20 ms selection resync. `fill()` selects and inserts in separate protocol steps, so a step that lands in between collapses the selection and the text is appended. **Clear does nothing:** `fill("")` presses Delete before ProseMirror handles `selectionchange`, so the editor blocks it. | `clearComposer` / `replaceComposerText` | | §1 `inbox-edit.spec.ts:560` | 39538411386, 39821769576 | Same as above. | Same | | §1 `mention-recipients.spec.ts:591` (both variants) | 39799090750, 40374286094, 39641311877 | Same | Same | | §1 `mention-recipients.spec.ts:691` (both variants) | 40348069929, 40556019361, 39601763602 | Same | Same | | §1 `messaging.spec.ts:3799` | 39827234118, 40445481983 | Same, plus a bare `#general` click and a `force: true` Reply | Same, plus `openGeneral` and `pressRowAction` | | §1 `messaging.spec.ts:4022` | 40389392495 | Same | Same | | §1 `messaging.spec.ts:4289` | 39822883395, 40557808800, 40566841699 | Same | Same | | §1 `messaging.spec.ts:4368` | 40574058918 | Same | Same | | §1 `remote-owned-mentions.spec.ts:463` (both variants) | 39618692657, 40374286094, 40382726903, 40389392495, 40582387515 | Same, after the Invite dialog closes | Same | | §3 `messaging.spec.ts` :3531, :3647, :4122, :4208, :4450 (root and reply) and :4633 (keyboard and button) | :3531 40583830861, 40595483966. :3647 40566841699, 40597484657. :4122 39641311877, 40368315739, 40368363357. :4208 39525092486, 39796569342, 40367495044. :4450 39796569342, 40348069929, 40373404267. :4633 39530486096, 40348069929, 40368315739, 40582387515, 40368363357 | The action bar is `pointer-events-none` until its row is hovered. Opening `#general` refreshes the head after the live subscription lands, and the timeline re-pins, so the row moves out from under the pointer. `force: true` only hid this. | `openGeneral`, then `pressRowAction` on Reply or more-actions. Six more messaging tests that used hover + `click({ force: true })` get the same change, and each first waits for its last row to contain the sent root. | | §18 `thread-follow.spec.ts:350` | 39529963092, 40348069929, 40583830861 (all 3), 40595483966; PR 39541263507 (all 3) | Same; the composer overlay intercepts the more-actions click. | Same | | §8 `thread-head-stale-edit.spec.ts:55` | 39525092486, 39618692657, 39796569342, 39822883395, 40011714848, 40281448810, 40368363357, 40389392495, 40445481983, 40574058918; PR 40031299575 (all 3) | Same (used `force: true`) | Same | | `file-attachment.spec.ts:165` | 1–2 CI failures (audit long tail) | A more-actions click on a moving timeline | Same | | `empty-edit-delete.spec.ts` (`openMoreActionsMenu`, `submitEmptyEdit`, the non-empty edit, `beforeEach`) | Covered under §11 in the desktop-races sibling | The same hover and `fill` races | The head-ready wait before emitting, `pressRowAction`, `clearComposer` and `replaceComposerText`. The autocomplete table rows keep their back-to-back keys, which is what they test. | | §9 `file-attachment.spec.ts:419` "upload progress floats above the dock and lifts Jump to latest" | 10 of 50 main jobs, including 40583830861, 40597484657, 40608383847 | A programmatic `scrollTop` keeps the bottom intent armed. The upload card resizes the content, the timeline re-pins, and the pill unmounts under `boundingBox()`. The fixed 250 ms wait read the card mid-slide. | `scrollTimelineUp()` leaves the bottom with a real wheel, and one `expect.poll` checks all three layout relations. | Local before/after counts are from the original `flakey-tests` commits (e286a6ae0, 816c7c35e, c66f411f3, 67185df6a), running `main`'s copy and the changed copy side by side under forcings: - **§1 at 4× CPU throttle:** 130/130 after; `main`'s copies ranged from 5/10 to 10/10. - **§1 with `selectionchange` delayed, a stretched resync, or a gap inside `fill()`:** `main`'s copies 0/5 to 4/5 and 0/20; after 65/65 and 20/20. - **§3 at 4× throttle:** `main` had 3 Reply-click timeouts at :4208 and 3 at :4633 (keyboard); after 120/120. - **`file-attachment:165`:** 0/3 before, 3/3 after. - **§9 unforced:** 4/5 before, 5/5 after. These were measured with the helper's first version, which waited two frames plus the 20 ms resync. The final state-based helper ran unforced only: 157/157 and then 196/196 over the touched specs. Its retry was not checked under a forced late collapse. :3531, :3647, :4122, :4450 root, :4633 button, `thread-head-stale-edit:55` and the six force-click tests were not force-reproduced; they get the fix that matches their CI signatures and reproduced siblings. Dropped: - `mention-recipients.spec.ts:933` (40608383847) fails on `flavors.handled`, which is unrelated. - `persistent-agent-audience.spec.ts:828` (`@Mor@Morg`) did not reproduce in 79 forced runs. ### Verification This branch was pushed without local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. **Hosted CI is the check.** It was pushed with `git push --no-verify`, because the pre-push hook runs typecheck and the unit suites. #8613's flake-check lane will also repeat the changed specs. - Every file except `empty-edit-delete.spec.ts` is byte-identical to its final version on `flakey-tests` (67185df6a). That file carries only this PR's hunks; see "Merge order". - Checked for this PR: `git diff --check`, biome on the partial spec (no changes, the same as on `main`), and a conflict-free merge with all six siblings onto `main`. ### Readiness Test-only change. AGENTS.md's readiness steps (agent review, then human confirmation) are still outstanding, so `buzz-review-completed` is not set. ### Merge order No required order. `desktop/tests/e2e/empty-edit-delete.spec.ts` is also edited by **flakey-tests-desktop-races**, which adds the autocomplete table after line 158. This PR changes imports, helpers, `beforeEach` and line 143. The hunks don't touch, so the two merge cleanly in either order, as checked with `git merge-tree`. Draft #8661 edits the same `submitEmptyEdit`/`beforeEach` lines; whichever lands second will need a rebase. ### Sibling PRs Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease + lane), #8677 (relay E2E helper + Windows shell), #8678 (mobile), #8679 (Tauri apt cache + Windows Rust cap), #8680 (desktop product races), #8681 (desktop E2E state gates). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary Test-side fixes for ten desktop E2E flake clusters and six long-tail rows from the 21–28 Oct 2026 CI audit. Each spec now waits for the state it assumes (a rendered row, a mounted listener, a finished animation, a held boot read), not a fixed delay or a one-shot probe. Two mock-bridge seams support this. The PR changes test code and mock-bridge seams only, apart from one product constant extracted with its value unchanged. It is one of seven PRs split out of the `flakey-tests` branch (siblings below). ### Related issue None found. - #8622, #8244 and #8386 touch some of these spec files, but not the tests fixed here. - §7 mode A (`navigation:472`) is left to open PR #6852, which already sets the width and measures in one polled evaluate. - Draft #8279 queues cold-launch nostr-bind links in the product. Its mock returns null, so the §12 listener seam is still needed. ### Mock-bridge seams - `__BUZZ_E2E_EVENT_LISTENER_COUNT__(event)` counts live **app** listeners per Tauri event. It wraps the mocked `plugin:event|listen`/`unlisten`. The spec-facing `__TAURI_INTERNALS__.listen` passes a marker target that the wrapper skips, so a spec's own subscription can't satisfy the wait. Tauri 2.11.1's mock ignores targets, so delivery is unchanged. - `deferFirstChannelsRead` arms the existing channels-read hold at boot. `__BUZZ_E2E_RELEASE_CHANNELS_READ__` releases it. - `CREATE_PROJECT_REFOCUS_DELAY_MS` moves to `createProjectFormFocus.ts` with its value unchanged (50 ms). Both `CreateProjectFormContent` and `project-commit-detail.spec` import it. ### Flaky tests fixed | Test | Failing runs | Cause | Change | |---|---|---|---| | §2 `overscroll-boundary.spec.ts:34` (and its sibling) | 39601763602, 40367495044, 40583830861, 40597484657, 40373404267 | The `message-timeline` test id first lands on the loading skeleton, which doesn't overflow, so the scroll lock rightly cancels the wheel. | Wait for a rendered row, and content taller than the viewport, before dispatching wheels. | | §4 `project-conversation-load-failure.spec.ts:73` | 40367495044 (all 3 attempts), 40608383847, 40583830861; PRs 39541263507, 40509460168 | The seeded root and reply were dated `now`. The panel's live REQ (`since = floor(now)`) still replayed the reply, so the error card never appeared. | Date both events a minute in the past, and correct the comment. | | §6 `workflows.spec.ts` `createWorkflow` cluster (:222, :365, :297, :525, :471, :406, :614, :997) | 14 of 50 main jobs; :222 ×4, e.g. 40353619362 | `AnimatePresence mode="wait"` keeps the trigger body mounted through its exit, so "Message text" could fill the trigger's field (:222 and the callers). For :297, `isVisible()` read "closed" one frame before the dialog opened its own channel list. | Wait for the step body's "Step details" control before filling. Create dialogs wait for the list; only the duplicate dialog clicks it. | | §12 `nostr-bind.spec.ts` (:156, :198, :228, :247, :260, :335, :417, :447) | 8 main jobs, each test once | The spec emitted `deep-link-nostr-bind` before the app subscribed, and the mock drops events that have no listener. | Wait for `__BUZZ_E2E_EVENT_LISTENER_COUNT__` to report the app listener before emitting. | | §15 `profile.spec.ts:1128` (integration) | 7 of 50 main jobs | The 320 ms motion-sampling window started before the click. | Start the window when `activeTab` reads "runtime". | | §17 `agent-availability.spec.ts:602`, both shutdown-failure variants (integration) | 6 jobs (4 disconnected-online, 2 failed-online) | The "shutdown refused" toast sits over the Delete row and pauses while the pointer is on it, so the retry click is intercepted. | Assert the toast, then retry with focus + Enter. The toaster has no close button. | | §19 `virtualization.spec.ts:247` "08 — cascading older pages…" | 4 of 50 main jobs; PR 40605736093 (all 3 attempts) | The 120 ms and 400 ms windows were timed from before the wheel bursts, and CDP can deliver wheels late. | Count the wheels the page receives. Each window opens at the first wheel and closes only after all have landed, then waits for still frames. **A second, product-side mode remains** (see below). | | §20 `message-copy-link.spec.ts:160` | 4 of 50 main jobs (5 failed attempts), e.g. 40595483966 | The huddle row reused `"d".repeat(64)`, which is a seeded ordinary `#general` message. | Use ids that no seed uses, and emit only after `waitForMockChannelHeadReady`. | | §21 `navigation.spec.ts:687` | 4 of 50 main jobs | The pending chip's `boundingBox()` was read once, and can be null while the row mounts. | Poll until the chip has a width. | | §23 `sidebar-snapshot.spec.ts:358`, `:388`, `:438` | PR 40427822003 (all 3 attempts); never on main | The 500 ms loading-state budget counted app boot, so a slow runner finished the channels read first. | Hold the first channels read with `deferFirstChannelsRead`, assert the loading state, then release it. | | `project-commit-detail.spec.ts:259` | 1–2 CI failures (audit long tail) | A 50 ms refocus timer can land mid-fill and type the description into the name field. | Once the name field is focused, wait out a page timer of `CREATE_PROJECT_REFOCUS_DELAY_MS`; equal-delay timers fire in order. | | `project-pr-review.spec.ts:3520` | 1–2 CI failures | The same timer pattern in `CreateProjectWorkItemDialog`. | Wait for `toBeFocused()` on the title, which only that timer can satisfy. | | `thread-load-failure.spec.ts:79` | 1–2 CI failures | `openWelcomeThread` branched on a one-shot `count()` and fell back to a hover-only Reply click. | Wait for the summary row, then click it. | | `virtualization.spec.ts:201` | 1–2 CI failures | The drop point sat 2 px above the target's bottom edge, after two fixed frames. | Drop at the centre, after dnd-kit's live region names the target. | | `channel-composer-overflow.spec.ts:217` | 1–2 CI failures | Same-second emits sort by id, and the clearance was checked once. | `createdAt: base + i` and `expect.poll`. | | `scroll-history.spec.ts:1289` | 1–2 CI failures | A fixed 250 ms wait before idle sampling. | Poll for coverage first. | Local before/after counts are from the original `flakey-tests` commits (e286a6ae0, 816c7c35e, 67185df6a). Each used a forcing that made `main`'s copy fail with CI's signature: | Test | Before | After | |---|---|---| | §2 | 3/15 | 15/15 | | §4 | 0/10 | 10/10 | | §6 :222 | 0/3 | 3/3 | | §6 :297 | 0/3 | 3/3 | | §12 | 0/3 | 3/3 | | §15 | 0/3 | 3/3 | | §17 | 1/4 | 4/4 | | §19 late-wheel modes | 0/3 | 3/3 and 2/3 | | §23 | 2/6 | 6/6 | | project-commit-detail | 0/3 | 3/3 | | project-pr-review | 0/3 | 3/3 | §20, §21, `thread-load-failure:79`, `virtualization:201`, `channel-composer-overflow:217` and `scroll-history:1289` were not reproduced; they get the fix that matches their CI signature. Not fixed (follow-up): `virtualization:247` also has a product-side mode. In one frame after a prepend, `scrollHeight` can drop 458–639 px while `scrollTop` moves only 50–114 px, so the anchor row jumps 382–457 px. That is lost anchor compensation, and this test can still fail on it. ### Verification This branch was pushed without local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. **Hosted CI is the check.** It was pushed with `git push --no-verify`, because the pre-push hook runs typecheck and the unit suites. #8613's flake-check lane will also repeat the changed specs. - Every file except `virtualization.spec.ts` is byte-identical to its final version on `flakey-tests` (67185df6a). That file carries only this PR's hunks; see "Merge order". - Checked for this PR: `git diff --check`, biome on the partial spec (no changes, the same as on `main`), and a conflict-free merge with all six siblings onto `main`. ### Readiness Test and mock-bridge change; the one product edit is a constant extraction with its value unchanged. AGENTS.md's readiness steps (agent review, then human confirmation) are still outstanding, so `buzz-review-completed` is not set. ### Merge order No required order. `desktop/tests/e2e/virtualization.spec.ts` is also edited by **flakey-tests-desktop-races**, which adds one test after line 929; this PR's hunks are all above line 551. They don't touch, so the two merge cleanly in either order, as checked with `git merge-tree`. ### Sibling PRs Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease + lane), #8677 (relay E2E helper + Windows shell), #8678 (mobile), #8679 (Tauri apt cache + Windows Rust cap), #8680 (desktop product races), #8682 (desktop E2E composer + row actions). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Jitter <2b2e6415e748c35180847a37aadae06a11e30dddc76b784e7fd9354c4eb42e7a@buzz.block.builderlab.xyz>
…cap (#8679) ## Summary Fixes the two CI workflow defects that turned `main` red most often in the 21–28 Oct 2026 audit: - **Desktop Tauri:** the lane now caches its apt `.deb`s. It no longer re-downloads 62 MB from a slow Ubuntu mirror on every run. - **Windows Rust:** the lane drops a duplicate `cargo check` and gets a 60 min cap, so a cold run can save its cache. It also updates #8613's `scripts/flake-check.test.mjs`, which breaks once the Tauri package list moves into an env var. There are no product code or test-behaviour changes. This is one of seven PRs split out of the `flakey-tests` branch (siblings below). ### Related issue None found. #8613 touched both workflows after the audit, but not these steps. No open PR changes how the Tauri packages are fetched, the Windows timeout, or its Check (workspace) step: - #8422, #8321 and #8373 only add packages to the list, and would now add them to `TAURI_APT_PACKAGES`. - #8414 adds two mesh-llm Tauri steps to the Windows job. If it lands, they must fit the cold budget below. ### CI failures fixed | Lane / test | Failing runs | Cause | Change | |---|---|---|---| | Desktop Tauri, "Install Tauri dependencies (Linux)" (`_ci-desktop.yml`) | 62 jobs in 51 run attempts, 22–27 Oct. On main: 39821769576, 39822883395, 39827234118, 40333705511, 40342468627, 40348069929, 40353619362, 40368315739, 40373404267, 40374519539, 40556742929 | All 3 `ci-apt-retry.sh` attempts hit their 420 s limit while `azure.archive.ubuntu.com` served the 62 MB at 20–65 KB/s. The 25.5 MB `libwebkit2gtk-4.1-0` alone takes a whole attempt, and each attempt loses its partial download. | The package list moves to a job-level `TAURI_APT_PACKAGES`. A new "Resolve Tauri apt archives" step builds the cache key from `apt-get install --print-uris -o Acquire::ForceHash=SHA256` (name, size and SHA256 per file). `actions/cache/restore` uses that key. The install step copies only the restored files whose SHA256 matches the signed index, because apt 2.8.3 reuses any same-size cached `.deb` without hashing it. A best-effort rebuild keeps exactly the listed files. `actions/cache/save` runs only on core-lane pushes after a successful rebuild; PR runs never save. | | Windows Rust (`_ci-rust.yml`) | All on main: 39796569342 (job 119885795086), 39821769576 (119975358772), 39827234118 (119994286695), 39865872891 (120128351939), 40402556064 (121852634685) | Every test passed. Cold steps take 41.3–43.5 min, so the 45 min cap cancelled the `Swatinem/rust-cache` post-run save 1.6–3.7 min in. No cache was written, and the next push was cold again. | Delete "Check (workspace)". The clippy step before it covers the same `--workspace --all-targets` set with `-D warnings`, which saves 2.0–2.8 min. Raise `timeout-minutes` from 45 to 60, with the measured breakdown in a comment: 39–41 min of steps, a ~4.5 min save and ~15 min of margin. | | `scripts/flake-check.test.mjs` "repeat invocations mirror the lanes they stand in for" | None in CI yet: it fails only with this change applied | #8613 added this test after `flakey-tests` branched. It checks that the advisory `flake-tauri` lane installs the same packages as the Desktop Tauri core lane, by regex-matching the inline `apt-get install -y --no-install-recommends \` list in `_ci-desktop.yml`. That list now lives in `TAURI_APT_PACKAGES`, so the regex matched nothing and the test failed with "Desktop Tauri apt packages". | Read `TAURI_APT_PACKAGES` from `_ci-desktop.yml` and compare package names (12 on each side) with `_ci-flake.yml`'s list. Renaming one package in `_ci-flake.yml` still fails the test. Giving the `flake-tauri` lane the same read-only `.deb` restore is a follow-up. | Local evidence is from the original `flakey-tests` commit (9aa768a3b): - The apt steps ran verbatim in Ubuntu 24.04 containers with apt 2.8.3, with the cache restore and the save step's `if:` emulated. - On a miss, the install fetches everything ("Need to get 254 MB"). On an exact hit with the network down, it fetches nothing ("Need to get 0 B/254 MB"). - On a fallback with one `.deb` missing and one corrupted at the same size, the hash check rejects the corrupt file, and apt re-fetches only the corrupt and missing files. - The key was identical in all four simulated jobs. - The Windows numbers come from the jobs API for the five runs above. On this branch, the split session ran the `scripts/` node tests: `ci-selection`, `ci-required-gates`, `ci-apt-retry` and `flake-check` on macOS gave 84 pass, 0 fail and 8 skipped (Linux-only). Before the `flake-check.test.mjs` fix, that test failed 1 case. The same session ran actionlint 1.7.12 with shellcheck, which was clean, and the rust-cache contract scripts, which passed. ### Verification This branch was pushed without further local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. **Hosted CI is the check.** It was pushed with `git push --no-verify` over SSH. The pre-push hook runs tests, and the HTTPS token lacks the `workflow` scope. Only hosted CI can show: - The runner's real package list (about 62 MB) and the same key in all three Tauri lanes. - The `actions/cache` round trip. The first PR run misses, because `main` has no key yet. After merge, the first push to `main` saves it. - A cold Windows run saving its cache within the new cap. The next toolchain bump or cache eviction will show it. Remaining risk: after Ubuntu updates a large package, runs fetch it from the mirror until a push to `main` saves a new key. ### Readiness CI-only change. AGENTS.md's readiness steps (agent review, then human confirmation) are still outstanding, so `buzz-review-completed` is not set. ### Merge order Independent of the other six. It merges cleanly onto `main`, and with every sibling, in any order. ### Sibling PRs Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease + lane), #8677 (relay E2E helper + Windows shell), #8678 (mobile), #8680 (desktop product races), #8681 (desktop E2E state gates), #8682 (desktop E2E composer + row actions). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#8677) ## Summary Fixes two CI flakes from the 21–28 Oct 2026 audit, both in test code with no product change: - The relay E2E helper `collect_until_eose` now drops repeat copies of an event id. - Two Windows shell tests drop a 5 s cap that a cold Git Bash start can exceed. This is one of seven PRs split out of the `flakey-tests` branch (siblings below). ### Related issue None found for these two. The buzz-terminal `passwd_shell()` `getpwuid` race (`env_fence_tests::resolve_shell_falls_through_to_passwd_not_the_default`, run 38590095289) is left to open PR #7687, which already replaces `getpwuid`. #8636 and #7976 edit `buzz-test-client/src/lib.rs`, and #7607 and #7915 edit `buzz-dev-mcp/src/shell.rs`, but none of them changes `collect_until_eose` or these timeouts. ### Flaky tests fixed | Test | Failing runs | Cause | Change | |---|---|---|---| | Relay E2E `e2e_project::test_project_tombstone_deletes_coordinate_and_spares_members` (e2e_project.rs:338, "left: 2 right: 1") | 40408678412#1, job 121879630394 (PR #8620, schema-only) | The EVENT handler sends OK, then fans out in a spawned task. `handle_req` registers the live subscription before its history query, and `req.rs`'s `seen_ids` dedups history rows only. If fan-out runs after the REQ is registered, the event arrives once live and once from history, both before EOSE, and `collect_until_eose` kept every frame. | `collect_until_eose` keeps the first copy of each event id, as production clients do. Its doc comment says a test of the relay's own dedup must read frames with `recv_event`. A new unit test, `collect_until_eose_keeps_one_copy_of_each_event_id`, uses a fake relay that sends the duplicate. | | Windows `buzz-dev-mcp` `shell::tests::basic_echo` (shell.rs:1026, exit 124) | 39675698081#1, job 119485363399 (green on re-run) | 124 is the shell tool's own timeout exit code. `basic_echo` capped the shell at 5 s, and it is one of the first tests in the process to start Git Bash, which can take longer than 5 s on a cold runner. | `timeout_ms: None` (the production 120 s default) in `basic_echo` and in `workdir_is_honored`, which has the same exposure. `timeout_fires` still covers timeouts. | Pre-check: no e2e test relies on the relay's own history dedup. Every exact-count assertion uses a single filter. The NIP-33 "only newest remains" assertions would see a failed replacement as two different ids, and this dedup keeps both. Local evidence is from the original `flakey-tests` commit (fc4518d39): - The new unit test: FAIL before, PASS after. - 13 of the 14 e2e files that call the helper, plus `e2e_thread_roots`, against an isolated local relay. All pass, except `e2e_relay` `test_unarchive_emits_member_added_notification`. That test fails by itself 3/5 with the original helper too, and CI's Relay E2E lane doesn't select it. - The Windows failure can't be reproduced on macOS. The Windows Rust lane is the check. Follow-up (out of scope): make the relay deliver each event id at most once per subscription, keeping the live/history overlap. ### Verification This branch was pushed without local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. **Hosted CI is the check.** It was pushed with `git push --no-verify`, because the pre-push hook runs clippy and unit tests. - Both files are byte-identical to their final versions on `flakey-tests` (67185df6a). - Checked for this PR: `git diff --check`, `rustfmt --check`, and a conflict-free merge with all six siblings onto `main`. ### Readiness Test-only change. AGENTS.md's readiness steps (agent review, then human confirmation) are still outstanding, so `buzz-review-completed` is not set. ### Merge order Independent of the other six. It merges cleanly onto `main`, and with every sibling, in any order. ### Sibling PRs Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease + lane), #8678 (mobile), #8679 (Tauri apt cache + Windows Rust cap), #8680 (desktop product races), #8681 (desktop E2E state gates), #8682 (desktop E2E composer + row actions). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… helper main does not compile its relay lib tests: #8676 renamed test_state_with_database_url to test_state_with_unreachable_database_url, and #8670 added two new calls to the old name in state.rs postgres_tests. These tests need a real database, so they now use test_state_with_database_url_and_pool_options with default pool options. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Signed-off-by: Logan Johnson <loganj@squareup.com>
…#8678) ## Summary Fixes three mobile flakes from the 21–28 Oct 2026 CI runs, in test code only: - In two iOS `RunnerTests` classes, late bridge callbacks no longer fail whichever test is running when they fire. - The waits on a busy test-host main thread now cap at 10 s. - Two Flutter avatar tests poll for the ready frame instead of sleeping a fixed 50 ms. This is one of seven PRs split out of the `flakey-tests` branch (siblings below). ### Related issue None found. ### Flaky tests fixed | Test | Failing runs | Cause | Change | |---|---|---|---| | `BuzzPushSnapshotEnrichmentTests` (RunnerTests) | 40362671989 (job 121712747802, main), 39681273328#1 (job 119504560518), 40658588384 (job 122715950541, PR #8640) | The 1 s waits ran out while the test host's Flutter app held the main thread, and the test returned and deleted its temp directory. The bridge callback's assertions then ran later and were blamed on whichever test was running (`ProfileSetupRecoveryTests`, `IdentityFilePluginTests`). | Callbacks only record the value and fulfill. Every assertion, including the `isRestricted` checks, runs after the wait, so a late callback writes into a dead local. The seven waits cap at 10 s (`callbackTimeout`). | | `ProfileSetupRecoveryTests` `testStartedEditorWithoutReadyRestoresProfileAndAllowsRetry` | 40585805357 (job 122477274956, PR #8568) | Engine release and editor dismissal must both happen within 2 s, on the main thread the host app keeps busy. | Both predicate waits cap at 10 s. | | Flutter `channel_detail_page_test.dart` "profile sheet shows the poster before autoplay and restores it on tap" and `settings_profile_header_test.dart` "shows the poster until the animated avatar is ready" | 40368315739 (job 121732711851, main) | After the image response is released, both tests slept a fixed 50 ms of real time before `pumpAndSettle`. A slower real decode left no frame scheduled, so the ready key was missing. | A new `WidgetHelpers.pumpUntilFound` pumps in 10 ms real-time steps until the ready key appears, capped at about 5 s. | Local evidence is from the original `flakey-tests` commit (53aaec97c), Xcode 27.1 on the iOS 26.2 simulator: - Enrichment, with `Self.complete` delayed 1.5 s and 4 s: 6 failures before, 107/107 after. With a 12 s delay, every failure lands in its owning test. A mutated error code is still caught. - Profile setup, with the engine release held 4 s: 2 failures before, 4/4 after. A leaked engine still fails. - Avatar tests: under load, 94/96 before and 96/96 after. With a 200 ms image delay, 0/5 before and 5/5 after. A widget that never signals ready still fails. Not included: - The planned `AppDelegate` / `Runner.xcscheme` change that keeps Flutter out of the RunnerTests host. It did not work locally: restored scene sessions and the `UIMainStoryboardFile` fallback, and a normal launch came up black afterwards. - `IdentityFilePluginTests` `testSyntheticDocumentsReadInPlaceAndRejectInvalidOrOversize` (40441255307#1). It is left unchanged and stays open. ### Verification This branch was pushed without local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. **Hosted CI is the check.** It was pushed with `git push --no-verify`, because the pre-push hook runs the Flutter tests. - Every file is byte-identical to its final version on `flakey-tests` (67185df6a). - Checked for this PR: `git diff --check`, and a conflict-free merge with all six siblings onto `main`. - #8613's flake-check lane will repeat the two changed Dart test files. ### Readiness Test-only change. AGENTS.md's readiness steps (agent review, then human confirmation) are still outstanding, so `buzz-review-completed` is not set. ### Merge order Independent of the other six. It merges cleanly onto `main`, and with every sibling, in any order. ### Sibling PRs Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease + lane), #8677 (relay E2E helper + Windows shell), #8679 (Tauri apt cache + Windows Rust cap), #8680 (desktop product races), #8681 (desktop E2E state gates), #8682 (desktop E2E composer + row actions). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Jitter <2b2e6415e748c35180847a37aadae06a11e30dddc76b784e7fd9354c4eb42e7a@buzz.block.builderlab.xyz>
…nity-storage-calculation * origin/main: fix(mobile): resolve iOS pushes on a fresh connection and retry a stale one (#8722) ci(loadgen): publish buzz-loadgen as ghcr.io/block/buzz-loadgen (#8717) test(relay): fix main's build after the test-state helper rename (#8718) fix(db): keep lease releases durable and deflake PostgreSQL lane tests (#8676) test: dedupe collect_until_eose by event id and uncap two shell tests (#8677) ci: cache the Tauri apt .debs and fit a cold Windows Rust run in its cap (#8679) test(desktop): gate flaky E2E specs on the state they assume (#8681) test(desktop): settle composer edits and press hover-only row actions by keyboard (#8682) Signed-off-by: Shu Hui <shu@squareup.com>
## Summary Fixes the four desktop product races behind recurring E2E flakes in the 21–28 Oct 2026 CI audit. Each fix has a regression test: - In the channel browser, Enter acted on a stale deferred list. - A stale autocomplete swallowed the Enter that submits an empty edit. - Same-second messages swapped order on screen. - The unread pill read "Jump to latest" instead of "N new messages". Two E2E specs that failed for the same ordering reason also get test-side fixes. This is one of seven PRs split out of the `flakey-tests` branch (siblings below). ### Related issue None found for these. Draft #8559 does not fix the voice-note flake. Draft #8661 (Refs #6733) closes the `#channel` list synchronously when an edit is cleared. §11 below also covers that case, plus the `@mention` and `:emoji` lists, so the two PRs overlap in `useChannelLinks.ts` and `empty-edit-delete.spec.ts`. ### Flaky tests fixed | Test | Failing runs | Cause | Change | |---|---|---|---| | §5 `channel-browser.spec.ts:402` "Enter with no matches jumps to create" | 15 of 50 main runs, e.g. 40597484657, 40582387515, 40566841699, 40367495044 | **Product.** The list renders from `useDeferredValue(query)`. An Enter that landed before the deferred render read the old list, so it opened a stale match or skipped the create row. | While the deferred query lags, Enter filters the live query with the same `selectBrowserChannels` the list renders from. | | §11 `empty-edit-delete.spec.ts:88`, `:111`, `:132` | :88 in 3 jobs, :111 in 3, :132 in 2. :111 failed all 3 attempts on main 39822883395 and 39799090750 | **Product.** Clearing the draft went through the 120 ms query debounce. The `#channel` list stayed open, so Enter re-inserted the suggestion instead of submitting the empty edit. | `useChannelLinks`, `useEmojiAutocomplete` and `useMentions` (typed `@` only) close as soon as the trigger is gone. Typing a query still debounces. `ChannelAutocomplete` gains `data-testid="channel-autocomplete"`. | | §10 `video-attachment.spec.ts:1469` "right-click menus expose distinct selectors…" | 10 of 50 main runs, e.g. 40528171327 | **Product.** The live cache merge (`sortMessages`, ascending id within a second) and the channel-window projection (descending id) disagreed, so two same-second rows swapped on screen as each writer landed. | A new `shared/lib/eventOrder.ts` holds `compareRelayOrder` and its exact reverse, `compareChronologicalOrder`. `sortMessages` and the window and reconciliation sorts all use the window's order, so older-page paging is unchanged. `sortEvents` keeps ascending id. The spec gives its three messages distinct seconds. | | §16 `voice-note.spec.ts:906` "starting a duplicate voice-note player pauses the other instance" | 6 of 50 main runs. All 3 attempts on PRs 39622488737 and 40309414135 | Same-second messages plus index-based card lookup. | Distinct seconds, and each card is found through its own message row. | | §13 `stream.spec.ts:477` "keeps scroll position when new messages arrive above the fold" (integration) | 7 of 50 main runs. All 3 attempts on PR 39546556833 | **Product.** Frozen rows came back reordered (7 of 12 instrumented runs). The selector ended older history at the snapshot's first id, so moved rows were both prepended and kept in the frozen tail. The duplicates filled the row count, `pendingCount` fell to 0, and the pill read "Jump to latest". Deleting a frozen row also dropped the whole freeze. | `useBufferedTimelineMessages` keys rows by `renderKey`, which is stable across an optimistic send and its ack (`frozenRowKeys`). It drops only deleted frozen rows, and ends older history at the first frozen row of any key. `stream.spec` is unchanged. | New regression tests: - `channel-browser.spec`: two forced Enter variants. - `empty-edit-delete.spec`: a `#`/`@`/`:` autocomplete table. - `autocompleteTriggerClose.jsdom-test`. - `projectChannelWindow.test`: the cache merge and the window agree in both arrival orders. - `channel-dense-second-reach.spec`: a RAF sampler sees one order. - `useBufferedTimelineMessages.test`: deletion, reorder and ack rows. - `virtualization.spec`: deleting a row while reading keeps the next arrival counted. - A `useThreadReplies.test` row: the thread fetch and a live append agree on same-second order. Local before/after counts are from the original `flakey-tests` commits (238c08c7c, c66f411f3): | Test | Before | After | |---|---|---| | channel-browser | 0/20 | 20/20 | | autocomplete table | 0/30 | 30/30 | | jsdom | 0/2 | 2/2 | | dense-second sampler | 0/10 | 10/10 | | buffered-timeline unit | 4/7 | 7/7 | | virtualization deletion | 0/10 | 10/10 | | `stream:477` against an isolated relay | 25/30, all failures "Jump to latest" | 30/30, then 20/20 and 10/10 | ### Verification This branch was pushed without local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. **Hosted CI is the check.** It was pushed with `git push --no-verify`, because the pre-push hook runs typecheck and the unit suites. #8613's flake-check lane will also repeat the changed specs and unit tests. - Every file except `empty-edit-delete.spec.ts` and `virtualization.spec.ts` is byte-identical to its final version on `flakey-tests` (67185df6a). Those two carry only this PR's hunks; see "Merge order". - Checked for this PR: `git diff --check`, biome on both partial specs (no changes, the same as on `main`), and a conflict-free merge with all six siblings onto `main`. ### Readiness These are **product fixes in the desktop app**. Under AGENTS.md "Before opening a PR", this needs an agent review, an agent run of the affected flows, and a **human in-app check** before `buzz-review-completed` is added. Suggested check, in `just dev` or a local build against any community: 1. **Channel browser:** open "Browse channels", type a name that matches nothing and press Enter straight away. The create-channel flow opens with that name. Then type an existing channel's exact name and press Enter straight away. That channel opens, not an earlier match. 2. **Empty-edit delete:** edit one of your own messages and type `#gen` so the channel list opens. Select all, delete, and press Enter at once. The empty edit is submitted (the delete-by-clearing flow) and no suggestion is re-inserted. Repeat with `@` (mention list) and `:sm` (emoji list). 3. **Same-second order:** send two short messages in quick succession, within one second. They appear in send order and never swap while they land. 4. **Unread pill:** in a busy channel, scroll up past the fold, and have a second client post one message. The pill reads "1 new message". While still scrolled up, delete one of your visible messages, then have the second client post again. The pill still counts the new message. ### Merge order No required order, but two files are shared: - `desktop/tests/e2e/empty-edit-delete.spec.ts` is also edited by **flakey-tests-desktop-e2e-actions**. This PR adds the autocomplete table after line 158. The sibling changes imports, helpers, `beforeEach` and line 143. - `desktop/tests/e2e/virtualization.spec.ts` is also edited by **flakey-tests-desktop-e2e-gates**. This PR adds one test after line 929. The sibling's hunks are all above line 551. The hunks don't touch, so each pair merges cleanly in either order, as checked with `git merge-tree`. Draft #8661 overlaps this PR in `useChannelLinks.ts` and `empty-edit-delete.spec.ts`; whichever lands second will need a rebase. ### Sibling PRs Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease + lane), #8677 (relay E2E helper + Windows shell), #8678 (mobile), #8679 (Tauri apt cache + Windows Rust cap), #8681 (desktop E2E state gates), #8682 (desktop E2E composer + row actions). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Jitter <2b2e6415e748c35180847a37aadae06a11e30dddc76b784e7fd9354c4eb42e7a@buzz.block.builderlab.xyz>
…vent-guards * origin/main: (29 commits) Restyle community invitations (#8728) feat(relay): add incremental artifact reads (#8523) fix(mobile): show agent DM typing and balance reply bubble padding (#8731) perf(relay): route author- and #p-scoped global subscriptions by selector (#8666) fix(desktop): fix the product races behind four E2E flakes (#8680) test(scripts): stop file-size fixtures leaking git repos and fsmonitor daemons (#8727) test(mobile): keep late callbacks and fixed sleeps from failing tests (#8678) fix(mobile): resolve iOS pushes on a fresh connection and retry a stale one (#8722) ci(loadgen): publish buzz-loadgen as ghcr.io/block/buzz-loadgen (#8717) test(relay): fix main's build after the test-state helper rename (#8718) fix(db): keep lease releases durable and deflake PostgreSQL lane tests (#8676) test: dedupe collect_until_eose by event id and uncap two shell tests (#8677) ci: cache the Tauri apt .debs and fit a cold Windows Rust run in its cap (#8679) test(desktop): gate flaky E2E specs on the state they assume (#8681) test(desktop): settle composer edits and press hover-only row actions by keyboard (#8682) test(desktop): measure inline video controls against their player (#8683) relay: open the search pool lazily so a dead reader can't block boot (#8662) fix(mobile): address #8650 second-pass review (#8691) Revert "fix(mobile): push every message in your own Solo chats (#8615)" (#8690) fix: don't notify workflow owners through their attribution tag (#8694) ... Signed-off-by: Ravneet Arora <rarora@squareup.com>
Summary
When the user-deletion worker dropped a lease heartbeat, its pooled connection could be left inside an open transaction. The worker's final lease
releasethen reported success but never committed, and a rerun saw the request as "leased by another worker" for up to 60 s. The heartbeat is now one autocommitUPDATE.The PR also fixes six PostgreSQL-lane test flakes from the 21–28 Oct 2026 CI audit. It moves one relay test off the 100 ms-acquire test helper before that test starts flaking the same way.
This is one of seven PRs split out of the
flakey-testsbranch (siblings below). It touches onlybuzz-db,buzz-deletionandbuzz-relay. Every relay file exceptconnection.rsandboot_lifecycle.rschanges only for the helper rename in the last row.Related issue
None found. Nothing on
main(eda26b9) or in an open PR covers these. G, a single hermitpgschema504 that needs a workflow change, is out of scope.Flaky tests fixed
buzz-deletionuser::worker::postgres_tests::dummy_executor_failure_retries_same_community_stage(:47) andnext_step_advances_one_handler_at_a_time_through_completion(:131)run_stage_with_heartbeatdrops the renewal future when the stage handler winsselect!. The heartbeat opened a transaction. sqlx 0.9 bumpstransaction_depthonly afterBEGINreturns, so abegin()dropped mid-BEGINqueues noROLLBACK, and the connection goes back to the pool inside an open transaction. The worker's autocommitreleasethen ran inside it and never committed.EntityDeletionStore::heartbeatis one autocommitUPDATE, guarded bylock_owned's ownership/status/expiry predicate. A superseded lease is never renewed. A new regression test,dropped_heartbeat_leaves_no_open_transaction_for_the_release, drops the real heartbeat mid-BEGINthrough a pausing proxy.buzz-dbsession_statement_timeouts_bound_writer_and_reader_sessions(runtime/tests.rs:3138,PoolTimedOut)READER_ACQUIRE_TIMEOUT.Db::newnever runs the boot ping.read_pool_boot_ping_for_tests+await_idle_floor).buzz-relaystartup_owns_the_dependency_sampler(boot_lifecycle.rs)checks_totalcounters.buzz-dbsample_writer_waits_for_shared_floor_lock_before_sampling_time(replica_fence.rs:1086) andreplica_floor_probe_waits_for_shared_writer_and_records_after_release(runtime/tests.rs:3452)pg_stat_activitybut ran outside the serializedpostgres-cluster-globalgroup. Connection churn from other tests masked activity (MaskedActivity { masked: 1 }).cluster_global_prefix.nextest show-config test-groups --profile postgres-cilists both in the group.buzz-dbstalled_reader_sessions_release_their_slots_for_new_connections(reader_pool_postgres_tests.rs:354)PoolTimedOut, for up to 5 s (#8472's precedent). The stall under test keeps the production budget.buzz-dbreadiness_check_cancellation_balances_waiter_and_inflight_connection(runtime/tests.rs:391,Elapsed)pg_sleep(5)after a fixed 25 ms. Recovery lands at server start + 5 s, but the bound is abort + 5 s, a ~22 ms margin locally.pg_stat_activityuntil the query is active, then abort. Usepg_sleep(1)within the unchanged 5 s bound.buzz-relayupgrade_would_deny_line_is_written_at_auth_while_open(connection.rs)test_state_with_database_urlagainst the real test database, the same shape as thePoolTimedOutflakes above.test_state_with_database_url_and_pool_optionswith sqlx's default timeout. The 100 ms helper is renamedtest_state_with_unreachable_database_url; all its remaining callers use closed ports.Local before/after counts are from the original
flakey-testscommit (1f0451915), on macOS with OrbStack Postgres:MaskedActivityunderpgbench -C: 2/50 iterations FAIL before. After, the lane passed 3× (613/613) but was not re-run under pgbench.Executecancellation: 10/10 FAIL before, 10/10 PASS after.Verification
This branch was pushed without local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. Hosted CI is the check. It was pushed with
git push --no-verify, because the pre-push hook runs clippy and unit tests.flakey-tests(67185df6a), where the counts above were measured.git diff --check,rustfmt --checkon the changed files, and a conflict-free merge with all six siblings ontomain.Readiness
This PR has a product fix (the
buzz-dblease heartbeat). Under AGENTS.md "Before opening a PR", it needs an agent review and a human check against a local relay beforebuzz-review-completedis added. Suggested check:buzz-admin user-deletions submit/approve.buzz-admin user-deletions rununtil it completes, then run it again at once.buzz-admin user-deletions inspect <id>shows no leftover lease. The original race is rare and is not expected to reproduce by hand; this check confirms releases still commit.Merge order
Independent of the other six. It merges cleanly onto
main, and with every sibling, in any order.Sibling PRs
Split from
flakey-tests(one PR per area): #8677 (relay E2E helper + Windows shell), #8678 (mobile), #8679 (Tauri apt cache + Windows Rust cap), #8680 (desktop product races), #8681 (desktop E2E state gates), #8682 (desktop E2E composer + row actions).🤖 Generated with Claude Code