[TheSentinel454]

What

Removes every production call to the SQL fence predicate community_write_allowed() from application SQL, so no application mutation depends on the boolean predicate anymore. This is one prerequisite for BUZZ-315 part B; #8322 (tolerating absent fence objects) is a separate one. The triggers and SQL functions stay in this PR because mixed-version pods still depend on them.

The predicate isn't inlined. It's a VOLATILE plpgsql function that takes the deletion lock and then reads lifecycle state on a fresh snapshot, and an inline expression would read state from the statement's start snapshot. A separate earlier check would race deletion. So each sweep now:

1. reads its candidate community IDs with a plain read, bounded the same way the original statement was,
2. opens one AdmittedTx per community (begin_community_sweep_transaction, a thin wrapper over begin_community_write_transaction that turns an admission refusal into "skip this tenant"; the four periodic sweeps go through sweep_admitted_communities, which also owns their error handling),
3. runs the original mutation scoped to community_id = $1 and commits.

Site	Change
runtime/push_enqueue.rs	Deleted the redundant predicate. This transaction already calls guard_transaction.
store/relay_invite.rs reap_expired_relay_invites	Keeps the global expiry order and the 1,000-row ceiling. Candidates are pre-filtered to active tenants so fenced invites can't fill the batch. Admission is still what authorizes the write.
store/workflow.rs prune_scheduled_workflow_fires_before	Prunes per admitted community and sums the row counts.
store/channel.rs reap_expired_ephemeral_channels	Archives per admitted community and returns the committed rows, so the relay still runs their system and discovery follow-ups. The candidate read has no lifecycle pre-filter (it has no LIMIT, so admission alone decides).
store/event.rs backfill_d_tags	Backfills per admitted community at startup.
store/push.rs claim_due_match_batch_with_loader	Picks the earliest due active community, admits it, and claims under the same FOR UPDATE OF q SKIP LOCKED and batch limit. It skips at most 16 tenants that fence between the read and admission.
store/push.rs reap_exhausted_matches	Reaps per admitted community.

Tenant failures. The four periodic sweeps (invite reap, workflow fire prune, channel reap, exhausted match reap) log a warning and continue when one tenant fails. That tenant's rows are left unmodified and stay eligible, so the next tick retries them. A tenant error doesn't abort processing of the other tenants already in the selected candidate page. Invite cleanup keeps its oldest-first 1,000-row selection, so if one tenant owns that entire page and its delete keeps failing, tenants outside the page wait. No persistent tenant-local failure for that delete is known in the current schema, and base aborted the whole page on any error. The returned count or rows cover the tenants that committed. Claim, enqueue and the startup backfill still return the error to the caller.

Database outages. A connection-level error (pool acquire timeout, closed pool, socket I/O; DbError::is_connection_unavailable) isn't tenant-local, so it stops the periodic sweep at that tenant instead of moving on. Without this, an outage that began after the candidate read cost one writer acquire timeout and one warning per remaining tenant, and invite cleanup runs inside the usage-metrics leader tick. Rows already committed are still returned, so reaped channels keep their follow-ups. Remaining tenants retry next tick.

Known limit (pre-existing, tracked separately):

• Claim at a locked head. claim_due_match_batch_with_loader returns None once an admitted tenant claims zero rows. If every due row in the oldest active tenant is row-locked, younger tenants aren't tried on that call. Base behaved the same way. This PR doesn't change it.

Tests

All tests use a one-session fixture (test_support::admission_fixture) with session_replication_role = replica, so the trigger backstop is off and only application admission protects the fenced tenants. The fixture shadows community_write_allowed with a function that raises, in a per-call admission_test_<uuid> schema.

• One test per site: a quiescing tenant's rows are untouched while an active tenant in the same sweep still progresses.
• Race tests for invite reap, channel reap and claim. These are the sites where the candidate read already filters out fenced tenants. Each test holds the tenant's deletion lock, waits until the sweep is blocked in admission, fences the tenant in that transaction, and asserts its rows are untouched. The claim test also asserts that a second active tenant is claimed in the same call.
• channel_reap_returns_committed_rows_after_earlier_tenant_error and periodic_reaps_continue_after_tenant_error_and_retry_retained_rows inject a per-tenant failure. They check that later tenants commit, that the failed tenant's rows are retained, and that the next call processes them.
• fleet_sweep_stops_at_first_unavailable_connection runs the sweep helper against a closed pool with three candidates and asserts only the first is attempted. fleet_sweep_returns_committed_results_when_database_drops_mid_page (PostgreSQL) commits the first tenant, closes the pool, and asserts the first tenant's result is returned and the sweep stops at the second of three candidates. connection_unavailable_covers_only_pool_and_io_failures pins the classification; the per-tenant failure tests above confirm a database-raised error still continues.

Falsification, each with the tests failing as expected:

Mutation	Result
Original SQL restored at invite sweep, push claim, d-tag backfill	0/3 pass (legacy SQL fence predicate called)
guard_community_write returns Ok(()) after taking the lock	race tests + channel reap test 0/4 pass
Per-tenant errors abort the sweep again	0/2 pass
Sweep helper continues instead of stopping on a connection-level error	fleet_sweep_stops_at_first_unavailable_connection and fleet_sweep_returns_committed_results_when_database_drops_mid_page fail
Sweep helper drops already-committed results when it stops (return Vec::new())	fleet_sweep_returns_committed_results_when_database_drops_mid_page fails
Sweep helper removed from the source-policy chokepoint check	serving_table_writes_expose_syntactic_chokepoint_or_guarded_tx_routes fails

The PostgreSQL tests are under postgres_tests modules with #[ignore = "requires Postgres"], which puts them in the .config/nextest.toml PostgreSQL filter. CI evidence is under Verification.

Verification

CI at head 5d37fbda (run 38925996275) is green. The PostgreSQL job (job 117062895683) passed 1017/1017. That includes named PASS lines for all 30 admission_postgres_tests, fleet_sweep_returns_committed_results_when_database_drops_mid_page, and both writer-pool tests listed below. Unit Tests (job 117058136482) has PASS lines for connection_unavailable_covers_only_pool_and_io_failures and fleet_sweep_stops_at_first_unavailable_connection.

Local gates ran on a Linux cloud workstation against private PostgreSQL 16 and Redis. The tree was 735170df, which is this head rebased on main c0ab789, so the gates built the merge result.

Gate	Result
cargo fmt --all -- --check	pass
cargo clippy --workspace --all-targets -- -D warnings	pass
cargo nextest run -p buzz-db --lib --tests	185 passed
scripts/test-postgres-test-discovery.sh	pass
PostgreSQL lane for buzz-db (excluding the two writer-pool tests below)	530 passed
PostgreSQL lanes for buzz-admin, buzz-audit, buzz-deletion, buzz-push-gateway, buzz-search, buzz-workflow	all pass
PostgreSQL lane for buzz-relay	413/426. The 13 group_state_postgres_tests fail with PoolTimedOut on unmodified main c0ab789 too (see below)
The three mutations in the falsification table above (continue ×2, return Vec::new())	each fails its target test

These failures happen only on the workstation, and each one passes in CI's PostgreSQL job at this head:

• writer_pool_metrics_stop_after_session_timeout_setup_failure stalls. It's an unchanged writer-pool configuration test, and it stalled the same way at earlier heads.
• The 13 relay group_state_postgres_tests fail at the same point, ensure community: Sqlx(PoolTimedOut), on unmodified main c0ab789. This PR doesn't touch buzz-relay.

What got simpler

• No application mutation calls the boolean predicate community_write_allowed() anymore. Dropping the fence objects still has other dependencies: serving-write lease acquisition calls assert_community_write_allowed() (store/deletion.rs:3068), catalog validation requires both functions (store/deletion.rs:1195-1198), and the triggers and compatibility SQL still use them. Those belong to buzz-db: tolerate absent universal write-fence objects (expand) #8322 and part B, not this PR.
• The claim's target CTE is gone, and every sweep uses the same admission-and-skip helper.
• The four periodic sweeps share one loop (sweep_admitted_communities) for admission, skip, continue-on-tenant-error and stop-on-outage, instead of four copies of it.
• The source policy treats that helper as a tenant-local chokepoint, so backfill_d_tags no longer needs a reviewed exception.
• The per-tenant loops add code, because each community now gets its own admitted transaction.

[github-actions]

🔐 Codex Security Review

Note: This is an automated, security-focused review generated by Codex.
Use it as a supplement to human review; false positives are possible.

Scope

• Exact PR diff: 2d0f78c6e1c43762aeb51efa633e9387fc301649...5d37fbda699f56231930d949643490e35c70d2c8
• Model: gpt-6.1-sol

💡 Click "edited" above to see earlier reviews for this PR.

---

Review Summary

Overall Risk: NONE

No concrete security, correctness, or reliability findings were identified in the authorized PR range.

Findings

No concrete security, correctness, or reliability findings were identified.

Notes

• Review used read-only inspection of the diff, surrounding source, callers, schema, tests, and history. Builds and tests were not run, as instructed.

---

_{Generated by Codex Security Review |
Requested by: @TheSentinel454 |
Workflow run}

[wpfleger96]

🤖 Thanks Luke. No blocking findings at 2c151217.

All seven former community_write_allowed() sites are still fenced. Six mutate through a community-scoped AdmittedTx. guard_community_write takes the shared deletion lock and then reads lifecycle in a separate statement, so under the writer pool's READ COMMITTED requirement it sees the same fresh post-lock state the SQL predicate did. Push enqueue already guards its transaction before the push lock and insert, so dropping the predicate there is safe. Every mutation is constrained to the admitted community_id, including both sides of the claim join. Counts and reaped channel rows only include tenants that committed. Channel follow-ups are built from those committed rows, so a failed tenant can't create follow-ups or suppress follow-ups for tenants that did commit.

The race tests hold the real exclusive community_deletion_lock_key, wait until the production sweep is blocked on it, fence the tenant in that same transaction, then commit. They would fail if admission were removed, and also if the lock were kept but the lifecycle re-read skipped. CI at this head (run 38589716836, PostgreSQL job 982/982) has named PASS lines for all 11 new admission tests, the 18 tests on the shared fixture, and enqueue_admits_active_tenant_and_rejects_quiescing_tenant. serving_table_writes_expose_syntactic_chokepoint_or_guarded_tx_routes passes in Rust Unit Tests.

Optional:

1. Invite cleanup can still be blocked by one tenant. reap_expired_relay_invites selects the globally oldest 1,000 expired invites before grouping by community (relay_invite.rs:186-207). If one tenant owns that whole page and its delete keeps failing, later tenants never enter the loop. I couldn't find a persistent tenant-local failure for this delete in the current schema, and base aborted the whole page on any failure, so this is still an improvement. The description's "one failing tenant can't block the tenants after it" is broader than what the code does, though. Suggested wording: "A tenant error does not abort processing of other tenants already in the selected candidate page; invite cleanup keeps its oldest-first 1,000-row selection." A cursor or per-tenant cap, plus a full-page regression test, would close it fully.
2. Systemic errors are also treated as tenant errors. The four periodic sweeps continue on every error, including pool acquire timeouts and connection loss. During a database outage each remaining tenant waits out the 3s writer acquire timeout and logs a warning. Invite cleanup runs inside the usage-metrics leader tick (buzz-relay/src/main.rs ~2472), so a failover can stall that tick for minutes. Consider continuing only on tenant-local errors and returning on pool or IO errors.
3. Claim stops at a locked head (pre-existing). claim_due_match_batch_with_loader returns None once an admitted tenant claims zero rows (push.rs:929-937). If every due row in the oldest active tenant is row-locked, younger tenants with due rows aren't tried on that call. Base behaved the same way, but the new loop could skip empty tenants within its 16-candidate bound.
4. Description. The Verification section cites older heads (b3313167, e0229c8a, 979/979). The current-head PostgreSQL job is 982/982 and also passes both writer-pool tests listed as stalled. The "admission tests via plain cargo test" row sits after a paragraph, so it doesn't render as part of the table. buzz-review-completed isn't in the description yet.
5. Scope of "unblocks dropping the fence functions and triggers." No application mutation calls community_write_allowed() anymore. However, assert_community_write_allowed() is still called when a serving-write lease is acquired (store/deletion.rs:3068). Catalog validation still requires both functions (store/deletion.rs:1195-1198), and the trigger and compatibility SQL paths also still use them. Consider narrowing that line to the boolean predicate, so the later drop change accounts for these remaining dependencies.