[TheSentinel454]

Summary

Moves the producer side of the two event follow-up triggers into application code. The triggers stay in place, so both producers run until a later change drops the triggers after this ships to every pod.

Builds on #8130 (AdmittedTx). Refs BUZZ-176.

Push match enqueue

New store::event_follow_up::enqueue_push_match replaces the events_enqueue_push_match trigger (schema/functions/enqueue_push_match_job.sql, migration 0023) and matches it exactly:

• Same kind allowlist (9, 40002, 45001, 45003), now PUSH_MATCH_KINDS.
• Takes the same SHARED buzz_push_gate:<community> advisory lock as its own statement, so under READ COMMITTED the lease check gets a snapshot taken after the lock is granted. Lease activations still take it EXCLUSIVE in push.rs.
• Same lease predicate (active AND endpoint_enabled AND expires_at > now), then INSERT push_match_queue ... ON CONFLICT DO NOTHING.
• Fails closed: any error rejects the event, as the trigger did.

It runs right after the INSERT, only when the insert affected a row. The consumer side (claims, lease_until, attempts, reaper) is already app-owned and is unchanged.

Channel TTL refresh

AdmittedTx now records every channel that received an event (except kind 9007 and channel-less events). AdmittedTx::commit() refreshes their ttl_deadline as the last statement before COMMIT, which keeps the commit-time behavior of the deferred events_refresh_channel_ttl trigger (migration 0024):

• One savepoint per channel, in sorted order, under the same SHARED buzz_channel_ttl:<community>:<channel> lock.
• Runs the same UPDATE with the same archived_at/deleted_at filters as its own statement after the lock, so its snapshot follows the lock grant. Its ttl_seconds IS NOT NULL qual filters a permanent channel before any tuple lock, so a permanent channel's row is never locked. An ephemeral channel's row lock is held through the remaining refreshes and the COMMIT round trip (the trigger held it only inside COMMIT).
• Lock keys come from one builder per protocol (channel::channel_ttl_lock_key, push::push_gate_lock_key), shared by the SHARED and EXCLUSIVE sides. The shared push-gate wait is observed through observe_advisory_lock(LockType::PushGate, …).
• On failure it rolls back the savepoint, logs a warning, and increments buzz_db_channel_ttl_refresh_failures_total{reason}, where reason is lock_timeout or error. The event still commits, matching the trigger's EXCEPTION WHEN OTHERS. A cancelled statement (57014) still rejects the transaction, because plpgsql never caught that either.
• While both producers run, an app refresh that hits lock_timeout is counted, and then the trigger retries it inside COMMIT and may succeed. So during the dual window the failure counter can over-report and the worst-case wait doubles. Both go away when the retirement PR drops the triggers, after which the counter is exact.

Call sites

Every production INSERT INTO events calls event_follow_up::after_admitted_insert:

• insert_event_on
• insert_event_with_thread_metadata_tx
• LockedMemberSnapshot::replace_member_event
• replace_addressable_event
• publish_nip43_membership_locked
• accept_lease_event (kind 30350, a no-op today, kept so the policy has no exceptions)

replace_parameterized_event_in_transaction_impl inserts inside the parameterized_replace savepoint. It enqueues the push job inside the savepoint, so a rollback to that savepoint also drops the job, and records the TTL channel only after the savepoint is released.

The two rolled-back verification probes (verify_floor_guard_behavior, verify_channel_roster_fence_behavior) are named exceptions, because they never commit.

Source policy

tests/observability_source.rs adds every_production_event_insert_runs_the_follow_up_hook. It scans every crates/*/src and fails on any function that inserts into events without the hook. Matching is normalized (letter case, whitespace, \ continuations, \n/\t/\r escapes, optional public.). The file is masked once first: comments (//, nested /* */) and string, raw-string (including br"/cr"), and char literals become spaces. #[cfg(test)] markers, fn signatures and the hook are all found in that masked view, so a marker in a comment (even a multiline block comment) strips nothing, an fn line inside a multiline string doesn't split a function, and a commented-out or quoted hook doesn't count. SQL is still matched in the original text. A table test pins the masker's output for each literal form. A fixture test pins what the scan rejects (no hook, push without TTL, a hook in a line comment, block comment, or string, lowercase or line-broken or escaped SQL, writers after test-only items or under comment mentions) and what it ignores (writers in #[cfg(test)] mod). A floor of seven hooked writers catches a scan that silently stops seeing writers.

Tests

src/store/event_follow_up_postgres_tests.rs is the contract suite from the BUZZ-176 discovery work, ported to the AdmittedTx API. It is a crate-internal #[cfg(test)] module because it runs SET LOCAL timeouts and in-transaction reads through the crate-private AdmittedTx::conn(). It runs on both the desired and migration schemas, in two arms:

• dual: production during the migration window, with the triggers and the app code both running.
• app-only: drops both triggers in the disposable database (and asserts no partition clones remain), the way the retirement migration will.

The trigger-only baseline is the base branch, where the same suite passed before this change. The suite also pins the two branches that differ from "swallow everything": a lock_timeout failure increments buzz_db_channel_ttl_refresh_failures_total{reason="lock_timeout"} exactly once, and a statement cancel (57014) during the refresh rejects the event with no row, job, or deadline change.

What got simpler

Nothing yet. This adds a parallel producer during the migration window. It is what lets the follow-up PR delete two plpgsql functions, two triggers, and the "keep in sync with migrations/00xx" duplication between schema/functions/ and push.rs.

Verification

Rebased onto main @ e9269cbdf. The head is now 184cbb2e8. All runs used a Linux cloud workstation with the pinned toolchain (rustc/clippy 1.95.0, hermit), and the tree was clean before and after each run.

Production source changed since 1eb11bde. The rebase took in #8178. That PR removed backfill_push_match_jobs and both of its call sites from push.rs, and this PR's event_follow_up.rs docs were reworded to match (51664d60c). So the evidence pinned to 1eb11bde and c5a06a82 below is history. It does not describe today's runtime tree byte for byte. 184cbb2e8 changes only test code (the char-literal masker in crates/buzz-db/tests/observability_source.rs).

Evidence at the current head 184cbb2e8:

• Delta review: clean, with no P0s.
• Source policy: cargo test --locked -p buzz-db --test observability_source passed 17/17 on macOS, run by the reviewer.
• Codex Security Review: re-ran over e9269cbdf…184cbb2e8 and reported risk NONE.
• Live E2E: an exact-head run is in progress.

Descendant evidence: #8210 at 6be64d87a is this head plus migration 0058, which drops the retired triggers. On the cloud workstation it passed fmt, clippy, policy, lib, buzz-db PG (486) and buzz-relay PG (378).

Check	SHA	Result
cargo fmt --check, cargo clippy -p buzz-db --all-targets -D warnings	fbc9a598	pass
cargo clippy (workspace)	1eb11bde	pass
source policy (--test observability_source), fmt, buzz-db clippy	c5a06a82	17/17, pass, pass
cargo test --locked -p buzz-db (non-PG)	c5a06a82	148 lib + 17 policy + 5 doctests
source policy (--test observability_source)	fbc9a598	16/16
cargo test --locked -p buzz-db (non-PG)	fbc9a598	148 lib + 16 policy + 5 doctests
contract suite (store::event_follow_up_postgres_tests, desired/migration × dual/app-only)	1eb11bde	4/4
buzz-db PG full	1eb11bde	464/464
buzz-relay PG	1eb11bde	378/378
GitHub CI	fbc9a598	53 pass, 0 fail (Codex security review pending)

Falsifications (each mutation made the named test fail, then was reverted):

• dropping the relay_members hook → policy names publish_nip43_membership_locked
• no-op push producer → app-only arms fail, dual arms pass (trigger covers)
• skipping the TTL refresh → app-only fails on the deadline; dual fails on the failure-metric assertion
• swallowing 57014 → all four arms fail
• dropping the metric increment → all four arms fail
• unanchored #[cfg(test)] match, or no escape handling → fixture test fails
• a #[cfg(test)] constructor that skips admission, added inside impl AdmittedTx (indented or as a column-0 mod) → admitted_tx_is_constructed_only_by_admitting_constructors fails; putting the strip back into the constructor scan → its fixture test fails
• at 7c0baa41: accept_lease_event's hook commented out and the real call made through an alias → policy names accept_lease_event (the same mutation passes under the fbc9a598 policy); hook matching on unmasked text → the comment/string fixtures fail
• at 9199992c/d405b2f6: no nested block-comment depth, no string-escape skip, no br"/cr" raw-literal prefix, or no raw-string branch at all in the masker → the follow-up fixture test fails
• at c5a06a82: finding #[cfg(test)] in the unmasked text → writer_after_block_commented_cfg_test is no longer reported; finding fn signatures in the unmasked text → writer_after_raw_string_example is no longer reported; no cr raw prefix → the masker table test fails on cr"
• unhooked writers appended to side_effects.rs (after test-only items, under a doc-comment mention, with an escaped newline) → policy names each

Live relay E2E: an independent reviewer ran this at 1eb11bde against a real relay and CLI, with PostgreSQL 17 and Redis 7, the follow-up triggers absent, and no partition clones.

• A kind 9 on a TTL channel advanced the shortened deadline by about 580 s. On a permanent channel, the deadline stayed NULL.
• Each accepted event produced exactly one pending push_match_queue row with the same event id.
• An exact signed replay was a no-op.
• While the exclusive TTL lock was held, the event and its job stayed invisible and the deadline didn't change. After release, both persisted once.
• Roster create/join/add/remove kept SQL and wire state in step (1→2→3→2), with one current 39002 head.
• NIP-33 replace, duplicate, stale, failure and retry cases all passed.

Limits: the push lease was a SQL fixture, and push consumers were disabled. Kind 30023 is global, so channel-scoped TTL on the parameterized path is covered only by the PG contracts. No live relay rerun was done at fbc9a598, because that delta is test-only.

Latency (measured at pre-rebase f8380927, same production delta; debug build on loopback, 1,000 samples per cell, EVENT→OK): head app-only p50/p99 3.80/4.58 ms (permanent) and 3.87/4.64 ms (TTL), against base trigger-only 3.49–3.53/4.00–4.03 ms and 3.56–3.62/4.30–4.46 ms. About +0.3 ms p50, from the extra client round trips (lock, UPDATE, savepoints, push lock/insert) that the triggers ran server-side. One debug loopback run, not a production p99.

Not measured: contention from many writers on one hot TTL channel, where each writer now waits on the channel row lock for about two more client round trips (RELEASE SAVEPOINT and COMMIT) than with the trigger. In review, a burst of 8 concurrent posts to one TTL channel committed with no deadlock or lock timeout. Watch send p99 and pg_locks waits on channels during rollout.

Known pre-existing, not touched here: api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo is flaky when Redis is present. It returns a 504 in 4 of 5 runs at base 745501962 and 2 of 5 at fbc9a598, always at the same assertion. No CI lane selects it.

[wpfleger96]

🤖 Thanks, Luke. Nothing here blocks.

I compared both producers with the trigger functions. The push enqueue uses the same kind allowlist, the same shared buzz_push_gate key (the Rust Uuid display matches ::text), the lock as its own statement before the lease check, the same lease predicate, and it fails closed. The TTL refresh uses the same shared lock key, the same ttl_seconds/archived_at/deleted_at filters, and skips 9007. The per-channel savepoint behaves like the plpgsql EXCEPTION block: rolling it back releases the advisory and row locks, and 57014 still rejects the event. Every non-test INSERT INTO events under crates/*/src is one of the seven hooked writers or one of the two rolled-back probes. Each ON CONFLICT DO NOTHING path runs the hook only after a real insert. The parameterized path enqueues inside parameterized_replace and records the channel only after release, which also covers replacements. commit() is the only way an AdmittedTx commits, and conn() is crate-private, so nothing outside the crate can wrap an insert in a savepoint that would leave a recorded channel behind. The deferred floor guard takes no locks, refreshes run in sorted order, and FK KEY SHARE doesn't conflict with the refresh's NO KEY UPDATE, so I don't see a new lock cycle.

All four contract arms pass by name in PostgreSQL Domain / PostgreSQL Tests, and the 16 source-policy tests run in Rust / Unit Tests. Against a local relay at this head, in both app-only and dual mode, a kind 9 advanced a TTL channel's deadline and left a permanent channel's NULL. Each message with an active lease got exactly one queue row. Missing, expired, and endpoint-disabled leases got none, and an exact replay changed nothing. Eight concurrent posts to one TTL channel all committed with no deadlock or lock timeout. Making enqueue_push_match a no-op fails both app-only arms and leaves both dual arms green, as expected. The full buzz-db Postgres lane is 464/464.

Optional

• observability_source.rs L1545–1549: runs_event_follow_ups matches raw text, and function_slices keeps comments and strings, so a writer whose hook is commented out (// crate::store::event_follow_up::after_admitted_insert() still passes and still counts toward the floor of seven. The seven writers you have today are covered anyway, because the app-only arms exercise every writer that carries a channel or a push kind. The gap is a future writer. Matching the hook only outside // comments and string literals (the lexer in cfg_test_item_len already skips both) plus fixtures for a commented-out hook and a hook name inside a string would close it.
• event_follow_up.rs L123–147: an ephemeral channel's row lock is now held through RELEASE SAVEPOINT and the COMMIT round trip, so writers to a busy TTL channel queue behind about two more client round trips each than with the trigger. The latency numbers in the description are single-client, so they don't measure that. The burst above was fine, but it's worth watching once this is on real pods.
• While both producers run, a refresh that hits lock_timeout is retried by the trigger inside COMMIT, so the worst case waits twice and the failure counter can count a refresh the trigger then completes. That goes away with the retirement PR, which might say so.
• The description has no buzz-review-completed line.

[TheSentinel454]

Thanks Will. (1) Fixed at 7c0baa41: the hook match now runs on comment- and literal-masked source, with fixtures for a commented-out hook, a block-commented hook, and a hook name in a string; cfg_test_item_len now shares that masker. (2) Agreed — noted in the description as a rollout watch item (send p99 / channels lock waits on hot TTL channels). (3) Noted in the description; the retirement PR will call out that the counter becomes exact once the triggers drop.

[github-actions]

🔐 Codex Security Review

Note: This is an automated, security-focused review generated by Codex.
Use it as a supplement to human review; false positives are possible.

Scope

• Exact PR diff: e9269cbdf66b0e2fdb588aa20aad65bcba3ca622...184cbb2e810af514ba84ec8dd811ae7db0aaf46c
• Model: gpt-5.6-sol

💡 Click "edited" above to see earlier reviews for this PR.

---

Review Summary

Overall Risk: NONE

No concrete security, correctness, or reliability defects were found in the authorized PR range.

Findings

No concrete security, correctness, or reliability findings were identified.

Notes

• Review used read-only source and history inspection; repository tests and builds were not executed as instructed.

---

_{Generated by Codex Security Review |
Requested by: @TheSentinel454 |
Workflow run}

[wpfleger96]

🤖 Thanks, Luke. The comment/string masking fixes the false pass I raised last time, and nothing here blocks.

runs_event_follow_ups now matches on masked text, so a hook in a // comment, a nested /* */, a string, a string with escaped quotes, or a raw string no longer counts. inserts_events_row still reads the original text, which it has to since the SQL is inside literals. If the masker gets something wrong, it hides a real hook rather than inventing one, so the test fails loudly instead of passing. All seven writers still have real hooks at this head (two in event.rs, two in replaceable.rs, one each in channel_members.rs, push.rs, and relay_members.rs), and the fixture test and the production scan both pass by name in Rust / Unit Tests.

Optional

• observability_source.rs L1427–1440 and L953–979: strip_cfg_test_items and function_slices still find attributes and signatures in the unmasked text, and the masker only runs on fragments afterward, so it can't see a comment or string that opened before the fragment. Two ways that slips, both older than this change and neither hitting today's writers:

  • a #[cfg(test)] at the start of a line inside a multiline /* */ gets treated as a real attribute. The item scan starts after the marker, never sees the opening /*, and strips the next production function through its closing brace, so a new writer can drop out while the floor of seven still holds.
  • a multiline raw string with a line starting fn example() { splits the function. If the quoted example contains the hook and the real insert comes after the literal, the second slice starts inside the string, sees the quoted hook as code, and passes.

  Masking the whole file once and finding attributes and signatures in that view (offsets match the original, so SQL checks can keep reading the original) would close both. Fixtures for both cases would pin it.

• L1710–1763: the fixtures cover less than the description says. Nothing pins cr", plain b"/c" strings, raw strings with zero or several #, or escaped char literals, so dropping cr handling wouldn't fail any test. A small table of inputs and their masked output would cover it, or the description could claim less.

[TheSentinel454]

Thanks Will, both done at c5a06a82. #[cfg(test)] markers and fn signatures are now found in a once-masked view of the file, with fixtures for a marker inside a multiline block comment and an fn line inside a multiline raw string. A table test now pins the masker's output for r"/r#"/r##", br"/cr", b"/c", and escaped char literals.

[wpfleger96]

🤖 Thanks, Luke. Both optional items from last round are in, and nothing here blocks.

strip_cfg_test_items and function_slices now find markers and signatures in one masked view of the whole file. A #[cfg(test)] inside a multiline /* */ strips nothing, and an fn line inside a raw string no longer splits a function. The new fixtures pin both, and masker_blanks_every_literal_form_and_keeps_code covers the literal forms. Locally, four edits to a real enqueue_push_match call each fail every_production_event_insert_runs_the_follow_up_hook: commenting it out, moving it into a string, moving it into a raw string, and deleting it. Each revert brings all 17 source tests back to green. All 17 also pass by name in Rust / Unit Tests.

Optional

• observability_source.rs L1560–L1571: with the whole file masked, a misread character literal can now hide a real signature. That changes what I said last round, that a masker mistake can only make the test fail. Char literals are recognized only as one byte or a two-byte escape, so 'é' and '\u{1f}' aren't consumed as tokens. In let _ = ['é','"'];, the closing quote, the comma, and the next quote read as a new char literal, the " opens a string, and the mask runs through the next function's signature. A hooked writer followed by an unhooked one then becomes one slice with no violation; the scanner at fbc9a598 flags it. The same misalignment can hide a real #[cfg(test)]. rustfmt writes that as 'é', '"', which masks correctly, so it only bites in code rustfmt doesn't format, and nothing today hits it. Consuming whole char tokens (UTF-8 characters and \x/\u{…} escapes, with lifetimes still falling through), plus a fixture for that input, would close it.
• The description has no buzz-review-completed line.

[wpfleger96]

🤖 Thanks, Luke. The char-literal fix from last round is in, the rebase onto #8178 is clean, and nothing here blocks.

char_literal_end now consumes whole char tokens: UTF-8 scalars, simple escapes, \x.., and \u{…}. Lifetimes and labels still fall through, and it can't slice off a char boundary. The three new fixtures ('é', '\u{1F600}', '\x7f') each fail if the old two-arm match is put back and pass with the new helper. All 17 source tests pass by name in Rust / Unit Tests. Locally at this head, the full PostgreSQL suite passes too, including the four follow-up contracts, all 18 push tests, and the TTL/activation-race tests.

Commits 2–11 are unchanged from the last round. Commit 1 only drops the backfill comment that #8178 removed, and the new "no wake was owed" wording in event_follow_up.rs matches main: activation still takes the push gate exclusive, and an event holds it shared to transaction end.

Optional

• observability_source.rs L1580–L1583: the 9-byte cap on escapes misses Unicode escapes with underscores, which Rust allows after each hex digit (reference). '\u{0_0_0_0_7_b}' returns None, so in a('\u{0_0_0_0_7_b}','{') {} its closing quote reads as the start of the next literal and the { stays unmasked. Inside a #[cfg(test)] item that brace can make the strip run into production code. Nothing hits it today, and the old masker had the same gap. Parsing the \u{…} grammar instead of capping bytes, plus a fixture with adjacent literals, would close it.
• The description has no buzz-review-completed line.