feat: firehose mode with in-band slot watermarks - #6
Conversation
- DlpSyncer::start_firehose delivers every delegation-record update on chain (no per-record subscriptions, no replay buffering) with lossless backpressure: sends wait for channel capacity because a silently dropped update would let a mirror consumer serve stale state forever. - New AccountUpdate::SlotAdvanced(slot) watermark, emitted in firehose mode in-band with record updates: receiving SlotAdvanced(s) proves every record update up to slot s was already delivered on the channel, which is what lets a consumer treat an unchanged mirror entry as current state. - Subscribed mode behavior is unchanged. - Pin helius-laserstream to the same rev as the magicblock-validator workspace so downstream builds compile a single laserstream copy. - Tests cover firehose delivery + watermarks and subscribed-mode filtering via a directly driven syncer (LaserStream handle now optional for tests).
|
Warning Review limit reached
Next review available in: 10 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
WalkthroughDlpSyncer now supports subscribed and firehose delivery modes. Firehose mode delivers all valid delegation updates with lossless backpressure and ChangesFirehose delivery
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f8932e36fb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/syncer.rs`:
- Around line 763-783: Extend
firehose_delivers_all_records_and_inband_watermarks to include a
transaction-derived undelegation update using a fixture containing an
undelegation instruction. Send that transaction through syncer.handle_update,
then assert AccountUpdate::Undelegated is received before the subsequent
SlotAdvanced watermark, while preserving the existing no-replay-buffer
assertion.
- Around line 400-407: Remove the Firehose SlotAdvanced emission from the raw
Laserstream Slot handling in the syncer update match. Do not treat the incoming
slot as a watermark; instead, only emit AccountUpdate::SlotAdvanced after
updates are buffered and ordered by slot and index, or route this behavior
through a source that guarantees the required ordering.
In `@src/types.rs`:
- Around line 37-40: Document the source-breaking nature of adding the public
AccountUpdate::SlotAdvanced variant alongside its declaration in src/types.rs.
Clearly state that exhaustive matches in downstream crates must add an arm for
SlotAdvanced, and communicate this API change through the project’s established
release or changelog documentation mechanism.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 92bcf749-ab4e-4c39-abf4-5762c36a8747
📒 Files selected for processing (5)
Cargo.tomlREADME.mdsrc/lib.rssrc/syncer.rssrc/types.rs
- The stream is not slot-ordered, so a bare slot notification cannot prove that updates at or before it were delivered. Firehose watermarks are now published RESUME_SAFETY_MARGIN_SLOTS behind the observed slot, keeping the SlotAdvanced contract sound under the same skew bound the resume logic already assumes. - Test the transaction-derived Undelegated firehose path and the no-op watermark inside the margin. - Bump to 0.2.0: the SlotAdvanced variant is a source-breaking addition for exhaustive matchers.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c979bb0ab4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- The skew margin is a budget, not a proven bound. A late update landing at or before a published watermark now voids all delivered state via SyncInterrupted (after delivering the update), so consumers rebuild from live updates instead of trusting a broken claim. Watermarks are also published monotonically. - Undelegate instructions resolve their account indexes against the full runtime ordering (static keys, then lookup-table loaded writable and readonly addresses), so versioned transactions that load the delegation program or record through an ALT no longer miss the Undelegated event. - Tests cover the violation path, the monotonic no-op inside the margin, and an ALT-loaded undelegation.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d1f13ee950
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Replay resumes from the last published watermark's slot, so redelivered updates at that slot are expected duplicates, not skew. The watermark is reset on every re-establishment and re-arms with the next slot notification; consumers merge watermarks monotonically, so the brief regression is invisible to them.
1 participant
Summary
Adds the delivery mode the magicblock-validator delegation-record mirror needs (first consumer landing separately): a single channel carrying every delegation-record update on chain, with the ordering and losslessness guarantees a mirror requires to treat its entries as authoritative for the live path.
DlpSyncer::start_firehose(endpoint, api_key)try_send— a silently dropped update would let a mirror consumer serve stale record state forever. A slow consumer backpressures the stream; the intended consumer is a fast in-memory map writer.start(subscribed mode) behavior is completely unchanged.AccountUpdate::SlotAdvanced(slot)Emitted in firehose mode from the slot-update arm, in-band on the same channel as
Delegated/Undelegated. That ordering is the point: receivingSlotAdvanced(s)proves every record update up to slotswas already delivered, so a consumer can conclude "this mirror entry is unchanged through slots" — the freshness proof a mirror lookup needs. An out-of-band watch channel could race ahead of undrained updates and was deliberately not used.SyncInterruptedsemantics are unchanged and still void all previously delivered state.Laserstream pin
Pinned
helius-laserstreamto rev295e8528— the same rev the magicblock-validator workspace pins — so the validator builds a single laserstream copy once it depends on this crate. (Previously this crate tracked branchv0.2.8+conn-fix, a different head of the same fork.)Tests
The syncer is now directly drivable in tests (
LaserStreamhandle is optional): coverage for firehose delivery + in-band watermarks + no replay buffering, and for subscribed-mode filtering staying unchanged. 9 unit tests + doc test pass; clippy/fmt clean.Summary by CodeRabbit
New Features
Documentation