Skip to content

Add the in-process event bus and the outbox - #2755

Merged
pournasserian merged 4 commits into
mainfrom
feat/serene-jennings-117876
Nov 3, 2026
Merged

pournasserian merged 4 commits into
mainfrom
feat/serene-jennings-117876

Conversation

@pournasserian

@pournasserian pournasserian commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Closes #2697

What

  • IEventBus, IEventHandler<TEvent> and IOutboxEventHandler<TEvent> in the SDK, with the event records of spec 4.11 that this issue names: the ten entry events, SettingsChangedEvent, SiteCreatedEvent, SiteUpdatedEvent, CacheInvalidatedEvent, and JobFailedEvent / JobDeadLetteredEvent, moved to the SDK. CmsEvent carries an id and an EventSource (site, audited actor and its kind, origin, correlation id). The bus fills Source from the current operation when the publisher left it out.
  • IEntryChangeSink is implemented by EntryChangeSink: it hears a change only after the outermost commit and maps it to an event. IEntryChangeStager lets it write the outbox rows inside the transaction, just before the commit.
  • Events published through IEventBus while the content service holds its transaction are buffered (deduplicated by EventId; a handler that writes during dispatch gets a buffer of its own and can neither repeat nor lose them) and dropped when the transaction or a savepoint rolls back. Outside any transaction they are delivered at once. Inside a transaction on CmsDbContext or SystemDbContext that the service did not begin, publishing throws FCMS2602, since its end cannot be seen. There is no API yet to stage rows inside a SystemDbContext transaction; Typed settings model persisted per site #2704 can add one when it needs it.
  • cms_outbox: one row per event and outbox handler, written in the same transaction, saved without saving anything else the caller tracked. A write inside a transaction the service did not begin also writes its rows inside it, so they commit or roll back with it; the handlers of this process still hear nothing of such a write, as in Add the content service with optimistic concurrency #2747.
  • OutboxDrainer claims a batch with a conditional update and a lease. Claiming costs no attempt: the worker starts an attempt, and takes its lease again, with one conditional update before each delivery, which matches nothing once another instance has the row, so a worker whose lease was taken over delivers nothing more and a row whose handler never ran is never failed. A delivery runs on a thread of its own and is cut off after DeliveryTimeout (1 minute, shorter than the 5-minute lease), logged as abandoned and counted as a failed attempt; an abandoned handler may still be running, since nothing can stop it. A failed delivery is retried with backoff and left as failed after MaxAttempts; a row whose started attempts are used up by workers that died is failed at the next claim; a row whose handler this instance lacks waits without using an attempt. The idempotency key is the row id.
  • The drain runs on its own outbox queue (one worker by default), so a hung handler holds up no other job. It is asked for after a commit that recorded rows, and a recurring run each minute (which also prunes delivered rows) catches the rest. The recurring run is registered only when an outbox handler exists, and retried until it is stored.
  • Rows are delivered in a sequence that rises with the order they were recorded in, so the events of one commit, and of one entry, arrive in order from one instance.
  • JobStore writes the outbox rows of JobFailedEvent / JobDeadLetteredEvent in the same transaction as the job's outcome, with ids derived from the job and the attempt, and the site of the job as the site of the event. BusJobEventSink replaces NullJobEventSink and runs the handlers of this process after the commit, under a system operation on the site of the job and in its default locale. A job worker waits for them at most JobEventTimeout (5 seconds); handlers that take longer are left running on a thread of their own and logged.
  • Handlers run with the operation of the event: outbox handlers as the system on the site of the event, in that site's default locale, with the actor and correlation id of the event; in-process handlers get the host's shutdown token, never the request's.
  • One scrubber, best effort and by shape, is applied to the error of a failed delivery and to the error carried by job events: a URL is reduced to scheme and host, a Bearer or Basic credential is masked, and so is the value of a key=value pair whose key names a secret (token, secret, password, key, authorization, signature and similar). A secret in any other shape is not recognised. cms_job.last_error is not changed by this PR.
  • IEventDeadLetters lists, retries and removes failed deliveries.
  • Migration AddOutbox for SQLite, PostgreSQL and SQL Server, per docs/system-migrations.md. No seed data.
  • cms.Events.AddHandler and cms.Events.AddOutboxHandler(name) register handlers. They run in [HookOrder] order, then by the IHookTiebreaker rank, then by registration.

Acceptance

  • Events from a rolled-back transaction are never dispatched. Tested for a failed validation, an after hook that throws, a failed commit, a failed stage of the outbox row, a savepoint rolled back inside the write, and a foreign transaction rolled back; none leaves a handler call or an outbox row.
  • An outbox subscriber receives the event after a simulated crash between commit and dispatch. The first instance commits with its job table unreachable; a second instance drains the row. Also for a write in a foreign transaction that commits.
  • Handler ordering follows [HookOrder] and plugin dependency order, for in-process handlers and for the start of outbox deliveries. The plugin host does not exist yet, so the dependency order is the existing IHookTiebreaker seam, tested with a stub. Outbox delivery order across instances and across retries is not guaranteed, as the SDK documentation says; the order of the events of one commit and of one entry holds from one instance.
  • All content, publishing, settings and site events from the catalogue exist as records. Identity, media and plugin events are outside this issue. Publishing, settings and site writes do not raise their events yet; that belongs to their issues.
  • SQLite only: the tests run on SQLite. The PostgreSQL and SQL Server migrations and the drain's conditional updates on them wait for Validate and complete the PostgreSQL and SQL Server providers #2711.

Tests of the failure paths

A handler that throws or cancels, or cannot be built, leaves the write and the other events alone; a crash between commit and dispatch; a commit that fails; a stage that fails and leaves nothing for the next write of the scope; a settle that fails; a lease that is taken over under a live worker; three instances draining together; a worker that dies on every attempt; a handler that never returns; a drain that is cancelled mid-batch; a handler the instance does not have; a job outcome whose event cannot be recorded; a handler that leaves a poisoned entity in the shared context; a handler that writes content while it is told.

Notes

  • If the handlers of one event type cannot be built (a constructor that throws), none of that type's in-process handlers run: the container builds them together. The failure is logged and reaches no caller, no other event and no drain request.
  • An outbox handler is matched on the exact event type, as an in-process handler is.
  • A handler can run again after a crash, or if it outlives its lease. Rows of a plugin that stays removed stay pending.
  • OutboxWriter and EventBus use DbContext from the service or a probe in Core.Content, because the architecture tests allow only the content and storage layers to touch CmsDbContext.
  • Hand-over for Publishing state machine: states, transitions, versions, diff, restore and reference integrity #2695, Bulk operations with per-item results and a filter cap #2703 and others is in the shared notes folder.

Dependencies

No new packages.

Events raised in the content service's transaction are held until the outermost commit and dropped on rollback; outbox handlers get their row in the same transaction and a drain job delivers it at least once. Closes #2697
Stage the outbox rows without saving what the caller tracked and also inside a foreign transaction; probe both contexts; renew the lease before each delivery and cut a delivery off before its lease ends; give the drain its own queue; record job events with the job outcome; stamp the source of a publish; run handlers under the operation of the event; order rows by a sequence; scrub stored errors; add a service for failed deliveries.
…eview round

Claim without using an attempt; run handlers on their own thread so a blocking one can be abandoned; keep a nested write from re-staging or losing the buffered events; bound the wait of the job runner for in-process job event handlers; scrub secrets from job event errors and stored errors; give job events the site of the job.
@pournasserian
pournasserian merged commit 16ea7a1 into main Nov 3, 2026
5 checks passed
@pournasserian
pournasserian deleted the feat/serene-jennings-117876 branch November 3, 2026 00:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

In-process event bus with after-commit dispatch and outbox

1 participant