From 8cdea942f2f2f63e1be3a2914d6512e1a4bffef8 Mon Sep 17 00:00:00 2001 From: Maxime Lamothe-Brassard Date: Wed, 30 Sep 2026 23:32:35 +0000 Subject: [PATCH 1/6] Document analysis completion workflows and latency alerts --- docs/email-security/automation.md | 87 +++++++++++++++++++++++++++ docs/email-security/pipeline.md | 65 +++++++++++++++++++- docs/email-security/rule-reference.md | 31 +++++++++- 3 files changed, 181 insertions(+), 2 deletions(-) diff --git a/docs/email-security/automation.md b/docs/email-security/automation.md index c7f92762e..da7467305 100644 --- a/docs/email-security/automation.md +++ b/docs/email-security/automation.md @@ -20,6 +20,7 @@ is one sensor, not ten thousand. |---|---| | `EMAIL_MESSAGE` | Once per message, at ingest. Carries the whole parsed model — headers, sender, recipients, body, links, attachments, authentication, hops — plus the enrichments and the verdict. It is the record that this mail arrived | | `EMAIL_VERDICT` | On **every** verdict decision: the rule pack's own, at ingest right after the `EMAIL_MESSAGE` (`revision/seq: 0`, `revision/mode: auto`), and then once per override afterwards (`seq: 1…`, mode `analyst`, `ai` or `detonation`) | +| `EMAIL_ANALYSIS_COMPLETE` | When the initial analysis window closes, including messages with nothing pending. Carries terminal outcomes, final verdict snapshot and timing | | `EMAIL_ACTION` | On every remediation outcome, including failures and skips, **and on every raw-message download** (`action: get_eml`), served or refused. Who asked, what was attempted, what happened | | `EMAIL_USER_REPORT` | When a message reaches the abuse mailbox and becomes a report | | `EMAIL_INGEST_ERROR` | When a message could not be fetched or processed. Coverage honesty: failures are visible, never silent | @@ -57,6 +58,21 @@ The MDM is deliberately *not* repeated here: it is already in the immutable `EMAIL_MESSAGE`, and copying it into every verdict change would multiply a year of telemetry by how often people change their minds. +### `EMAIL_ANALYSIS_COMPLETE` + +This event uses the same identity and `revision` paths as `EMAIL_VERDICT`, so +triage reads `event/revision/verdict`, not the MDM's `event/verdict/verdict`. +`event/results` maps each armed analysis (`detonation`, `attachment_scan`) to +`completed`, `changed_verdict`, `skipped`, `shed`, `failed`, or `timed_out`. +`event/completed_at` and `event/timing` describe the completed window. Empty +results mean no delayed work was needed. The completion payload does not include +the seq-0 `analysis` snapshot or `after_complete`. + +Start an AI or analyst triage workflow on this event when it needs the initial +analysis results. Continue handling later `EMAIL_VERDICT` escalations: analyses +that finish beyond the deadline carry `event/after_complete: true`. Completion +never means that a failed or timed-out analysis cleared the message. + ### A message that joins a campaign late Clustering runs while a message is being ingested, so two copies of one attack @@ -285,3 +301,74 @@ Two conventions make this pleasant to keep in git: Onboarding a new tenant is then: subscribe the extension, write the secret, write the provider record, apply the policy directory, run the connection test. + +## Triage after initial analysis + +This platform D&R detection reports suspicious or malicious messages after the +initial evidence window closes. The final verdict is a snapshot at completion; +read `results` when your triage needs to distinguish an examined message from a +deadline, capacity refusal or analysis failure. Use `msg_uuid` as the workflow's +idempotency key when dispatching external work. + +```yaml +# Detect +event: EMAIL_ANALYSIS_COMPLETE +op: and +rules: +- op: exists + path: event/revision/verdict + truthy: true +- op: or + rules: + - op: is + path: event/revision/verdict + value: malicious + case sensitive: false + - op: is + path: event/revision/verdict + value: suspicious + case sensitive: false +``` + +```yaml +# Respond +- action: report + name: email-analysis-triage +``` + +The initial `EMAIL_MESSAGE` remains useful for immediate containment and +content rules. Waiting for completion is a workflow choice; it does not prevent +the existing ingest-time automations from containing an already malicious message. + +## Alerting on provider delivery delays + +This example reports a provider notification lag over five minutes. The +existence guard excludes messages whose notification time is unknown. Adjust +the threshold to your own operating expectations; this is an example rule, +not a built-in alert. + +```yaml +# Detect +op: and +rules: +- op: is + path: routing/event_type + value: EMAIL_ANALYSIS_COMPLETE +- op: exists + path: event/timing/provider_lag_ms +- op: is greater than + path: event/timing/provider_lag_ms + value: 300000 +``` + +```yaml +# Respond +- action: report + name: email-analysis-provider-lag +``` + +Compare `provider_lag_ms` with `queue_ms` and `processing_ms` to separate delay +before notification from delay inside processing. The same pattern can alarm +on another present timing field. `analysis_ms` includes the delayed analysis +window; `end_to_end_ms` ends at the initial verdict. Check `clock_skew` before +interpreting clamped measurements. diff --git a/docs/email-security/pipeline.md b/docs/email-security/pipeline.md index d5a0983f2..8f43aed15 100644 --- a/docs/email-security/pipeline.md +++ b/docs/email-security/pipeline.md @@ -112,7 +112,7 @@ the message at the provider. | | | |---|---| | **Synchronous, one pass per message** | Fetch, parse, every enrichment including attachment explosion, matching, scoring, the verdict, campaign clustering, persistence, and the emission of `EMAIL_MESSAGE` followed by the engine's own `EMAIL_VERDICT` (`revision/seq: 0`) | -| **Later, and recorded as such** | Verdict revisions, remediation outcomes, campaign membership added when a later message joins the cluster | +| **Later, and recorded as such** | Link detonation, deferred attachment scans, analysis completion, verdict revisions, remediation outcomes, campaign membership added when a later message joins the cluster | A revision does **not** rewrite `EMAIL_MESSAGE`. The original event stands as the record of what the engine decided at ingest, a revision row is appended with its @@ -135,6 +135,69 @@ revision: a message nobody has overridden still reports zero revisions. the gaps named. A slow enrichment degrades one signal. A blocking one degrades coverage, which is the thing you bought. +## Knowing when initial analysis has finished + +The initial `EMAIL_VERDICT` (`revision/seq: 0`) includes +`analysis: {pending: [...], complete: false}` when delayed work is outstanding. +The closed set of pending kinds is `detonation` and `attachment_scan`. With +nothing outstanding, it carries `pending: []` and `complete: true`. + +`EMAIL_ANALYSIS_COMPLETE` closes that initial window for every live ingested +message, including messages with no delayed work. It carries the final +`revision/verdict`, `revision/score`, `revision/seq`, and known message identity +fields, plus `results` and `timing`. Use this event to start triage that needs the +initial batch of evidence. A completion is a processing fact, **not a safety +verdict**. + +| Result | Meaning | +|---|---| +| `completed` | The analysis finished without changing the verdict | +| `changed_verdict` | The analysis committed a verdict revision | +| `skipped` | The analysis was no longer applicable | +| `shed` | Capacity admission refused the work | +| `failed` | The analysis failed | +| `timed_out` | The initial analysis deadline expired before a terminal result | + +Only armed kinds appear in `results`; `{}` means no delayed work was needed. +Pending work is durable. A collector restart or lost in-memory task cannot leave +the window open forever: unresolved work becomes `timed_out` at the configured +deadline, which defaults to 20 minutes. Recovery publishes queued completion +snapshots after a service interruption. + +Evidence that arrives after this boundary can still change the verdict. Its +`EMAIL_VERDICT` carries `after_complete: true`; it does not rewrite the completion +snapshot. Keep a revision handler alongside completion-based triage for those +later changes. A later escalation re-runs post-verdict rules and automations; +a downgrade does not automatically restore mail. + +## Processing latency + +The MDM's `timestamps` includes optional `notified`, the time the provider's +notification reached LimaCharlie. Historical or periodically discovered mail +may have no notification time. The seq-0 verdict and completion event include +absolute `sent`, `received`, `notified`, `ingested`, `decided`, and `completed` +instants where applicable, with these integer millisecond measurements: + +| Timing field | Interval | +|---|---| +| `provider_lag_ms` | Provider received → notification reached LimaCharlie | +| `queue_ms` | Notification reached LimaCharlie → processing began | +| `processing_ms` | Processing began → initial verdict decided | +| `end_to_end_ms` | Provider received → initial verdict decided | +| `analysis_ms` | Initial verdict decided → initial analysis window completed | + +`provider_lag_ms` and `queue_ms` are **absent** when `notified` is unknown. A +measured zero is present as `0`. Negative intervals clamp to zero and set +`clock_skew: true`; investigate clock differences before treating those zeros as +fast processing. `sent` comes from the sender's untrusted Date header and is +never used for these calculations. In coalesced Gmail notifications, `notified` +is the batch's receiver observation, not a claim of one notification per message. + +The message drawer shows this timeline and each analysis outcome. The API and +`limacharlie mailsec message get ` expose the same `analysis` +and `timing` state. See [Events & Automation](automation.md#triage-after-initial-analysis) +for completion triage and provider-delay alerts. + ## Rules are organization-owned The organization’s enabled `dr-mail` records are the complete rule set. Defaults diff --git a/docs/email-security/rule-reference.md b/docs/email-security/rule-reference.md index e8cb83b3a..f541c263c 100644 --- a/docs/email-security/rule-reference.md +++ b/docs/email-security/rule-reference.md @@ -14,7 +14,7 @@ have different wrappers and validation rules. | `dr-mail`, either phase | `sender/email/domain/root` | | `dr-mail`, `post_verdict` only | `verdict/verdict` | | `dr-general` on `EMAIL_MESSAGE` | `event/sender/email/domain/root` | -| `dr-general` on `EMAIL_VERDICT` | `event/revision/verdict` | +| `dr-general` on `EMAIL_VERDICT` or `EMAIL_ANALYSIS_COMPLETE` | `event/revision/verdict` | | Inside `scope` with `path: links` | `href_url/domain/root` | The MDM is the root of a mail rule. Do not add `mdm/` or `event/`. The Hive @@ -214,6 +214,7 @@ D&R rule on `EMAIL_MESSAGE`. Message-list filtering does not accept `mail_type`. |---|---|---| | `sent` | timestamp | When set | | `received` | timestamp | Always | +| `notified` | timestamp | When the provider notification reached LimaCharlie; absent for mail without a known notification | | `ingested` | timestamp | Always | ### Headers @@ -689,3 +690,31 @@ lookup failure must not be treated as evidence against a message. When present, `code` is the stable identifier for a recovered failure. Prefer it over matching the human-readable `message`, which may change. + +## Analysis status and timing on platform events + +These fields belong to `EMAIL_VERDICT` and `EMAIL_ANALYSIS_COMPLETE`, rather +than the MDM rule root. A `dr-mail` scoring rule cannot wait for completion; +use a platform `dr-general` rule on the emitted completion event. + +| Path | Available on | Meaning | +|---|---|---| +| `event/analysis/pending` | Seq-0 `EMAIL_VERDICT` | Array of `detonation` and/or `attachment_scan`; empty when none outstanding | +| `event/analysis/complete` | Seq-0 `EMAIL_VERDICT` | Whether there was no outstanding work in that snapshot | +| `event/results/` | `EMAIL_ANALYSIS_COMPLETE` | `completed`, `changed_verdict`, `skipped`, `shed`, `failed`, or `timed_out` | +| `event/completed_at` | `EMAIL_ANALYSIS_COMPLETE` | When the initial analysis window was durably closed | +| `event/revision/verdict`, `event/revision/score`, `event/revision/seq` | Both | Initial decision or final completion snapshot | +| `event/after_complete` | Later `EMAIL_VERDICT` | True when a revision was decided after the completion boundary | +| `event/timing/received`, `ingested`, `decided` | Both | Required absolute processing instants | +| `event/timing/sent`, `notified` | Both, when known | Sender Date header and notification arrival; sent is untrusted | +| `event/timing/completed` | Completion | Absolute completion instant | +| `event/timing/provider_lag_ms` | Both, when notified known | Received → notified, integer ms | +| `event/timing/queue_ms` | Both, when notified known | Notified → ingested, integer ms | +| `event/timing/processing_ms` | Both | Ingested → initial decided, integer ms | +| `event/timing/end_to_end_ms` | Both | Received → initial decided, integer ms | +| `event/timing/analysis_ms` | Completion | Initial decided → completed, integer ms | +| `event/timing/clock_skew` | When true | At least one negative interval was clamped to zero | + +Missing optional intervals are absent, never fabricated zero. A completion with +`failed`, `shed`, or `timed_out` results does not classify the message as benign. +See [completion triage and delay rules](automation.md#triage-after-initial-analysis). From 0ed073675394bb7fe9867076a76a5f26936477c9 Mon Sep 17 00:00:00 2001 From: Maxime Lamothe-Brassard Date: Thu, 1 Oct 2026 00:46:48 +0000 Subject: [PATCH 2/6] Document operator rejudge escalation responses --- docs/email-security/pipeline.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/email-security/pipeline.md b/docs/email-security/pipeline.md index 8f43aed15..5ce36aec3 100644 --- a/docs/email-security/pipeline.md +++ b/docs/email-security/pipeline.md @@ -168,7 +168,10 @@ Evidence that arrives after this boundary can still change the verdict. Its `EMAIL_VERDICT` carries `after_complete: true`; it does not rewrite the completion snapshot. Keep a revision handler alongside completion-based triage for those later changes. A later escalation re-runs post-verdict rules and automations; -a downgrade does not automatically restore mail. +a downgrade does not automatically restore mail. An explicit operator re-judge +that escalates a message also runs responses against its newly judged evidence, +including a re-judge of historical mail. Dry runs do not act; initial historical +ingestion still does not run automations. ## Processing latency From 11ed9a38ef9edcf8f005331b33493fc6e087098e Mon Sep 17 00:00:00 2001 From: Maxime Lamothe-Brassard Date: Thu, 1 Oct 2026 01:15:34 +0000 Subject: [PATCH 3/6] Clarify that rejudge containment finishes asynchronously --- docs/email-security/pipeline.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/email-security/pipeline.md b/docs/email-security/pipeline.md index 5ce36aec3..a842674f8 100644 --- a/docs/email-security/pipeline.md +++ b/docs/email-security/pipeline.md @@ -170,8 +170,10 @@ snapshot. Keep a revision handler alongside completion-based triage for those later changes. A later escalation re-runs post-verdict rules and automations; a downgrade does not automatically restore mail. An explicit operator re-judge that escalates a message also runs responses against its newly judged evidence, -including a re-judge of historical mail. Dry runs do not act; initial historical -ingestion still does not run automations. +including a re-judge of historical mail. Re-judge responses run asynchronously +after the stored correction: a changed count does not mean containment has +finished. Dry runs do not act; initial historical ingestion still does not run +automations. ## Processing latency From 4ea8eda3ad940a1d24b54d0f50a688ffb5d15690 Mon Sep 17 00:00:00 2001 From: Maxime Lamothe-Brassard Date: Thu, 1 Oct 2026 01:22:59 +0000 Subject: [PATCH 4/6] Document stable completion identity for idempotent triage --- docs/email-security/pipeline.md | 4 +++- docs/email-security/rule-reference.md | 1 + 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/email-security/pipeline.md b/docs/email-security/pipeline.md index a842674f8..02b7f583c 100644 --- a/docs/email-security/pipeline.md +++ b/docs/email-security/pipeline.md @@ -147,7 +147,9 @@ message, including messages with no delayed work. It carries the final `revision/verdict`, `revision/score`, `revision/seq`, and known message identity fields, plus `results` and `timing`. Use this event to start triage that needs the initial batch of evidence. A completion is a processing fact, **not a safety -verdict**. +verdict**. Its `completion_id` is the message's `msg_uuid` and stays the same +for that immutable snapshot. Delivery retries after an interruption can repeat +the snapshot; use `completion_id` for idempotent triage or duplicate suppression. | Result | Meaning | |---|---| diff --git a/docs/email-security/rule-reference.md b/docs/email-security/rule-reference.md index f541c263c..65a22f5a9 100644 --- a/docs/email-security/rule-reference.md +++ b/docs/email-security/rule-reference.md @@ -699,6 +699,7 @@ use a platform `dr-general` rule on the emitted completion event. | Path | Available on | Meaning | |---|---|---| +| `event/completion_id` | `EMAIL_ANALYSIS_COMPLETE` | Stable completion identity, equal to `msg_uuid`; use it to suppress retried delivery | | `event/analysis/pending` | Seq-0 `EMAIL_VERDICT` | Array of `detonation` and/or `attachment_scan`; empty when none outstanding | | `event/analysis/complete` | Seq-0 `EMAIL_VERDICT` | Whether there was no outstanding work in that snapshot | | `event/results/` | `EMAIL_ANALYSIS_COMPLETE` | `completed`, `changed_verdict`, `skipped`, `shed`, `failed`, or `timed_out` | From ee0d0472704b89fbe92db63de1390a6a6304c894 Mon Sep 17 00:00:00 2001 From: Maxime Lamothe-Brassard Date: Thu, 1 Oct 2026 01:49:52 +0000 Subject: [PATCH 5/6] Document completion retry suppression and historical scope --- docs/email-security/automation.md | 17 +++++++++++++++-- docs/email-security/pipeline.md | 19 +++++++++++-------- docs/email-security/rule-reference.md | 5 +++++ 3 files changed, 31 insertions(+), 10 deletions(-) diff --git a/docs/email-security/automation.md b/docs/email-security/automation.md index da7467305..5696d5b56 100644 --- a/docs/email-security/automation.md +++ b/docs/email-security/automation.md @@ -307,8 +307,9 @@ the provider record, apply the policy directory, run the connection test. This platform D&R detection reports suspicious or malicious messages after the initial evidence window closes. The final verdict is a snapshot at completion; read `results` when your triage needs to distinguish an examined message from a -deadline, capacity refusal or analysis failure. Use `msg_uuid` as the workflow's -idempotency key when dispatching external work. +deadline, capacity refusal or analysis failure. Completion delivery is at least +once. Use `completion_id` as the workflow's idempotency key when dispatching +external work. Initial historical backfill emits no completion event. ```yaml # Detect @@ -334,8 +335,20 @@ rules: # Respond - action: report name: email-analysis-triage + suppression: + max_count: 1 + period: 720h + is_global: true + keys: + - email-analysis-triage + - '{{ .event.completion_id }}' ``` +The example suppresses repeated reports for the same completion for 30 days +across sensors within the organization. After the suppression period expires, +the same identity can report again. External workflows that require durable +idempotency should retain their own completion identities for their retry horizon. + The initial `EMAIL_MESSAGE` remains useful for immediate containment and content rules. Waiting for completion is a workflow choice; it does not prevent the existing ingest-time automations from containing an already malicious message. diff --git a/docs/email-security/pipeline.md b/docs/email-security/pipeline.md index 02b7f583c..e362aea78 100644 --- a/docs/email-security/pipeline.md +++ b/docs/email-security/pipeline.md @@ -7,10 +7,10 @@ message from the moment your provider says it exists to the moment somebody decides what to do about it, and it is explicit about which parts happen in one pass and which parts can happen later. -The short version: **everything that produces the first verdict happens -synchronously, in one pass, per message.** There is no queue of half-judged mail -and no second job that fills in the answer. Anything that changes a verdict -afterwards is recorded as a *revision*, not as a late arrival. +The first verdict uses the evidence available within the bounded initial pass. +Delayed attachment scans and link detonation can add evidence afterwards; a +changed verdict is recorded as a *revision*. Wait for `EMAIL_ANALYSIS_COMPLETE` +when your workflow needs the initial analysis window to close. ## The stages @@ -142,14 +142,17 @@ The initial `EMAIL_VERDICT` (`revision/seq: 0`) includes The closed set of pending kinds is `detonation` and `attachment_scan`. With nothing outstanding, it carries `pending: []` and `complete: true`. -`EMAIL_ANALYSIS_COMPLETE` closes that initial window for every live ingested -message, including messages with no delayed work. It carries the final +`EMAIL_ANALYSIS_COMPLETE` closes that initial window for every message emitted +through the live lane, including re-drives that emit and messages with no delayed +work. Initial historical backfill emits no `EMAIL_*` events and has no completion +event; a later live notification can promote that message into the live lane. It carries the final `revision/verdict`, `revision/score`, `revision/seq`, and known message identity fields, plus `results` and `timing`. Use this event to start triage that needs the initial batch of evidence. A completion is a processing fact, **not a safety verdict**. Its `completion_id` is the message's `msg_uuid` and stays the same -for that immutable snapshot. Delivery retries after an interruption can repeat -the snapshot; use `completion_id` for idempotent triage or duplicate suppression. +for that immutable snapshot. Delivery is at least once: retries after an interruption can repeat +the snapshot. Use `completion_id` for idempotent triage or the +[completion suppression example](automation.md#triage-after-initial-analysis). | Result | Meaning | |---|---| diff --git a/docs/email-security/rule-reference.md b/docs/email-security/rule-reference.md index 65a22f5a9..b005b369a 100644 --- a/docs/email-security/rule-reference.md +++ b/docs/email-security/rule-reference.md @@ -716,6 +716,11 @@ use a platform `dr-general` rule on the emitted completion event. | `event/timing/analysis_ms` | Completion | Initial decided → completed, integer ms | | `event/timing/clock_skew` | When true | At least one negative interval was clamped to zero | +Completion delivery is at least once; key response suppression on +`event/completion_id`, as shown in the [triage example](automation.md#triage-after-initial-analysis). +Completion covers emitted live messages, including emitting re-drives. Initial +historical backfill emits no `EMAIL_*` events and no completion event. + Missing optional intervals are absent, never fabricated zero. A completion with `failed`, `shed`, or `timed_out` results does not classify the message as benign. See [completion triage and delay rules](automation.md#triage-after-initial-analysis). From b9a4c6a13155a11b201e0714f38deb629f5f9efb Mon Sep 17 00:00:00 2001 From: Maxime Lamothe-Brassard Date: Thu, 1 Oct 2026 23:24:22 +0000 Subject: [PATCH 6/6] Document attachment scan pending for unloaded YARA rules Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/email-security/pipeline.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/email-security/pipeline.md b/docs/email-security/pipeline.md index e362aea78..d61a80b91 100644 --- a/docs/email-security/pipeline.md +++ b/docs/email-security/pipeline.md @@ -142,6 +142,12 @@ The initial `EMAIL_VERDICT` (`revision/seq: 0`) includes The closed set of pending kinds is `detonation` and `attachment_scan`. With nothing outstanding, it carries `pending: []` and `complete: true`. +`attachment_scan` is also pending when your organization's custom YARA rules were +not yet loaded while the message was processed, for example just after a deploy or +restart. The attachments are rescanned with your rules once they load, whatever the +verdict or direction, and a match revises the verdict. If the rules cannot be loaded +before the deadline, the result is `timed_out`, never a clean `completed`. + `EMAIL_ANALYSIS_COMPLETE` closes that initial window for every message emitted through the live lane, including re-drives that emit and messages with no delayed work. Initial historical backfill emits no `EMAIL_*` events and has no completion