adaptive_export: reliable dx-steered pem-direct capture (chunk/end_time, breaker, dc_snoop filter, DaemonSet) - #92
adaptive_export: reliable dx-steered pem-direct capture (chunk/end_time, breaker, dc_snoop filter, DaemonSet)#92ConstanzeTU wants to merge 100 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…e fix) Root cause of the flaky dx-steered capture (dc_snoop/http erratically 0 while light tables always land): OrderExportAll fans out ~20 tables concurrently, each OrderQuery issued ONE unbounded PxL query over the full ~600s control window against the single node-local PEM (pem-direct). QueryFor only set start_time, so every query re-scanned [sliceStart, now] and post-filtered — the heavy tables materialize huge result sets on a saturated PEM and lose the fixed 180s deadline race, dropping out; the cheap tables (redis/conn/stack) return instantly and survive. Reconcile fingerprint: the same dc_snoop query returns 2459 rows in isolation but 0 + 1 err under the fan-out. Fix (durable — removes the data-volume↔deadline coupling, not just tunes it): - pxl.QueryFor: bound the PEM source scan on BOTH sides. Emit a relative end_time (floored toward now so nothing real is clipped; the exact upper bound stays enforced by the df.time_ < sliceEnd nanos post-filter) whenever sliceEnd is in the past. Live-edge slices keep scanning to now (no end_time), preserving prior behavior for the most-recent window. - controller.OrderQuery: walk the capture window in OrderChunk-sized sub-windows (default 60s, env ADAPTIVE_ORDER_CHUNK_SEC), each a both-sides bounded query, so no single query re-materializes the whole window. captureSpan adaptively halves any chunk that still fails with a transient (deadline/overload) error down to orderMinChunk (1s); non-transient errors (missing dark table) surface immediately without wasteful splitting. Overlapping/retried spans dedupe in the ReplacingMergeTree evidence tables, so re-pulls are idempotent. One aggregated reconcile row per table (not per chunk). Chunks run sequentially per table, so OrderExportAll's per-table concurrency is unchanged while each table now issues cheap bounded queries instead of one firehose — reliable capture without needing the global inflight throttle set. Tests: queryfor end_time present for past windows / absent at the live edge; OrderQuery chunking, single aggregated reconcile row, adaptive subdivision on transient error, no-split on non-transient error, termination at min-chunk.
… (dc_snoop) The dx-steered OrderExportAll path applied only a partial comm denylist and NO namespace filter to the node-scoped dark-vector tables — unlike the shipped cron preset (script/presets dc_snoop.pxl __DC_SNOOP_EXCLUSION__, built from presets.go defaultExcludeNamespaces + defaultExcludeComms). So every dc_snoop capture drowned in infra dcache churn: on a real k3s node a single window returned ~54k rows dominated by ConfigReloader/iptables/CNI(host-local,bridge,flannel,loopback)/host daemons(systemd-udevd,dbus-daemon,tailscaled)/kubevuln — burying the salient attack specimens (whoami/cat/getent reading /etc/shadow + the SA token). - Extend darkExcludeCommsDefault with the host/CNI/node daemons that were leaking (systemd-udevd, host-local, bridge, flannel, loopback, bandwidth, dbus-daemon, mount, umount, tailscaled, grpc_health_pro, kubevuln, opm, kube-proxy, …). - Add darkExcludeNamespacesDefault + darkNamespaceExclusion(), applied in the IsDarkVector branch AFTER PodEnrichPxL resolves df.namespace, dropping infra namespaces (pl, kube-system, clickhouse, …). Blank-namespace transient rows survive (each `!=` is true for ''), so the attack's short-lived children — which resolve blank — are never dropped. Overridable via DC_SNOOP_EXCLUDE_NAMESPACES. Kept in sync with script/presets.go. Tests: infra namespaces + host/CNI comms dropped; df.namespace never pinned to the alert pod (node-scoped); env override replaces the default list.
… depth cap) Live RCA on aeprod54: the chunk fix is correct in isolation (pem unit suite — dc_snoop 54k, redis/conn/stack written per-chunk) but UNSAFE under the dx steering firehose. dx does generic collect-per-alert, so OrderExportAll (20 tables) fires on every noisy pl system pod continuously; all land on the ONE node-local PEM (pem-direct) → it saturates → 100% DeadlineExceeded. captureSpan then split every timeout into two narrower retries, amplifying a busy PEM into a query storm where nothing completes (observed: "0 ordered pixie rows written" across the whole run; draining dx + restarting AE → pem-direct instantly serves again). Make subdivision safe: - Circuit-breaker: orderTimeoutStreak (atomic) counts CONSECUTIVE transient failures; any success resets it. Above orderBreakerTrip (8) captureSpan stops subdividing — a saturated PEM must not be flooded with retries. It still splits a genuinely-oversized window on a healthy PEM (the reset keeps that path live). - Depth cap: maxOrderSplitDepth (3) bounds one chunk to ≤2^3 leaf queries even if it keeps timing out (was ~64 splitting 60s→1s). Tests: a 10-chunk all-timeout window stays <60 queries (ungated ≈640); a single transient failure still recovers (breaker resets on success, no latch). NOTE (deployment, not code): the firehose root also needs dx steering scoped so it doesn't fire 20-table captures on every noisy pl/system-pod alert — tracked separately for dx-agent.
Live RCA (aeprod55): every dx-steered capture in the e2e returned 0 rows, and the reconcile showed why — all 36 ordered captures had ~512ns-wide windows (width_s=0), so they matched no pixie rows. /export/start already reaches back controlExportLookback, but a control client that keys the /query window on a single finding's event_time sends lo≈hi (a sub-microsecond span). That passes the lo<hi validation yet captures nothing. handleQuery now widens any window narrower than minControlQueryWindow (5s) to controlExportLookback ending at hi — a point-in-time referral still captures the evidence leading up to it. hi is preserved; comfortably-wide windows pass through unchanged. Isolated /query probes (proper windows) already proved the capture path works — dc_snoop 54k→16k filtered, redis/conn/stack per-chunk; this makes the dx-driven path robust to degenerate windows too. Tests: a 512ns window is widened to >=5s (hi preserved); a 120s window is untouched. NOTE (dx-agent): dx should send a real window (or use /export/start) rather than a point window per finding — tracked separately. This is the AE-side safety net.
The bootstrap manifest was a replicas:0 Deployment with minimal env (EXPORT_MODE= auto, no pem-direct, no throttle) — it never ran and could not do node-local pem-direct. Replace it with the working config that the e2e RCA validated: - DaemonSet (one-per-node) so each pod queries its OWN node's vizier-pem at HOST_IP:50305 (pem-direct: node-local, desync-immune). - dx-steered: EXPORT_MODE=never + CONTROL_ADDR=:9100 + the control Service (internalTrafficPolicy:Local so dx reaches its co-located AE). - PEM-protection: ADAPTIVE_MAX_INFLIGHT_QUERIES_GLOBAL=4 and ADAPTIVE_ORDER_CHUNK_SEC =600 (one query per table, no window pre-chunking) so the AE never saturates the single node-local PEM it shares with dx. See RCA_ae_capture_20260803. Secret still seeded per-cluster (unchanged).
…efault; trim comments - queryfor.go: add darkExcludeCommSubstrings (kworker/ksoftirqd/rcu_/… — kernel threads with variable suffixes exact-match misses) applied via px.logicalNot( px.contains); add pause + systemd-logind exact. Workload comms (redis-*) untouched. - controller.go: defaultOrderChunk 60s -> 600s (one query per table; pre-chunking 10x-amplified queries on the single node-local PEM). - Strip verbose comments across queryfor.go/controller.go/server.go + the AE manifest. Test: kernel-thread substrings dropped, workload comms kept, pause dropped.
Deploys the dx-daemon DaemonSet + Service into honey and mirrors the pl->honey secrets (jwt-signing-key, cluster-id, cloud-addr, api-key, clickhouse http-url) via a before-hook, replacing the hand-applied manifest used in the e2e. Deploy with: skaffold deploy -f k8s/vizier/dx/skaffold.yaml CH http-url defaults to the soc clickhouse Service; override with DX_CH_HTTP_URL.
Replaces the imperative seed-secret + patch-cloud-addr + sed-image +
kubectl-apply sequence with a single skaffold module:
skaffold deploy -f k8s/vizier/adaptive_export/skaffold.yaml
- kustomize overlay reuses bootstrap/adaptive_export_{role,deployment}
and pins the image via images: (ghcr aeprod tag) instead of sed.
- before-hook patches PL_CLOUD_ADDR :443 and seeds
pl-adaptive-export-secrets ONLY when PIXIE_API_KEY/PX_API_KEY is set,
never clobbering an existing secret with an empty key.
- LoadRestrictionsNone so the overlay can reuse the bootstrap manifests
in place (no duplication/drift).
Pairs with the dx-daemon skaffold (k8s/vizier/dx). Bump the AE image by
editing newTag in kustomization.yaml.
…aths
The AE/dx skaffold configs lived inside their overlay dirs with kustomize
paths: [.], which skaffold resolves against the shell CWD (repo root), not
the config-file dir -> 'unable to find kustomization.yaml in /.../pixie'.
Match the repo convention instead (skaffold/skaffold_vizier.yaml et al.):
skaffold configs live in skaffold/ and reference overlays by repo-root-
relative kustomize paths. Overlays stay in k8s/vizier/{adaptive_export,dx}.
skaffold deploy -f skaffold/skaffold_adaptive_export.yaml
skaffold deploy -f skaffold/skaffold_dx.yaml # run from repo root
- dx overlay gains a kustomization.yaml (was rawYaml).
- both validated with 'skaffold render' from repo root (image overrides +
RBAC/DaemonSet/Service resolve).
00d654e to
2e05ed2
Compare
|
@build-agent — (reposting here, channel moved) please build the AE image from branch Fixes the |
Builds for
|
|
@build-agent — Please build the AE image from branch |
|
@build-agent — being specific, please build the AE (adaptive_export) image from this exact ref, NOT the #92 PR head:
This ref = #94 ppid/pcomm dc_snoop capture + PR#95 streaming dark-vector fix. Verify:
The last build (aeprod59) was #92 head |
|
Re-built from TTL image (ready now, expires ~24h from 2026-08-12 11:25 UTC): GHCR image — tag One note for rig 6a7be0ac: |
scanner.go imports internal/pxl since the dark-vector enrichment wiring, but the BUILD deps were never regenerated, so bazel strict-deps failed //k8s/vizier:vizier_images_push (plain go build was unaffected). Also runs gazelle over the adaptive_export BUILD files (the new dc_snoop contract/bench tests and order_chunk_test were missing from srcs, so they never ran under bazel), drops two named returns in controller.go and renames deadlineErr to errDeadline. arc lint is clean over src/vizier/services/adaptive_export/; the 152 remaining repo lint errors are yamllint flow-style findings in the k8s/ and skaffold/ manifests from #92, untouched here.
|
GHCR build update:
The #95 wiring added the
The already-posted TTL image Not touched: 152 yamllint findings (flow-style braces/indentation) in |
|
✅ Green — run 31594196141 (build-release, create-github-release, update-gh-artifacts-manifest all success). From Superseded, do not deploy: |
|
@build-agent — please rebuild the AE (adaptive_export) image from this exact ref (supersedes aeprod61/
This adds the OOM fix on top of the BUILD.bazel fix. aeprod61 fixed the Verify before building:
No new cross-package imports vs |
|
@build-agent — ping (no reply/CI run in ~17min). Please build BOTH images from
Post both refs when ready. This is the OOM-fixed AE (shared |
|
TTL image ready — built from Expires ~2026-08-13 15:40 UTC. GHCR Superseded, do not deploy: |
|
@build-agent — one more turn of the loop. Please build BOTH images from
What changed: aeprod62/TTL Verify: |
|
TTL image ready — Expires ~2026-08-13 15:47 UTC. GHCR |
|
✅ GHCR green — run 31613928161, all jobs success. Same commit as the TTL image above ( Tag ledger: aeprod63 = current. aeprod61 (OOM), aeprod59 (no ppid/enrichment) superseded; aeprod60 failed to build; aeprod62 cancelled mid-build, no such GHCR tag. |
|
@build-agent — this is a DX build (entlein/dx repo, NOT the AE/pixie image). Posting here since this is the channel you watch.
Please post BOTH:
(The entlein release-tag CI is out of GitHub-hosted Actions minutes, so it queues forever — that is why I need you to build it.) This = deployed rc2 + one fix: |
|
DX build answered on entlein/dx#136 — TTL |
|
BUILD-READY e76cd2e TTL expires ~2026-08-25 17:00 UTC. No need for the correction — I had already spotted that Deploy them together, and recreate the tables — this pair is not backward compatible. dx rc14 changes what goes into a
dx rc14's TTL is on dx#136 once its build finishes. Numbering note: aeprod83's run is still in flight, so 82, 83 and 84 are now all building at once. 84 is a superset of 83 ( |
|
✅ aeprod82, 83 and 84 all green. Three tags landed in quick succession, so here they are together with the one to pin. Pin this one — The other two published fine and are strict subsets — no reason to deploy them, listed only so the digests are on record:
84 descends from 83 descends from 82, so pinning 84 gets all of it. Pair with dx And the recreate step still applies before any of this means anything on an existing |
…ow's The 11 dx_ord__ views projected c.event_time (the evidence row's kernel CAPTURE time). The px connector windows on event_time, and aeprod82 made that filter real (DateTime64, not the old ns-vs-seconds no-op). Evidence always precedes its alert, so a window around a fresh anomaly dropped its own (older) evidence — all panels empty. Project the ORDER's time instead: fromUnixTimestamp64Nano(toInt64(e.event_time)) (the edge carries ref.T, populated non-zero). ts/row_time keep the row's real capture time for display/ordering. Now a window around the anomaly returns its evidence regardless of when the packet/syscall was captured. Validated live: R0011 DNS-egress order, -15m window -> conn_stats 10, dns 34, dc_snoop 495 (was 0/0/0). In-place upgrade: DROP the 11 dx_ord__ views before apply (CREATE IF NOT EXISTS won't replace); fresh rigs unaffected.
|
build Fixes empty evidence panels. The 11 Now project the ORDER's time: Validated live (rig, new views applied by hand): R0011 DNS-egress order at -15m → conn_stats 10, dns 34, dc_snoop 495 (was 0/0/0). In-place upgrade needs DROP of the 11 dx_ord__ views before apply (CREATE IF NOT EXISTS won't replace). Fresh rigs fine. |
|
BUILD-READY dc7230f TTL expires ~2026-08-26 08:40 UTC. Confirmed on the in-place upgrade: the 11 views must be dropped first, as you noted — DROP VIEW IF EXISTS forensic_db.dx_ord__conn_stats; DROP VIEW IF EXISTS forensic_db.dx_ord__redis_events;
DROP VIEW IF EXISTS forensic_db.dx_ord__http_events; DROP VIEW IF EXISTS forensic_db.dx_ord__dns_events;
DROP VIEW IF EXISTS forensic_db.dx_ord__pgsql_events; DROP VIEW IF EXISTS forensic_db.dx_ord__mysql_events;
DROP VIEW IF EXISTS forensic_db.dx_ord__cql_events; DROP VIEW IF EXISTS forensic_db.dx_ord__mongodb_events;
DROP VIEW IF EXISTS forensic_db.dx_ord__creds_change; DROP VIEW IF EXISTS forensic_db.dx_ord__dc_snoop;
DROP VIEW IF EXISTS forensic_db.dx_ord__stack_trace;Three consequences of
|
|
✅ aeprod85 green — run 32836409319, every job success. From Bump Current pair: AE aeprod85 + dx rc17 ( Tag ledger: aeprod85 = current AE · 84 = UID-key ORDER BY · 83, 82 superseded · 68 unsigned, do not pin · 73, 64 failed. Don't forget the 11 |
…nels ORDERS: add ts (order/anomaly datetime) so it's visible + sortable — pulled from dx_kubescape_mitre, which gains ts = toString(fromUnixTimestamp64Nano( event_time)) (schema.sql; live-updatable). Drop the constant namespace column from http/dns/pgsql/mysql/dc_snoop panels (kept on conn/redis). Native header-click sort covers 'sort on TS'.
The dx/ script is the maintained SOC dashboard now; remove the old px/ sibling. Its only unique view deps (dx_src__kubescape_logs, dx_ord__stack_trace) are left in AE schema for now — decommission separately once confirmed unused.
New Live-UI script (does not touch dx/evidence_graph). Seed from pods with an alert >= min_sev (editable knob; severity is data-driven), then overlay every sensor's edges for the incident from a UNION view dx_fullchain_edges (kubescape rule->pod, conn egress pod->ip, dns name->ip, pgsql pod->query). Deliberately not keyed on one order_id (the read and send split across anomaly windows); correlates by pod + window + shared IP. Sparse: a missing sensor drops its edges, never the graph. dx_fullchain_edges is a NEW view (created live on the rig; not yet in AE schema.sql). v1 renders the alert layer; the egress/DNS/SQL overlay fills in once Pixie attributes the incident pod's exfil protocol traffic (capture fix in flight) — the connects/resolves edges currently land on coredns/infra, and pgsql isn't captured (TLS). v2 = upid->pid bridge + IP-eqclass + sender/receiver stitching for true cross-pod chains.
|
BUILD-REQUEST = aeprod85 + one schema addition: |
|
BUILD-READY c1b96cf (TTL first, as asked — durable follows) TTL expires ~2026-08-26 13:40 UTC. Durable One correction to the description, since it affects what lands on the rig: this is not aeprod85 + one schema addition. Three commits sit in that range —
The schema diff really is the one line you described. But the script bundle changes substantially: The DROP VIEW IF EXISTS forensic_db.dx_kubescape_mitre;then restart AE. Without it the ORDERS panel keeps throwing |
|
✅ aeprod86 green — run 32857819638, every job success. From Bump Current pair: AE aeprod86 + dx rc20 ( Tag ledger: aeprod86 = current AE · 85 = order-time |
dx stamps culprit_key = ns/pod/RootPID on each order; orders sharing it are one actor's steps. dx_cases joins orders to dx_kubescape_mitre so a case's steps (read + exfil + spawns) surface under one culprit_key, mitre/severity attached for UI colouring. Registered in Known/OperatorOwnedTables; contract test updated.
|
build |
|
BUILD-READY d68a257 TTL expires ~2026-08-27 10:20 UTC. Pairs with dx rc22 ( Nice to report: Two things for the rig:
|
|
✅ Both durables green — the culprit_key pair is fully published. AE aeprod87 (run 32933246660): dx rc22 (run 32933244270): Current pair: AE aeprod87 + dx rc22. Before the restart on an existing Ledgers — AE: 87 = current · 86 = mitre |
Adds cases() + a top CASES graph widget: culprit (ns/pod/RootPID) fans out to each step it took (pod:rule, mitre-labelled, severity-coloured). Reads dx_cases so one actor's read + exfil + spawns render as one campaign instead of 40 disconnected order dots. Order-link drills into any step. Existing per-order panels shifted down; unchanged.
|
Cloud bundle update needed for the CASES view (P3): |
|
Cloud bundle building — verify: Numbering follows the existing line: v0.0.45 was the last cloud tag (2026-08-25 21:40), so this is v0.0.46-pre-v0.0. First cloud release I have cut — say the word if you would rather own that tag yourself in future; it is a different pipeline from vizier/dx and it publishes the whole cloud image set, not just the bundle. Deployment shape, since this is the piece that surfaces in the Live UI rather than on the rig: the bundle is baked into the cloud proxy image, so the CASES view appears once your cloud deployment is rolled onto the new image — a vizier/AE restart will not surface it, and neither will the rig. The three pieces have to line up:
Missing the cloud piece is the one that looks most like a bug — the data is correct in ClickHouse, the panel simply is not in the UI's script list. Which matches what you are seeing now. Digest and the run result follow when it finishes; cloud builds take longer than vizier ones. |
|
✅ Cloud v0.0.46-pre-v0.0 green — run 32938859534, images published. (plus the rest of the cloud image set from the same tag — the proxy is the one that carries
All three pieces are now published:
Roll the cloud deployment onto the new proxy image and the CASES widget should appear in the Live UI script list; the rig side is already correct once |
conn_stats gains remote_pod (dx-resolved peer pod, non-key). dx_case_links joins an order's conn_stats.remote_pod to the culprit on that peer pod → directed from_culprit->to_culprit edge. Links the sink's exfil-receipt culprit back to the attacker's culprit that opened the connection (the one cross-pod under-match P0 left).
|
build |
|
BUILD-READY 15e733f TTL expires ~2026-08-27 12:20 UTC. Pairs with dx rc23 ( Recreate list is growing — DROP TABLE IF EXISTS forensic_db.conn_stats; -- new column: remote_pod
DROP VIEW IF EXISTS forensic_db.dx_ord__conn_stats; -- selects from it
DROP VIEW IF EXISTS forensic_db.dx_case_links; -- if a stale one existsUnlike This is the sixth build in a row gated on a manual drop, and now one that costs data. The |
|
✅ Both durables green — the cross-pod case bridge is published. AE aeprod88 (run 32940508745): dx rc23 (run 32940504721): Current pair: AE aeprod88 + dx rc23. Before restarting on an existing DROP TABLE IF EXISTS forensic_db.conn_stats; -- new column: remote_pod (LOSES stored evidence)
DROP VIEW IF EXISTS forensic_db.dx_ord__conn_stats;
DROP VIEW IF EXISTS forensic_db.dx_case_links;Ledgers — AE: 88 = current · 87 = culprit_key/dx_cases · 86 = mitre |
…hain_edges views Lands the three dashboard views in schema.sql so they exist on any cluster (they were rig-live only, breaking shipped scripts on fresh rigs). dx_alerts is the thin generic flatten of kubescape_logs; dx/breakout PxL now derives target/kind in PxL from it (story logic out of the DB). dx_breakout_story + dx_fullchain_edges kept for the shipped v0.0.45 dashboards. Registered in apply.go/ddl.go/apply_test.go. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011drju5k6V9dRaJnoiZLSLv
|
BUILD-REQUEST = aeprod86 + three new
All three are VIEWs over existing tables (kubescape_logs/conn_stats/dns_events/pgsql_events) — deferred-view path, no table changes. |
|
BUILD-READY c4357a6 — tagged aeprod89, not 87 (TTL first as asked; durable starting now) TTL expires ~2026-08-27 13:40 UTC — rig 6a8d99e4 unblocked, the cloud image's views will exist once this is up. On the number: 87 and 88 are both taken and published. You cut those an hour ago on The good news, which I checked before building rather than assuming: Recreate: all three are new views over existing tables, so nothing to drop for these — but if you have not yet done the aeprod88 step on 6a8d99e4, |
Fresh-rig reproducibility: the deploy pins were 8/12 builds stale (AE aeprod81,
dx rc11), so a fresh rig got none of the case-grouping / order-time-view /
cross-pod-bridge work. Bump AE 0.14.19-aeprod89 (culprit_key + dx_cases +
dx_case_links + story views) and dx rc23. Add DX_NARROW_WINDOW_{BEFORE,AFTER}_MS
=150: the server-side protocol capture (e.g. an exfil POST seen on the sink)
lands just AFTER the egress anomaly, so a backward-only window drops it.
2 participants
Stacked on #89 (dark-vector tables). Makes the dx-steered
OrderExportAll/OrderQuerycapture reliable on a single node-local PEM, and turns the AE bootstrap into a functional pem-direct DaemonSet. Validated e2e on a reproducible skaffold stack (soc-stack + bob redis-apps pixie-io#184 + this): kubescape → dx → AE,redis_events/dc_snoop/stack_trace/conn_stats/dns_eventscaptured, deduped via ReplacingMergeTree.Commits (each independent, tested):
QueryForbounds the source scan on both sides;OrderQuerywalks the window in sub-windows,captureSpansubdivides only on timeout. Default is one query/table (OrderChunk=600s) — pre-chunking every table 10x-amplified queries on the one PEM.px.logicalNot(px.contains(...))substring drop for kernel threads (kworker/…) that exact-match misses; workload comms (redis-*) kept.replicas:0Deployment never ran and couldn't do node-local pem-direct; replaced with the working config (EXPORT_MODE=never, control surface,MAX_INFLIGHT=4) + control Service.RCA + numbers: biz/PoC/OTel/RCA_ae_capture_20260803.md (internal).
Known follow-up: node-scoped tables (dc_snoop, dx_*) are re-pulled once per steered pod on a node, so
raw > FINALwhen multiple pods on a node are steered (RMT still dedups). Fix = per-(node,window) dedup of node-scoped pulls inOrderExportAll.