fix(ae): honour an intentionally tight /query window instead of widening to 600s - #101
Closed
ConstanzeTU wants to merge 92 commits into
Closed
fix(ae): honour an intentionally tight /query window instead of widening to 600s#101ConstanzeTU wants to merge 92 commits into
ConstanzeTU wants to merge 92 commits into
Conversation
…e fix) Root cause of the flaky dx-steered capture (dc_snoop/http erratically 0 while light tables always land): OrderExportAll fans out ~20 tables concurrently, each OrderQuery issued ONE unbounded PxL query over the full ~600s control window against the single node-local PEM (pem-direct). QueryFor only set start_time, so every query re-scanned [sliceStart, now] and post-filtered — the heavy tables materialize huge result sets on a saturated PEM and lose the fixed 180s deadline race, dropping out; the cheap tables (redis/conn/stack) return instantly and survive. Reconcile fingerprint: the same dc_snoop query returns 2459 rows in isolation but 0 + 1 err under the fan-out. Fix (durable — removes the data-volume↔deadline coupling, not just tunes it): - pxl.QueryFor: bound the PEM source scan on BOTH sides. Emit a relative end_time (floored toward now so nothing real is clipped; the exact upper bound stays enforced by the df.time_ < sliceEnd nanos post-filter) whenever sliceEnd is in the past. Live-edge slices keep scanning to now (no end_time), preserving prior behavior for the most-recent window. - controller.OrderQuery: walk the capture window in OrderChunk-sized sub-windows (default 60s, env ADAPTIVE_ORDER_CHUNK_SEC), each a both-sides bounded query, so no single query re-materializes the whole window. captureSpan adaptively halves any chunk that still fails with a transient (deadline/overload) error down to orderMinChunk (1s); non-transient errors (missing dark table) surface immediately without wasteful splitting. Overlapping/retried spans dedupe in the ReplacingMergeTree evidence tables, so re-pulls are idempotent. One aggregated reconcile row per table (not per chunk). Chunks run sequentially per table, so OrderExportAll's per-table concurrency is unchanged while each table now issues cheap bounded queries instead of one firehose — reliable capture without needing the global inflight throttle set. Tests: queryfor end_time present for past windows / absent at the live edge; OrderQuery chunking, single aggregated reconcile row, adaptive subdivision on transient error, no-split on non-transient error, termination at min-chunk.
… (dc_snoop) The dx-steered OrderExportAll path applied only a partial comm denylist and NO namespace filter to the node-scoped dark-vector tables — unlike the shipped cron preset (script/presets dc_snoop.pxl __DC_SNOOP_EXCLUSION__, built from presets.go defaultExcludeNamespaces + defaultExcludeComms). So every dc_snoop capture drowned in infra dcache churn: on a real k3s node a single window returned ~54k rows dominated by ConfigReloader/iptables/CNI(host-local,bridge,flannel,loopback)/host daemons(systemd-udevd,dbus-daemon,tailscaled)/kubevuln — burying the salient attack specimens (whoami/cat/getent reading /etc/shadow + the SA token). - Extend darkExcludeCommsDefault with the host/CNI/node daemons that were leaking (systemd-udevd, host-local, bridge, flannel, loopback, bandwidth, dbus-daemon, mount, umount, tailscaled, grpc_health_pro, kubevuln, opm, kube-proxy, …). - Add darkExcludeNamespacesDefault + darkNamespaceExclusion(), applied in the IsDarkVector branch AFTER PodEnrichPxL resolves df.namespace, dropping infra namespaces (pl, kube-system, clickhouse, …). Blank-namespace transient rows survive (each `!=` is true for ''), so the attack's short-lived children — which resolve blank — are never dropped. Overridable via DC_SNOOP_EXCLUDE_NAMESPACES. Kept in sync with script/presets.go. Tests: infra namespaces + host/CNI comms dropped; df.namespace never pinned to the alert pod (node-scoped); env override replaces the default list.
… depth cap) Live RCA on aeprod54: the chunk fix is correct in isolation (pem unit suite — dc_snoop 54k, redis/conn/stack written per-chunk) but UNSAFE under the dx steering firehose. dx does generic collect-per-alert, so OrderExportAll (20 tables) fires on every noisy pl system pod continuously; all land on the ONE node-local PEM (pem-direct) → it saturates → 100% DeadlineExceeded. captureSpan then split every timeout into two narrower retries, amplifying a busy PEM into a query storm where nothing completes (observed: "0 ordered pixie rows written" across the whole run; draining dx + restarting AE → pem-direct instantly serves again). Make subdivision safe: - Circuit-breaker: orderTimeoutStreak (atomic) counts CONSECUTIVE transient failures; any success resets it. Above orderBreakerTrip (8) captureSpan stops subdividing — a saturated PEM must not be flooded with retries. It still splits a genuinely-oversized window on a healthy PEM (the reset keeps that path live). - Depth cap: maxOrderSplitDepth (3) bounds one chunk to ≤2^3 leaf queries even if it keeps timing out (was ~64 splitting 60s→1s). Tests: a 10-chunk all-timeout window stays <60 queries (ungated ≈640); a single transient failure still recovers (breaker resets on success, no latch). NOTE (deployment, not code): the firehose root also needs dx steering scoped so it doesn't fire 20-table captures on every noisy pl/system-pod alert — tracked separately for dx-agent.
Live RCA (aeprod55): every dx-steered capture in the e2e returned 0 rows, and the reconcile showed why — all 36 ordered captures had ~512ns-wide windows (width_s=0), so they matched no pixie rows. /export/start already reaches back controlExportLookback, but a control client that keys the /query window on a single finding's event_time sends lo≈hi (a sub-microsecond span). That passes the lo<hi validation yet captures nothing. handleQuery now widens any window narrower than minControlQueryWindow (5s) to controlExportLookback ending at hi — a point-in-time referral still captures the evidence leading up to it. hi is preserved; comfortably-wide windows pass through unchanged. Isolated /query probes (proper windows) already proved the capture path works — dc_snoop 54k→16k filtered, redis/conn/stack per-chunk; this makes the dx-driven path robust to degenerate windows too. Tests: a 512ns window is widened to >=5s (hi preserved); a 120s window is untouched. NOTE (dx-agent): dx should send a real window (or use /export/start) rather than a point window per finding — tracked separately. This is the AE-side safety net.
The bootstrap manifest was a replicas:0 Deployment with minimal env (EXPORT_MODE= auto, no pem-direct, no throttle) — it never ran and could not do node-local pem-direct. Replace it with the working config that the e2e RCA validated: - DaemonSet (one-per-node) so each pod queries its OWN node's vizier-pem at HOST_IP:50305 (pem-direct: node-local, desync-immune). - dx-steered: EXPORT_MODE=never + CONTROL_ADDR=:9100 + the control Service (internalTrafficPolicy:Local so dx reaches its co-located AE). - PEM-protection: ADAPTIVE_MAX_INFLIGHT_QUERIES_GLOBAL=4 and ADAPTIVE_ORDER_CHUNK_SEC =600 (one query per table, no window pre-chunking) so the AE never saturates the single node-local PEM it shares with dx. See RCA_ae_capture_20260803. Secret still seeded per-cluster (unchanged).
…efault; trim comments - queryfor.go: add darkExcludeCommSubstrings (kworker/ksoftirqd/rcu_/… — kernel threads with variable suffixes exact-match misses) applied via px.logicalNot( px.contains); add pause + systemd-logind exact. Workload comms (redis-*) untouched. - controller.go: defaultOrderChunk 60s -> 600s (one query per table; pre-chunking 10x-amplified queries on the single node-local PEM). - Strip verbose comments across queryfor.go/controller.go/server.go + the AE manifest. Test: kernel-thread substrings dropped, workload comms kept, pause dropped.
Deploys the dx-daemon DaemonSet + Service into honey and mirrors the pl->honey secrets (jwt-signing-key, cluster-id, cloud-addr, api-key, clickhouse http-url) via a before-hook, replacing the hand-applied manifest used in the e2e. Deploy with: skaffold deploy -f k8s/vizier/dx/skaffold.yaml CH http-url defaults to the soc clickhouse Service; override with DX_CH_HTTP_URL.
Replaces the imperative seed-secret + patch-cloud-addr + sed-image +
kubectl-apply sequence with a single skaffold module:
skaffold deploy -f k8s/vizier/adaptive_export/skaffold.yaml
- kustomize overlay reuses bootstrap/adaptive_export_{role,deployment}
and pins the image via images: (ghcr aeprod tag) instead of sed.
- before-hook patches PL_CLOUD_ADDR :443 and seeds
pl-adaptive-export-secrets ONLY when PIXIE_API_KEY/PX_API_KEY is set,
never clobbering an existing secret with an empty key.
- LoadRestrictionsNone so the overlay can reuse the bootstrap manifests
in place (no duplication/drift).
Pairs with the dx-daemon skaffold (k8s/vizier/dx). Bump the AE image by
editing newTag in kustomization.yaml.
…aths
The AE/dx skaffold configs lived inside their overlay dirs with kustomize
paths: [.], which skaffold resolves against the shell CWD (repo root), not
the config-file dir -> 'unable to find kustomization.yaml in /.../pixie'.
Match the repo convention instead (skaffold/skaffold_vizier.yaml et al.):
skaffold configs live in skaffold/ and reference overlays by repo-root-
relative kustomize paths. Overlays stay in k8s/vizier/{adaptive_export,dx}.
skaffold deploy -f skaffold/skaffold_adaptive_export.yaml
skaffold deploy -f skaffold/skaffold_dx.yaml # run from repo root
- dx overlay gains a kustomization.yaml (was rawYaml).
- both validated with 'skaffold render' from repo root (image overrides +
RBAC/DaemonSet/Service resolve).
dx image 0.3.0-public3 -> 0.4.0-ssotforest-rc6 (forest scope + evidence_graph + isTableAbsent; broker no longer blinds the verdict). AE aeprod57 -> aeprod63 (upid + OOM firehose-collapse + px.any dark-export fixes). Add DX_FOREST_SCOPE, DX_PRECORRELATE_GRAPH, DX_EVIDENCE_GRAPH_CH so dx populates forensic_db.dx_evidence_graph. Verified live on a pemdq1 rig: dc_snoop + evidence_graph populate; DX_BENCH=pemdirect (already set) keeps dx off the shared broker so AE's export doesn't DeadlineExceed.
…nce_graph Root cause of 'attack fires but dx_evidence_graph stays empty': with DX_PRECORRELATE_GRAPH=1 the workup pulls the per-anomaly full-evidence set into memory and at the 1Gi limit dx is OOM-killed (exit 137) mid-workup, BEFORE writing the graph — then crash-loops, so no edges ever land. Reproduced on a pemdq1 rig: dx received the referral (comm=ls/sh rule=R0001) then died OOMKilled x4. 2Gi clears it (verified: graph 23->34, 32 malignant). Request 256Mi->512Mi. Operational note (not a manifest change): the vector->dx sink can wedge when the dx pod bounces (findings stop arriving, no referral) — bounce the node-01 vector pod after any dx redeploy. Also seen: transient node-01 PEM restart -> pemdirect 'connection refused' -> BLIND verdicts (edges still write via generic-malignant).
…flood Fresh-PG validation of the 2Gi memory fix surfaced a SECOND bug: the bobctl attacks/redis-oss.yaml kill-chain fires ~31 distinct comm/rule anomalies on redis-master-0, and dx with 4 concurrent workers SIGSEGVs (exit 139, no Go panic = hard crash in the concurrent workup/pemdirect path) — crash-loops, graph stays empty. Serializing workups (DX_WORKERS=1) eliminates it: restarts=0, graph 0->32 (30 malignant) on a fresh pemdq1 rig with the full kill-chain. The 2Gi fix (previous commit) handles OOM; this handles the concurrency crash. Root fix for the race (so >1 worker is safe) tracked separately.
…aid to 4 The DX_WORKERS=1 workaround is no longer needed: the crash was a nil qes.Timing deref in pxapi handleStats (fixed in rc8 via pixie@6422c0508782 + dx nil-rs guard + TriagePull recover), not a dx concurrency defect (race-detector floods clean). Restore the default 4 workers and the fixed image. Memory stays 2Gi (real precorrelate need).
Sync the entlein/dx#138 fixes into the skaffold-deployed lab manifest: - image rc8 -> optdbg2 (non-garble; rc8 predates the manifest+pushdown code and the garble release crashes exit 139 under load — fault 2, unresolved) - memory 2Gi -> 3Gi (fault 1: 1.3GB/round peak, OOM below) - DX_FOREST_PUSHDOWN=1 + depth 4 (fault 3: PxL lineage pushdown frees the PEM so AE exports dc_snoop under load; validated restarts=0 over 6+ rounds, dc_snoop 0->1777).
…ult 2 fixed) rc13 = garble -literals (dropped -tiny, the SIGSEGV cause). Obfuscated + survives the kill-chain (restarts=0). Replaces the non-garble optdbg2 debug tag.
… + metrics (#97) The trigger's strict forward-only high-water-mark on the content event_time could silently halt AE forever (F8/AE-9, loadtest E8): one far-future row jumped the cursor past all real data, and out-of-order / clock-skewed / restart-buried rows were dropped with no signal. Fix: - Bounded lookback (ADAPTIVE_TRIGGER_LOOKBACK_SEC, default 300; 0 = legacy strict HWM): each poll scans [watermark-lookback, inf) and a bounded insertion-ordered LRU of row fingerprints (dedup.go) makes re-seen rows exactly-once. Includes in-window paging (catchup floor) so backlogs wider than PollLimit still drain. - Wall-clock poison clamp (ADAPTIVE_TRIGGER_MAX_SKEW_SEC, default 3600): a normalized event_time past now+skew is emitted once but never advances the cursor; an already-poisoned persisted watermark is clamped at load, so E8 recovers with no manual ALTER TABLE + restart. - Metrics on the default prometheus registry (metrics.go), served via the shared services/metrics /metrics handler in cmd/main.go (AE_PPROF_ADDR mux + optional AE_METRICS_ADDR listener): ae_trigger_watermark_ns{table,hostname}, ae_trigger_below_watermark_total, ae_trigger_event_time_rejected_total. normalizeEventTimeNanos stays as the first line of defense; the monotonic happy path with LOOKBACK=0 is byte-identical to before (existing suite runs unchanged). New tests: late-arrival exactly-once, below-lookback bound, E8 poison non-halt + self-recovery, strict-mode regression, dedup LRU unit tests. Fixes #97
Flip the dx→AE control surface (:9100) to secure-by-default so the bearer
JWT + control payloads no longer cross the CNI in cleartext.
- TLS default-ON. Mounted /certs/server.{crt,key} (service-tls-certs) win;
else AE self-generates an ephemeral in-memory ECDSA P-256 self-signed cert
(1y, SAN localhost/127.0.0.1/::1/node) so TLS works with zero extra secrets.
Plaintext ONLY via explicit CONTROL_INSECURE=true (loud WARN).
- Auth default-ON whenever PL_JWT_SIGNING_KEY is present (drops the extra
CONTROL_REQUIRE_AUTH gate). No key + no CONTROL_INSECURE => fail-closed:
the control HTTP surface refuses to start; the rest of AE keeps running.
- CONTROL_TLS / CONTROL_REQUIRE_AUTH become deprecated no-ops (warn if set);
CONTROL_TLS_CERT/KEY kept as overrides; new CONTROL_INSECURE opt-out.
control/tls.go: TLSConfig(cert,key,hosts) + selfSignedCert + certToPEM helper.
control/tls_test.go: self-gen serves TLS /healthz, TLS rejects unauthenticated,
mounted-cert load path, plaintext opt-out path. BUILD.bazel srcs updated.
Manifests: AE deployment already mounts /certs + PL_JWT_SIGNING_KEY (no
CONTROL_TLS to drop) — added a secure-by-default note. dx-daemon
AE_CONTROL_ADDR http:// -> https:// (dx client skip-verifies).
Stacks on #92 (fix/ae-protocol-export-pxexport); does not touch #97 code.
…' error) %(taggerdate:raw) is empty for a lightweight release tag → create_manifest_update emits 'timestamp: ,' → jq syntax error → the vizier release-metadata step fails even though the image built + pushed. Fall back to the tagged commit's committer date.
Standalone GraphWidget bundle (no src/ui changes) rendering the dx evidence graph with drill-down: graph edges -> investigation manifest -> consulted raw forensic rows. Reads forensic_db in ClickHouse via px.DataFrame(clickhouse_dsn). - evidence_graph: severity-weighted pod->pod edges; px.Pod() stamps ST_POD_NAME so nodes deep-link to px/pod via the widget's built-in deepLinkURLFromSemanticType. - investigation_detail: manifest row(s), case_window bounds via px.pluck_int64. - consulted_rows: demo.md 'H3 dc_snoop reconstruction for the alert pod. - vis.json: Graph over evidence_graph (edgeWeightColumn=confidence, edgeColorColumn=max_severity, edgeHoverInfo=investigation_id/condition/criteria/ edge_kind), plus manifest + consulted-rows Table widgets; investigation_id var is the zoom. - README: 3-level zoom, load-into-UI steps, clickhouse_dsn feasibility (YES) + templated-read / hostname-partition / ns-start_time caveats. Static-validated only; needs live-UI validation on a cluster carrying forensic_db.
Enhance the existing widget in place (drop the parallel dx/evidence_graph bundle): - drill-able pod nodes (px.Pod -> ST_POD_NAME -> double-click deep-links to px/pod via the GraphWidget's built-in deepLinkURLFromSemanticType; NO src/ui change) - L2 investigation_detail: the manifest (verdict / case_window / evidence_hash / findings) - L3 consulted_rows: the raw forensic rows dx considered (demo.md §H reconstruction) - investigation_id vis variable = the zoom key; keeps the forensic_analyst read DSN. Needs live-UI validation on a cluster with populated dx_evidence_graph/manifest.
…orensic_db L3 consulted_rows referenced df.pod/namespace/container which exist in NO raw table (dc_snoop has only time_,pid,comm,t,file,hostname,event_time) -> the 'Column pod not found' compile error that killed the whole view. Project the event_time+hostname intersection (present in every raw_table), filter by hostname (host_filter, was pod_filter/PX_POD). raw_table default dc_snoop -> kubescape_logs: the px ClickHouseSourceNode only returns rows for UInt64 event_time; dc_snoop/redis_events/conn_stats are DateTime64 and read back 0 despite millions in CH. kubescape_logs (and the UInt64 dx_evidence_* graph/manifest tables) are the readable ones.
… not event_time/hostname The defensive event_time+hostname+investigation_id projection returned content-free rows (a nanosecond int + node name + blank) -- looked like random fields. Fix: consulted_rows pins to kubescape_logs (the one px-readable table) and px.plucks the real evidence from the RuntimeK8sDetails / RuntimeProcessDetails JSON columns: namespace/pod/container + comm/cmdline + RuleID + the alert message. Drops the raw_table var (dc_snoop/redis_events are DateTime64-unreadable anyway). Validated live: redis/redis-master-0 R0001/R0002/R0006/R0008/R0010/R0011 with real messages.
Re-point L1/L2 to the deterministic uniqueID join (dx rc15 carries the kubescape uniqueID into the manifest seed). Graph edges = subject_pod ->[process]-> target (process now surfaced: cat/ln/getent); consulted findings join on uniqueID, not the lossy RuleID@timestamp. Reads the dx_kubescape_anomalies + dx_anomaly_findings views (deduped).
… per anomaly dx's consulted findings reference benign background rows (proven: closest redis row is PING/CLIENT LIST, never the attack), so they can't surface the payload. The real command lives in kubescape_logs RuntimeProcessDetails.processTree.cmdline, keyed by uniqueID. Re-point consulted_findings at the dx_anomaly_findings view (now built from that process tree): rule / comm / parent / the actual cmdline / alert -- e.g. R0010 -> '/usr/bin/cat /etc/shadow', R1008 -> 'sh -c getent hosts xmr.pool.minergate.com'.
…ld -> rc18 L2 (consulted_records) now joins each finding to its actual record: exact (time_) join for pixie redis_events/conn_stats (rc18 makes finding.time_ == source.time_), row_identity content for dc_snoop/dns/process. Reveals the COMPLETE evidence set per anomaly -- attack (cat /etc/shadow, anomalous.dns.query, mnt_payload/drifted_bob) AND benign (PING, 127.0.0.1, proc/self/stat) -- the pre-correlation completeness guarantee. Bump k8s/vizier/dx/dx-daemon.yaml to dx rc18 (the per-row timestamp fix chain rc15-18).
PodEnrichPxL set namespace + pod on the native (socket_tracer) tables but
NOT hostname — only the stack_trace branch stamped it. So conn_stats /
http_events / dns_events / redis_events (and every protocol table) landed
in ClickHouse with an EMPTY hostname, even though hostname is the LEADING
ORDER BY column on all of them.
Consequences that this fixes:
* px reads of these tables filter WHERE hostname=<node>; empty hostname
matched nothing (the reads only worked via a join that sourced hostname
elsewhere).
* the pixie-io#136 order-UUID pre-correlation views could not expose a real
hostname without an order-JOIN, and that join blocked the (hostname,
event_time) primary-key pushdown (validated on rig 6a841cf7, CH 24.8).
Fix: PodEnrichPxL's native path also emits
df.hostname = px.upid_to_node_name(df.upid)
— the same UDF stack_trace already uses; valid on any upid-bearing table.
Dark-vector tables (raw pid, no upid) are unchanged; their node stamping
is a separate follow-up. Tests: statement-count oracles +1 line.
Two AE changes for the order-UUID pre-correlation dashboard: 1. dx_order_seeds table (schema.sql + KnownTables + OperatorOwnedTables): dx INSERTs one row per referral (evidence-loss fix — dx coalesces same-pod anomalies so most write no manifest). The dx_anomaly_orders view windows every uniqueID from this. ReplacingMergeTree ORDER BY (unique_id, rule_id), 30d TTL. 2. PodEnrichPxL dark-vector branch stamps hostname via px.upid_to_node_name on the same process_stats upid pod/ns already come from, so dc_snoop (and the other dark tables) carry hostname and become px-readable. Transient pids that miss process_stats resolve blank — same accepted limitation as pod/ns.
…line stack_diff window was event_time +/-300s (600s attack) with an unbounded ~6h baseline -> too wide + asymmetric. Now: ATTACK = [event_time-30s, event_time+30s] and a MATCHED 60s BASELINE immediately before it [event_time-90s, event_time-30s), computed as Int64 offsets from dx_orders_win.lo (no float division, so it compares against px.time_to_int64 row_time). delta = attack - baseline is now like-for-like. px-verified on a recent order (real redis stacks). NOTE: only populated for attacks within Pixie profiler retention (~1h); older attacks have no native profiler stacks.
…od/ns-minus-stat)
…megraph The stack_trace table and the differential flamegraph both pulled the native profiler across all pods (~1.5s each). The table showed unreadable raw folded stacks; the flamegraph supersedes it. Removing it halves the profiler cost. Flamegraph height 7->4 (little content at +/-30s).
…ble back Perf comparison across all prior versions (px, fresh data): dc_snoop _ord 7.7s vs _bridge 3.8s; CH stacks empty vs native stacks working (303/12 rows). No single prior version was both fast and functional. This is the measured best: _bridge for dc_snoop (needs dx_base__dc_snoop passthrough), native profiler for both stack panels (table + ±30s diff), order-centric MITRE graph, deep-links. DDL to bake: dx_base__dc_snoop, dx_kubescape_mitre (LIMIT 1 BY uniqueID,rule), dx_src__kubescape_mitre (+pid/ppid).
… pid/ppid + dx_base__dc_snoop Bakes the three rig-only DDL the dx/evidence_graph cloud script needs: dx_kubescape_mitre LIMIT 1 BY uniqueID,rule (graph edges); pid/ppid on dx_src__kubescape_mitre (kubescape panel); dx_base__dc_snoop passthrough (dc_snoop bridge fast path). Additive; px/dx_evidence_graph unaffected.
The view was added to schema.sql but not to KnownTables/OperatorOwnedTables, and Apply only iterates OperatorOwnedTables (apply.go:150) — so the dc_snoop panel's fast bridge path would have found no dx_base__dc_snoop on any cluster, fresh or upgraded, with nothing in the logs to say why. Every sibling view (dx_src__*, dx_ord__*, the MITRE trio) is registered in both lists; this one was missed. Tail guard extended to match.
The _bridge dc_snoop read dx_base__dc_snoop (raw SELECT * FROM dc_snoop), whose hostname column is empty. px shards ClickHouse reads by the PEM hostname, so an empty hostname reads as 0 rows -> the dc_snoop panel was silently empty on real data. dx_ord__dc_snoop inherits a real hostname from the order/edge join and returns the full row set (~4s, 2946 rows for a redis order). Drop the now-dead _bridge helper.
…) so raw table is px-readable
… 0.5.0-keepset-rc8 (dc_snoop fullpath collapse)
The ClickHouse source paginated with LIMIT/OFFSET at batch_size 1024, so a 2.66M-row forensic fan-out view (e.g. dx_ord__dc_snoop) took ~2600 pages and OFFSET's O(N) per-page skip made deep pages seconds each — the views were effectively unusable under load. Replace OFFSET with keyset pagination on the ORDER BY timestamp column and raise batch_size to 131072. Each page seeks via WHERE <ts> >= cursor instead of skipping. Correctness details: - Peek one row past the batch (LIMIT batch+1) so a full batch is emitted whole when the boundary falls between distinct timestamps. - When a timestamp group straddles the boundary, defer the entire trailing group to the next page; a group is never split, so ClickHouse's unstable intra-group ordering across queries cannot skip or duplicate rows. - The cursor column need not be in the projection: it is appended to the SELECT and stripped before the batch is emitted. - Falls back to OFFSET when there is no timestamp column, the query has its own ORDER BY, or the column type is not an integer/DateTime.
This reverts commit 9fe3154.
New Live-UI script (separate graph + order_id dropdown). Reconstructs the DNS activity in an order's alert window: querier -> resolver:53 edges plus the answer tree (name -> CNAME, name -> A). Reads a CH exploding view (dx_dns_resolve: ARRAY JOIN over resp_body answers), time-windowed via dx_orders_win.lo/hi (not edge-linked -- the resolution runs on coredns / cluster-DNS, not the attacked pod). Renders the C2/miner resolution chain (e.g. xmr.pool.minergate.com -> pool.minergate.com -> 49.12.80.x). Requires the forensic_db.dx_dns_resolve view (AE schema).
Add forensic_db.dx_dns_resolve (DNS resolution edges exploded from
dns_events.resp_body via ARRAY JOIN) for the dx/dns_resolve UI panel.
Drop 9 views verified unused (no dx Go ref incl. PR#139, no pxl ref, no
view depends on them, never read/written per query_log):
dx_base__dc_snoop, dx_evidence_graph_malignant, and the 7
dx_src__{conn_stats,dc_snoop,dns_events,http_events,mysql_events,
pgsql_events,redis_events} views (UI reads dx_ord__* instead).
Kept: dx_kubescape_anomalies, dx_anomaly_orders (dx refs),
dx_src__kubescape_logs, dx_src__stack_trace, dx_ord__stack_trace.
schema.sql + KnownTables + OperatorOwnedTables + apply_test want-list
kept consistent; go test green.
Resolve query-edge endpoints to k8s identities so the DNS hops connect: remote_addr (e.g. 10.43.0.10) via px.ip_to_service_id and the coredns querier pod via px.pod_name_to_service_name both collapse to kube-system/kube-dns -> client -> kube-dns -> upstream chains through one node. External resolvers fall back to px.nslookup (100.100.100.100 -> magicdns, 77.42.3.29 -> firenode-eu-11). Add ts_ns (event_time nanoseconds) to the edges table + graph hover. Answer edges (CNAME/A) pass through unchanged. No AE/view change.
Add ts = toString(dns_events.event_time) to the dx_dns_resolve view (e.g. 2026-08-22 19:04:27.115852400) and surface it in the pxl/edges + graph hover instead of the raw int64 ns. event_time stays int64-ns as the connector cursor / window-filter key. Qualified dns_events.event_time so the int64 event_time alias doesn't shadow the raw DateTime64.
Tested live: ts renders human UTC ns (2026-08-24 12:40:57.450165144), event_time stays Int64 ns for the px connector cursor, view queryable, AE clean. NOTE: in-place upgrade needs DROP VIEW forensic_db.dx_dns_resolve first — AE's CREATE VIEW IF NOT EXISTS won't replace a changed view (fresh rigs unaffected).
…ing to 600s /query widened ANY window under 5s to controlExportLookback (600s). A deliberate +/-50ms span around an anomaly was therefore inflated 6000x, so caller-side narrowing could not work at all: a chatty protocol (pgsql/mysql) came back with tens of thousands of rows per referral, which no analyst can read. The original rationale — "a point window keyed on one finding's timestamp matches no pixie rows" — holds only for a DEGENERATE window. It is now known that kubescape's BaseRuntimeMetadata.timestamp IS the kernel event time (bpf_ktime_get_boot_ns, converted to wall clock exactly once, never re-stamped by the queue/dedup/exporter path) and that it reaches AE as nanos end-to-end, so a millisecond-wide span around an anomaly is meaningful evidence, not noise. Drop the floor to 1ms — still catches a sub-microsecond point window (which matches nothing) while honouring any intentional millisecond span. ADAPTIVE_MIN_QUERY_WINDOW_MS re-arms a larger floor per deployment. Tests: the existing 512ns-window widening test still passes; adds a regression test that a +/-50ms pgsql window reaches the runner unchanged.
…filter) The px ClickHouse connector forwards start_time as WHERE event_time >= <seconds>, but the dx_ord__* fan-out views flattened event_time to UInt64 nanoseconds, so the comparison (ns >= seconds) was always true — start_time was silently a no-op and the connector paged the ENTIRE fan-out view (e.g. 25.3M-row dx_ord__pgsql_events), OOMing ClickHouse. Keep event_time as the base DateTime64(9) in all 8 dx_ord__* views so the forwarded filter windows the pull to the query's start_time. Verified in CH: DateTime64 >= seconds-int returns only in-window rows (500/1000), UInt64-ns returns all (1000/1000). _ord pxl drops event_time, so no pxl change; row_time stays int64-ns for any precise use.
…tables
Bridging (mirrors dx evidencegraph.UIDColsByTable):
- /dx/rows allowlist + bridgedPushSkip gain cql_events, mongodb_events and
creds_change, so dx-stamped rows are accepted and steer-all stops writing an
un-stamped second copy of them.
- schema.sql: those three tables gain `unique_id String DEFAULT ''` (the join key
the bridge views need) and get dx_ord__cql_events / dx_ord__mongodb_events /
dx_ord__creds_change. creds_change was previously reachable ONLY via steer-all,
with no order linkage at all.
Phantom tables removed (dx_vfs_events/dx_unlink/dx_dlookup/dx_mprotect/dx_bpf/
dx_ptrace): AE deploys dc_snoop + creds_change tracepoints only, so these six
never had a producer — every steer-all fan-out query on them failed PxL compile
("Table not found") on every referral. Dropped from schema.sql, apply.go, ddl.go
and the pxl builtins/dark-vector sets.
Test updates are contract-list adjustments, not weakened assertions:
- darkVectorTables / builtinTables count / dark-vs-native split drop the removed
names.
- TestCaptureSpanDoesNotSplitNonTransient keyed its "non-transient error" case on
dx_bpf; with the table gone OrderQuery rejects it before the querier runs, so
nothing subdivided. Re-pointed at a known table with a non-transient error —
the invariant under test is the error class, not the table.
NOTE for in-place upgrades: AE uses CREATE TABLE/VIEW IF NOT EXISTS and does not
re-run schema apply on restart, so an existing forensic_db needs the three
unique_id columns ALTERed in and the new views created manually. Fresh rigs are
unaffected.
dx_ord__{cql_events,mongodb_events,creds_change} were added to schema.sql but
not to KnownTables/OperatorOwnedTables, and Apply only iterates
OperatorOwnedTables — so the three views would never exist on any cluster and
the cql/mongodb/creds_change bridge would silently produce nothing, exactly as
dx_base__dc_snoop did on aeprod77. Tail guard extended.
An object declared in schema.sql but absent from OperatorOwnedTables is never
created — Apply iterates that list — and absent from KnownTables it cannot be
resolved by DDL/Columns. Both failures are silent: no boot error, no log line,
just panels that return nothing. It has bitten twice (dx_base__dc_snoop on
aeprod77, dx_ord__{cql,mongodb,creds_change} on aeprod83), caught on a rig both
times.
TestEverySchemaObjectIsRegistered parses every
'CREATE TABLE|VIEW IF NOT EXISTS forensic_db.<name>' out of the embedded
schema and requires it in KnownTables, and in OperatorOwnedTables unless it is
soc-owned (alerts, kubescape_logs — which it also asserts stay OUT, tying the
ownership boundary to the same source of truth).
TestEveryRegisteredNameHasDDL pins the reverse drift: a listed name with no
CREATE behind it, which today only surfaces when DDL() is called for it.
Verified the guard actually guards: removing dx_ord__creds_change from
OperatorOwnedTables reproduces exactly the aeprod83 bug and the test fails with
that name; restored, it passes. gazelle-registered in BUILD.bazel so it runs
under bazel too.
pgsql/mysql/cql/mongodb ordered on (hostname,event_time) alone: 1.19M rows shared 280842 unique_ids, so unique_id was not row-unique and the dx_ord__ join fanned out. redis/http/dns lacked the body columns for the same reason. ORDER BY now mirrors evidencegraph.UIDColsByTable exactly. Existing forensic_db needs table recreation to pick up the new sort keys.
…ow's The 11 dx_ord__ views projected c.event_time (the evidence row's kernel CAPTURE time). The px connector windows on event_time, and aeprod82 made that filter real (DateTime64, not the old ns-vs-seconds no-op). Evidence always precedes its alert, so a window around a fresh anomaly dropped its own (older) evidence — all panels empty. Project the ORDER's time instead: fromUnixTimestamp64Nano(toInt64(e.event_time)) (the edge carries ref.T, populated non-zero). ts/row_time keep the row's real capture time for display/ordering. Now a window around the anomaly returns its evidence regardless of when the packet/syscall was captured. Validated live: R0011 DNS-egress order, -15m window -> conn_stats 10, dns 34, dc_snoop 495 (was 0/0/0). In-place upgrade: DROP the 11 dx_ord__ views before apply (CREATE IF NOT EXISTS won't replace); fresh rigs unaffected.
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
/querywidened any window narrower than 5 s tocontrolExportLookback(600 s):So a deliberate ±50 ms span around an anomaly was inflated 6000×. Caller-side narrowing was impossible — a chatty wire protocol (pgsql/mysql) returned tens of thousands of rows per referral, which no analyst can read.
Why the original rationale no longer applies
The comment justified it as "a point window keyed on one finding's timestamp matches no pixie rows." That is true only for a degenerate window.
A node-agent code review established that
BaseRuntimeMetadata.timestampis the kernel event time: every gadget stampsbpf_ktime_get_boot_nsin the eBPF program, it is converted boot→wall exactly once, and the ordered event queue / enrichment / dedup / exporters add delivery lag only — they never re-stamp the value. It reaches AE as unix nanos end-to-end (kubescape_logs.event_timeUInt64 ns → dx referral int64 ns → control API ns).So a millisecond-wide span around an anomaly is meaningful evidence, not noise. Residual error is the boot→wall offset sampling (sub-ms) plus cross-clock skew against Pixie's own estimate — comfortably inside ±50 ms.
Change
Widen only a degenerate window; drop the floor from 5 s to 1 ms, which still catches a sub-microsecond point window (matches nothing) while honouring any intentional millisecond span.
ADAPTIVE_MIN_QUERY_WINDOW_MSre-arms a larger floor per deployment.Tests
TestQueryWidensNarrowWindow(512 ns window) still passes — a true point window is still widened.TestQueryHonoursIntentionalTightWindow: a ±50 ms pgsql window reaches the runner unchanged,lopreserved.internal/controlpackage green. (internal/e2e TestLoad_DataPlaneExactReproducible_L1fails on this branch and on the base commit — pre-existing, unrelated.)Scope / follow-up
This unblocks the
/querypath only./export/start(steer-all) cannot be narrowed today — it receives justt_end, no anomaly anchor, so it can only reach back a fixedcontrolExportLookback. Narrowing that path needs the anchor added to the request; deliberately left out of this PR.Companion change in dx pins pgsql/mysql collection to ±50 ms (
DX_NARROW_WINDOW_MS).