From 5265d0ed3ebe244e8822ec342f9fdd0e10605ec9 Mon Sep 17 00:00:00 2001 From: wankai123 Date: Wed, 2 Sep 2026 17:05:35 +0800 Subject: [PATCH 1/2] Add BanyanDB trace tail sampling monitoring Add otel-rules/banyandb/banyandb-trace-sampling.yaml, 41 rules covering the banyandb_trace_pipeline_* / banyandb_trace_tst_pipeline_* catalog a sampler plugin chain emits. Modeled at Service scope with `group` kept as a metric label rather than at Endpoint scope, so one cluster-wide page can render per-group series and cluster totals alike. Add the e2e case test/e2e-v2/cases/banyandb/trace-sampling, registered in the e2e-test matrix. It runs a real sw-trace-sampler chain on the plugin-capable BanyanDB image and asserts the 17 rules a healthy, fully-dropping run can produce; the other 24 cover failure counters, the keep verdict, FINALIZE-only state and back-pressure guards that cannot occur in a short healthy run. The existing BanyanDB self-observability case cannot cover any of this: its cluster runs no sampler plugins, so the wire families never register. Fix a cross-rule CounterWindow key collision in the v2 MAL engine. rate() / increase() / irate() resolved their lower bound from a process-wide window keyed on the counter name plus its post-.sum(...) label set, with nothing identifying the evaluating rule, so two rules reading one wire family and dropping the label they filtered on collapsed onto one slot and differenced against each other's values. The window is now keyed by owning rule as well. Auditing the shipped rules gives 10 colliding keys over ~25 rules. Fix meter_rabbitmq_node_outgoing_messages_total double-counting rabbitmq_global_messages_delivered_get_auto_ack_total, which appeared twice among the six summed delivery terms. --- .github/workflows/skywalking.yaml | 6 + docs/en/banyandb/dashboards-banyandb.md | 130 +++++++++- docs/en/banyandb/tail-sampling.md | 7 + docs/en/changes/changes.md | 5 + .../rabbitmq/rabbitmq-node.data.yaml | 2 +- .../oap/meter/analyzer/v2/dsl/Sample.java | 9 +- .../meter/analyzer/v2/dsl/SampleFamily.java | 3 + .../v2/dsl/counter/CounterWindow.java | 8 +- .../oap/meter/analyzer/v2/dsl/counter/ID.java | 9 + .../v2/dsl/CrossRuleCounterWindowTest.java | 124 ++++++++++ .../v2/dsl/counter/CounterWindowTest.java | 8 +- .../banyandb/banyandb-trace-sampling.yaml | 223 ++++++++++++++++++ .../otel-rules/rabbitmq/rabbitmq-node.yaml | 2 +- .../trace-sampling/docker-compose.yml | 137 +++++++++++ .../cases/banyandb/trace-sampling/e2e.yaml | 59 +++++ .../expected/active-samplers.yml | 59 +++++ .../expected/labels-group-lane-p.yml | 47 ++++ .../expected/labels-group-p.yml | 45 ++++ .../expected/labels-group-plugin-p.yml | 48 ++++ .../expected/labels-group-plugin-result.yml | 47 ++++ .../labels-group-plugin-rule-verdict.yml | 50 ++++ .../expected/labels-group-plugin-rule.yml | 48 ++++ .../expected/labels-group-plugin.yml | 45 ++++ .../expected/labels-group-result.yml | 46 ++++ .../trace-sampling/expected/labels-group.yml | 44 ++++ .../trace-sampling/otel-collector-config.yaml | 53 +++++ .../trace-sampling/trace-sampling-cases.yaml | 102 ++++++++ 27 files changed, 1350 insertions(+), 16 deletions(-) create mode 100644 oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/CrossRuleCounterWindowTest.java create mode 100644 oap-server/server-starter/src/main/resources/otel-rules/banyandb/banyandb-trace-sampling.yaml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/docker-compose.yml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/e2e.yaml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/expected/active-samplers.yml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-lane-p.yml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-p.yml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-p.yml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-result.yml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-rule-verdict.yml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-rule.yml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin.yml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-result.yml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group.yml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/otel-collector-config.yaml create mode 100644 test/e2e-v2/cases/banyandb/trace-sampling/trace-sampling-cases.yaml diff --git a/.github/workflows/skywalking.yaml b/.github/workflows/skywalking.yaml index df5785c71c23..c177cc086e3d 100644 --- a/.github/workflows/skywalking.yaml +++ b/.github/workflows/skywalking.yaml @@ -387,6 +387,12 @@ jobs: config: test/e2e-v2/cases/storage/banyandb/pipeline-plugin/zipkin-keep/e2e.yaml - name: BanyanDB monitoring config: test/e2e-v2/cases/banyandb/e2e.yaml + # The meter_banyandb_trace_sampling_* rules, which the case above cannot cover: its + # cluster runs no sampler plugins, so the wire families never register. Needs the + # BanyanDB "-plugins" host and "-plugins-carrier" images, published only for the + # SW_BANYANDB_COMMIT pinned in test/e2e-v2/script/env. + - name: BanyanDB trace sampling monitoring + config: test/e2e-v2/cases/banyandb/trace-sampling/e2e.yaml - name: Storage MySQL config: test/e2e-v2/cases/storage/mysql/e2e.yaml - name: Storage PostgreSQL diff --git a/docs/en/banyandb/dashboards-banyandb.md b/docs/en/banyandb/dashboards-banyandb.md index 9d5f9274785c..909bda3697fc 100644 --- a/docs/en/banyandb/dashboards-banyandb.md +++ b/docs/en/banyandb/dashboards-banyandb.md @@ -84,8 +84,8 @@ and renders it on the `Layer: BANYANDB` dashboards in the Horizon UI: The metric source expressions mirror the upstream BanyanDB Grafana boards, so the SkyWalking dashboards stay in lockstep with the BanyanDB catalog. The rule files are -`otel-rules/banyandb/banyandb-service.yaml`, `banyandb-instance.yaml`, `banyandb-endpoint.yaml` and -`banyandb-instance-relation.yaml`. +`otel-rules/banyandb/banyandb-service.yaml`, `banyandb-instance.yaml`, `banyandb-endpoint.yaml`, +`banyandb-instance-relation.yaml` and `banyandb-trace-sampling.yaml`. The instance and endpoint catalogs are **category-separated**: the rule name carries a role prefix (instance scope) or a data-type prefix (endpoint scope) so that a human can read a metric name and know @@ -307,6 +307,132 @@ Each edge kind carries the same four facets: > `lifecycle_last_run` / `lifecycle_last_run_success` / `lifecycle_migration_cycles`. The migration > *traffic* (throughput / latency / error / bytes) above is already per-edge. +### Trace tail sampling scope — sampler plugins (`meter_banyandb_trace_sampling_*`) + +BanyanDB 0.11+ can run an ordered chain of **sampler plugins** during trace merge and finalization, +dropping whole traces after their fragments have already landed — tail sampling inside the storage +tier. See [Trace Tail Sampling](tail-sampling.md) for how a trace is judged and how the chain is +configured from `bydb.yml`. + +These rules live in their own file, `otel-rules/banyandb/banyandb-trace-sampling.yaml`, mirroring the +upstream **[BanyanDB — Trace Sampling Plugins](https://github.com/apache/skywalking-banyandb/blob/main/docs/operation/grafana-fodc-trace-plugin.json)** +board, which upstream also ships separately from its two core boards. The reason is the same in both +places: the plugin chain is **optional**. On a cluster with no sampler configured none of the wire +families are registered and **every metric in this section stays absent** — that is the healthy +default, not a broken scrape. + +> **Service scope with `group` as a label.** Every family here is keyed on `group`, so the natural +> scope would be Endpoint (see [Endpoint scope](#endpoint-scope--per-group-meter_banyandb_endpoint_)). +> They are modeled at **Service** scope instead, with `group` kept as a metric *label*, because +> sampling is presented as one cluster-wide page: a labeled metric renders every group as a series +> and still gives cluster totals when the label is aggregated away, whereas an Endpoint-scope metric +> cannot be rolled back up to the cluster — OAP does no cross-scope rollup. Each metric keeps exactly +> the labels its upstream panel splits on, listed in the **Labels** column below. + +Two wire prefixes feed these rules, and the split is not cosmetic: plugin lifecycle, telemetry-host +safety, finalization and the plugin-published business metrics are `banyandb_trace_pipeline_*`, while +per-merge execution, batching and trace retention outcomes are `banyandb_trace_tst_pipeline_*` (the +trace time-series table is where merge runs). + +**Pipeline status and reconciliation:** + +| Unit | Metric | Labels | Description | +| ---- | ------ | ------ | ----------- | +| Count | `trace_sampling_active_samplers` | `group` | Sampler plugins registered for the group. `0` means nothing is filtering | +| percent | `trace_sampling_drop_ratio` | `group` | Committed drop ratio — dropped / evaluated traces | +| c/m | `trace_sampling_plugin_load_failures` | `group`, `name`, `reason` | Plugin load failures. Reconciliation fails open, so the chain keeps running without the plugin | +| c/m | `trace_sampling_register_rate` / `trace_sampling_update_rate` | `group`, `result` | Pipeline registrations / updates (`result` = `success` or `rejected`) | +| c/m | `trace_sampling_remove_rate` | `group` | Explicit pipeline removals | + +**Plugin execution and batching** (`data` containers): + +| Unit | Metric | Labels | Description | +| ---- | ------ | ------ | ----------- | +| o/s | `trace_sampling_plugin_execution_rate` | `group`, `plugin_name`, `result` | `Decide` calls/s. `result` = `success`, `decide_error`, `length_mismatch`, `panic` or `late` | +| ms | `trace_sampling_plugin_decide_latency_p99` | `group`, `plugin_name` | p99 `Decide` wall time, successful calls only | +| ms | `trace_sampling_plugin_decide_latency` | `group`, `plugin_name` | Mean `Decide` wall time, successful calls only | +| o/s | `trace_sampling_chain_batch_rate` | `group`, `result` | Chain batches/s. `result` = `success`, `timeout` or `circuit_open` | +| ms | `trace_sampling_plugin_time_per_trace` | `group` | Whole-chain plugin time per evaluated trace — the sampling overhead | +| Count | `trace_sampling_batch_size_p99` | `group` | p99 traces presented per successful chain batch | +| c/m | `trace_sampling_link_bypasses` | `group`, `plugin_name`, `reason` | Individual links bypassed after `decide_error`, `length_mismatch` or `panic` | + +> `trace_sampling_plugin_time_per_trace` is deliberately **not** split by `plugin_name`. The upstream +> panel divides a per-plugin numerator by a per-group denominator with PromQL's `on (group) +> group_left`; MAL arithmetic inner-joins on exact label equality and has no `group_left` equivalent, +> so the numerator collapses `plugin_name` and the metric reports the chain total. The per-plugin +> split is `trace_sampling_plugin_decide_latency`. + +**Sampling outcomes** — the trace-level denominators: + +| Unit | Metric | Labels | Description | +| ---- | ------ | ------ | ----------- | +| traces/s | `trace_sampling_traces_evaluated` | `group` | Traces evaluated by a completed, batch-aligned chain. The denominator for every ratio here | +| traces/s | `trace_sampling_traces_retained` | `group` | Traces retained by plugin verdict or by a safety fallback | +| traces/s | `trace_sampling_traces_dropped` | `group` | Traces removed from core and secondary-index output | +| traces/s | `trace_sampling_traces_immature` | `group` | Traces not yet eligible — their fragments are still inside the maturity boundary | +| o/s | `trace_sampling_sidx_pruned` | `group` | Secondary-index entries pruned after a confirmed drop | + +**Fail-open and bounded-retention safety.** Every metric below *retains* data that a sampler proposed +dropping, so a drop ratio under target is explained here before it is explained by plugin logic: + +| Unit | Metric | Labels | Description | +| ---- | ------ | ------ | ----------- | +| c/m | `trace_sampling_oversized_traces_bypassed` | `group` | Traces retained because their staged representation exceeded the per-trace budget | +| c/m | `trace_sampling_ambiguous` | `group` | Traces whose fragment state could not produce an unambiguous drop decision | +| c/m | `trace_sampling_plugin_errors` | `group`, `reason` | Chain-level errors that forced fail-open retention | +| o/s | `trace_sampling_guard_bloom_probes` | `group` | Bloom-filter probes checking for trace fragments outside the selected parts | +| c/m | `trace_sampling_guard_deferred` | `group` | Drops deferred because the fragment guard found a possible external fragment | +| c/m | `trace_sampling_guard_budget_exhausted` | `group` | Fragment-guard checks that exhausted their work budget | +| c/m | `trace_sampling_guard_publication_rejected` | `group` | Publications rejected after guard validation went stale | +| c/m | `trace_sampling_guard_lossless_retry` | `group` | Rejected publications retried without dropping traces | +| c/m | `trace_sampling_guard_bypassed` | `group` | Merges retained unsampled because the guard was unavailable or inapplicable | + +**Drop-set capacity and finalization.** The per-merge dropped-trace-ID set is memory-bounded; when it +hits the ceiling the merge retains the rest, which is what makes a drop ratio plateau under load: + +| Unit | Metric | Labels | Description | +| ---- | ------ | ------ | ----------- | +| c/m | `trace_sampling_capped_merges` | `group`, `lane` | Merges whose drop-ID set reached its ceiling | +| c/m | `trace_sampling_traces_retained_by_ceiling` | `group` | Proposed drops retained after the ceiling was reached | +| Count | `trace_sampling_drop_set_entries_p99` | `group`, `lane` | p99 dropped trace IDs held per merge — read against the budget for headroom | +| Bytes | `trace_sampling_drop_set_budget_bytes` | `group` | Resolved per-merge memory budget for confirmed dropped trace IDs | +| Count | `trace_sampling_finalize_rounds` | `group` | Highest finalization round count across the group's cooled shards | +| Status | `trace_sampling_finalize_terminal` | `group` | `1` when a cooled shard can run no further finalization round — no more tail sampling for that data | + +**First-party sampler decisions** (`sw-trace-sampler` / `zipkin-trace-sampler`): + +| Unit | Metric | Labels | Description | +| ---- | ------ | ------ | ----------- | +| o/s | `trace_sampling_sampler_decisions` | `group`, `plugin_name`, `verdict`, `rule` | Trace evaluations attributed to the first matching rule. `verdict` = `keep` or `drop` | +| rows/s | `trace_sampling_sampler_rows` | `group`, `plugin_name` | Span / segment rows carried by evaluated traces | +| rows/s | `trace_sampling_sampler_rows_dropped` | `group`, `plugin_name`, `rule` | Rows belonging to traces the sampler proposed dropping | +| percent | `trace_sampling_sampler_dropped_row_ratio` | `group`, `plugin_name` | Proposed dropped-row ratio | +| c/m | `trace_sampling_sampler_row_count_unavailable` | `group`, `plugin_name` | Evaluations whose metadata-only projection exposed no row count — these are missing from the row denominators | + +> These are **proposed**, not committed. A trace can be evaluated across several merge or finalization +> rounds, and the safety guards above can retain a trace the plugin voted to drop — so always read +> `trace_sampling_sampler_decisions{verdict=drop}` against `trace_sampling_traces_dropped`, and treat +> `trace_sampling_sampler_dropped_row_ratio` as incomplete while +> `trace_sampling_sampler_row_count_unavailable` is nonzero. `rule` is one of the fixed built-in values +> (`duration`, `error`, `healthy_sample`, `healthy_rejected`, `no_keep_rule`, `decode_failure_*`) or a +> positional `tag_00`–`tag_31` slot following `keepTagRules` order; tag keys and values are never +> copied into the label, which is what keeps the label set bounded. + +> **Adding a rule that divides two counters?** `rate()` / `increase()` keep their lower bound per +> (rule, wire family, label set). Two rules reducing one family to the same label set therefore no +> longer share a window — but check anyway that the operands of a ratio are the reductions you think +> they are, and verify the result against known inputs rather than trusting it by inspection. + +**Plugin telemetry host safety.** The host caps what a plugin may publish (100 label-value series per +instrument) and rate-limits its logs. Nonzero here means a plugin's *own* telemetry is being +truncated — its business metrics above are then incomplete, though the sampling itself is unaffected: + +| Unit | Metric | Labels | Description | +| ---- | ------ | ------ | ----------- | +| c/m | `trace_sampling_plugin_telemetry_series_rejected` | `group`, `plugin_name` | Series rejected by host safety bounds | +| c/m | `trace_sampling_plugin_log_dropped` | `group`, `plugin_name` | Plugin log lines dropped by the host rate limiter | +| c/m | `trace_sampling_plugin_telemetry_panic` | `group`, `plugin_name` | Panics recovered inside the plugin telemetry adapter | + ## Customizations You can customize your own metrics / expressions. The metric definitions and expression rules are in diff --git a/docs/en/banyandb/tail-sampling.md b/docs/en/banyandb/tail-sampling.md index 7a8e35cde417..a7191aa369f8 100644 --- a/docs/en/banyandb/tail-sampling.md +++ b/docs/en/banyandb/tail-sampling.md @@ -173,6 +173,13 @@ Watch these on the data node's own metrics endpoint: | `banyandb_trace_pipeline_sampler_load_failed{group,name,reason}` | a plugin was rejected — non-zero means the config is not doing what it says | | `banyandb_trace_pipeline_sampler_register_total{group,result}` | registration outcomes, including `rejected` | +The whole sampling catalog is also collected by the OAP itself when the BanyanDB self-observability +setup is in place — the same three above become `meter_banyandb_trace_sampling_active_samplers`, +`_plugin_load_failures` and `_register_rate`, alongside the drop ratio, per-plugin `Decide` latency, +the fail-open guard counters and the first-party samplers' own decision metrics. That is the better +place to operate from, because it shows what the plugins *proposed* next to what storage *committed*. +See [Trace tail sampling scope](dashboards-banyandb.md#trace-tail-sampling-scope--sampler-plugins-meter_banyandb_trace_sampling_). + **Expect a CPU cost on merge.** BanyanDB can normally copy a single-block trace through a merge as raw bytes without decoding it. A sampler that projects any tag column — which every useful configuration does — disables that fast path, so blocks are decoded in full during merges that diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index afd15f9713f4..d222c3e1eb70 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -8,11 +8,16 @@ * Support hot-reloading the BanyanDB TLS trust CA. The file at `sslTrustCAPath` is now watched, and a change rebuilds the gRPC channel so the new CA takes effect without restarting the OAP. Previously the CA was only re-read after the certificate in use had already caused requests to fail, which meant a rotation was paid for with an outage. The replacement channel is created before the old one is released, so a failed rebuild leaves the current channel serving; requests already in flight finish on the old channel. The replacement is not health-checked before the swap — a gRPC channel connects lazily — so rotating to a CA that does not validate the server interrupts traffic until valid material is written back; the storage doc describes the old+new overlap procedure that avoids this. Note the replacement re-picks an address from `targets`, so the OAP may connect to a different node after a rotation. * Fix `MultipleFilesChangeMonitor` being able to silently disable every file watch in the OAP. Its registry of monitors was a plain `ArrayList` that `scanChanges()` iterated from the scheduler thread without holding the lock that `start()` / `stop()` take, so starting a monitor while a scan was in flight could raise a `ConcurrentModificationException` from the iterator. That exception escapes past the per-monitor catch, and an uncaught exception cancels a `scheduleAtFixedRate` task permanently — after which no secrets file, keystore, or TLS certificate is ever reloaded again, with nothing in the log to say so. The registry is now copy-on-write. The failure log in the same scan loop also now names the monitor that failed instead of printing an empty `gourp = `. * Fix `MultipleFilesChangeMonitor` never honouring its watching period. `lastCheckTimestamp` was declared and compared against, but never assigned, so the guard always measured against `0` and passed — every registered monitor re-stat'd its watched files on each 200ms tick of the shared scheduler thread, and the `watchingPeriodInSec` constructor argument had no effect at all. This affects every file watch in the OAP: the ElasticSearch storage secrets / truststore / keystore watch, the BanyanDB credentials and trust CA watches, and the TLS certificate watches behind each OAP HTTP and gRPC server, all of which ask for 10 seconds. Change detection is now paced as configured, which also means it is no longer near-instant: a rotated file is picked up within the requested period rather than within ~200ms. +* Add BanyanDB trace tail sampling metrics to the BanyanDB self-observability layer, in a new `otel-rules/banyandb/banyandb-trace-sampling.yaml` rule file. It covers the whole `banyandb_trace_pipeline_*` / `banyandb_trace_tst_pipeline_*` catalog a sampler plugin chain emits — pipeline reconciliation, per-plugin `Decide` execution rate and latency, chain batching, the trace-level evaluated / retained / dropped / immature outcomes, every fail-open guard and bounded-retention counter, drop-set capacity and finalization state, the plugin telemetry-host safety bounds, and the first-party `sw-trace-sampler` / `zipkin-trace-sampler` decision and row metrics. The plugin chain is optional, and the metrics follow it: on a cluster with no sampler configured the wire families are never registered, so every metric here stays absent rather than reading zero. Modeled at Service scope with `group` kept as a metric label rather than at Endpoint scope, so one cluster-wide page can render per-group series and cluster totals alike — OAP does no cross-scope rollup, so an Endpoint-scope metric could not have been aggregated back up to the cluster. +* Fix a second `CounterWindow` key collision in the v2 MAL engine, this time ACROSS rules. `rate()` / `increase()` / `irate()` resolve their lower bound from a process-wide window keyed on the counter's own name plus its post-`.sum(...)` label set, with nothing identifying the rule doing the evaluation. Two rules that read one wire family, tell their streams apart with `tagEqual(...)`, and then `.sum(...)` away the label they filtered on therefore collapse onto one window slot and difference against each other's values. The queue is ordered by (timestamp, value), so the smaller counter wins the lower-bound lookup and still reads correctly while its partner is inflated by the gap between them — which is why this went unnoticed. A collision needs the discriminating label to be DROPPED by the `.sum(...)`: where it survives, the rules' label values differ and the window keeps them apart. Auditing the shipped rules on that basis gives 10 colliding keys over ~25 rules — `meter_activemq_cluster_gc_parallel_young_collection_count` reported ~9000/min of young-gen collections from a completely idle broker (differencing against the old-gen counter); MySQL `commands_*` / `tps` rate against each other; so do the GenAI gateway input/output token rates, four Envoy `cluster_*` counters, APISIX matched/unmatched instance bandwidth, and BanyanDB's own `network_recv` / `network_sent`, which drop the `kind` label that separates bytes-received from bytes-sent on one interface. Measured against two live scrapes of the demo cluster's FODC proxy, that last pair was wrong on every interface: `network_sent` read a flat 0 B/s and `network_recv` read large negative values (down to -778 MB/s) from differencing against the sent counter, where both now match the byte delta exactly. No rule changes were needed for any of these -- each rule already reduces to the labels it should; only the window key was wrong. The window is now keyed by (owning rule, counter name, labels). This is the complement of the within-rule collision fixed earlier by keying on the counter's own name: neither name alone is sufficient, because the two collisions are independent. `RunningContext.metricName` — written on every rule evaluation and read by nobody since that earlier fix — is what supplies the rule identity, so no code generation or MAL syntax changes. Note the whole-rule-set comparison suite could not have caught this: it resets the shared window before every rule, the one condition under which the collision cannot appear. +* Fix `meter_rabbitmq_node_outgoing_messages_total` double-counting one of its terms. The rule summed six delivery-rate terms but `rabbitmq_global_messages_delivered_get_auto_ack_total` appeared twice, so auto-ack `basic.get` deliveries were counted once more than the other four delivery paths and the reported outgoing rate ran high whenever polling consumers were in use. The duplicate term is removed, leaving the five distinct families (redelivered, consume auto/manual ack, get auto/manual ack). +* Add an e2e case for the BanyanDB trace tail sampling metrics, `test/e2e-v2/cases/banyandb/trace-sampling`. The existing BanyanDB self-observability case cannot cover them: its cluster runs no sampler plugins, and the wire families a sampler emits are registered on first observation, so they never appear. This case runs a real `sw-trace-sampler` chain on the plugin-capable (`-plugins`) BanyanDB image with the `.so` delivered by the `-plugins-carrier` image, scrapes the data node's own Prometheus endpoint through an OTel collector, and asserts the 17 `meter_banyandb_trace_sampling_*` rules that a healthy, fully-dropping run can produce. The remaining 24 rules in the file cover failure counters, the keep verdict, FINALIZE-only state and back-pressure guards that five traces in one shard never reach, and are deliberately not asserted: an empty result satisfies a presence check just as well when the expression is misspelled. #### UI * Add a Virtual GenAI evaluation-record page and evaluation-score chart in Horizon UI, so operators can inspect evaluation result, level, reason, judge model, timestamp, trace linkage, and the `gen_ai_model_evaluation_score_ppm` trend for evaluated records. #### Documentation +* Document the BanyanDB trace tail sampling metrics in the BanyanDB self-observability dashboard catalog, and point the "Operating it" section of the trace tail sampling guide at them — the OAP-collected metrics show what the sampler plugins *proposed* next to what storage *committed*, which the data node's raw metrics endpoint alone does not. All issues and pull requests are [here](https://github.com/apache/skywalking/issues?q=milestone:11.1.0) diff --git a/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/rabbitmq/rabbitmq-node.data.yaml b/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/rabbitmq/rabbitmq-node.data.yaml index 5df8e7738fb1..e598403dcebf 100644 --- a/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/rabbitmq/rabbitmq-node.data.yaml +++ b/oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/rabbitmq/rabbitmq-node.data.yaml @@ -203,7 +203,7 @@ expected: - labels: cluster: 'rabbitmq::test-cluster' node: test-node - value: 150.0 + value: 125.0 meter_rabbitmq_node_consumer_total: entities: - scope: SERVICE_INSTANCE diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/Sample.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/Sample.java index f22d837f77e6..aeea14859093 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/Sample.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/Sample.java @@ -46,14 +46,15 @@ Sample newValue(Function transform) { return toBuilder().value(transform.apply(value)).build(); } - Sample increase(String range, String metricName, Function2 transform) { - Tuple2 i = CounterWindow.INSTANCE.increase(metricName, labels, value, Duration.parse(range).toMillis(), timestamp); + Sample increase(String range, String owner, String metricName, Function2 transform) { + Tuple2 i = CounterWindow.INSTANCE.increase( + owner, metricName, labels, value, Duration.parse(range).toMillis(), timestamp); double nv = transform.apply(i._2, i._1); return newValue(ignored -> nv); } - Sample increase(String metricName, Function2 transform) { - Tuple2 i = CounterWindow.INSTANCE.pop(metricName, labels, value, timestamp); + Sample increase(String owner, String metricName, Function2 transform) { + Tuple2 i = CounterWindow.INSTANCE.pop(owner, metricName, labels, value, timestamp); double nv = transform.apply(i._2, i._1); return newValue(ignored -> nv); } diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/SampleFamily.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/SampleFamily.java index f07ccc8bf38c..98c5ce29e74c 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/SampleFamily.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/SampleFamily.java @@ -431,6 +431,7 @@ public SampleFamily increase(String range) { Arrays.stream(samples) .map(sample -> sample.increase( range, + this.context.getMetricName(), sample.getName(), (lowerBoundValue, unused) -> sample.value - lowerBoundValue )) @@ -448,6 +449,7 @@ public SampleFamily rate(String range) { Arrays.stream(samples) .map(sample -> sample.increase( range, + this.context.getMetricName(), sample.getName(), (lowerBoundValue, lowerBoundTime) -> { final long timeDiff = (sample.timestamp - lowerBoundTime) / 1000; @@ -466,6 +468,7 @@ public SampleFamily irate() { this.context, Arrays.stream(samples) .map(sample -> sample.increase( + this.context.getMetricName(), sample.getName(), (lowerBoundValue, lowerBoundTime) -> { final long timeDiff = (sample.timestamp - lowerBoundTime) / 1000; diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/CounterWindow.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/CounterWindow.java index bf64b7d68bac..824581de35b3 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/CounterWindow.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/CounterWindow.java @@ -45,8 +45,8 @@ public class CounterWindow { private final Map> lastElementMap = new ConcurrentHashMap<>(); private final Map>> windows = new ConcurrentHashMap<>(); - public Tuple2 increase(String name, ImmutableMap labels, Double value, long windowSize, long now) { - ID id = new ID(name, labels); + public Tuple2 increase(String owner, String name, ImmutableMap labels, Double value, long windowSize, long now) { + ID id = new ID(owner, name, labels); Queue> window = windows.computeIfAbsent(id, unused -> new PriorityQueue<>()); synchronized (window) { window.offer(Tuple.of(now, value)); @@ -71,8 +71,8 @@ public Tuple2 increase(String name, ImmutableMap l } } - public Tuple2 pop(String name, ImmutableMap labels, Double value, long now) { - ID id = new ID(name, labels); + public Tuple2 pop(String owner, String name, ImmutableMap labels, Double value, long now) { + ID id = new ID(owner, name, labels); Tuple2 element = Tuple.of(now, value); Tuple2 result = lastElementMap.put(id, element); diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/ID.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/ID.java index b9bb2340857b..ec819b61cf42 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/ID.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/ID.java @@ -28,6 +28,15 @@ @ToString class ID { + /** + * The rule that is evaluating, i.e. the output metric name of the MAL rule + * ({@code RunningContext.metricName}). Without it, two rules that reduce the SAME wire family + * to the SAME label set share one window and difference against each other's values; with it + * but without {@link #name}, several families inside ONE rule collide instead. The window is + * per (rule, family, labels) because those two collisions are independent. + */ + private final String owner; + private final String name; private final ImmutableMap labels; diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/CrossRuleCounterWindowTest.java b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/CrossRuleCounterWindowTest.java new file mode 100644 index 000000000000..e522200aa12a --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/CrossRuleCounterWindowTest.java @@ -0,0 +1,124 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.v2.dsl; + +import com.google.common.collect.ImmutableMap; +import java.util.HashMap; +import java.util.Map; +import org.apache.skywalking.oap.meter.analyzer.v2.dsl.counter.CounterWindow; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +/** + * The cross-rule half of the {@link CounterWindow} key, and the companion to + * {@link BanyanDBErrorRateReproTest}, which pins the within-rule half. + * + *

Two SHIPPED rules (otel-rules/activemq/activemq-cluster.yaml) read one wire family, + * {@code java_lang_GarbageCollector_CollectionCount}, tell the generations apart with + * {@code tagEqual('name', ...)}, and then {@code .sum(['cluster','service_instance_id'])} — which + * drops the very {@code name} label that distinguished them. Both therefore reduce to the same + * family plus the same label set, and keying the window on those alone puts them in one slot, + * where each differences against the other's value. Because the queue is ordered by + * (timestamp, value) the SMALLER counter wins {@code peek()} and reads a correct 0, so only its + * partner looks wrong — which is why this survived so long. + * + *

Note the whole-rule-set comparison suite cannot catch this: it calls + * {@code CounterWindow.INSTANCE.reset()} before every rule, so each one is evaluated in isolation, + * the one condition under which the collision cannot appear. + * + *

Counters are FROZEN across scrapes, so every increase must be exactly 0 for both rules. + */ +public class CrossRuleCounterWindowTest { + + private static final String GROUP_BY = "['cluster','service_instance_id']"; + + // Verbatim from otel-rules/activemq/activemq-cluster.yaml (value part). + private static final String OLD_GC = + "java_lang_GarbageCollector_CollectionCount.tagEqual('name','PS MarkSweep')" + + ".sum(" + GROUP_BY + ").increase(\"PT1M\")"; + private static final String YOUNG_GC = + "java_lang_GarbageCollector_CollectionCount.tagEqual('name','PS Scavenge')" + + ".sum(" + GROUP_BY + ").increase(\"PT1M\")"; + + // Deliberately far apart: a collision differences one against the other, so the error is the + // gap between them and cannot be mistaken for rounding. + private static final double OLD_GEN_VALUE = 10; + private static final double YOUNG_GEN_VALUE = 9000; + + @BeforeEach + void resetWindow() { + CounterWindow.INSTANCE.reset(); + } + + private static Sample s(final double value, final long ts, final String gcName) { + return Sample.builder() + .name("java_lang_GarbageCollector_CollectionCount") + .labels(ImmutableMap.of( + "cluster", "c1", "service_instance_id", "broker-1", "name", gcName)) + .value(value) + .timestamp(ts) + .build(); + } + + private static Map scrape(final long ts) { + final Map m = new HashMap<>(); + m.put("java_lang_GarbageCollector_CollectionCount", + SampleFamilyBuilder.newBuilder( + s(OLD_GEN_VALUE, ts, "PS MarkSweep"), + s(YOUNG_GEN_VALUE, ts, "PS Scavenge") + ).build()); + return m; + } + + private static double maxAbs(final Result r) { + double max = 0; + if (r.isSuccess() && r.getData() != SampleFamily.EMPTY) { + for (final Sample out : r.getData().samples) { + max = Math.max(max, Math.abs(out.getValue())); + } + } + return max; + } + + @Test + void frozenCounters_twoRulesOverOneFamily_bothMustBeZero() { + final Expression oldGc = DSL.parse( + "meter_activemq_cluster_gc_parallel_old_collection_count", OLD_GC); + final Expression youngGc = DSL.parse( + "meter_activemq_cluster_gc_parallel_young_collection_count", YOUNG_GC); + + final StringBuilder trace = new StringBuilder("\n"); + long ts = 1_700_000_000_000L; + for (int scrape = 0; scrape < 6; scrape++, ts += 10_000L) { + // Both rules see the SAME map instance, and BOTH run on this scrape before either + // sees the next one -- that is what MetricConvert.toMeter does, looping its analyzers + // per scrape. Do not "simplify" this to run one rule through every scrape and then the + // other: each rule would then drain its own entry before the other adds one, the + // collision would not occur, and this test would pass while asserting nothing. + final Map in = scrape(ts); + final double old = maxAbs(oldGc.run(in)); + final double young = maxAbs(youngGc.run(in)); + trace.append("scrape ").append(scrape) + .append(" old=").append(old).append(" young=").append(young).append('\n'); + assertEquals(0.0, old, 1e-9, "old-gen increase must be 0 on frozen counters" + trace); + assertEquals(0.0, young, 1e-9, "young-gen increase must be 0 on frozen counters" + trace); + } + } +} diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/CounterWindowTest.java b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/CounterWindowTest.java index 3e1a24f3541d..619afbf3e437 100644 --- a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/CounterWindowTest.java +++ b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/CounterWindowTest.java @@ -47,7 +47,7 @@ public static List> parameters() { public void testPT15S() { double[] actuals = parameters().stream().mapToDouble(e -> { Tuple2 increase = CounterWindow.INSTANCE.increase( - "test", ImmutableMap.builder().build(), e._2, + "test-rule", "test", ImmutableMap.builder().build(), e._2, Duration.parse("PT15S").getSeconds() * 1000, e._1 ); return e._2 - increase._2; @@ -60,7 +60,7 @@ public void testPT15S() { public void testPT35S() { double[] actuals = parameters().stream().mapToDouble(e -> { Tuple2 increase = CounterWindow.INSTANCE.increase( - "test", ImmutableMap.builder().build(), e._2, + "test-rule", "test", ImmutableMap.builder().build(), e._2, Duration.parse("PT35S").getSeconds() * 1000, e._1 ); return e._2 - increase._2; @@ -73,7 +73,7 @@ public void testPT35S() { public void testPT1M() { double[] actuals = parameters().stream().mapToDouble(e -> { Tuple2 increase = CounterWindow.INSTANCE.increase( - "test", ImmutableMap.builder().build(), e._2, + "test-rule", "test", ImmutableMap.builder().build(), e._2, Duration.parse("PT1M").getSeconds() * 1000, e._1 ); return e._2 - increase._2; @@ -86,7 +86,7 @@ public void testPT1M() { public void testPT2M() { double[] actuals = parameters().stream().mapToDouble(e -> { Tuple2 increase = CounterWindow.INSTANCE.increase( - "test", ImmutableMap.builder().build(), e._2, + "test-rule", "test", ImmutableMap.builder().build(), e._2, Duration.parse("PT2M").getSeconds() * 1000, e._1 ); return e._2 - increase._2; diff --git a/oap-server/server-starter/src/main/resources/otel-rules/banyandb/banyandb-trace-sampling.yaml b/oap-server/server-starter/src/main/resources/otel-rules/banyandb/banyandb-trace-sampling.yaml new file mode 100644 index 000000000000..d135fa49ecb6 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/otel-rules/banyandb/banyandb-trace-sampling.yaml @@ -0,0 +1,223 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# BanyanDB trace tail sampling (trace pipeline plugins), Service scope = one BanyanDB cluster. +# +# BanyanDB 0.11+ can run an ordered chain of sampler plugins during trace merge and +# finalization, dropping whole traces after their fragments have landed -- tail sampling +# inside the storage tier. The plugin chain is OPTIONAL: on a cluster with no sampler +# configured none of the families below are registered and every metric here stays absent. +# Source expressions mirror the upstream BanyanDB Grafana "Trace Sampling Plugins" board +# (docs/operation/grafana-fodc-trace-plugin.json), which upstream ships as a SEPARATE +# optional board for exactly that reason -- hence a separate rule file here rather than more +# rules bolted onto banyandb-endpoint.yaml. +# +# SERVICE scope with `group` kept as a LABEL. Every family below is keyed on `group`, so the +# natural SkyWalking scope would be Endpoint (a group is an Endpoint under the cluster +# Service -- see banyandb-endpoint.yaml). It is modeled at Service scope instead because the +# sampling dashboard is one cluster-wide page: `group` survives as a metric label rather than +# becoming the entity, so a single page can render every group as a series and still show +# cluster totals by aggregating the label away. Aggregating an Endpoint-scope metric back up +# to the cluster is NOT possible -- OAP does no cross-scope rollup -- so the label is the only +# modeling that serves a cluster page. Consequence to respect when adding rules: `group` (and +# every other retained label) MUST stay in the .sum()/.max() group-by, and the entity key is +# ['cluster'] alone. +# +# Label retention mirrors each upstream panel's `by (...)` clause -- `group` always, plus the +# one discriminator that panel splits on (`plugin_name`, `result`, `verdict`, `rule`, `reason`, +# `lane`, `name`). Keeping only those bounds the labeled-value cardinality: BanyanDB rejects a +# sampler configured with more than 32 tag rules, so `rule` is capped at 40 fixed values and +# never carries a tag key or value. +# +# Family-name traps (verified against the upstream metric catalog, +# docs/operation/observability/metrics.md "Trace Plugin Metrics"): +# - TWO prefixes, not one. Plugin lifecycle / telemetry-host / finalization / plugin-defined +# business metrics are banyandb_trace_pipeline_*; per-merge execution, batching and trace +# retention outcomes are banyandb_trace_tst_pipeline_* (the _tst_ infix -- the trace +# time-series table is where merge runs). Dropping or adding _tst_ silently yields no data. +# - the three histograms reach MAL as THREE sample families each: the bare name (buckets, +# carrying `le`), plus _sum and _count synthesized by PrometheusMetricConverter. That is +# what makes the mean-latency and per-trace rules below expressible without PromQL's +# `group_left`, which MAL has no equivalent of. +# - banyandb_trace_pipeline_sampler_load_failed carries `name` (the plugin that failed to +# load), NOT `plugin_name` -- a plugin that never loaded has no chain identity. +# - execution_duration_seconds is SECONDS on the wire; .histogram() rescales seconds buckets +# to the meter system's millisecond buckets, so the p99 is in ms. +# - batch_traces and drop_set_entries are histograms of COUNTS, not durations. They must use +# .histogram('le', TimeUnit.MILLISECONDS) so the scale factor is 1 and the bucket bounds +# stay raw trace / entry counts; the default .histogram() would multiply them by 1000. +filter: "{ tags -> tags.job_name == 'banyandb-monitoring' }" +expSuffix: service(['cluster'], Layer.BANYANDB) +metricPrefix: meter_banyandb +metricsRules: + # ---- Pipeline status ---- + # Registered sampler plugins per group. A per-node gauge replicated on every data node that + # holds a shard of the group, so .max() (NOT .sum(), which would multiply by node count). + - name: trace_sampling_active_samplers + exp: banyandb_trace_pipeline_sampler_active_count.max(['cluster', 'group']) + # Effective drop ratio (%) = dropped / evaluated. Both operands reduce to the identical + # ['cluster','group'] label set so MAL's inner join matches; safeDiv yields 0 while a group + # is idle. This is the COMMITTED ratio after BanyanDB's safety guards, which is why it can sit + # below what the samplers proposed -- compare against trace_sampling_sampler_decisions. + - name: trace_sampling_drop_ratio + exp: banyandb_trace_tst_pipeline_traces_dropped.sum(['cluster', 'group']).increase('PT1M').safeDiv(banyandb_trace_tst_pipeline_traces_evaluated.sum(['cluster', 'group']).increase('PT1M')) * 100 + # Plugin load failures (c/m), split by the failing plugin and reason. Reconciliation fails + # open: a plugin that will not load leaves the chain running without it. + - name: trace_sampling_plugin_load_failures + exp: banyandb_trace_pipeline_sampler_load_failed.sum(['cluster', 'group', 'name', 'reason']).rate('PT1M') * 60 + # Pipeline reconciliation (c/m). register/update carry result=success|rejected. + - name: trace_sampling_register_rate + exp: banyandb_trace_pipeline_sampler_register_total.sum(['cluster', 'group', 'result']).rate('PT1M') * 60 + - name: trace_sampling_update_rate + exp: banyandb_trace_pipeline_sampler_update_total.sum(['cluster', 'group', 'result']).rate('PT1M') * 60 + - name: trace_sampling_remove_rate + exp: banyandb_trace_pipeline_sampler_remove_total.sum(['cluster', 'group']).rate('PT1M') * 60 + + # ---- Plugin execution and batching ---- + # Decide() calls/s per plugin, split by result (success|decide_error|length_mismatch|panic|late). + # `late` means the plugin returned after the host had already abandoned its worker. + - name: trace_sampling_plugin_execution_rate + exp: banyandb_trace_tst_pipeline_plugin_executions.sum(['cluster', 'group', 'plugin_name', 'result']).rate('PT1M') + # p99 Decide() wall time (ms) per plugin, successful calls only -- an errored or panicking + # call's duration says nothing about the plugin's cost. `le` stays in the group-by for + # .histogram(); seconds buckets are rescaled to ms. + - name: trace_sampling_plugin_decide_latency_p99 + exp: banyandb_trace_tst_pipeline_plugin_execution_duration_seconds.tagEqual('result', 'success').sum(['le', 'cluster', 'group', 'plugin_name']).histogram().histogram_percentile([99]) + # Mean Decide() wall time (ms) per plugin = rate(_sum) / rate(_count) over the same window. + # Both are counters, so a WINDOWED mean needs rate() on both operands; dividing the raw + # cumulative pair would give an all-time average instead. + - name: trace_sampling_plugin_decide_latency + exp: banyandb_trace_tst_pipeline_plugin_execution_duration_seconds_sum.tagEqual('result', 'success').sum(['cluster', 'group', 'plugin_name']).rate('PT1M').safeDiv(banyandb_trace_tst_pipeline_plugin_execution_duration_seconds_count.tagEqual('result', 'success').sum(['cluster', 'group', 'plugin_name']).rate('PT1M')) * 1000 + # Chain batches/s, split by result (success|timeout|circuit_open). circuit_open means sampling + # is currently BYPASSED for the group. + - name: trace_sampling_chain_batch_rate + exp: banyandb_trace_tst_pipeline_plugin_batches.sum(['cluster', 'group', 'result']).rate('PT1M') + # Whole-chain plugin time per evaluated trace (ms) -- the sampling overhead per trace. + # DELIBERATELY NOT split by plugin_name: the upstream panel divides a per-plugin numerator by + # a per-group denominator with `on (group) group_left`, and MAL arithmetic inner-joins on exact + # label equality with no group_left equivalent. Collapsing plugin_name on the numerator makes + # both sides ['cluster','group'] and yields the chain total, which is the operational number + # anyway; the per-plugin split lives in trace_sampling_plugin_decide_latency. + - name: trace_sampling_plugin_time_per_trace + exp: banyandb_trace_tst_pipeline_plugin_execution_duration_seconds_sum.tagEqual('result', 'success').sum(['cluster', 'group']).rate('PT1M').safeDiv(banyandb_trace_tst_pipeline_traces_evaluated.sum(['cluster', 'group']).rate('PT1M')) * 1000 + # p99 traces presented per successful chain batch (Count, unitless). TimeUnit.MILLISECONDS + # pins the bucket scale factor to 1 -- these buckets are trace counts, not durations. + - name: trace_sampling_batch_size_p99 + exp: banyandb_trace_tst_pipeline_plugin_batch_traces.tagEqual('result', 'success').sum(['le', 'cluster', 'group']).histogram('le', TimeUnit.MILLISECONDS).histogram_percentile([99]) + + # ---- Sampling outcomes ---- + # The trace-level denominators and verdict outcomes (traces/s). evaluated is the denominator + # for every ratio here; retained + dropped account for evaluated traces, while immature traces + # were never eligible (their fragments are still inside the maturity boundary). + - name: trace_sampling_traces_evaluated + exp: banyandb_trace_tst_pipeline_traces_evaluated.sum(['cluster', 'group']).rate('PT1M') + - name: trace_sampling_traces_retained + exp: banyandb_trace_tst_pipeline_traces_retained.sum(['cluster', 'group']).rate('PT1M') + - name: trace_sampling_traces_dropped + exp: banyandb_trace_tst_pipeline_traces_dropped.sum(['cluster', 'group']).rate('PT1M') + - name: trace_sampling_traces_immature + exp: banyandb_trace_tst_pipeline_traces_immature.sum(['cluster', 'group']).rate('PT1M') + + # Fail-open and bounded-retention safety events (c/m). Every one of these RETAINS data that a + # sampler proposed dropping, so a drop ratio below target is explained here before it is + # explained by plugin logic. Read them alongside trace_sampling_drop_ratio. + - name: trace_sampling_oversized_traces_bypassed + exp: banyandb_trace_tst_pipeline_oversized_traces_bypassed.sum(['cluster', 'group']).rate('PT1M') * 60 + - name: trace_sampling_ambiguous + exp: banyandb_trace_tst_pipeline_ambiguous.sum(['cluster', 'group']).rate('PT1M') * 60 + # Chain-level errors that forced fail-open retention, split by reason. Distinct from + # trace_sampling_link_bypasses, which counts the per-LINK causes inside a chain that can + # still report result=success overall. + - name: trace_sampling_plugin_errors + exp: banyandb_trace_tst_pipeline_plugin_errors.sum(['cluster', 'group', 'reason']).rate('PT1M') * 60 + - name: trace_sampling_link_bypasses + exp: banyandb_trace_tst_pipeline_plugin_link_bypasses.sum(['cluster', 'group', 'plugin_name', 'reason']).rate('PT1M') * 60 + + # Fragment guard -- the check that a trace has no fragment outside the parts being merged. + # bloom_probes is its work rate (o/s); the rest are the outcomes that hold a drop back (c/m). + - name: trace_sampling_guard_bloom_probes + exp: banyandb_trace_tst_pipeline_guard_bloom_probes.sum(['cluster', 'group']).rate('PT1M') + - name: trace_sampling_guard_deferred + exp: banyandb_trace_tst_pipeline_guard_deferred.sum(['cluster', 'group']).rate('PT1M') * 60 + - name: trace_sampling_guard_budget_exhausted + exp: banyandb_trace_tst_pipeline_guard_budget_exhausted.sum(['cluster', 'group']).rate('PT1M') * 60 + - name: trace_sampling_guard_publication_rejected + exp: banyandb_trace_tst_pipeline_guard_publication_rejected.sum(['cluster', 'group']).rate('PT1M') * 60 + - name: trace_sampling_guard_lossless_retry + exp: banyandb_trace_tst_pipeline_guard_lossless_retry.sum(['cluster', 'group']).rate('PT1M') * 60 + - name: trace_sampling_guard_bypassed + exp: banyandb_trace_tst_pipeline_guard_bypassed.sum(['cluster', 'group']).rate('PT1M') * 60 + # Secondary-index entries pruned after a confirmed drop (o/s) -- the drop's index-side work. + - name: trace_sampling_sidx_pruned + exp: banyandb_trace_tst_pipeline_sidx_pruned.sum(['cluster', 'group']).rate('PT1M') + + # ---- First-party sampler plugin decisions (sw-trace-sampler / zipkin-trace-sampler) ---- + # These are plugin-published business metrics under the host's banyandb_trace_pipeline_plugin_ + # prefix, so they report what the sampler PROPOSED, not what storage committed -- a trace can + # be evaluated across several merge or finalization rounds, and the guards above can retain a + # trace the plugin voted to drop. Always read them against trace_sampling_traces_dropped. + # `rule` is a fixed built-in value or a positional tag_00..tag_31 slot; tag keys and values + # are never copied into the label, which is what keeps this bounded. + - name: trace_sampling_sampler_decisions + exp: banyandb_trace_pipeline_plugin_trace_sampler_decisions_total.sum(['cluster', 'group', 'plugin_name', 'verdict', 'rule']).rate('PT1M') + - name: trace_sampling_sampler_rows + exp: banyandb_trace_pipeline_plugin_trace_sampler_rows_total.sum(['cluster', 'group', 'plugin_name']).rate('PT1M') + - name: trace_sampling_sampler_rows_dropped + exp: banyandb_trace_pipeline_plugin_trace_sampler_rows_dropped_total.sum(['cluster', 'group', 'plugin_name', 'rule']).rate('PT1M') + # Proposed dropped-row ratio (%). `rule` is collapsed on the numerator so both operands reduce + # to ['cluster','group','plugin_name'] and MAL's inner join matches; the per-rule breakdown is + # trace_sampling_sampler_rows_dropped. Incomplete while row_count_unavailable is nonzero -- + # those traces are missing from the denominator. + # Dividing two independently-windowed counters is only sound because rate()/increase() keep + # their lower bound per (rule, family, labels): this denominator reduces to the same family and + # label set as trace_sampling_sampler_rows, and before the CounterWindow key carried the owning + # rule the two shared one window and this rule reported 34% where the truth was 28%. + - name: trace_sampling_sampler_dropped_row_ratio + exp: banyandb_trace_pipeline_plugin_trace_sampler_rows_dropped_total.sum(['cluster', 'group', 'plugin_name']).increase('PT1M').safeDiv(banyandb_trace_pipeline_plugin_trace_sampler_rows_total.sum(['cluster', 'group', 'plugin_name']).increase('PT1M')) * 100 + - name: trace_sampling_sampler_row_count_unavailable + exp: banyandb_trace_pipeline_plugin_trace_sampler_row_count_unavailable_total.sum(['cluster', 'group', 'plugin_name']).rate('PT1M') * 60 + + # ---- Drop-set capacity and finalization ---- + # The per-merge dropped-trace-ID set is memory-bounded. When it hits the ceiling the merge + # stops accepting drops and RETAINS the rest, so these three read together explain a drop + # ratio that plateaus under load. + - name: trace_sampling_capped_merges + exp: banyandb_trace_tst_pipeline_merges_ceiling_reached.sum(['cluster', 'group', 'lane']).rate('PT1M') * 60 + - name: trace_sampling_traces_retained_by_ceiling + exp: banyandb_trace_tst_pipeline_traces_retained_by_ceiling.sum(['cluster', 'group']).rate('PT1M') * 60 + # p99 dropped-IDs held per merge (Count, unitless -- TimeUnit.MILLISECONDS pins the scale to 1), + # read against the resolved byte budget to see the headroom before the ceiling bites. + - name: trace_sampling_drop_set_entries_p99 + exp: banyandb_trace_tst_pipeline_drop_set_entries.sum(['le', 'cluster', 'group', 'lane']).histogram('le', TimeUnit.MILLISECONDS).histogram_percentile([99]) + # Per-node gauges: .max() per group, same reason as trace_sampling_active_samplers. + - name: trace_sampling_drop_set_budget_bytes + exp: banyandb_trace_tst_pipeline_drop_set_budget_bytes.max(['cluster', 'group']) + # Finalization state. finalize_terminal is 1 when a cooled shard can run no further + # finalization round -- no more tail sampling will happen for that data. + - name: trace_sampling_finalize_rounds + exp: banyandb_trace_pipeline_finalize_rounds.max(['cluster', 'group']) + - name: trace_sampling_finalize_terminal + exp: banyandb_trace_pipeline_finalize_terminal.max(['cluster', 'group']) + + # ---- Plugin telemetry host safety (c/m) ---- + # The host caps what a HostAware plugin may publish (100 label-value series per instrument) + # and rate-limits its logs. Nonzero here means a plugin's own telemetry is being truncated, + # which makes the plugin's business metrics incomplete -- not the sampling itself. + - name: trace_sampling_plugin_telemetry_series_rejected + exp: banyandb_trace_pipeline_plugin_telemetry_series_rejected_total.sum(['cluster', 'group', 'plugin_name']).rate('PT1M') * 60 + - name: trace_sampling_plugin_log_dropped + exp: banyandb_trace_pipeline_plugin_log_dropped_total.sum(['cluster', 'group', 'plugin_name']).rate('PT1M') * 60 + - name: trace_sampling_plugin_telemetry_panic + exp: banyandb_trace_pipeline_plugin_telemetry_panic_total.sum(['cluster', 'group', 'plugin_name']).rate('PT1M') * 60 diff --git a/oap-server/server-starter/src/main/resources/otel-rules/rabbitmq/rabbitmq-node.yaml b/oap-server/server-starter/src/main/resources/otel-rules/rabbitmq/rabbitmq-node.yaml index c1edc7380d73..b176c0b3801e 100644 --- a/oap-server/server-starter/src/main/resources/otel-rules/rabbitmq/rabbitmq-node.yaml +++ b/oap-server/server-starter/src/main/resources/otel-rules/rabbitmq/rabbitmq-node.yaml @@ -45,7 +45,7 @@ metricsRules: - name: unacknowledged_messages exp: rabbitmq_queue_messages_unacked.sum(['cluster', 'node']) - name: outgoing_messages_total - exp: rabbitmq_global_messages_redelivered_total.sum(['cluster', 'node']).rate('PT1M') + rabbitmq_global_messages_delivered_consume_auto_ack_total.sum(['cluster', 'node']).rate('PT1M') + rabbitmq_global_messages_delivered_consume_manual_ack_total.sum(['cluster', 'node']).rate('PT1M') + rabbitmq_global_messages_delivered_get_auto_ack_total.sum(['cluster', 'node']).rate('PT1M') + rabbitmq_global_messages_delivered_get_auto_ack_total.sum(['cluster', 'node']).rate('PT1M') + rabbitmq_global_messages_delivered_get_manual_ack_total.sum(['cluster', 'node']).rate('PT1M') + exp: rabbitmq_global_messages_redelivered_total.sum(['cluster', 'node']).rate('PT1M') + rabbitmq_global_messages_delivered_consume_auto_ack_total.sum(['cluster', 'node']).rate('PT1M') + rabbitmq_global_messages_delivered_consume_manual_ack_total.sum(['cluster', 'node']).rate('PT1M') + rabbitmq_global_messages_delivered_get_auto_ack_total.sum(['cluster', 'node']).rate('PT1M') + rabbitmq_global_messages_delivered_get_manual_ack_total.sum(['cluster', 'node']).rate('PT1M') - name: consumer_total exp: rabbitmq_consumers.sum(['cluster', 'node']) - name: channel_total diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/docker-compose.yml b/test/e2e-v2/cases/banyandb/trace-sampling/docker-compose.yml new file mode 100644 index 000000000000..00305adcbdeb --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/docker-compose.yml @@ -0,0 +1,137 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Trace tail sampling self-observability e2e: the meter_banyandb_trace_sampling_* rule file +# (otel-rules/banyandb/banyandb-trace-sampling.yaml), which the parent case cannot cover because +# its cluster runs no sampler plugins. +# +# Sampling metrics only exist while a sampler plugin chain is actually executing, so this case +# needs the plugin-capable host, the sampler .so, trace data, and a filtering merge. The topology +# and its tuning are taken from cases/storage/banyandb/pipeline-plugin/sw (the behavioural case +# for the same feature); the addition here is the OTel collector that turns the data node's own +# :2121 Prometheus endpoint into OAP meters, and assertions on the resulting metrics rather than +# on which traces survived. +# +# It is a single `standalone` node, not the parent's 2-node cluster: the sampling rules are +# service-scoped on `cluster` alone, so a second node adds startup cost and no coverage. That +# also keeps the plugin host off the cluster discovery path, which matters because the distroless +# `-plugins` image cannot run the bydbctl healthcheck peers wait on. +services: + # Carrier init: the "-plugins-carrier" image ships the sampler .so at /plugins. Copy them into a + # shared volume and exit; banyandb waits for completion. Mirrors the initContainer + emptyDir + # delivery in test/plugin-sidecar (banyandb repo). + plugin-carrier: + image: "ghcr.io/apache/skywalking-banyandb:${SW_BANYANDB_COMMIT}-plugins-carrier" + command: sh -c "cp /plugins/*.so /shared/ && ls -l /shared" + volumes: + - plugins:/shared + networks: + - e2e + + # The plugin-capable (CGO / distroless-glibc) "-plugins" host, defined inline rather than via + # `extends` because base-compose's banyandb services use the "-testing" image (Alpine/musl, + # which cannot dlopen a Go plugin) and a bydbctl healthcheck this image has no binary for. + # No healthcheck and no published ports: the image has no shell, and infra-e2e's port-readiness + # probe execs /bin/sh inside the container. Nothing needs host access to the node here -- the + # collector scrapes :2121 over the compose network and every assertion goes through OAP -- so + # unlike the behavioural cases this needs no socat sidecar. + banyandb: + image: "ghcr.io/apache/skywalking-banyandb:${SW_BANYANDB_COMMIT}-plugins" + # max-merge-parts=2 + a short flush timeout make a filtering merge fire as soon as a second + # part lands, so the chain runs (and its metrics register) within the test's lifetime; the + # mocker writes 5 traces 30s apart, i.e. one part each. + command: standalone --measure-metadata-cache-wait-duration 1m --stream-metadata-cache-wait-duration 1m --trace-pipeline-native-plugin-enabled=true --trace-pipeline-trusted-plugin-dir=/plugins --trace-max-merge-parts=2 --trace-flush-timeout=500ms + volumes: + - plugins:/plugins + depends_on: + plugin-carrier: + condition: service_completed_successfully + networks: + - e2e + + # Scrapes the node's own Prometheus endpoint and injects the `cluster` identity label the + # service-scoped sampling rules key on (see otel-collector-config.yaml). + otel-collector: + image: otel/opentelemetry-collector:${OTEL_COLLECTOR_VERSION} + networks: + - e2e + command: [ "--config=/etc/otel-collector-config.yaml" ] + volumes: + - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml + expose: + - 55678 + depends_on: + banyandb: + condition: service_started + + oap: + extends: + file: ../../../script/docker-compose/base-compose.yml + service: oap + environment: + SW_STORAGE: banyandb + SW_STORAGE_BANYANDB_TARGETS: "banyandb:17912" + SW_STORAGE_BANYANDB_TRACE_PIPELINE_ENABLED: "true" + # One shard so every segment lands in the same shard: the in-merge filter runs per shard, and + # spreading a handful of segments over the default 2 shards can leave each with a single part, + # which never reaches max-merge-parts and so never merges. + SW_STORAGE_BANYANDB_TRACE_SHARD_NUM: "1" + # Make every mocked trace fail EVERY sure-keep rule, so the chain reports a full drop and the + # dropped/evaluated counters, the row counters and the drop-set all populate: + # - the mocked trace envelope is 6000ms, so a 60s threshold never fires; + # - every mocked trace carries an is_error span (ServiceBMock), so keepErrors must be off, + # otherwise all 5 are kept and the drop-side families never register; + # - healthySampleRate 0 removes the probabilistic keep, which would otherwise decide the + # drop/keep split by hash(trace_id) and make the assertions flaky run to run; + # - bydb.yml ships no keepTagRules. + # The keep side of the verdict (traces_retained) therefore cannot register in the same run -- + # the mocker's 5 traces are identical apart from their IDs, so no sampler config splits them + # deterministically. cases/storage/banyandb/pipeline-plugin/sw-keep covers the keep path. + SW_STORAGE_BANYANDB_TRACE_SAMPLER_DURATION_THRESHOLD_MS: "60000" + SW_STORAGE_BANYANDB_TRACE_SAMPLER_KEEP_ERRORS: "false" + SW_STORAGE_BANYANDB_TRACE_SAMPLER_HEALTHY_SAMPLE_RATE: "0" + # Keeps freshly written traces out of the filter briefly; past it all parts are permanently + # cold (isMergeHot is evaluated against the data timestamp, which AgentDataMock computes once + # before its write loop) while the remaining writes keep supplying merge triggers. + SW_STORAGE_BANYANDB_TRACE_PIPELINE_MERGE_GRACE_SECONDS: "20" + ports: + - 12800 + depends_on: + banyandb: + condition: service_started + + # Sends a fixed set of trace segments straight to OAP's gRPC receiver; the jar is built by CI's + # `mvnw -f test/e2e-v2/java-test-service/pom.xml package` step before the e2e runs. The traces + # are only fuel for the sampler here -- nothing asserts their content. MINUS_DAYS=0 keeps them in + # the hot stage, where merges run. + trace-mocker: + image: "eclipse-temurin:17-jre" + volumes: + - ../../../java-test-service/trace-mocker/target/trace-mocker-2.0.0.jar:/trace-mocker-2.0.0.jar + command: [ "java", "-jar", "/trace-mocker-2.0.0.jar" ] + environment: + OAP_HOST: oap + MINUS_DAYS: 0 + networks: + - e2e + depends_on: + oap: + condition: service_healthy + +volumes: + plugins: + +networks: + e2e: diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/e2e.yaml b/test/e2e-v2/cases/banyandb/trace-sampling/e2e.yaml new file mode 100644 index 000000000000..61b85e052508 --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/e2e.yaml @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Verifies the meter_banyandb_trace_sampling_* rule file end to end: a real sampler plugin chain +# runs inside the data node, its own :2121 Prometheus families are scraped into OAP as OTel +# metrics, and the MAL rules in otel-rules/banyandb/banyandb-trace-sampling.yaml turn them into +# queryable service-scoped meters. The behavioural half of the same feature -- which traces the +# sampler keeps or deletes -- is covered by cases/storage/banyandb/pipeline-plugin/*; this case +# asserts only that the metrics exist and carry values. + +setup: + env: compose + file: docker-compose.yml + timeout: 25m + init-system-environment: ../../../script/env + steps: + - name: set PATH + command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH + - name: install yq + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq + - name: install swctl + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl + +# No trigger: the trace-mocker service feeds OAP on its own, and the sampler runs off that. + +verify: + # Nothing can pass until the mocker has written enough parts for a merge to fire (it writes the + # first trace after 10s, then one every 30s), the segments have aged past the 20s merge grace, a + # filtering merge has run the chain, the collector has scraped the resulting families (5s), and + # OAP has aggregated them into a stored minute bucket. Budget accordingly. + retry: + count: 60 + interval: 10s + cases: + - includes: + - ./trace-sampling-cases.yaml + +cleanup: + on: always + collect: + on: failure + output-dir: $SW_INFRA_E2E_LOG_DIR/banyandb-trace-sampling + items: + - service: banyandb + paths: + - /tmp/trace/ + - /tmp/measure/ diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/expected/active-samplers.yml b/test/e2e-v2/cases/banyandb/trace-sampling/expected/active-samplers.yml new file mode 100644 index 000000000000..d7eb6e975b3c --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/expected/active-samplers.yml @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The gate metric, and the one place the VALUES are asserted exactly rather than with notEmpty: +# both are fixed by configuration, not by timing. sw_trace has exactly one sampler plugin declared in +# bydb.yml so it must reach 1, and sw_zipkinTrace has pipeline.enabled=false so it must stay 0 -- the +# family is registered for it either way, which is what makes 0 an assertion rather than an absence. +# +# Asserting both together is what gives this teeth: a chain accidentally attached to every trace group +# (or a rule that summed across groups instead of keeping `group` as a label) would report 1 for +# sw_zipkinTrace too, and a sampler that failed to load would report 0 for sw_trace. Both stay +# non-empty series in either case, so a presence-only check would pass through both faults. +# +# `contains` over the buckets means "some bucket carries this value", which is what is wanted: the +# earliest buckets in the query window predate the plugin loading and are null. +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: group + value: sw_trace + {{- end }} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: "1" + owner: null + traceid: null + {{- end }} + - metric: + labels: + {{- contains .metric.labels }} + - key: group + value: sw_zipkinTrace + {{- end }} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: "0" + owner: null + traceid: null + {{- end }} + {{- end }} +error: null diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-lane-p.yml b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-lane-p.yml new file mode 100644 index 000000000000..96ebed2b960e --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-lane-p.yml @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Drop-set occupancy percentile, split by the lane the entries were recorded in. lane=fast is the +# in-merge path (the FINALIZE sweep records into a different lane, and is not enabled here). +# Label keys and values are exact: every one of them is fixed by this case's setup -- the storage +# group names come from bydb.yml, plugin_name from the configured sampler, and rule / verdict / lane / +# result are the sampler's own vocabulary. Bucket ids and metric values use notEmpty because they are +# genuinely dynamic: the ids are epoch-millis and the values are rates, ratios and percentiles whose +# magnitude depends on where the merge burst falls relative to the minute boundary (a wire-true 100% +# drop was observed stored as 58 then 41 across adjacent buckets, and every rate decays to 0 once the +# mocker stops), so no fixed figure is correct for them. +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: group + value: sw_trace + - key: lane + value: fast + - key: p + value: "99" + {{- end }} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ notEmpty .value }} + owner: null + traceid: null + {{- end }} + {{- end }} +error: null diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-p.yml b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-p.yml new file mode 100644 index 000000000000..b3e24467da2f --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-p.yml @@ -0,0 +1,45 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Chain-level (not per-plugin) percentile: batch size in traces. Same `p` label as above; this rule +# pins its histogram unit explicitly because the family counts traces rather than measuring seconds. +# Label keys and values are exact: every one of them is fixed by this case's setup -- the storage +# group names come from bydb.yml, plugin_name from the configured sampler, and rule / verdict / lane / +# result are the sampler's own vocabulary. Bucket ids and metric values use notEmpty because they are +# genuinely dynamic: the ids are epoch-millis and the values are rates, ratios and percentiles whose +# magnitude depends on where the merge burst falls relative to the minute boundary (a wire-true 100% +# drop was observed stored as 58 then 41 across adjacent buckets, and every rate decays to 0 once the +# mocker stops), so no fixed figure is correct for them. +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: group + value: sw_trace + - key: p + value: "99" + {{- end }} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ notEmpty .value }} + owner: null + traceid: null + {{- end }} + {{- end }} +error: null diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-p.yml b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-p.yml new file mode 100644 index 000000000000..4bf53c8bb515 --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-p.yml @@ -0,0 +1,48 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# A histogram_percentile result: OAP splits the OTLP histogram into buckets + _sum + _count and the +# rule reassembles it, which adds the `p` label naming the percentile. p=99 is what the rule asks for, +# so asserting it catches a rule that silently computed a different percentile. +# Label keys and values are exact: every one of them is fixed by this case's setup -- the storage +# group names come from bydb.yml, plugin_name from the configured sampler, and rule / verdict / lane / +# result are the sampler's own vocabulary. Bucket ids and metric values use notEmpty because they are +# genuinely dynamic: the ids are epoch-millis and the values are rates, ratios and percentiles whose +# magnitude depends on where the merge burst falls relative to the minute boundary (a wire-true 100% +# drop was observed stored as 58 then 41 across adjacent buckets, and every rate decays to 0 once the +# mocker stops), so no fixed figure is correct for them. +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: group + value: sw_trace + - key: p + value: "99" + - key: plugin_name + value: sw-trace-sampler + {{- end }} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ notEmpty .value }} + owner: null + traceid: null + {{- end }} + {{- end }} +error: null diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-result.yml b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-result.yml new file mode 100644 index 000000000000..261fae56aa6b --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-result.yml @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Per-plugin execution outcome. success distinguishes a chain that ran cleanly from one that timed out +# or panicked, which the same rule would report under a different result. +# Label keys and values are exact: every one of them is fixed by this case's setup -- the storage +# group names come from bydb.yml, plugin_name from the configured sampler, and rule / verdict / lane / +# result are the sampler's own vocabulary. Bucket ids and metric values use notEmpty because they are +# genuinely dynamic: the ids are epoch-millis and the values are rates, ratios and percentiles whose +# magnitude depends on where the merge burst falls relative to the minute boundary (a wire-true 100% +# drop was observed stored as 58 then 41 across adjacent buckets, and every rate decays to 0 once the +# mocker stops), so no fixed figure is correct for them. +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: group + value: sw_trace + - key: plugin_name + value: sw-trace-sampler + - key: result + value: success + {{- end }} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ notEmpty .value }} + owner: null + traceid: null + {{- end }} + {{- end }} +error: null diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-rule-verdict.yml b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-rule-verdict.yml new file mode 100644 index 000000000000..7b5759660d14 --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-rule-verdict.yml @@ -0,0 +1,50 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The full decision key: which plugin, under which rule, reached which verdict. verdict=drop is the +# assertion that the sampler actually decided to delete rather than merely executing -- a keep-configured +# run reports verdict=keep from the same rule, so this value is the one that distinguishes them. +# Label keys and values are exact: every one of them is fixed by this case's setup -- the storage +# group names come from bydb.yml, plugin_name from the configured sampler, and rule / verdict / lane / +# result are the sampler's own vocabulary. Bucket ids and metric values use notEmpty because they are +# genuinely dynamic: the ids are epoch-millis and the values are rates, ratios and percentiles whose +# magnitude depends on where the merge burst falls relative to the minute boundary (a wire-true 100% +# drop was observed stored as 58 then 41 across adjacent buckets, and every rate decays to 0 once the +# mocker stops), so no fixed figure is correct for them. +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: group + value: sw_trace + - key: plugin_name + value: sw-trace-sampler + - key: rule + value: no_keep_rule + - key: verdict + value: drop + {{- end }} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ notEmpty .value }} + owner: null + traceid: null + {{- end }} + {{- end }} +error: null diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-rule.yml b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-rule.yml new file mode 100644 index 000000000000..8c124ea2715c --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-rule.yml @@ -0,0 +1,48 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Row-level drops attributed to the keep rule that decided them. no_keep_rule is the sampler's label +# for "no sure-keep rule matched", which is the only outcome this case's sampler config can produce: +# the duration threshold is above the mocked envelope, keepErrors is off and healthySampleRate is 0. +# Label keys and values are exact: every one of them is fixed by this case's setup -- the storage +# group names come from bydb.yml, plugin_name from the configured sampler, and rule / verdict / lane / +# result are the sampler's own vocabulary. Bucket ids and metric values use notEmpty because they are +# genuinely dynamic: the ids are epoch-millis and the values are rates, ratios and percentiles whose +# magnitude depends on where the merge burst falls relative to the minute boundary (a wire-true 100% +# drop was observed stored as 58 then 41 across adjacent buckets, and every rate decays to 0 once the +# mocker stops), so no fixed figure is correct for them. +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: group + value: sw_trace + - key: plugin_name + value: sw-trace-sampler + - key: rule + value: no_keep_rule + {{- end }} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ notEmpty .value }} + owner: null + traceid: null + {{- end }} + {{- end }} +error: null diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin.yml b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin.yml new file mode 100644 index 000000000000..864709590302 --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin.yml @@ -0,0 +1,45 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Group plus the plugin that produced the measurement. plugin_name is the name from the bydb.yml +# pipeline config, so it proves the metric is attributed to the sampler that actually ran. +# Label keys and values are exact: every one of them is fixed by this case's setup -- the storage +# group names come from bydb.yml, plugin_name from the configured sampler, and rule / verdict / lane / +# result are the sampler's own vocabulary. Bucket ids and metric values use notEmpty because they are +# genuinely dynamic: the ids are epoch-millis and the values are rates, ratios and percentiles whose +# magnitude depends on where the merge burst falls relative to the minute boundary (a wire-true 100% +# drop was observed stored as 58 then 41 across adjacent buckets, and every rate decays to 0 once the +# mocker stops), so no fixed figure is correct for them. +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: group + value: sw_trace + - key: plugin_name + value: sw-trace-sampler + {{- end }} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ notEmpty .value }} + owner: null + traceid: null + {{- end }} + {{- end }} +error: null diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-result.yml b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-result.yml new file mode 100644 index 000000000000..08de268c6d0a --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-result.yml @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Group plus the outcome of the operation being counted. success is asserted specifically: these two +# rules also expose failure/panic results, and a chain that was erroring would report a different +# result value while still producing a non-empty series. +# Label keys and values are exact: every one of them is fixed by this case's setup -- the storage +# group names come from bydb.yml, plugin_name from the configured sampler, and rule / verdict / lane / +# result are the sampler's own vocabulary. Bucket ids and metric values use notEmpty because they are +# genuinely dynamic: the ids are epoch-millis and the values are rates, ratios and percentiles whose +# magnitude depends on where the merge burst falls relative to the minute boundary (a wire-true 100% +# drop was observed stored as 58 then 41 across adjacent buckets, and every rate decays to 0 once the +# mocker stops), so no fixed figure is correct for them. +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: group + value: sw_trace + - key: result + value: success + {{- end }} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ notEmpty .value }} + owner: null + traceid: null + {{- end }} + {{- end }} +error: null diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group.yml b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group.yml new file mode 100644 index 000000000000..cb8be570bce2 --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group.yml @@ -0,0 +1,44 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Series keyed by storage group alone: the trace-level outcome counters, the per-trace plugin cost and +# the drop-set budget gauge. Only sw_trace has a sampler chain configured, so it is the only group that +# reports these at all. +# Label keys and values are exact: every one of them is fixed by this case's setup -- the storage +# group names come from bydb.yml, plugin_name from the configured sampler, and rule / verdict / lane / +# result are the sampler's own vocabulary. Bucket ids and metric values use notEmpty because they are +# genuinely dynamic: the ids are epoch-millis and the values are rates, ratios and percentiles whose +# magnitude depends on where the merge burst falls relative to the minute boundary (a wire-true 100% +# drop was observed stored as 58 then 41 across adjacent buckets, and every rate decays to 0 once the +# mocker stops), so no fixed figure is correct for them. +debuggingtrace: null +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: group + value: sw_trace + {{- end }} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ notEmpty .value }} + owner: null + traceid: null + {{- end }} + {{- end }} +error: null diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/otel-collector-config.yaml b/test/e2e-v2/cases/banyandb/trace-sampling/otel-collector-config.yaml new file mode 100644 index 000000000000..52ad40999a1d --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/otel-collector-config.yaml @@ -0,0 +1,53 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Trace-sampling rule file only (banyandb-trace-sampling.yaml), whose entity is +# service(['cluster']) -- so `cluster` is the only identity label the rules under test need. +# The SWIP-15 instance/endpoint identity labels (container_name / node_role / node_type / +# pod_name) are deliberately NOT injected here: this is a single `standalone` node, which runs +# the liaison and data roles in one process and so has no single honest node_role value +# (banyandb-database.proto has ROLE_META / ROLE_DATA / ROLE_LIAISON and no standalone member). +# The parent case (../otel-collector-config.yaml) exercises those rules on a real 2-node cluster +# where the values are real. `group` / `plugin_name` / `rule` / `verdict` / `lane` are not +# injected either -- they are emitted by the pipeline metrics themselves. +receivers: + prometheus: + config: + scrape_configs: + - job_name: "banyandb-monitoring" + scrape_interval: 5s + static_configs: + - targets: ["banyandb:2121"] + labels: + cluster: e2e-banyandb-sampling + +processors: + batch: + +exporters: + otlp: + endpoint: oap:11800 + tls: + insecure: true + +service: + pipelines: + metrics: + receivers: + - prometheus + processors: + - batch + exporters: + - otlp diff --git a/test/e2e-v2/cases/banyandb/trace-sampling/trace-sampling-cases.yaml b/test/e2e-v2/cases/banyandb/trace-sampling/trace-sampling-cases.yaml new file mode 100644 index 000000000000..b97372fe1cc7 --- /dev/null +++ b/test/e2e-v2/cases/banyandb/trace-sampling/trace-sampling-cases.yaml @@ -0,0 +1,102 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Trace tail sampling self-observability metrics, from +# oap-server/server-starter/src/main/resources/otel-rules/banyandb/banyandb-trace-sampling.yaml. +# Every rule is Service scope -- expSuffix is service(['cluster'], Layer.BANYANDB) -- so the +# entity is the `cluster` label the collector injects (e2e-banyandb-sampling) and `group` stays a +# metric label rather than becoming an Endpoint. That is why one query per rule covers every +# storage group at once. +# +# The 17 rules asserted here are the ones a HEALTHY, fully-dropping sampler run can produce. The +# other 24 in the file are absent by design, not broken, and are NOT asserted: +# - failure counters (plugin_load_failures, plugin_errors, ambiguous, sampler_row_count_ +# unavailable, plugin_telemetry_* , plugin_log_dropped) -- Prometheus registers a labeled +# counter on first observation, so nothing is exposed until something actually fails; +# - the keep verdict (traces_retained) -- see the sampler config in docker-compose.yml: the +# mocker's traces are identical apart from their IDs, so no config splits them deterministically +# and this run drops all of them. cases/storage/banyandb/pipeline-plugin/sw-keep covers keeps; +# - lifecycle churn (update_rate, remove_rate) -- the pipeline config is pushed once and never +# changed mid-run; +# - FINALIZE-only metrics (finalize_rounds, finalize_terminal) -- segments are daily, so a +# segment cannot settle inside a test, and enabledEvents here is MERGE; +# - the guard / ceiling / bypass families (guard_*, oversized_traces_bypassed, link_bypasses, +# sidx_pruned, capped_merges, traces_retained_by_ceiling, traces_immature) -- these are +# back-pressure and fail-open paths that 5 traces in one shard never reach. +# Asserting those would mean asserting an empty result, which passes just as well when the rule +# is misspelled. dashboards-banyandb.md documents the full catalog. +cases: + # ---- Sampler reconciliation: the chain is loaded and registered for the group ---- + # active_samplers is the gate for reading anything else on the page: 0 means nothing is + # filtering. It returns two series here (sw_trace = 1, sw_zipkinTrace = 0, the group whose + # pipeline is left disabled), which is also the per-group label proof. + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_active_samplers --service-name=e2e-banyandb-sampling + expected: expected/active-samplers.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_register_rate --service-name=e2e-banyandb-sampling + expected: expected/labels-group-result.yml + + # ---- Plugin execution: the chain ran, and what it cost ---- + # These come from the trace_tst_pipeline_plugin_* families, i.e. the telemetry host that wraps + # each plugin, so they prove the Decide() call actually executed rather than merely loaded. + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_plugin_execution_rate --service-name=e2e-banyandb-sampling + expected: expected/labels-group-plugin-result.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_plugin_decide_latency --service-name=e2e-banyandb-sampling + expected: expected/labels-group-plugin.yml + # The percentile rules are the histogram path: OAP splits the OTLP histogram into buckets + + # _sum + _count, and histogram().histogram_percentile([99]) reassembles it. A `p` label appears + # on the result (p=99) that no other rule has, so these also cover that shape. + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_plugin_decide_latency_p99 --service-name=e2e-banyandb-sampling + expected: expected/labels-group-plugin-p.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_chain_batch_rate --service-name=e2e-banyandb-sampling + expected: expected/labels-group-result.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_batch_size_p99 --service-name=e2e-banyandb-sampling + expected: expected/labels-group-p.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_plugin_time_per_trace --service-name=e2e-banyandb-sampling + expected: expected/labels-group.yml + + # ---- Trace-level outcomes: what the chain decided ---- + # evaluated is the denominator every other outcome is read against; dropped is the verdict this + # run is configured to produce. drop_ratio is their safeDiv -- the metric whose two increase() + # operands sharing one process-wide CounterWindow slot produced a wrong ratio before the window + # was keyed by owning rule as well as counter name. + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_traces_evaluated --service-name=e2e-banyandb-sampling + expected: expected/labels-group.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_traces_dropped --service-name=e2e-banyandb-sampling + expected: expected/labels-group.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_drop_ratio --service-name=e2e-banyandb-sampling + expected: expected/labels-group.yml + + # ---- The first-party sw-trace-sampler's own metrics (row granularity) ---- + # Emitted by the plugin itself through the telemetry host, not by the pipeline, so these prove + # the plugin's metric channel works, and they are keyed by the `rule` that decided + # (no_keep_rule here) and by `verdict` -- the labels the Sampler Decisions panel reads. + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_sampler_decisions --service-name=e2e-banyandb-sampling + expected: expected/labels-group-plugin-rule-verdict.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_sampler_rows --service-name=e2e-banyandb-sampling + expected: expected/labels-group-plugin.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_sampler_rows_dropped --service-name=e2e-banyandb-sampling + expected: expected/labels-group-plugin-rule.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_sampler_dropped_row_ratio --service-name=e2e-banyandb-sampling + expected: expected/labels-group-plugin.yml + + # ---- Drop-set capacity: the bounded retention structure the drops are recorded in ---- + # budget_bytes is a per-node gauge reduced with .max() (not .sum(), which would multiply the + # budget by the node count); entries_p99 is a COUNT histogram, so its rule pins the unit with + # histogram('le', TimeUnit.MILLISECONDS) to keep the default seconds-to-ms rescale from + # multiplying entry counts by 1000. + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_drop_set_budget_bytes --service-name=e2e-banyandb-sampling + expected: expected/labels-group.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_trace_sampling_drop_set_entries_p99 --service-name=e2e-banyandb-sampling + expected: expected/labels-group-lane-p.yml From 4db13b975dde79e3f50f366dc4f71040e56d697e Mon Sep 17 00:00:00 2001 From: wankai123 Date: Wed, 2 Sep 2026 17:12:27 +0800 Subject: [PATCH 2/2] changes --- docs/en/changes/changes.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index d222c3e1eb70..f094b4c0b898 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -11,7 +11,6 @@ * Add BanyanDB trace tail sampling metrics to the BanyanDB self-observability layer, in a new `otel-rules/banyandb/banyandb-trace-sampling.yaml` rule file. It covers the whole `banyandb_trace_pipeline_*` / `banyandb_trace_tst_pipeline_*` catalog a sampler plugin chain emits — pipeline reconciliation, per-plugin `Decide` execution rate and latency, chain batching, the trace-level evaluated / retained / dropped / immature outcomes, every fail-open guard and bounded-retention counter, drop-set capacity and finalization state, the plugin telemetry-host safety bounds, and the first-party `sw-trace-sampler` / `zipkin-trace-sampler` decision and row metrics. The plugin chain is optional, and the metrics follow it: on a cluster with no sampler configured the wire families are never registered, so every metric here stays absent rather than reading zero. Modeled at Service scope with `group` kept as a metric label rather than at Endpoint scope, so one cluster-wide page can render per-group series and cluster totals alike — OAP does no cross-scope rollup, so an Endpoint-scope metric could not have been aggregated back up to the cluster. * Fix a second `CounterWindow` key collision in the v2 MAL engine, this time ACROSS rules. `rate()` / `increase()` / `irate()` resolve their lower bound from a process-wide window keyed on the counter's own name plus its post-`.sum(...)` label set, with nothing identifying the rule doing the evaluation. Two rules that read one wire family, tell their streams apart with `tagEqual(...)`, and then `.sum(...)` away the label they filtered on therefore collapse onto one window slot and difference against each other's values. The queue is ordered by (timestamp, value), so the smaller counter wins the lower-bound lookup and still reads correctly while its partner is inflated by the gap between them — which is why this went unnoticed. A collision needs the discriminating label to be DROPPED by the `.sum(...)`: where it survives, the rules' label values differ and the window keeps them apart. Auditing the shipped rules on that basis gives 10 colliding keys over ~25 rules — `meter_activemq_cluster_gc_parallel_young_collection_count` reported ~9000/min of young-gen collections from a completely idle broker (differencing against the old-gen counter); MySQL `commands_*` / `tps` rate against each other; so do the GenAI gateway input/output token rates, four Envoy `cluster_*` counters, APISIX matched/unmatched instance bandwidth, and BanyanDB's own `network_recv` / `network_sent`, which drop the `kind` label that separates bytes-received from bytes-sent on one interface. Measured against two live scrapes of the demo cluster's FODC proxy, that last pair was wrong on every interface: `network_sent` read a flat 0 B/s and `network_recv` read large negative values (down to -778 MB/s) from differencing against the sent counter, where both now match the byte delta exactly. No rule changes were needed for any of these -- each rule already reduces to the labels it should; only the window key was wrong. The window is now keyed by (owning rule, counter name, labels). This is the complement of the within-rule collision fixed earlier by keying on the counter's own name: neither name alone is sufficient, because the two collisions are independent. `RunningContext.metricName` — written on every rule evaluation and read by nobody since that earlier fix — is what supplies the rule identity, so no code generation or MAL syntax changes. Note the whole-rule-set comparison suite could not have caught this: it resets the shared window before every rule, the one condition under which the collision cannot appear. * Fix `meter_rabbitmq_node_outgoing_messages_total` double-counting one of its terms. The rule summed six delivery-rate terms but `rabbitmq_global_messages_delivered_get_auto_ack_total` appeared twice, so auto-ack `basic.get` deliveries were counted once more than the other four delivery paths and the reported outgoing rate ran high whenever polling consumers were in use. The duplicate term is removed, leaving the five distinct families (redelivered, consume auto/manual ack, get auto/manual ack). -* Add an e2e case for the BanyanDB trace tail sampling metrics, `test/e2e-v2/cases/banyandb/trace-sampling`. The existing BanyanDB self-observability case cannot cover them: its cluster runs no sampler plugins, and the wire families a sampler emits are registered on first observation, so they never appear. This case runs a real `sw-trace-sampler` chain on the plugin-capable (`-plugins`) BanyanDB image with the `.so` delivered by the `-plugins-carrier` image, scrapes the data node's own Prometheus endpoint through an OTel collector, and asserts the 17 `meter_banyandb_trace_sampling_*` rules that a healthy, fully-dropping run can produce. The remaining 24 rules in the file cover failure counters, the keep verdict, FINALIZE-only state and back-pressure guards that five traces in one shard never reach, and are deliberately not asserted: an empty result satisfies a presence check just as well when the expression is misspelled. #### UI * Add a Virtual GenAI evaluation-record page and evaluation-score chart in Horizon UI, so operators can inspect evaluation result, level, reason, judge model, timestamp, trace linkage, and the `gen_ai_model_evaluation_score_ppm` trend for evaluated records.