Skip to content

Add BanyanDB trace tail sampling monitoring - #14052

Merged
wankai123 merged 2 commits into
apache:masterfrom
wankai123:tail-sampling-monitoring
Sep 2, 2026
Merged

Add BanyanDB trace tail sampling monitoring#14052
wankai123 merged 2 commits into
apache:masterfrom
wankai123:tail-sampling-monitoring

Conversation

@wankai123

Copy link
Copy Markdown
Member

Add otel-rules/banyandb/banyandb-trace-sampling.yaml, 41 rules covering the banyandb_trace_pipeline_* / banyandb_trace_tst_pipeline_* catalog a sampler plugin chain emits. Modeled at Service scope with group kept as a metric label rather than at Endpoint scope, so one cluster-wide page can render per-group series and cluster totals alike.

Add the e2e case test/e2e-v2/cases/banyandb/trace-sampling, registered in the e2e-test matrix. It runs a real sw-trace-sampler chain on the plugin-capable BanyanDB image and asserts the 17 rules a healthy, fully-dropping run can produce; the other 24 cover failure counters, the keep verdict, FINALIZE-only state and back-pressure guards that cannot occur in a short healthy run. The existing BanyanDB self-observability case cannot cover any of this: its cluster runs no sampler plugins, so the wire families never register.

Fix a cross-rule CounterWindow key collision in the v2 MAL engine. rate() / increase() / irate() resolved their lower bound from a process-wide window keyed on the counter name plus its post-.sum(...) label set, with nothing identifying the evaluating rule, so two rules reading one wire family and dropping the label they filtered on collapsed onto one slot and differenced against each other's values. The window is now keyed by owning rule as well. Auditing the shipped rules gives 10 colliding keys over ~25 rules.

Fix meter_rabbitmq_node_outgoing_messages_total double-counting rabbitmq_global_messages_delivered_get_auto_ack_total, which appeared twice among the six summed delivery terms.

  • If this is non-trivial feature, paste the links/URLs to the design doc.

  • Update the documentation to include this new feature.

  • Tests(including UT, IT, E2E) are added to verify the new feature.

  • If it's UI related, attach the screenshots below.

  • If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.

  • Update the CHANGES log.

Add otel-rules/banyandb/banyandb-trace-sampling.yaml, 41 rules covering the
banyandb_trace_pipeline_* / banyandb_trace_tst_pipeline_* catalog a sampler plugin chain
emits. Modeled at Service scope with `group` kept as a metric label rather than at Endpoint
scope, so one cluster-wide page can render per-group series and cluster totals alike.

Add the e2e case test/e2e-v2/cases/banyandb/trace-sampling, registered in the e2e-test
matrix. It runs a real sw-trace-sampler chain on the plugin-capable BanyanDB image and
asserts the 17 rules a healthy, fully-dropping run can produce; the other 24 cover failure
counters, the keep verdict, FINALIZE-only state and back-pressure guards that cannot occur
in a short healthy run. The existing BanyanDB self-observability case cannot cover any of
this: its cluster runs no sampler plugins, so the wire families never register.

Fix a cross-rule CounterWindow key collision in the v2 MAL engine. rate() / increase() /
irate() resolved their lower bound from a process-wide window keyed on the counter name
plus its post-.sum(...) label set, with nothing identifying the evaluating rule, so two
rules reading one wire family and dropping the label they filtered on collapsed onto one
slot and differenced against each other's values. The window is now keyed by owning rule
as well. Auditing the shipped rules gives 10 colliding keys over ~25 rules.

Fix meter_rabbitmq_node_outgoing_messages_total double-counting
rabbitmq_global_messages_delivered_get_auto_ack_total, which appeared twice among the six
summed delivery terms.
@wankai123 wankai123 added this to the 11.1.0 milestone Sep 2, 2026
@wankai123 wankai123 added so11y Self Observability database BanyanDB - SkyWalking native database labels Sep 2, 2026
@wankai123
wankai123 requested a lite review from Copilot September 2, 2026 09:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core v2 MAL rate/increase/irate windowing semantics that can affect many existing meter rules, so it warrants final human verification despite added regression coverage.

Pull request overview

This PR extends SkyWalking’s self-observability for BanyanDB by adding a dedicated MAL/OTel ruleset and an end-to-end e2e case for trace tail-sampling (sampler plugin chain) metrics, while also fixing a correctness bug in the v2 MAL counter windowing logic and correcting a RabbitMQ node metric overcount.

Changes:

  • Add banyandb-trace-sampling.yaml (41 rules) to expose BanyanDB trace tail-sampling pipeline/plugin metrics at Service scope (cluster), retaining group as a metric label.
  • Add a new e2e-v2 case (test/e2e-v2/cases/banyandb/trace-sampling) and register it in the GitHub Actions e2e matrix to validate key healthy “fully-dropping” metric families end-to-end.
  • Fix v2 MAL CounterWindow key collisions across rules by including the owning rule (output metric name) in the window key; fix RabbitMQ outgoing message delivery double-counting.
File summaries
File Description
test/e2e-v2/cases/banyandb/trace-sampling/trace-sampling-cases.yaml Defines swctl queries/assertions for the trace sampling monitoring e2e scenario.
test/e2e-v2/cases/banyandb/trace-sampling/otel-collector-config.yaml Collector config to scrape BanyanDB Prometheus metrics and inject cluster label for rule evaluation.
test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group.yml Expected shape for group-labeled time series results.
test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-result.yml Expected shape for group+result-labeled time series results.
test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin.yml Expected shape for group+plugin_name-labeled time series results.
test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-rule.yml Expected shape for group+plugin_name+rule-labeled time series results.
test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-rule-verdict.yml Expected shape for group+plugin_name+rule+verdict-labeled time series results.
test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-result.yml Expected shape for group+plugin_name+result-labeled time series results.
test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-plugin-p.yml Expected shape for histogram percentile results with p label (per plugin).
test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-p.yml Expected shape for histogram percentile results with p label (chain-level).
test/e2e-v2/cases/banyandb/trace-sampling/expected/labels-group-lane-p.yml Expected shape for histogram percentile results split by lane.
test/e2e-v2/cases/banyandb/trace-sampling/expected/active-samplers.yml Expected output asserting sampler activation (1 vs 0) per trace group.
test/e2e-v2/cases/banyandb/trace-sampling/e2e.yaml New e2e-v2 harness for trace sampling monitoring validation (retry/timeout tuned for merge + scrape + bucket).
test/e2e-v2/cases/banyandb/trace-sampling/docker-compose.yml Compose topology for plugin-capable BanyanDB + OTel collector + OAP + trace mocker.
oap-server/server-starter/src/main/resources/otel-rules/rabbitmq/rabbitmq-node.yaml Removes a duplicated delivery counter term in outgoing_messages_total.
oap-server/server-starter/src/main/resources/otel-rules/banyandb/banyandb-trace-sampling.yaml Adds the new BanyanDB trace tail sampling rule catalog (service-scoped, group retained as label).
oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/CrossRuleCounterWindowTest.java Regression test proving cross-rule counter window isolation for increase() on frozen counters.
oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/CounterWindowTest.java Updates unit test calls for the new (owner, name, labels, ...) CounterWindow API.
oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/SampleFamily.java Passes rule identity (context.metricName) into increase/rate/irate windowing calls.
oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/Sample.java Extends counter window calls to include owning rule for proper keying.
oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/ID.java Adds owner field to CounterWindow key material (rule identity).
oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/dsl/counter/CounterWindow.java Keys counter windows by (owner, counter name, labels) to avoid cross-rule collisions.
oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/rabbitmq/rabbitmq-node.data.yaml Updates expected value for RabbitMQ outgoing messages after de-duplication.
docs/en/changes/changes.md Changelog entries for the new BanyanDB sampling metrics, CounterWindow fix, and RabbitMQ rule correction.
docs/en/banyandb/tail-sampling.md Points operators to the OAP-collected sampling metric catalog and anchor link in dashboards doc.
docs/en/banyandb/dashboards-banyandb.md Documents the full trace tail sampling metric catalog and adds the rule file to the BanyanDB list.
.github/workflows/skywalking.yaml Registers the new BanyanDB trace sampling monitoring e2e case in CI matrix.
Review details
  • Files reviewed: 27/27 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@wankai123
wankai123 merged commit e27cc27 into apache:master Sep 2, 2026
460 of 463 checks passed
@wankai123
wankai123 deleted the tail-sampling-monitoring branch September 2, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

database BanyanDB - SkyWalking native database so11y Self Observability

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants