Add a span watchers for collecting extra search metrics on-demand

[mapno]

What this PR does:

Adds a framework for adding span watchers in the TraceQL engine for collecting extra query metrics on-demand. It works for both search and metrics queries.

Watchers are enabled per-tenant via the experimental span_pruning_awareness override (operator-controlled, not user-facing). Its only user checks whether the matched spans include span-pruning summary spans, i.e. spans carrying the aggregation.is_summary attribute. Example:

overrides:
  defaults:
    read:
      span_pruning_awareness: true

Checklist

• Tests updated
• Documentation added
• Changelog entry added under .chloggen/ (run make chlog-new, or make chlog-new FILENAME=<name> to override the default branch-name file; see .chloggen/README.md)

[oleg-kozlyuk-grafana]

Would it make sense to design the syntax with single parameter for multiple observers? e.g. observe=summary,.... This would match the implementation you already have

[mdisibio]

I do like this idea for multiple observers. But I wouldn't spend too much time thinking about it. The final vision for this is going to be a global config option, so that Tempo can be always aware of the span pruning aggregation data without the user having to do it in every query. In that case the hint won't be used, and actually might need to be the inverse, somehow to disable it, which would have to be a different set of hints.

[mapno]

Agreed. I've replaced it by a per-tenant report_attributes override.

[Copilot]

Pull request overview

This PR introduces a span-watcher framework in the TraceQL engine to collect additional, on-demand per-query metrics based on operator-configured watched attributes (via the per-tenant watch_attributes override). It wires watchers into both TraceQL search and metrics query execution paths and exposes results via AdditionalMetrics.

Changes:

• Add SpanWatcher abstractions plus an initial “attribute presence” watcher and watcher orchestration (spanWatchers).
• Integrate watchers into TraceQL search execution and metrics query evaluation (including request-scoped aggregation across metrics sub-pipelines).
• Add per-tenant override plumbing (watch_attributes) with docs, changelog entry, and tests.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/traceql/watch.go	Introduces watcher interfaces, attribute-presence watcher, and spanWatchers coordinator.
pkg/traceql/watch_test.go	Unit tests for watcher behavior and search integration coverage.
pkg/traceql/storage.go	Updates Condition.CallBack documentation used by storage predicates.
pkg/traceql/options.go	Adds WithWatchers compile option and stores watchers in compile options.
pkg/traceql/engine.go	Installs watchers for search and emits watcher-derived AdditionalMetrics.
pkg/traceql/engine_metrics.go	Makes watchers request-scoped for metrics and reports watcher stats once across sub-pipelines.
pkg/traceql/engine_metrics_test.go	Adds metrics watcher tests incl. “count once across sub-pipelines” coverage.
modules/querier/querier.go	Appends watcher compile options for TraceQL backend search.
modules/querier/querier_query_range.go	Appends watcher compile options for metrics query-range path.
modules/overrides/watchers.go	Converts per-tenant WatchAttributes specs into TraceQL compile options (warn+skip invalid).
modules/overrides/runtime_config_overrides.go	Exposes WatchAttributes from runtime overrides manager.
modules/overrides/interface.go	Adds WatchAttributes(userID) to overrides interface.
modules/overrides/config.go	Adds watch_attributes override schema and WatchAttribute struct.
modules/overrides/config_test.go	Updates legacy overrides test data to include WatchAttributes.
modules/overrides/config_legacy.go	Adds WatchAttributes to legacy/new override conversions.
modules/livestore/instance_search.go	Appends watcher compile options for live-store search and query-range paths.
docs/sources/tempo/configuration/_index.md	Documents watch_attributes override configuration.
.chloggen/span-pruning-telemetry.yaml	Adds changelog entry describing watcher framework and config.

[mdisibio]

Thinking down the road, a list of plain watch attributes might not be too useful, because we probably will want specialized logic for each one? I like the structure of the watchers and callbacks in the engine, but a better config option might be a way to enforce hints by default, i.e. this one to report, or future ones like span_pruning_awareness or extrapolate. Thoughts?

[mapno]

I wasn't very convinced about the config either. I've updated it to be span_pruning_awareness.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.