feat: add better instrumentation to querier methods using new metrics stats by javiermolinar · Pull Request #7571 · grafana/tempo

javiermolinar · 2026-07-02T14:47:06Z

What this PR does:
Add span traces for all querier methods with standardized attributes and the new stats implemented in #7504

Which issue(s) this PR fixes:
Fixes #

Checklist

Tests updated
Documentation added
Changelog entry added under .chloggen/ (run make chlog-new, or make chlog-new FILENAME=<name> to override the default branch-name file; see .chloggen/README.md)

… stats

Copilot

Pull request overview

This PR adds shared OpenTelemetry span helpers for the querier and wires them into several querier request paths to attach request parameters and the new “stats metrics” fields onto spans.

Changes:

Introduces start*/finish* helpers to consistently set querier span attributes and attach SearchMetrics / MetadataMetrics / TraceByIDMetrics to spans.
Updates multiple querier methods (trace-by-id, search/tags/tag-values, query-range) to use the new span helpers and record errors via a shared finishQuerierSpan.
Adds focused unit tests validating that span attributes and AdditionalMetrics are exported as span attributes.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
modules/querier/span_attrs.go	New helper functions to start/end spans and attach request attributes + metrics.
modules/querier/span_attrs_test.go	Tests that span attributes + metrics are correctly emitted (including AdditionalMetrics).
modules/querier/querier.go	Replaces ad-hoc span code with shared helpers across several querier RPC methods.
modules/querier/querier_query_range.go	Adds spans around QueryRange execution paths using the shared helpers.
.chloggen/instrument-querier-methods.yaml	Adds a changelog entry for the querier instrumentation change.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-07-02T15:21:18Z

⚠️ The workflow jobs listed below don't declare a permissions: block and may break when the organization's default GITHUB_TOKEN permissions are restricted to read-only.

Expand for findings

warning[excessive-permissions]: overly broad permissions
  --> ./.github/workflows/deploy-pr-preview.yml:13:3
   |
13 | /   deploy-pr-preview:
14 | |     if: "!github.event.pull_request.head.repo.fork"
15 | |     uses: grafana/writers-toolkit/.github/workflows/deploy-preview.yml@main # zizmor: ignore[unpinned-uses]
16 | |     with:
...  |
31 | |       title: ${{ github.event.pull_request.title }}
   | |                                                    ^
   | |                                                    |
   | |____________________________________________________this job
   |                                                      default permissions used due to no permissions: block
   |
   = note: audit confidence → Medium
   = help: audit documentation → https://docs.zizmor.sh/audits/#excessive-permissions

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

+	ctx, span, tenantID, err := startTagsBlockSpan(ctx, "Querier.internalTagsSearchBlockV2", req)
+	if err != nil {
+		return nil, fmt.Errorf("error extracting org id in Querier.BackendSearch: %w", err)
+	}
+	defer func() { finishQuerierSpan(span, err, resp.GetMetrics()) }()


stoewer

I left a few minor comments. Otherwise this looks good 👍

stoewer · 2026-07-03T15:17:00Z


-func (q *Querier) SearchTagValuesBlocks(ctx context.Context, req *tempopb.SearchTagValuesBlockRequest) (*tempopb.SearchTagValuesResponse, error) {
+func (q *Querier) SearchTagValuesBlocks(ctx context.Context, req *tempopb.SearchTagValuesBlockRequest) (resp *tempopb.SearchTagValuesResponse, err error) {
+	ctx, span, _, err := startTagValuesBlockSpan(ctx, "Querier.SearchTagValuesBlocks", req)


The call to q.internalTagValuesSearchBlock(ctx, req) later in this function creates a span with the name Querier.internalTagsSearchBlockV2 that carries similar (identical?) information. This is a bit redundant. Should we remove the internal one?

The same pattern with similarly duplicated spans occurs in SearchTagsBlocks, SearchTagsBlocksV2, and SearchTagsBlocksV2 as well

stoewer · 2026-07-03T15:27:31Z

-	resp, err := q.store.SearchTagValues(ctx, meta, req, opts)
+	resp, err = q.store.SearchTagValues(ctx, meta, req, opts)
 	if err != nil {
 		return &tempopb.SearchTagValuesResponse{}, err


I think this overwrites the named return value resp that is used in the deferred call to finishQuerierSpan() above. But readerWriter.SearchTagValues() populates metrics also when the function returns an error. Due to this overwrite, the metrics are not captured by the deferred call to finishQuerierSpan()

I think the main culprit here are the named return values (resp *tempopb.SearchTagValuesResponse, err error) as those tend to cause maintenance hazards in longer functions. Not a must, but finding an alternative pattern to capture the error and stats in the finish call would be nice (but not worth it, if it makes the function complex in other ways)

stoewer · 2026-07-03T15:34:38Z

+	)
+}
+
+func setQuerierSpanMetrics(span oteltrace.Span, metrics any) {


The type switch without default case can silently do nothing when additional metrics structs are added in the future. Instead of one function with a type switch, this function can be refactored into 3 small separate function for each type

feat: add better instrumentation to querier methods using new metrics…

868fabd

… stats

Copilot AI review requested due to automatic review settings July 2, 2026 14:47

javiermolinar requested review from carles-grafana, electron0zero, ie-pham, knylander-grafana, mapno, mattdurham, mdisibio, oleg-kozlyuk-grafana, ruslan-mikhailov, stoewer, yvrhdn, zalegrala and zhxiaogg as code owners July 2, 2026 14:47

Copilot started reviewing on behalf of javiermolinar July 2, 2026 14:47 View session

This comment has been minimized.

Sign in to view

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread modules/querier/querier.go Outdated

Comment thread modules/querier/querier.go

github-advanced-security AI found potential problems Jul 2, 2026

View reviewed changes

Comment thread modules/querier/span_attrs.go Dismissed

Comment thread modules/querier/span_attrs.go Dismissed

Comment thread modules/querier/span_attrs.go Dismissed

Comment thread modules/querier/span_attrs.go Dismissed

recover error from missing tenant

bdb0bd2

This comment has been minimized.

Sign in to view

pass the metrics in the outer methods

83e431b

Copilot AI review requested due to automatic review settings July 2, 2026 15:11

Copilot started reviewing on behalf of javiermolinar July 2, 2026 15:12 View session

This comment has been minimized.

Sign in to view

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread modules/querier/span_attrs.go

Comment thread modules/querier/span_attrs_test.go

Comment thread modules/querier/querier.go Outdated

Comment thread modules/querier/querier.go Outdated

Potential fix for pull request finding

504a260

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 2, 2026 15:19

Potential fix for pull request finding

1da7988

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot started reviewing on behalf of javiermolinar July 2, 2026 15:20 View session

This comment has been minimized.

Sign in to view

Copilot AI reviewed Jul 2, 2026

View reviewed changes

stoewer reviewed Jul 3, 2026

View reviewed changes

Uh oh!

Conversation

javiermolinar commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

github-actions Bot commented Jul 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

stoewer left a comment

Choose a reason for hiding this comment

Uh oh!

stoewer Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

stoewer Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

stoewer Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

javiermolinar commented Jul 2, 2026 •

edited

Loading