Skip to content

Document the around_test instrumentation hook - #116

Open
KyleOps wants to merge 2 commits into
inferno-framework:mainfrom
KyleOps:docs/instrumenting-test-execution
Open

Document the around_test instrumentation hook#116
KyleOps wants to merge 2 commits into
inferno-framework:mainfrom
KyleOps:docs/instrumenting-test-execution

Conversation

@KyleOps

@KyleOps KyleOps commented Jul 31, 2026

Copy link
Copy Markdown

Companion documentation for inferno-core#797, which adds TestRunner#around_test. Raised at @karlnaden's request on that PR, which suggested either the Debugging page or a new page under Advanced Features. I went with a new page, and linked to it from Debugging, because the two cover different concerns: Debugging is about investigating a single test interactively while writing it, this is about observing runs in a deployed instance.

What the page covers

  • Why instrumentation applied around the runner is not useful for a long run: Inferno executes a whole run in one call stack, so a tracing library that instruments the background worker sees one long-lived operation rather than one per test.
  • How to override the hook, with a small logging/timing example.
  • The contract an override has to honour: call super exactly once, return that value unchanged, let exceptions propagate. Each with the consequence of getting it wrong, since all three fail silently.
  • A worked OpenTelemetry example producing one bounded trace per test.
  • Where to put the override so it loads in every process that runs tests.

Two things worth flagging for review

The span-naming section is the part I would most like checked. The example deliberately gives the span a constant name and puts identity in attributes. This is not a style preference: tracing backends treat the span name as a low-cardinality dimension and generate metrics keyed on it, so naming a span after the test produces one metric series per test, per suite, per suite version. On our deployment of a large test kit that was over 1,400 distinct span names, and every one of those series is useless by construction, because a test emits a single span per run and rate() or increase() over one observation returns zero. An earlier draft of the inferno-core PR had the interpolated form, and it seemed worth documenting the trap rather than shipping an example that leads people into it.

The page notes that the hook wraps all of #run_test, which includes persisting the result along with its messages and requests. A duration measured with the hook is the wall time a user waited, which is usually what you want, but it is not purely test execution. Flagged because it surprised us: for request-heavy tests that persistence is a meaningful share of the elapsed time.

Also

docs/advanced-test-features/index.md gets an entry for the new page, and docs/getting-started/debugging.md gets a short closing section pointing at it.

Happy to fold this into the Debugging page instead, or to cut the span-naming section back, if you would rather the page stayed shorter.

KyleOps added 2 commits July 29, 2026 08:56
Inferno Core runs a whole test run in one call stack, so instrumentation
applied around the runner, such as tracing of the background worker, covers
the entire run rather than individual tests. Inferno::TestRunner#around_test
provides a seam for per-test instrumentation.

Add an Advanced Features page covering the hook: why run-scoped
instrumentation is not useful for a long run, how to override the hook, the
contract an override has to honour, and a worked OpenTelemetry example that
produces one trace per test. Link to it from the debugging page, which
covers the separate concern of investigating a single test interactively.
The worked example named each span after the test it wrapped. Tracing backends
treat the span name as a low-cardinality dimension and generate metrics keyed on
it, so that pattern produces one metric series per test, per suite, per suite
version. Measured on a deployment of a large test kit it produced over 1,400
distinct span names, and every one of those series was useless by construction:
a test emits a single span per run, so a per-test series holds one observation
and rate() or increase() over it returns zero.

Name the span with a constant and move identity into attributes, which carry no
such constraint. Add the short id and title while there, since those are what a
user reads back when reporting a slow test.

Also note two things the example would otherwise leave a reader to discover: that
an override recording the outcome should reserve error status for 'error' rather
than 'fail', since a conformance failure is the expected outcome of testing a
non-conformant system, and that the block covers persisting the result and its
requests as well as executing the test.
@KyleOps

KyleOps commented Jul 31, 2026

Copy link
Copy Markdown
Author

"Keep the test out of the span name" section might be too opinionated but it is a bit of a gotcha so I think it is worth it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant