Document the around_test instrumentation hook - #116
Open
KyleOps wants to merge 2 commits into
Open
Conversation
Inferno Core runs a whole test run in one call stack, so instrumentation applied around the runner, such as tracing of the background worker, covers the entire run rather than individual tests. Inferno::TestRunner#around_test provides a seam for per-test instrumentation. Add an Advanced Features page covering the hook: why run-scoped instrumentation is not useful for a long run, how to override the hook, the contract an override has to honour, and a worked OpenTelemetry example that produces one trace per test. Link to it from the debugging page, which covers the separate concern of investigating a single test interactively.
The worked example named each span after the test it wrapped. Tracing backends treat the span name as a low-cardinality dimension and generate metrics keyed on it, so that pattern produces one metric series per test, per suite, per suite version. Measured on a deployment of a large test kit it produced over 1,400 distinct span names, and every one of those series was useless by construction: a test emits a single span per run, so a per-test series holds one observation and rate() or increase() over it returns zero. Name the span with a constant and move identity into attributes, which carry no such constraint. Add the short id and title while there, since those are what a user reads back when reporting a slow test. Also note two things the example would otherwise leave a reader to discover: that an override recording the outcome should reserve error status for 'error' rather than 'fail', since a conformance failure is the expected outcome of testing a non-conformant system, and that the block covers persisting the result and its requests as well as executing the test.
Author
|
"Keep the test out of the span name" section might be too opinionated but it is a bit of a gotcha so I think it is worth it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion documentation for inferno-core#797, which adds
TestRunner#around_test. Raised at @karlnaden's request on that PR, which suggested either the Debugging page or a new page under Advanced Features. I went with a new page, and linked to it from Debugging, because the two cover different concerns: Debugging is about investigating a single test interactively while writing it, this is about observing runs in a deployed instance.What the page covers
superexactly once, return that value unchanged, let exceptions propagate. Each with the consequence of getting it wrong, since all three fail silently.Two things worth flagging for review
The span-naming section is the part I would most like checked. The example deliberately gives the span a constant name and puts identity in attributes. This is not a style preference: tracing backends treat the span name as a low-cardinality dimension and generate metrics keyed on it, so naming a span after the test produces one metric series per test, per suite, per suite version. On our deployment of a large test kit that was over 1,400 distinct span names, and every one of those series is useless by construction, because a test emits a single span per run and
rate()orincrease()over one observation returns zero. An earlier draft of the inferno-core PR had the interpolated form, and it seemed worth documenting the trap rather than shipping an example that leads people into it.The page notes that the hook wraps all of
#run_test, which includes persisting the result along with its messages and requests. A duration measured with the hook is the wall time a user waited, which is usually what you want, but it is not purely test execution. Flagged because it surprised us: for request-heavy tests that persistence is a meaningful share of the elapsed time.Also
docs/advanced-test-features/index.mdgets an entry for the new page, anddocs/getting-started/debugging.mdgets a short closing section pointing at it.Happy to fold this into the Debugging page instead, or to cut the span-naming section back, if you would rather the page stayed shorter.