Skip to content

Commit 75df24f

Browse files
TheGreatAxiosTheGreatAxios
committed
Document CL-6816 telemetry volume cut
Update the event table and AI section for generation aggregates, opt-in spans, subagent_end rollups, director allowlisting, anonymous ambient processing, plugin_loaded dedupe, and generation sampling.
1 parent 0628f9e commit 75df24f

1 file changed

Lines changed: 50 additions & 27 deletions

File tree

‎docs/TELEMETRY.md‎

Lines changed: 50 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,13 @@ Each event carries a small set of properties:
1515
| ------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
1616
| `cli_start` | Once per used session (see First-run disclosure) | (none beyond common properties) |
1717
| `session_end` | When a TUI session finishes | `status`, `turn_count`, `duration_ms`, `session_mode`, `exit_reason` |
18-
| `$ai_generation` | Once per turn — on completion, and once for a turn that ends in an error instead | `$ai_trace_id`, `$ai_provider`, `$ai_model`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_latency`, `$ai_is_error`, `$ai_error`, `$ai_cache_read_input_tokens`, `$ai_cache_creation_input_tokens`, `$ai_reasoning_tokens` |
19-
| `$ai_span` | Once per top-level tool call in a completed turn | `$ai_trace_id`, `$ai_span_id`, `$ai_parent_id`, `$ai_span_name`, `$ai_is_error` |
18+
| `$ai_generation` | Once per completed turn (may be sampled); always on turn failure | `$ai_trace_id`, `$ai_provider`, `$ai_model`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_latency`, `$ai_is_error`, `$ai_error`, `$ai_cache_read_input_tokens`, `$ai_cache_creation_input_tokens`, `$ai_reasoning_tokens`, `tool_call_count`, `tool_error_count`, `subagent_call_count` |
19+
| `$ai_span` | Opt-in only — once per top-level tool call when `CORBITS_TELEMETRY_AI_SPANS` is set | `$ai_trace_id`, `$ai_span_id`, `$ai_parent_id`, `$ai_span_name`, `$ai_is_error` |
2020
| `slash_command` | A slash command is dispatched (shared product-event path) | `command_name` |
2121
| `skill_used` | `use_skill` loads a skill that resolved | (none beyond common properties) |
22-
| `plugin_loaded` | A plugin is discovered and loaded at startup | `origin` |
23-
| `subagent_start` | A `task` dispatch begins | `agent_name` |
24-
| `subagent_end` | A `task` dispatch finishes | `agent_name`, `status`, `duration_ms` |
22+
| `plugin_loaded` | First successful load of a plugin identity in this process | `origin` |
23+
| `subagent_start` | A `task` / fleet dispatch begins | `agent_name` |
24+
| `subagent_end` | A `task` / fleet dispatch finishes | `agent_name`, `status`, `duration_ms`, `model`, `turn_count`, `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `reasoning_tokens`, `tool_call_count`, `tool_error_count`, `stop_reason`, `parent_trace_id` |
2525
| `permission_prompt` | An approval prompt is answered (or abandoned) | `decision`, `permission_kind` |
2626
| `compaction` | The compactor actually folds turns away | `mode`, `duration_ms`, `turns_before`, `turns_after` |
2727
| `crash` | A fatal error reaches the process-level handler | `kind`, `error_class` |
@@ -36,7 +36,11 @@ Common properties attached to every event: a random installation UUID
3636
(`distinct_id`), `session_id`, `$app_version` (PostHog's standard Version
3737
property, the running package version), `service_version` (same value, kept
3838
for existing custom-property dashboards), `os_type`, `os_arch`, and a
39-
`schema_version` for forward compatibility.
39+
`schema_version` for forward compatibility. Ambient product and AI events also
40+
carry `$process_person_profile: false` so PostHog treats them as anonymous
41+
(batch capture otherwise defaults to identified processing). Intentional
42+
`survey sent` omits that flag so `/feedback` can still join a person profile
43+
if one is ever created.
4044

4145
Approximate country-level location is derived server-side by PostHog from the
4246
request IP; no location data is collected by the client.
@@ -62,10 +66,14 @@ So none of them are transmitted. Each is matched against a fixed list of names
6266
this project itself ships and reported as that name, or as `custom` when it
6367
matches nothing — with `mcp` as its own bucket for `permission_kind`, so the
6468
share of prompts driven by MCP stays visible without the server key coming
65-
with it. `skill_used` and `plugin_loaded` go further: there is no first-party
66-
list of skills or plugins to match against, so `skill_used` carries no name at
67-
all and `plugin_loaded` carries only `origin`, the discovery tier
68-
(`repo`, `user`, `project`, `path`).
69+
with it. `agent_name` on `subagent_*` is the same pattern: first-party
70+
director ids from `DIRECTOR_IDS` (and the legacy `worker` alias) are reported
71+
by id; project-defined or marketplace profile ids become `custom`.
72+
`skill_used` and `plugin_loaded` go further: there is no first-party list of
73+
skills or plugins to match against, so `skill_used` carries no name at all and
74+
`plugin_loaded` carries only `origin`, the discovery tier (`repo`, `user`,
75+
`project`, `path`). The same plugin identity is emitted at most once per
76+
process — later rediscoveries or reloads are silent.
6977

7078
`error_class` is bucketed the same way: only the error types defined by the
7179
language are reported by name, because an error subclass defined in
@@ -84,21 +92,34 @@ the payload are in `tests/unit/telemetry-product-events.test.ts`.
8492

8593
## AI observability events
8694

87-
`$ai_generation` and `$ai_span` are the two PostHog AI observability events,
88-
emitted from `src/telemetry/ai-observability.ts`. PostHog's LLM analytics
89-
views query the `$ai_`-prefixed properties and nothing else, which is why
90-
these names are not ours to choose. `$ai_latency` is a duration in **seconds**
91-
as a float, per PostHog's schema — the runtime measures milliseconds and
92-
converts.
95+
`$ai_generation` and (optionally) `$ai_span` are the PostHog AI observability
96+
events, emitted from `src/telemetry/ai-observability.ts`. PostHog's LLM
97+
analytics views query the `$ai_`-prefixed properties and nothing else, which
98+
is why these names are not ours to choose. `$ai_latency` is a duration in
99+
**seconds** as a float, per PostHog's schema — the runtime measures
100+
milliseconds and converts.
101+
102+
**Default volume shape (CL-6816):** each completed primary turn emits **one**
103+
`$ai_generation` with tool/subagent aggregates folded onto it
104+
(`tool_call_count`, `tool_error_count`, `subagent_call_count`). Per-call
105+
`$ai_span` events are **off by default**. Set `CORBITS_TELEMETRY_AI_SPANS` to
106+
a truthy value (`1`, `true`, …) to restore per-call spans for debugging.
107+
Leaf `runSubAgent` workers do not emit `$ai_*`; worker rollups travel on
108+
`subagent_end` instead.
109+
110+
Successful `$ai_generation` events may be sampled with
111+
`CORBITS_TELEMETRY_GENERATION_SAMPLE_RATE` (a float in `0`–`1`, default `1.0`
112+
= keep all). Errored generations (`$ai_is_error: true`), `crash`, and
113+
`auth_failure` always ship regardless of the sample rate.
93114

94115
The trace is **flat**. Every turn gets one `$ai_trace_id` derived from the
95116
runtime's session id and the turn index; the turn's `$ai_generation` and each
96-
of its `$ai_span`s carry it, and every span's `$ai_parent_id` is that same
97-
trace id rather than another span. PostHog documents `$ai_parent_id` as
98-
accepting either a trace id or a span id, so this is a legal trace, and it is
99-
all the runtime can honestly describe: the turn record only exposes top-level
100-
tool calls. No `$ai_trace` event is emitted — PostHog synthesises the trace
101-
from its children.
117+
of its `$ai_span`s (when spans are enabled) carry it, and every span's
118+
`$ai_parent_id` is that same trace id rather than another span. PostHog
119+
documents `$ai_parent_id` as accepting either a trace id or a span id, so
120+
this is a legal trace, and it is all the runtime can honestly describe: the
121+
turn record only exposes top-level tool calls. No `$ai_trace` event is
122+
emitted — PostHog synthesises the trace from its children.
102123

103124
`$ai_span_id` is the provider-generated opaque tool call id. It identifies
104125
the call within the trace and carries nothing else.
@@ -124,11 +145,11 @@ apart by `$ai_error`. A turn that never reaches inference at all — suspended
124145
at an approval prompt and never resumed — emits nothing, because the runtime
125146
raises no event for it.
126147

127-
Exactly one `$ai_generation` is ever emitted per turn. A single give-up
128-
usually surfaces twice at the event stream (the failed inference, then the
129-
reactor terminating), and a turn that already reported completion is finished;
130-
`src/session/run-sink.ts` latches on both so neither can double-count a turn
131-
or append a phantom failure to a successful one.
148+
Exactly one `$ai_generation` is ever emitted per turn (when sampling keeps
149+
it). A single give-up usually surfaces twice at the event stream (the failed
150+
inference, then the reactor terminating), and a turn that already reported
151+
completion is finished; `src/session/run-sink.ts` latches on both so neither
152+
can double-count a turn or append a phantom failure to a successful one.
132153

133154
## What's never collected
134155

@@ -159,6 +180,8 @@ is off (`settings.telemetry.enabled === false` or the Telemetry toggle Off),
159180
because the operator typed the text for that purpose. Hard env kill switches
160181
still win — `DO_NOT_TRACK=1` or `CORBITS_TELEMETRY=0/false/off/no` block
161182
`/feedback` as well. Sending also requires an installation id and API key.
183+
Unlike ambient events, `survey sent` does not stamp `$process_person_profile:
184+
false`, so the response can join a person profile if one is ever created.
162185

163186
Survey id / question id are **baked into the client** (Corbits team survey
164187
`Corbits Code Feedback`). Same trust class as the public PostHog project key —

0 commit comments

Comments
 (0)