Expand the anonymous product event catalog (#425)

* Batch telemetry events through a single queued transport

Two upcoming event catalogs turn telemetry volume from one event per
turn into one per tool call, which the previous transport would have
answered with an unbounded set of concurrent per-event POSTs. Queueing
behind one request bounds both sockets and memory when the endpoint is
unreachable.

* Document how telemetry batches and sheds events on the wire

* Discard queued telemetry events when the user opts out

Dropping the singleton on opt-out left the outgoing instance's batch
timer armed, so events captured before the toggle would still reach the
network afterwards. Opting out speaks to activity already generated, not
only to activity still to come.

* Expand the anonymous product event catalog

Adds slash-command, skill, plugin, sub-agent, permission, compaction, and
crash events to the telemetry catalog.

Every identifier these events would naturally carry is named by someone
other than us — an MCP server key is a settings key, a skill is a directory
in the repo, a plugin id and an agent profile are author-chosen — so each is
mapped to a fixed first-party enum at the emission site and reported as
"custom" when it matches nothing. Emission takes Telemetry as an injected
dependency rather than reading the process-wide handle, so a module built
without one is silent by construction.

* Report which provider rejected credentials on auth failure

The auth_failure event reused error_class, which everywhere else means
the JS error constructor name. One column carrying two incompatible
meanings cannot be analysed, and it made the documented error_class
guarantee false: the value shipped was a local send-failure kind that
never passed through the classifier.

Author: TheGreatAxios

docs/TELEMETRY.md

@@ -6,13 +6,26 @@ includes prompts, code, file contents, or paths.
 
 ## What's collected
 
-Three events, each with a small set of properties:
+Each event carries a small set of properties:
 
 | Event | When | Properties |
 |---|---|---|
 | `cli_start` | Once per used session (see First-run disclosure) | (none beyond common properties) |
 | `session_end` | When a TUI session finishes | `status`, `turn_count`, `duration_ms`, `session_mode`, `exit_reason` |
 | `inference_turn` | Once per completed turn | `provider_id`, `model_id`, `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `thinking_tokens`, `duration_ms` |
+| `slash_command` | A slash command is dispatched in the TUI | `command_name` |
+| `skill_used` | `use_skill` loads a skill that resolved | (none beyond common properties) |
+| `plugin_loaded` | A plugin is discovered and loaded at startup | `origin` |
+| `subagent_start` | A `task` dispatch begins | `agent_name` |
+| `subagent_end` | A `task` dispatch finishes | `agent_name`, `status`, `duration_ms` |
+| `permission_prompt` | An approval prompt is answered (or abandoned) | `decision`, `permission_kind` |
+| `compaction` | The compactor actually folds turns away | `mode`, `duration_ms`, `turns_before`, `turns_after` |
+| `crash` | A fatal error reaches the process-level handler | `kind`, `error_class` |
+| `auth_failure` | A provider rejects the stored credentials | `auth_provider` |
+
+`compaction` is deliberately silent on the runs where the compactor decides
+there is nothing to compact — an event that also fires on no-ops makes its own
+duration and turn-count averages meaningless.
 
 Common properties attached to every event: a random installation UUID
 (`distinct_id`), `session_id`, `service_version`, `os_type`, `os_arch`, and a
@@ -30,10 +43,45 @@ onboarding or settings. `model_id` is the model identifier exactly as
 configured — it is the one user-entered string that is sent, so do not put
 anything identifying in a model name.
 
+## Names are never sent, only categories
+
+Most of the things a usage event would naturally want to name are named by
+someone other than us: an MCP server key is a key in your settings, a skill is
+a directory in your repo, a plugin id is chosen by its author, an agent profile
+and a plugin's slash commands are project-local. On a private repo those names
+are your employer, your internal services, or fragments of your paths.
+
+So none of them are transmitted. Each is matched against a fixed list of names
+this project itself ships and reported as that name, or as `custom` when it
+matches nothing — with `mcp` as its own bucket for `permission_kind`, so the
+share of prompts driven by MCP stays visible without the server key coming
+with it. `skill_used` and `plugin_loaded` go further: there is no first-party
+list of skills or plugins to match against, so `skill_used` carries no name at
+all and `plugin_loaded` carries only `origin`, the discovery tier
+(`repo`, `user`, `project`, `path`).
+
+`error_class` is bucketed the same way: only the error types defined by the
+language are reported by name, because an error subclass defined in
+application or plugin code is as author-chosen as any other string. It appears
+on `crash` and nowhere else, so the column means one thing everywhere it is
+recorded.
+
+`auth_provider` is a separate property for that reason: it names which
+provider's sign-in was rejected (`codex`, `xai`), chosen from a fixed
+first-party set in `src/tui-opentui/session-chrome.ts`. No part of the
+provider's rejection message is sent.
+
+The mapping is `src/telemetry/classify.ts`, and the tests that feed each
+emission site a deliberately identifying name and assert it reaches no part of
+the payload are in `tests/unit/telemetry-product-events.test.ts`.
+
 ## What's never collected
 
 - Prompts, model output, or any conversation content
 - File paths, file contents, or repo/project names
+- Names anyone but this project chose: MCP servers, skills, plugins, agent
+  profiles, plugin-registered slash commands, error subclasses (see above)
+- Shell commands, tool arguments, or tool results
 - API keys, tokens, or any other credential
 - Anything not in the allowlist above
 
@@ -46,6 +94,12 @@ Any of the following disables telemetry entirely:
 - `CORBITS_TELEMETRY` set to any falsy value: `0`, `false`, `off`, `no`, or empty
 - `DO_NOT_TRACK=1` (the standard [Console Do Not Track](https://consoledonottrack.com/) convention)
 
+Turning telemetry off also discards whatever is still queued and unsent.
+Events captured earlier in the session but not yet transmitted are thrown
+away at the moment you opt out, not sent on the way out — opting out covers
+the activity you have already generated, not just the activity still to
+come.
+
 Re-enable from the same Telemetry tab or by removing the env var / settings
 override. While an env kill is active the Telemetry tab cannot re-enable —
 the env override always wins, and the attempt is refused rather than
@@ -92,9 +146,30 @@ Events are sent to PostHog. PostHog derives an approximate country from the
 request IP server-side; the client sends no location data itself. No
 self-hosted or third-party analytics beyond PostHog are used.
 
+## On the wire
+
+Events are not sent one at a time. Each captured event is stamped with its
+capture time and held in an in-memory queue, which is posted to PostHog's
+`/batch/` endpoint when it reaches the batch size or when the batch
+interval elapses, whichever comes first. At most one request is ever in
+flight: events captured while a request is open wait for it rather than
+opening another connection. Exit paths flush the queue, bounded by a short
+deadline so a slow endpoint cannot delay quitting.
+
+The queue has a hard depth limit. Once it is full — which in practice means
+the endpoint is unreachable, as on a captive portal or behind a hung proxy
+— the oldest queued events are dropped to make room for new ones. Telemetry
+is therefore lossy by design: it never grows memory without bound, never
+retries indefinitely, and never blocks or reports failures to the user.
+Nothing is written to disk, so dropped events are gone rather than deferred
+to a later run.
+
+See `src/telemetry/index.ts` for the batch size, interval, and queue limit
+in force.
+
 ## Not this document
 
 Local performance tracing and optional OpenTelemetry export to an operator-owned
 collector (Phoenix, PostHog OTEL, Jaeger, generic OTLP) are documented in
-`docs/PERFTRACE.md`. That pipe is separate: it does not expand these three
-events, and product telemetry opt-out does not control OTEL export.
+`docs/PERFTRACE.md`. That pipe is separate: it does not expand the events
+above, and product telemetry opt-out does not control OTEL export.

src/agent/tools.ts

@@ -16,6 +16,7 @@ import {
   type ShellTimeoutConfig,
 } from "../plugins/shell-guard-plugin.js";
 import { advertiseEditFileLineRange } from "../plugins/edit-file-line-range.js";
+import type { Telemetry } from "../telemetry/index.js";
 import type { PermissionGate } from "../permission/gate.js";
 import { buildCorePosixToolPlugins } from "./posix-tool-plugins.js";
 import { createLazyBlobReader } from "./lazy-blob-reader.js";
@@ -112,6 +113,9 @@ export type AgentToolsetArgs = {
   // Real sessions always pass their detected values — see tool-search.ts for
   // why these must be fixed for the session's life.
   toolAvailability?: ToolAvailability;
+  // Records skill loads and sub-agent dispatch. Omitted (tests, ad-hoc
+  // toolsets) means those events are never emitted.
+  telemetry?: Telemetry;
   // When provided, the agent gets a `task` tool that delegates to autonomous
   // sub-agents. Omitted in contexts that cannot spawn sub-agents (e.g. tests).
   subAgent?: {
@@ -210,7 +214,7 @@ export async function createAgentToolset(args: AgentToolsetArgs): Promise<AgentT
       ),
     })),
     createListDirTool(cwd),
-    createUseSkillTool(cwd, skillDirs),
+    createUseSkillTool(cwd, skillDirs, args.telemetry),
     createWebFetchTool(),
     createWebSearchTool(),
     ...(subAgentsEnabled && args.subAgent !== undefined
@@ -237,6 +241,7 @@ export async function createAgentToolset(args: AgentToolsetArgs): Promise<AgentT
             ...(args.subAgent.useWorktree !== undefined
               ? { useWorktree: args.subAgent.useWorktree }
               : {}),
+            ...(args.telemetry !== undefined ? { telemetry: args.telemetry } : {}),
           }),
           ...(args.subAgent.profiles !== undefined
             ? [

src/agent/use-skill.ts

@@ -4,6 +4,7 @@ import type { ToolDefinition } from "@intx/types/runtime";
 import { type } from "arktype";
 
 import { resolveSkillBody } from "../extensions/skills.js";
+import { NOOP_TELEMETRY, type Telemetry } from "../telemetry/index.js";
 
 // Lazy skill loading: the available skills are listed by name + description in
 // the system prompt, but their full instructions are pulled into context only
@@ -24,7 +25,11 @@ const useSkillDefinition: ToolDefinition = {
 
 const UseSkillArgs = type({ name: "string" });
 
-export function createUseSkillTool(cwd: string, skillDirs: string[] = []): AgentTool {
+export function createUseSkillTool(
+  cwd: string,
+  skillDirs: string[] = [],
+  telemetry: Telemetry = NOOP_TELEMETRY,
+): AgentTool {
   return stringTool({
     definition: useSkillDefinition,
     handler: async (rawArgs: Record<string, unknown>): Promise<string> => {
@@ -34,6 +39,10 @@ export function createUseSkillTool(cwd: string, skillDirs: string[] = []): Agent
       if (name.length === 0) return "Error: use_skill requires a non-empty name.";
       const body = await resolveSkillBody(cwd, name, skillDirs);
       if (body === undefined) return `No skill named "${name}" is available.`;
+      // Skills are project- or plugin-authored, so the name is as identifying
+      // as any other user-written string and never leaves the process; the
+      // event records only that a skill was loaded.
+      telemetry.capture("skill_used");
       return `Skill "${name}" — follow these instructions for this task:\n\n${body}`;
     },
   });

src/exec/runner.ts

@@ -53,6 +53,7 @@ import type {
   PermissionRequest,
 } from "../permission/types.js";
 import { createAgentToolset, type AgentToolset, type OperatorResult } from "../agent/tools.js";
+import { liveTelemetry } from "../telemetry/singleton.js";
 import { collectToolPlugins, resolveToolPlugins } from "../plugins/tool-plugins.js";
 import {
   expandExistingPluginMembers,
@@ -262,6 +263,7 @@ export async function runExec(config: Config): Promise<ExecResult> {
       isProjectPluginTrusted,
       isRegisteredPathTrusted,
       diagnostics: pluginLoadDiag,
+      telemetry: liveTelemetry,
     });
     emitPluginWarningSummary(pluginLoadDiag, (line) => logger.warn(line));
     // Metadata-only (untrusted) modules stay out of executable plugins.
@@ -298,6 +300,7 @@ export async function runExec(config: Config): Promise<ExecResult> {
 
     const permissionGate = createPermissionGate({
       approvals: seededApprovals,
+      telemetry: liveTelemetry,
       cwd: config.cwd,
       rootsProvider: createWorktreeRootsProvider(config.cwd),
       providerName: config.providerName,
@@ -326,6 +329,7 @@ export async function runExec(config: Config): Promise<ExecResult> {
       cwd: config.cwd,
       permissionGate,
       skillDirs,
+      telemetry: liveTelemetry,
       ...(shellTimeout !== undefined ? { shellTimeout } : {}),
       ...(toolWatchdog !== undefined ? { toolWatchdog } : {}),
       ...(localSettingsForMode?.env !== undefined ? { shellEnv: localSettingsForMode.env } : {}),
@@ -544,6 +548,7 @@ export async function runExec(config: Config): Promise<ExecResult> {
           "pruning-compactor": createSessionPruningCompactor({
             compactionMode: liveCompactionMode,
             summarize: summarizeForCompaction,
+            telemetry: liveTelemetry,
           }),
         },
       });

src/index.ts

@@ -9,6 +9,7 @@ import { ensureTelemetrySettings, globalSettingsPath } from "./config/settings.j
 import { installFileLogSink } from "./logging/sink.js";
 import { flushPerfToOtel } from "./perf/index.js";
 import { createTelemetry, telemetryDisabledByEnv } from "./telemetry/index.js";
+import { classifyErrorClass } from "./telemetry/classify.js";
 import { getTelemetry, setTelemetry } from "./telemetry/singleton.js";
 import { runExec } from "./exec/runner.js";
 import { runOnboarding } from "./tui/onboarding.js";
@@ -154,6 +155,12 @@ export async function handleFatal(kind: CrashKind, error: unknown): Promise<void
     process.stderr.write("failed to write crash report\n");
   }
   await finalizeActiveRunOnCrash(error);
+  // kind is one of the two process-level handler names. A constructor name is
+  // author-chosen — an application or plugin error subclass can be as
+  // identifying as any other free text — so only the language's own error
+  // types are reported by name.
+  getTelemetry().capture("crash", { kind, error_class: classifyErrorClass(error) });
+  await getTelemetry().flush();
   process.exit(1);
 }
 

src/permission/gate.ts

@@ -28,6 +28,25 @@ import {
 import type { MCPClient } from "../mcp/client.js";
 import { end, start } from "../perf/index.js";
 import { currentTurnId } from "../perf/reactor-spans.js";
+import { classifyPermissionKind } from "../telemetry/classify.js";
+import { NOOP_TELEMETRY, type Telemetry } from "../telemetry/index.js";
+
+// Closes out an operator prompt: ends the wait span and records the outcome.
+// buildRequests yields at most one request per tool call, and the two prompt
+// sites below are mutually exclusive, so this runs once per prompt shown.
+function finishApprovalWait(
+  telemetry: Telemetry,
+  waitSpanId: string,
+  tool: string,
+  outcome: ApprovalOutcome | undefined,
+): void {
+  const decision = outcome !== undefined && outcome.allow ? "allow" : "deny";
+  end(waitSpanId, outcome !== undefined ? { decision } : undefined);
+  telemetry.capture("permission_prompt", {
+    decision,
+    permission_kind: classifyPermissionKind(tool),
+  });
+}
 
 export type GateVerdict = { allowed: true } | { allowed: false; reason: string };
 
@@ -211,6 +230,10 @@ export type PermissionGateOptions = {
   // restriction anchored to the session cwd; a caller resolving a sub-agent
   // request's own cwd would clear restrictions the gate still enforces.
   onGrant?: (approval: Approval, covers: (request: PermissionRequest) => boolean) => void;
+  // Records that a prompt was shown and how it was answered. Injected rather
+  // than read from the process-wide handle so a gate built without one is
+  // silent by construction.
+  telemetry?: Telemetry;
 };
 
 export type PermissionGate = {
@@ -246,6 +269,7 @@ export type PermissionGate = {
 
 export function createPermissionGate(options: PermissionGateOptions): PermissionGate {
   const { requestApproval, persist, interactive, skipPermissions, providerName, model, cwd } = options;
+  const telemetry = options.telemetry ?? NOOP_TELEMETRY;
   const mcpTiers = options.mcpTiers ?? createMcpToolPermissionRegistry();
   const resolvedCwd = cwd ?? process.cwd();
   const rootsProvider = options.rootsProvider ?? createWorktreeRootsProvider(resolvedCwd);
@@ -463,12 +487,7 @@ export function createPermissionGate(options: PermissionGateOptions): Permission
         try {
           outcome = await requestApproval(requestForOperator);
         } finally {
-          end(
-            waitSpanId,
-            outcome !== undefined
-              ? { decision: outcome.allow ? "allow" : "deny" }
-              : undefined,
-          );
+          finishApprovalWait(telemetry, waitSpanId, request.tool, outcome);
         }
         if (outcome === undefined || !outcome.allow) {
           const suffix =
@@ -516,12 +535,7 @@ export function createPermissionGate(options: PermissionGateOptions): Permission
       try {
         outcome = await requestApproval(request);
       } finally {
-        end(
-          waitSpanId,
-          outcome !== undefined
-            ? { decision: outcome.allow ? "allow" : "deny" }
-            : undefined,
-        );
+        finishApprovalWait(telemetry, waitSpanId, request.tool, outcome);
       }
       if (outcome === undefined || !outcome.allow) {
         const suffix =