Skip to content

fix(analytics): accurate log/cost reporting and always-on cost display - #530

Open
8nevil8 wants to merge 17 commits into
mainfrom
fix/analytics-log-accuracy-and-cost-display
Open

fix(analytics): accurate log/cost reporting and always-on cost display#530
8nevil8 wants to merge 17 commits into
mainfrom
fix/analytics-log-accuracy-and-cost-display

Conversation

@8nevil8

@8nevil8 8nevil8 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes analytics-reporting accuracy gaps in codemie analytics and adds token-usage propagation to CodeMie's live session metrics for anthropic-subscription (and, more broadly, any claude-agent) sessions, so the backend can derive accurate cost/usage figures server-side.

Changes

Analytics accuracy (original scope)

  • hadLog correctness: loadAgentSessionFile now verifies the correlated/native transcript path still exists on disk before treating a session as hadLog=true. A coding agent's own retention (e.g. Claude Code's default 30-day cleanup) can rotate a transcript away after the session ran while the correlation record still names it — 122 of 291 real Claude sessions in one report were misreported this way.
  • Ownership marker scan window: hasOwnershipMarker's legacy-session scan widened from a 4KB/10-line budget to a 256KB byte-bounded scan, now with a full-file fallback past that budget so a larger hookAdditionalContext blob can't reproduce the same misclassification at a higher threshold.
  • Active duration + always-on cost: sessions now surface CodeMie's own tracked activeDurationMs alongside the existing wall-clock duration. Cost enrichment now always runs for the plain console command, not just --report. --export json/--export csv now include cost and active-duration data consistent with the console output.
  • Test cleanup: the skills-setup integration test's Global scope installs into the real ~/.claude/skills/ and never cleaned that up — now removed in afterAll.

Live metrics token propagation (new scope)

  • MetricDelta gains a tokens field ({ input, output, cacheRead?, cacheCreation? }), captured by claude.metrics-processor.ts from the Anthropic usage object on every completed turn — previously read only to detect stream completion and discarded. Numeric fields are coerced defensively (Number(...) || 0).
  • Live /v1/metrics payload (SessionLifecycleAttributes, ToolUsageAttributes) gains four new flat, optional, snake_case fields: input_tokens, output_tokens, cache_read_tokens, cache_creation_tokens — additive only, never a dict/map shape, to stay compatible with the backend's documented behavior of silently dropping payloads containing unrecognized/dict-shaped fields. ToolUsageAttributes carries incremental since-last-sync totals; SessionLifecycleAttributes carries whole-session cumulative totals.
  • No cost is ever computed or sent by the client — only raw token counts, same pattern as the existing Codex integration.
  • MetricsSender.sendSessionEnd gains new optional trailing parameters (mirroring the existing activeDurationMs precedent); DesktopTelemetryRuntime's session-end path now also sums and sends token totals, matching the CLI hook path.
  • Resilience: MetricsWriter.readAll() now skips a single malformed metrics-delta line instead of aborting the entire session-end metrics send.
  • Debug-log and dry-run projections in MetricsApiClient now include the four new token fields.
  • Statusline: the Claude Code statusline now shows cumulative session token totals (input/output/cache) read from the metrics-delta file, alongside the existing live context-window snapshot.

Testing

  • Unit/integration tests added for each fix and each new capture/propagation path (token coercion, corrupt-delta handling, desktop-telemetry parity, native-loader fallback, export column coverage, activeDurationMs passthrough, cost-enrichment on the plain path)
  • npm run typecheck, lint, and secrets scan passed via pre-commit hooks on each commit
  • Full qa-gates run (license-check, lint, typecheck, build, unit, integration, secrets, commitlint) passed

Checklist

  • Code follows project standards
  • CI is green (npm run ci)
  • No merge conflicts with main

8nevil8 and others added 17 commits September 3, 2026 19:33
…Log=true

The cost enricher's loadAgentSessionFile trusted a correlation record's
agentSessionFile path without checking it still exists. A coding agent's own
retention (e.g. Claude Code's default 30-day cleanup) can rotate a transcript
away long after the session ran; the correlation record still names it. 122 of
291 real Claude sessions in one report claimed hadLog=true against a deleted
file, misrepresenting the Coverage table's "Native log" count.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
…tart payloads

hasOwnershipMarker's bounded transcript scan for legacy sessions without a
sidecar marker used a 4KB byte budget capped to the first 10 lines. A
SessionStart hook can inject a large hookAdditionalContext blob (CLAUDE.md,
memory files, ...) as one of the transcript's first lines — observed up to
~100KB on real sessions — pushing codemie_session_start past both limits.
153 CodeMie-owned Claude sessions were misreported as native-external for
exactly this reason. Scan is now 256KB, bounded by bytes read rather than
line count, since one large early line can consume most of a small budget
by itself.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
Two related gaps in the plain console command (not --report):

- Session duration only showed the wall-clock span (endTime - startTime),
  which counts idle/overnight gaps and can overstate active work by an
  order of magnitude for a resumed session. Now also surfaces CodeMie's own
  tracked activeDurationMs when the session JSON carries it, labeled
  separately from the wall-clock span.
- Cost enrichment previously only ran when --report was passed, so the
  plain console command — the one most invocations use — never computed or
  displayed cost at all. It now always enriches from correlated logs when
  no source (OTEL) cost is present, and both `Est. Cost` and each session's
  `Cost:` line are populated on every path; --report reuses the same result
  rather than re-enriching.

Also relabels the HTML report's "Files touched" stat to "File operations"
to distinguish it from the report's separate unique-files-changed columns.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
…tion test

The "setup skills" wizard's Global scope installs into the real
~/.claude/skills/ (claude-skill-generator.ts hardcodes os.homedir() there,
since .claude/ is Claude Code's own directory, not something CODEMIE_HOME
redirection reaches). That install escapes the temp CODEMIE_HOME used
elsewhere in this suite and was never cleaned up, leaking a skill dir into
the real user environment on every run. Removes it in afterAll by matching
the on-disk slug prefix (`${skillName}-${project}-global`) rather than
duplicating the exact slug algorithm.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
Read the already-detected completedMsg.message.usage object in
claude.metrics-processor.ts into a new MetricDelta.tokens field
(input/output/cacheRead/cacheCreation), following the codebase's existing
additive optional field convention. No cost/money computation.

Part of anthropic-live-metrics-tokens (task 1).

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
Add input_tokens/output_tokens/cache_read_tokens/cache_creation_tokens as
flat optional numeric fields to SessionLifecycleAttributes/ToolUsageAttributes,
and sum MetricDelta.tokens across deltas in metrics-aggregator.ts, spreading
each field conditionally (only when its total is > 0). tool_counts stays
banned; no schema_version bump.

Part of anthropic-live-metrics-tokens (task 2).

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
Add an optional trailing tokens parameter to MetricsSender.sendSessionEnd
(source-compatible with existing 6-arg callers) and read+sum the session's
metrics-delta file in hook.ts:sendSessionEndMetrics via MetricsWriter.readAll(),
which already tolerates a missing/empty file by returning []. A missing delta
file yields a zeroed tokens object rather than a thrown error.

Part of anthropic-live-metrics-tokens (task 3).

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
…gging

Add the four token fields to the sendRequest debug-log allow-list and to the
sendSessionEnd dry-run log projection, so operators can see token totals in
logs without them landing in real request bodies unless the attributes
already carry them.

Part of anthropic-live-metrics-tokens (task 4, final task).

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
usage.input_tokens/output_tokens/cache_read_input_tokens/cache_creation_input_tokens
were read via `?? 0` with no numeric guard; a non-numeric value would corrupt
downstream token sums via string concatenation.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
MetricsWriter.readAll() re-threw on a single unparseable JSONL line, which
propagated out of hook.ts's sendSessionEndMetrics and dropped the entire
session-end metrics send (status, duration, tokens) rather than just the
bad line.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
DesktopTelemetryRuntime.sendSessionEndMetric called sender.sendSessionEnd
with undefined tokens, silently diverging from the CLI hook path which
already sums MetricsWriter deltas before sending.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
…dget

hasOwnershipMarker's bounded scan returned false outright when the
codemie_session_start marker fell beyond its 256KB budget, reproducing
the same misreported-as-native-external bug at a higher size threshold.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
--export json/csv serialized the plain analytics tree without the
costIndex/costSummary merge the console path already applies, so exported
files silently omitted Est. Cost / per-session Cost and (for CSV) active
duration versus what the console just showed.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
…chment

activeDurationMs passthrough (aggregator) and unconditional cost enrichment
on the plain (non-report) analytics path had no asserting test, so a
regression in either could silently drop console output.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
cli-misc-coverage.test.ts asserted the pre-fix CSV header/row shape; the
CR-003 fix-up added Active Duration (s) and Cost (USD) columns to
AnalyticsExporter.exportCSV, which this pinned-behavior test hadn't caught up to.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_0146TJLVz6A1KcRoGRMn3MJx
… cumulative

Rename the context-window snapshot pair to `ctx in:/out:` (window occupancy) and the cumulative uncached input from `in:` to `new:`, so the two are no longer both labeled `in:`.
Anthropic's input_tokens excludes cached tokens, so cumulative `new` reads far smaller than cache reads (cR) by design; the labels now make that explicit instead of looking like a miscount.

Generated with AI

Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
Claude-Session: https://claude.ai/code/session_01NMpJkSvuRbb7MMC25XDY1Q
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants