diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml new file mode 100644 index 0000000..b6a4a5a --- /dev/null +++ b/.github/workflows/tests.yml @@ -0,0 +1,18 @@ +name: Token usage tests +on: + pull_request: + push: + branches: [main] +jobs: + tests: + runs-on: ubuntu-latest + strategy: + matrix: + python: ['3.11', '3.14'] + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python }} + - run: python3 -m unittest discover -s tests -v + - run: node --check dashboard/static/app.js diff --git a/CLAUDE.md b/CLAUDE.md index db225aa..b14a2e5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,7 +8,7 @@ This file is auto-loaded by Claude Code into its context whenever you open this ## What is this project -`token-usage` is a Claude Code skill that does local-first analytics on your own Claude Code transcripts (`~/.claude/projects/**/*.jsonl`). Three interfaces: +`token-usage` is a shared skill for local-first analytics on Claude Code transcripts (`~/.claude/projects/**/*.jsonl`) and Codex rollouts (`CODEX_HOME` or `~/.codex`, sessions and archived_sessions). Three interfaces: - **CLI** — `scripts/count_tokens.py`, `scripts/analyze.py` - **Browser dashboard** — `dashboard/server.py` + static frontend @@ -26,7 +26,7 @@ These are non-negotiable. Don't suggest changes that violate them without explic 1. **Pure stdlib only.** No `numpy`, `pandas`, `scipy`, `requests`, `flask`, `fastapi`, `pyyaml`, etc. on the Python side. The dashboard frontend uses Chart.js loaded from a CDN — that's the only external runtime dependency in the whole project. 2. **Zero install steps.** A user who clones the repo into `~/.claude/skills/token-usage/` should be able to run any script with `python3 scripts/foo.py` immediately. No `pip install -r requirements.txt`. No `npm install`. -3. **Don't touch `~/.claude/projects/**/*.jsonl`.** Read-only, ever. Those are the user's transcripts. +3. **Don't touch Claude or Codex transcript files.** Read-only, ever. Those are the user's transcripts. 4. **Chinese UI strings stay Chinese.** Don't "translate" 用法画像 / 突发型 / 晚高峰 etc. to English. The user-facing language is Chinese; code/comments are English. 5. **No telemetry, no analytics, no outbound calls.** Only outbound call permitted: the local `claude` CLI subprocess from the AI 解读 button (server.py `h_interpret`). @@ -40,6 +40,8 @@ README.md English landing page (GitHub default). README.zh-CN.md Chinese landing page. scripts/ + codex_records.py Codex request/counter parsing, mirrors and fork-history dedup. + transcript_index.py Disposable per-file SQLite/memory parsing cache for dashboard. token_stats.py CORE — JSONL parsing, dedup-by-message-id (field-wise max), time-window filtering, bucket aggregation, billing-equiv (5m/1h ephemeral cache split). Imported by everything else. @@ -74,6 +76,10 @@ docs/ **The single source of truth for token math is `scripts/token_stats.py`.** Don't reinvent dedup or billing-equiv logic anywhere else — import from there. +Codex cache is included in input and reasoning is included in output. Never add these subsets again. Pricing/billing-equivalent values cover Claude only; expose the unpriced Codex scope. See [Codex accounting](docs/codex-usage.md). Run `python3 -m unittest discover -s tests -v` for regression tests. + +Dashboard persistence is a rebuildable statistics cache, not a transcript store. Keep per-file identities and raw counter candidates until global dedup; caching only final per-file totals breaks copied/forked/mirrored records. See [index lifecycle](docs/performance.md) and [ADR 001](docs/decisions/001-transcript-index.md). + --- ## How the analysis modules talk to each other diff --git a/README.md b/README.md index cfae5c0..6c8df25 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # token-usage -> Local-first analytics for your Claude Code token consumption. +> Local-first analytics for your Claude Code and Codex token consumption. **English** · [简体中文](./README.zh-CN.md) -A [Claude Code](https://claude.com/claude-code) skill that reads JSONL transcripts under `~/.claude/projects/`, aggregates per model / project / day, and surfaces patterns and AI-generated interpretations through four interfaces: +A local skill for Claude Code and Codex that reads JSONL transcripts under `~/.claude/projects/` and `${CODEX_HOME:-~/.codex}/sessions/` / `archived_sessions/`, aggregates per source / model / project / day, and surfaces patterns and optional AI-generated interpretations through four interfaces: - **CLI** — `count_tokens.py` for one-shot reports in any time range - **Browser dashboard** — charts + a **Work-efficiency panel** (throughput per active minute, CDF + KDE) + a **Patterns panel** (Markov / ACF / change-point) + an in-page **🤖 AI 解读** button @@ -13,6 +13,8 @@ A [Claude Code](https://claude.com/claude-code) skill that reads JSONL transcrip Zero dependencies (pure stdlib, no `numpy` / `pandas` / `chart.js` server-side). Chart.js is loaded from CDN by the browser only. +By default both sources are counted. Use `--source claude|codex|all` on the CLIs, or the dashboard's source selector. Codex cache/reasoning subsets are never counted twice; cost and billing-equivalent estimates currently cover Claude only. See [Codex accounting and tests](docs/codex-usage.md). + --- ## Screenshots @@ -47,6 +49,8 @@ Claude Code's built-in `~/.claude/stats-cache.json` lags by one day and gives yo git clone https://github.com/WholeNightCoding/token-usage ~/.claude/skills/token-usage ``` +For shared discovery by Codex, clone into `~/.agents/skills/token-usage` instead; invoke scripts from that actual folder. Codex's data directory can be overridden with `--codex-home` or `CODEX_HOME`. + The skill is auto-discovered by Claude Code. Just open Claude Code in any project and ask: > *"How many tokens did I use today?"* @@ -98,9 +102,12 @@ python3 ~/.claude/skills/token-usage/dashboard/server.py # → opens http://127.0.0.1:8787/ in your default browser ``` +The dashboard keeps a rebuildable local statistics index across restarts and reparses only changed files. The first build reads all history. Use `--cache-dir PATH` or `--no-cache` to control disk caching; see [performance and cache lifecycle](docs/performance.md). + What you get: -- **4 KPI cards** — total / billing-equiv / USD estimate / last-1h rate +- **4 overview cards** — total / Claude Code / Codex / last-1h rate, with the selected scope and each source's share. Excluded sources display “—” instead of zero. +- **Claude cost reference** — expandable API cost and billing-equivalent estimates; excludes Codex and is not a subscription bill. - **Daily trend** — stacked bar chart by model, with totals labeled - **By model / Top 10 projects** — doughnut + bar - **Realtime** — last-Nh line chart, auto-refreshes every 10s, configurable bucket size (1 min → 4 hour) diff --git a/README.zh-CN.md b/README.zh-CN.md index 8af638b..dec7f6d 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -1,10 +1,10 @@ # token-usage -> Claude Code token 用量本地化分析工具。 +> Claude Code 与 Codex token 用量本地化分析工具。 [English](./README.md) · 简体中文 -一个 [Claude Code](https://claude.com/claude-code) skill —— 直接读 `~/.claude/projects/` 下的 JSONL transcript,按 model / project / day 聚合,并通过四个界面把数据变成可读的洞察: +一个同时支持 Claude Code 与 Codex 的本地 skill —— 读取 `~/.claude/projects/` 和 `${CODEX_HOME:-~/.codex}/sessions/`、`archived_sessions/` 下的 JSONL,按来源 / model / project / day 聚合,并通过四个界面把数据变成可读的洞察: - **CLI** — `count_tokens.py` 任意时间区间一次性出报告 - **浏览器仪表板** — 图表 + **工作效率面板**(按活跃分钟算吞吐 + CDF/KDE 分布图)+ **Patterns 面板**(Markov / ACF / 变化点检测)+ 内嵌 **🤖 AI 解读** 按钮 @@ -13,6 +13,8 @@ 零 pip 依赖(纯 stdlib,连 numpy/pandas 都没有)。Chart.js 只在浏览器端通过 CDN 加载。 +默认统计两者;命令行可用 `--source claude|codex|all`,面板可选“全部 / Claude Code / Codex”。Codex 缓存和推理子集只记一次,金额与计费折算目前只包含 Claude。详见 [Codex 统计口径与验证](docs/codex-usage.md)。 + --- ## 截图 @@ -47,6 +49,8 @@ Claude Code 自带的 `~/.claude/stats-cache.json` 有两个不爽的地方:** git clone https://github.com/WholeNightCoding/token-usage ~/.claude/skills/token-usage ``` +希望 Codex 发现共享 skill 时,可安装到 `~/.agents/skills/token-usage`;脚本从实际安装目录执行。Codex 数据目录可通过 `--codex-home` 或 `CODEX_HOME` 指定。 + Claude Code 会自动发现这个 skill。打开任意项目,问它: > *「我今天用了多少 token?」* @@ -87,6 +91,9 @@ python3 ~/.claude/skills/token-usage/scripts/count_tokens.py \ # JSON 输出(管道用) python3 ~/.claude/skills/token-usage/scripts/count_tokens.py --json + +# 只看 Codex +python3 ~/.claude/skills/token-usage/scripts/count_tokens.py --source codex --today ``` 输出列:`MODEL | MSGS | INPUT | OUTPUT | CACHE_READ | CACHE_CREATE | TOTAL`,外加总计和按权重 `input=1×, cache_read=0.1×, cache_create_5m=1.25×, cache_create_1h=2×, output=5×` 算的「计费等效输入 token」估算。 @@ -98,9 +105,12 @@ python3 ~/.claude/skills/token-usage/dashboard/server.py # → 自动打开 http://127.0.0.1:8787/ ``` +仪表板用本机统计索引复用历史解析结果,重启后也保留,只重新解析变化的文件。首次建立索引仍需读取历史。可以用 `--cache-dir PATH` 或 `--no-cache` 控制磁盘缓存,详见[性能与索引生命周期](docs/performance.md)。 + 包含: -- **4 张 KPI 卡** — 总量 / 计费等效 / 美元估算 / 最近 1 小时速率 +- **4 张概览卡** — 总用量 / Claude Code / Codex / 最近一小时速率,标明筛选范围和来源占比;未纳入筛选的来源显示“—”,不误显示为零。 +- **Claude 费用参考** — 展开查看 API 金额和计费等效 token 的估算,不含 Codex,也不是订阅账单。 - **Daily trend** — 按 model 堆叠的柱状图,柱顶标总计 - **By model / Top 10 projects** — 甜甜圈 + 柱状 - **Realtime** — 最近 N 小时折线图,每 10 秒自动刷新,桶大小可选(1 min → 4 hour) diff --git a/SKILL.md b/SKILL.md index d0ab06f..75f89fa 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,84 +1,58 @@ --- name: token-usage -description: Use when the user asks how many Claude Code tokens were consumed — today, yesterday, this week, this month, on a specific date, in a rolling window (last Nh/Nd), or across a custom date/time range — or wants a per-model breakdown with cache-read vs. cache-create split. Also applies when `~/.claude/stats-cache.json` is stale (it only updates through the previous day) or when the user wants a billing-equivalent estimate from local transcripts. Also triggers for visual/browser-based token usage monitoring — "dashboard", "监控板", "面板", "show me in a browser", live rate, per-project breakdown — via the local web UI at `dashboard/server.py`. +description: Report local Claude Code and Codex token usage for today, a date, a rolling window, or a custom range, with model, project, day, and source breakdowns. Use the local dashboard for visual monitoring and work-efficiency analysis; use the CLI for one-shot questions. --- # Token Usage Reporter -## Overview +Read local transcripts without modifying them. Default scope is both Claude Code and Codex; use `--source claude` or `--source codex` when the user names one. These are local recorded tokens, not account quota percentages or subscription charges. -Aggregates Claude Code token `usage` records from local JSONL transcripts under `~/.claude/projects/**/*.jsonl` for any time range. More accurate than `~/.claude/stats-cache.json`, which lags by one day. +## One-shot reports -**Core principle:** parse raw transcripts → filter by `timestamp` → dedupe by `message.id` → sum per model. - -## When to Use - -Trigger symptoms: -- "How many tokens did I use today / yesterday / this week / last 6 hours?" -- "Token usage between 2026-04-01 and 2026-04-15" -- "Breakdown by model" or "how much was cache vs. new input?" -- Questions about cost / billing-equivalent tokens for a period -- `stats-cache.json` doesn't include today's data -- User wants a **visual dashboard / monitoring page** (e.g. "token 监控板", "打开 token 面板", "in a browser", per-project top-N, live rate) → use the dashboard, not a one-shot CLI table - -**Do NOT use for:** message/session counts (those are in `stats-cache.json` — read it directly). - -## Quick Reference - -Script: `~/.claude/skills/token-usage/scripts/count_tokens.py` (invoke with `python3`). - -| Range flag | Scope | -|---|---| -| `--today` (default) | Today, local tz | -| `--yesterday` | Yesterday, local tz | -| `--this-week` | Mon–Sun of current week | -| `--this-month` | Current calendar month | -| `--all` | Entire history (all transcripts) | -| `--date YYYY-MM-DD` | One specific day | -| `--last 7d` / `12h` / `30m` | Rolling window ending now | -| `--from X --to Y` | Explicit range, `YYYY-MM-DD` or `YYYY-MM-DD HH:MM[:SS]` | - -Modifiers: `--by-day` (per-day table for multi-day ranges), `--json` (machine-readable), `--projects-dir PATH` (override transcript root). - -## Dashboard (visual, browser-based) - -Use when the user wants ongoing monitoring, live rate, per-project breakdown, or just a more navigable view than a CLI table. +Run from this skill's actual directory; do not assume it lives in `~/.claude/skills` or `~/.codex/skills`. ```bash -python3 ~/.claude/skills/token-usage/dashboard/server.py -# default: 127.0.0.1:8787, auto-opens browser -# flags: --port N, --host ADDR (keep 127.0.0.1 unless asked), --no-open +python3 scripts/count_tokens.py --today +python3 scripts/count_tokens.py --source codex --yesterday +python3 scripts/count_tokens.py --source claude --this-week +python3 scripts/count_tokens.py --from 2026-10-01 --to 2026-10-07 --by-day +python3 scripts/count_tokens.py --source codex --last 12h --json ``` -Shows: total + billing-equiv + USD estimate (pay-as-you-go) + last-1h rate cards, stacked daily-trend line, per-model doughnut, top-10 projects bar, realtime 1h line (auto 10s refresh), a **work-efficiency panel** (active hours, % of wall-clock, throughput while actually working, 30-min slot rate distribution, per-day breakdown — answers "how productive am I during the hours I'm at the keyboard?"), and a model×project detail table. Range selector: Today / Yesterday / This week / This month / 7d / 30d / All time / Custom… (from–to date pickers; All time auto-clamps to the first day with activity). +Range flags: `--today` (default), `--yesterday`, `--this-week`, `--this-month`, `--all`, `--date YYYY-MM-DD`, `--last 7d|12h|30m`, or `--from X --to Y`. Dates/times use local timezone. JSON retains the existing `by_model` and `total` fields, and adds `by_source`, reasoning-output and pricing-scope metadata. -The efficiency panel reports rates against **active minutes** (minutes that consumed tokens), not wall clock — so a 20-min work block isn't averaged against the 40 idle minutes around it. Also available as a CLI: `python3 ~/.claude/skills/token-usage/scripts/work_efficiency.py [days]`. +Sources: -Server: same data source as CLI (numbers match), in-memory cached + mtime-invalidated, binds `127.0.0.1` only, no auth. +- Claude Code: `~/.claude/projects/**/*.jsonl`; override with `--projects-dir`. +- Codex CLI/Desktop: `${CODEX_HOME:-~/.codex}/sessions/**/*.jsonl` and `archived_sessions/**/*.jsonl`; override with `--codex-home` (the home containing both folders). -**Do NOT launch the dashboard just to answer a one-shot CLI-style question.** Use `count_tokens.py` instead for "how many tokens yesterday" — it's faster and fits better in chat. +## Dashboard -## Canonical Example +Use when the user asks for a monitoring page, charts, live rates or navigable analysis. Do not launch it for a one-shot token-count question. ```bash -python3 ~/.claude/skills/token-usage/scripts/count_tokens.py --from 2026-04-01 --to 2026-04-21 --by-day +python3 dashboard/server.py +# 127.0.0.1:8787; --no-open, --port N, --projects-dir PATH, --codex-home PATH ``` -Output columns: `MODEL | MSGS | INPUT | OUTPUT | CACHE_READ | CACHE_CREATE | TOTAL`, plus grand total and a **billing-equivalent input tokens** estimate using weights `input=1×, cache_read=0.1×, cache_create_5m=1.25×, cache_create_1h=2×, output=5×`. CACHE_CREATE in the table is the 5m+1h sum; the billing-equiv splits them out via the per-record `cache_creation.ephemeral_1h_input_tokens` field. +The source selector (全部 / Claude Code / Codex) applies to totals, trends, models, projects, realtime, efficiency and Patterns. The dashboard reuses a local per-file statistics index across refreshes and restarts; only changed transcripts are reparsed. The first index build still reads all history. Use `--cache-dir PATH` to choose its directory or `--no-cache` for memory only; see [index lifecycle](docs/performance.md). Keep it bound to loopback unless the user requests another binding. -## Reading the output +Other reports use the same scanner: -- **Scope**: covers Claude Code only. Anthropic API direct calls, Claude.ai web/desktop usage are NOT in these transcripts — totals will be lower than your actual Anthropic billing. -- **Sub-agent dedup is automatic**: same `message.id` appearing in parent + sub-agent transcripts is merged via field-wise max. The numbers you see are post-dedup; no double-counting. -- `` model rows are cache-only compaction events; always sum to zero, safe to ignore. -- `cache_create_5m` is priced at 1.25× input, `cache_create_1h` at 2×; billing-equiv splits them via `cache_creation.ephemeral_1h_input_tokens` per record. +```bash +python3 scripts/work_efficiency.py 7 --source codex +python3 scripts/analyze.py --days 7 --source codex --html report.html +``` -(Internals — dedup-by-max, UTC handling, subagent scanning — see CLAUDE.md.) +The existing optional AI interpretation button uses the local `claude` CLI, regardless of the selected statistics source; ordinary statistics require only Python's standard library. -## Common Mistakes +## Interpret correctly -- **Reporting `cache_read` as new-input cost.** It's 0.1× in billing; cite billing-equiv when cost is the question. -- **Using `stats-cache.json` for "today".** It only updates through the previous day. -- **Pre-converting dates to UTC.** The script handles local-tz; pass the user's date as-is. -- **Filtering by file mtime.** Transcripts append throughout a session; always filter by the JSONL `timestamp` field. +- Claude Code: dedupe parent/subagent and streaming snapshots by message ID with field-wise maxima; cache creation retains the 5-minute/1-hour split. +- Codex: prefer per-response `token_usage_record` when present, dedupe response IDs, and suppress mirrored `token_count` events. Older rollouts use cumulative-counter differences, reading preceding counters before applying a time window. Repeated snapshots, counter resets and inherited fork history are handled separately. +- Codex cached input is a subset of input; the INPUT column shows the non-cached remainder. Reasoning output is a subset of OUTPUT. Never add either subset again to total tokens. +- Amounts and billing-equivalent values use the existing Claude pricing configuration only. Codex tokens are marked unpriced. Never apply Claude rates to Codex, or present the API estimate as the user's subscription bill. Mixed-source efficiency disables the billing view and retains raw throughput. +- Missing/deleted/local-unrecorded history cannot be reconstructed. Cloud conversations without local rollout files are outside the report. +- Filter by event timestamp, not file modification date. Do not use `stats-cache.json` to answer today's usage. +For format/dedup details and validation commands, see [docs/codex-usage.md](docs/codex-usage.md). diff --git a/dashboard/server.py b/dashboard/server.py index f511065..405df13 100644 --- a/dashboard/server.py +++ b/dashboard/server.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Local web dashboard for Claude Code token usage.""" +"""Local web dashboard for Claude Code and Codex token usage.""" from __future__ import annotations import argparse @@ -13,6 +13,7 @@ import time import webbrowser from datetime import datetime, timedelta, timezone +from pathlib import Path from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from urllib.parse import parse_qs, urlparse @@ -20,6 +21,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(HERE), "scripts")) sys.path.insert(0, os.path.dirname(HERE)) import token_stats as ts # noqa: E402 +from transcript_index import TranscriptIndex, default_cache_dir # noqa: E402 import work_efficiency as we # noqa: E402 from analysis.features import time_series_features # noqa: E402 from analysis.seasonal import hour_of_day, day_of_week # noqa: E402 @@ -28,15 +30,18 @@ STATIC_DIR = os.path.join(HERE, "static") DEFAULT_PORT = 8787 +PROJECTS_ROOT = ts.PROJECTS_DIR +CODEX_ROOT = ts.CODEX_HOME -# Whole-corpus records cache: one scan feeds every endpoint. Invalidated by a -# signature over (path, mtime, size) of all JSONL files — so new activity reloads -# automatically, but switching ranges never rescans disk. +# Whole-corpus snapshots feed every endpoint. A file-index underneath reuses +# unchanged parsing; only changed transcripts are reopened, and persisted +# entries survive restarts. Cross-file accounting still runs in token_stats. _EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc) _FAR_FUTURE = datetime(9999, 1, 1, tzinfo=timezone.utc) _records_lock = threading.Lock() _records_cache: dict = {"signature": None, "records": []} +_transcript_index = TranscriptIndex() # Separate lock so concurrent cold-cache callers don't all rescan in parallel # (thundering herd). With the GIL, parallel scan_records calls serialize anyway # but waste CPU repeating the same work — the page issues 3 fetches at once on @@ -53,10 +58,10 @@ def _corpus_signature(): """Cheap fingerprint of all transcripts — just stat calls, no reads.""" out = [] - for f in ts.list_transcript_files(): + for f in ts.list_transcript_files(PROJECTS_ROOT, codex_home=CODEX_ROOT): try: st = os.stat(f) - out.append((f, st.st_mtime_ns, st.st_size)) + out.append((f, st.st_mtime_ns, st.st_size, st.st_dev, st.st_ino, st.st_ctime_ns)) except OSError: continue out.sort() @@ -65,6 +70,8 @@ def _corpus_signature(): def get_all_records(): """Return the full record list, scanning disk only when files change.""" + with _records_lock: + observed_records = _records_cache["records"] sig = _corpus_signature() # Fast path: cache hit, no scan needed. with _records_lock: @@ -74,15 +81,36 @@ def get_all_records(): # wait, then see the populated cache via the inner re-check. with _scan_lock: with _records_lock: - if _records_cache["signature"] == sig: + # Logs can grow while this request waits. Share the scan that + # finished during the wait instead of queuing another full read + # for each slightly different signature. Later requests still + # check disk and refresh normally. + if (_records_cache["signature"] == sig or + _records_cache["records"] is not observed_records): return _records_cache["records"] - records = ts.scan_records(_EPOCH, _FAR_FUTURE) + paths = [entry[0] for entry in sig] + records = ts.scan_records(_EPOCH, _FAR_FUTURE, PROJECTS_ROOT, files=paths, + codex_home=CODEX_ROOT, file_cache=_transcript_index) + _transcript_index.prune(paths) with _records_lock: _records_cache["records"] = records _records_cache["signature"] = sig return records +def source_for(q: dict) -> str: + source = q.get('source', ['all'])[0] + if source not in ts.SOURCES: + raise HTTPError(400, {'error': 'source must be all, claude, or codex'}) + return source + + +def source_records(q: dict): + source = source_for(q) + records = get_all_records() + return records if source == 'all' else [r for r in records if r.source == source] + + def _range_cache_key(q: dict): """Stable key for the current query's range. @@ -104,10 +132,10 @@ def _range_cache_key(q: dict): else: key = ("custom", q.get("from", [""])[0], q.get("to", [""])[0]) minute = int(time.time()) // 60 if ROLLING else None - return key + (minute,) + return key + (source_for(q), minute) -def resolve_qs_range(q: dict): +def resolve_qs_range(q: dict, corpus=None): """Parse from/to/range query params. Returns (start_local, end_local, label).""" if "range" in q: name = q["range"][0] @@ -116,7 +144,7 @@ def resolve_qs_range(q: dict): # Clamp to the first day with any activity — otherwise by_day / # efficiency iterate dense days from 1970 (20k+ empty rows, and # active-vs-wall-clock percentages lose all meaning). - records = get_all_records() + records = corpus if corpus is not None else source_records(q) if records: first = min(r.t_utc for r in records).astimezone(ts.local_tz()) start = first.replace(hour=0, minute=0, second=0, microsecond=0) @@ -136,25 +164,27 @@ def _filter_by_window(records, start_utc: datetime, end_utc: datetime): return [r for r in records if start_utc <= r.t_utc < end_utc] -def records_for(q: dict): - start, end, label = resolve_qs_range(q) +def records_for(q: dict, corpus=None): + corpus = source_records(q) if corpus is None else corpus + start, end, label = resolve_qs_range(q, corpus) start_utc = start.astimezone(timezone.utc) end_utc = end.astimezone(timezone.utc) - records = _filter_by_window(get_all_records(), start_utc, end_utc) + records = _filter_by_window(corpus, start_utc, end_utc) return records, {"from": start.isoformat(), "to": end.isoformat(), "label": label} # --- handlers --- def h_summary(q): - records, rng = records_for(q) + corpus = source_records(q) + records, rng = records_for(q, corpus) totals = ts.aggregate_totals(records) usd = ts.usd_estimate(records) now_local = datetime.now(ts.local_tz()) rt_start_utc = (now_local - timedelta(hours=1)).astimezone(timezone.utc) now_utc = now_local.astimezone(timezone.utc) - rt_records = _filter_by_window(get_all_records(), rt_start_utc, now_utc) + rt_records = _filter_by_window(corpus, rt_start_utc, now_utc) rt_tokens = sum(r.total for r in rt_records) return { @@ -162,6 +192,9 @@ def h_summary(q): "total": totals["all"], "billing_equiv": totals["billing_equiv"], "est_usd": round(usd, 2), + 'unpriced_tokens': totals['unpriced_tokens'], + 'billing_scope': 'claude', + 'by_source': ts.aggregate_by_source(records), "rate_1h_tokens": rt_tokens, "rate_per_min": int(rt_tokens / 60) if rt_tokens else 0, } @@ -191,7 +224,7 @@ def h_realtime(q): delta = ts.parse_duration(window) start = now_local - delta records = _filter_by_window( - get_all_records(), + source_records(q), start.astimezone(timezone.utc), now_local.astimezone(timezone.utc), ) @@ -228,12 +261,14 @@ def h_efficiency(q): Reuses the cached records corpus + the same range resolver as other endpoints (range=7d / range=this-week / from=…&to=…). """ - start, end, label = resolve_qs_range(q) + corpus = source_records(q) + start, end, label = resolve_qs_range(q, corpus) start_utc = start.astimezone(timezone.utc) end_utc = end.astimezone(timezone.utc) - records = _filter_by_window(get_all_records(), start_utc, end_utc) + records = _filter_by_window(corpus, start_utc, end_utc) result = we.compute_efficiency(records, start, end) result["range"]["label"] = label + result['billing_supported'] = all(r.source == 'claude' for r in records) and source_for(q) != 'codex' return result @@ -251,14 +286,15 @@ def h_dashboard(q): if hit is not None: return hit - records, rng = records_for(q) + corpus = source_records(q) + records, rng = records_for(q, corpus) totals = ts.aggregate_totals(records) usd = ts.usd_estimate(records) now_local = datetime.now(ts.local_tz()) rt_start_utc = (now_local - timedelta(hours=1)).astimezone(timezone.utc) rt_records = _filter_by_window( - get_all_records(), + corpus, rt_start_utc, now_local.astimezone(timezone.utc), ) @@ -273,10 +309,14 @@ def h_dashboard(q): "total": totals["all"], "billing_equiv": totals["billing_equiv"], "est_usd": round(usd, 2), + 'unpriced_tokens': totals['unpriced_tokens'], + 'billing_scope': 'claude', "rate_1h_tokens": rt_tokens, "rate_per_min": int(rt_tokens / 60) if rt_tokens else 0, }, "by_day": ts.aggregate_by_day(records), + 'source': source_for(q), + 'by_source': ts.aggregate_by_source(records), "by_model": ts.aggregate_by_model(records), "by_project": ts.aggregate_by_project(records, limit=limit), "detail": ts.aggregate_detail(records), @@ -379,8 +419,9 @@ def _compute_patterns(q): start_local = now_local - timedelta(days=days) # --- features / markov window (uses user-chosen bucket) --- + corpus = source_records(q) recs = _filter_by_window( - get_all_records(), + corpus, start_local.astimezone(timezone.utc), end_local.astimezone(timezone.utc), ) @@ -418,7 +459,7 @@ def _compute_patterns(q): # --- changepoint: daily totals, dense, over cp_days_eff ending at end_local --- cp_start = end_local - timedelta(days=cp_days_eff) recs_cp = _filter_by_window( - get_all_records(), + corpus, cp_start.astimezone(timezone.utc), end_local.astimezone(timezone.utc), ) @@ -436,6 +477,7 @@ def _compute_patterns(q): "bucket_sec": bucket_sec, "cp_days": cp_days_eff, "tz": now_local.strftime("%z"), + 'source': source_for(q), } if custom_range: params["from"] = start_local.date().isoformat() @@ -477,7 +519,7 @@ def h_interpret(q): model = os.environ.get("TOKEN_USAGE_LLM_MODEL", "claude-sonnet-4-6") prompt = ( - "你是一个数据分析师。下面是某 Claude Code 用户最近一段时间的 token 用量时序分析结果" + "你是一个数据分析师。下面是某用户最近一段时间的 Claude Code / Codex token 用量时序分析结果" "(包含描述统计、突发性、自相关、马尔可夫状态转移、变化点检测等指标)。\n\n" "请用中文写一份简洁的解读报告(500-800 字),结构如下:\n\n" "## TL;DR\n一句话总结用户画像(如「晚高峰个人项目型 / 高黏性 / 强 24h 周期」等)。\n\n" @@ -586,7 +628,9 @@ def do_GET(self): url = urlparse(self.path) if url.path in ROUTES: try: - result = ROUTES[url.path](parse_qs(url.query)) + query = parse_qs(url.query) + source_for(query) + result = ROUTES[url.path](query) except CLIENT_GONE: self.close_connection = True return @@ -610,6 +654,8 @@ def do_GET(self): RELOAD_WATCH_FILES = [ os.path.abspath(__file__), os.path.join(os.path.dirname(HERE), "scripts", "token_stats.py"), + os.path.join(os.path.dirname(HERE), "scripts", "codex_records.py"), + os.path.join(os.path.dirname(HERE), "scripts", "transcript_index.py"), os.path.join(os.path.dirname(HERE), "scripts", "work_efficiency.py"), ] @@ -644,13 +690,23 @@ def _reload_watcher(poll_sec: float = 1.0): def main(): + global PROJECTS_ROOT, CODEX_ROOT, _transcript_index ap = argparse.ArgumentParser() ap.add_argument("--port", type=int, default=DEFAULT_PORT) ap.add_argument("--host", default="127.0.0.1") ap.add_argument("--no-open", action="store_true") + ap.add_argument('--projects-dir', default=ts.PROJECTS_DIR) + ap.add_argument('--codex-home', default=ts.CODEX_HOME) + ap.add_argument('--cache-dir', help='directory for disposable local statistics index') + ap.add_argument('--no-cache', action='store_true', help='use memory cache only') ap.add_argument("--no-reload", action="store_true", help="disable auto-reload on Python source changes") args = ap.parse_args() + PROJECTS_ROOT = os.path.expanduser(args.projects_dir) + CODEX_ROOT = os.path.expanduser(args.codex_home) + cache_dir = (None if args.no_cache else Path(args.cache_dir).expanduser() + if args.cache_dir else default_cache_dir(PROJECTS_ROOT, CODEX_ROOT)) + _transcript_index = TranscriptIndex(cache_dir) srv = ThreadingHTTPServer((args.host, args.port), Handler) url = f"http://{args.host}:{args.port}/" @@ -665,6 +721,10 @@ def main(): srv.serve_forever() except KeyboardInterrupt: print("\nStopping.") + finally: + srv.server_close() + with _scan_lock: + _transcript_index.close() if __name__ == "__main__": diff --git a/dashboard/static/app.js b/dashboard/static/app.js index 0029058..82c1ef8 100644 --- a/dashboard/static/app.js +++ b/dashboard/static/app.js @@ -1,6 +1,7 @@ 'use strict'; const state = { + source: 'all', range: 'this-week', // named range, or 'custom' (then customFrom/To apply) customFrom: '', customTo: '', @@ -91,16 +92,26 @@ function rangeQS() { const p = new URLSearchParams(); if (state.customFrom) p.set('from', state.customFrom); if (state.customTo) p.set('to', state.customTo); + p.set('source', state.source); return p.toString(); } - return 'range=' + encodeURIComponent(state.range); + return 'range=' + encodeURIComponent(state.range) + '&source=' + state.source; } function rangeKey() { return state.range === 'custom' - ? `custom:${state.customFrom}:${state.customTo}` - : state.range; + ? `${state.source}:custom:${state.customFrom}:${state.customTo}` + : `${state.source}:${state.range}`; } +$('#source').addEventListener('change', () => { + state.source = $('#source').value; + refreshAll(); + refreshRealtime(); + refreshPatterns(patternsParamsFromUI()); + $('#interpret-dismiss')?.click(); + lastPatternsQS = ''; +}); + // ---- dashboard data: client-side cache keyed by range ---- // Re-clicking a range served <60s ago returns instantly with zero network. // Range buttons also keep the previous panels visible while fresh data loads. @@ -444,11 +455,33 @@ function formatRtTooltipTitle(iso) { } // ---- panel updaters (reuse chart instances) ---- -function updateSummary(s) { - $('#card-total').textContent = fmt(s.total); +function updateSummary(data) { + const s = data.summary; + const source = data.source; + const sourceLabel = source === 'all' ? '全部来源' : source === 'claude' ? 'Claude Code' : 'Codex'; + const rangeLabels = { + today: '今天', yesterday: '昨天', 'this-week': '本周', 'this-month': '本月', + '7d': '近 7 天', '30d': '近 30 天', all: '全部历史', custom: '自定义日期', + }; + $('#card-total').textContent = fmt(s.total); + $('#card-total').title = `${fmtInt(s.total)} tokens`; + $('#card-total-hint').textContent = `tokens · ${rangeLabels[state.range]} · ${sourceLabel}`; + for (const provider of ['claude', 'codex']) { + const included = source === 'all' || source === provider; + const total = data.by_source[provider].all; + const value = $(`#card-${provider}`); + value.textContent = included ? fmt(total) : '—'; + value.title = included ? `${fmtInt(total)} tokens` : '未纳入当前筛选'; + $(`#card-${provider}-hint`).textContent = !included ? '未纳入当前筛选' + : total === 0 ? '该时间段暂无记录' + : `tokens · 占当前总量 ${(total / s.total * 100).toFixed(1)}%`; + } + $('#card-rate').textContent = fmtInt(s.rate_per_min); + $('#card-rate-hint').textContent = `tokens / 分钟 · ${sourceLabel}`; + $('#card-usd-wrap').hidden = source === 'codex'; $('#card-billing').textContent = fmt(s.billing_equiv); - $('#card-usd').textContent = '$' + (s.est_usd ?? 0).toFixed(2); - $('#card-rate').textContent = fmtInt(s.rate_per_min); + $('#card-billing').title = `${fmtInt(s.billing_equiv)} 计费等效 token`; + $('#card-usd').textContent = '$' + (s.est_usd ?? 0).toFixed(2); } function updateDaily(rows) { @@ -686,7 +719,7 @@ function renderEffPerDay(rows) { ${escapeHtml(r.date)} ${hrs.toFixed(1)}h ${fmt(r.tokens)} - ${fmt(r.billing_equiv)} + ${effState.data?.billing_supported === false ? '—' : fmt(r.billing_equiv)} ${fmt(r.rate_per_hr)}/h `; }).join(''); @@ -694,17 +727,24 @@ function renderEffPerDay(rows) { function updateEfficiency(data) { effState.data = data; + if (data.billing_supported === false) effState.mode = 'raw'; + document.querySelectorAll('.eff-dist-toggle button').forEach(btn => { + btn.disabled = btn.dataset.effMode === 'bill' && data.billing_supported === false; + btn.classList.toggle('active', btn.dataset.effMode === effState.mode); + btn.title = btn.disabled ? 'Codex 尚未配置计价,请查看 raw 用量' : ''; + }); const s = data.summary; $('#eff-active-hours').textContent = s.active_hours.toFixed(1) + 'h'; $('#eff-active-pct').textContent = (s.active_pct * 100).toFixed(1) + '%'; $('#eff-rate-raw').textContent = fmt(s.avg_rate_raw_per_hr) + '/h'; - $('#eff-rate-bill').textContent = fmt(s.avg_rate_bill_per_hr) + '/h'; + $('#eff-rate-bill').textContent = data.billing_supported === false ? '—' : fmt(s.avg_rate_bill_per_hr) + '/h'; const sd = data.slot_distribution; $('#eff-slot-meta').innerHTML = `${fmtInt(sd.active_slots)} / ${fmtInt(sd.total_slots)} 个 30 分钟槽有活动` + ` (${(sd.active_slot_pct * 100).toFixed(1)}% 占用率)  ·  ` - + `中位 ${fmt(sd.raw_per_hr.median)}/h (raw) / ${fmt(sd.bill_per_hr.median)}/h (bill)`; + + `中位 ${fmt(sd.raw_per_hr.median)}/h (raw)` + + (data.billing_supported === false ? ' · Codex 未配置计价,计费视图不可用' : ` / ${fmt(sd.bill_per_hr.median)}/h (bill)`); renderEffDistribution(); renderEffPerDay(data.per_day); @@ -837,7 +877,7 @@ async function refreshAll({ force = false } = {}) { // Guard against races: if the user switched range while this was in flight, // drop the result. if (key !== rangeKey()) return; - updateSummary(data.summary); + updateSummary(data); updateDaily(data.by_day); updateModel(data.by_model); updateProject(data.by_project); @@ -864,9 +904,11 @@ function fmtWindowMinutes(min) { } async function refreshRealtime() { + const source = state.source; const windowMin = computeRealtimeWindowMinutes(); - const url = `/api/realtime?bucket=${state.rtBucketSec}&window=${windowMin}m`; + const url = `/api/realtime?bucket=${state.rtBucketSec}&window=${windowMin}m&source=${source}`; const res = await fetchJSON(url); + if (source !== state.source) return; updateRealtime(res.rows); const lbl = $('#realtime-range-label'); if (lbl) lbl.textContent = `— last ${fmtWindowMinutes(windowMin)}, ${state.rtLabel} buckets (auto 10s)`; @@ -887,11 +929,12 @@ let cpChart; const patternsCacheMap = new Map(); // qs -> { data, fetchedAt } const PATTERNS_TTL_MS = 60_000; let lastPatternsQS = ''; // remember the most recent successful query string +let patternsRequestId = 0; function patternsParamsFromUI() { const range = $('#patterns-range')?.value || '7d'; const bucket = $('#patterns-bucket')?.value || '1800'; - const params = { bucket }; + const params = { bucket, source: state.source }; if (range === 'custom') { const f = $('#patterns-from')?.value || ''; const t = $('#patterns-to')?.value || ''; @@ -946,28 +989,33 @@ async function fetchPatterns(force = false, params = null) { } async function refreshPatterns(params = null) { + const requestId = ++patternsRequestId; + const source = state.source; + params = { ...(params || patternsParamsFromUI()), source }; const panel = $('#patterns-panel'); const applyBtn = $('#patterns-apply'); panel?.classList.add('is-loading'); - let prevApplyText = null; if (applyBtn) { - prevApplyText = applyBtn.textContent; applyBtn.disabled = true; applyBtn.textContent = '⏳ 分析中…'; } try { const data = await fetchPatterns(true, params); + if (source !== state.source || requestId !== patternsRequestId) return; renderPatterns(data); const lbl = $('#patterns-summary-label'); if (lbl && data?.params) lbl.textContent = patternsLabelFromParams(data.params); } catch (e) { + if (requestId !== patternsRequestId) return; console.warn('patterns load failed:', e); showPatternsError(e?.message || String(e)); } finally { - panel?.classList.remove('is-loading'); - if (applyBtn) { - applyBtn.disabled = false; - applyBtn.textContent = prevApplyText || '应用'; + if (requestId === patternsRequestId) { + panel?.classList.remove('is-loading'); + if (applyBtn) { + applyBtn.disabled = false; + applyBtn.textContent = '应用'; + } } } } @@ -1440,6 +1488,7 @@ function setInterpretActions({ copy, regen, dismiss }) { } async function runInterpret() { + const source = state.source; if (interpretInFlight) return; const btn = $('#patterns-interpret'); const body = $('#interpret-body'); @@ -1457,8 +1506,11 @@ async function runInterpret() { $('#interpret-section')?.scrollIntoView({ behavior: 'smooth', block: 'start' }); try { - const res = await fetch('/api/interpret' + qs); + const query = new URLSearchParams(qs); + query.set('source', source); + const res = await fetch('/api/interpret?' + query); const data = await res.json(); + if (source !== state.source) return; if (!res.ok || data.error) { const msg = data.error || `HTTP ${res.status}`; const hint = data.hint ? `
${renderMarkdown(data.hint)}` : ''; diff --git a/dashboard/static/index.html b/dashboard/static/index.html index 9772565..eab6d74 100644 --- a/dashboard/static/index.html +++ b/dashboard/static/index.html @@ -1,9 +1,9 @@ - + - Claude Code Token Usage + Claude Code / Codex Token Usage
-

Claude Code Token Usage

+

Claude Code / Codex Token Usage

+