Repository navigation
feat: ask to review model instructions on first use of a model (0.35.0) #46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,6 +4,24 @@ All notable changes to this project will be documented in this file. | |
|
|
||
| The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). | ||
|
|
||
| ## [0.35.0] - 2026-10-08 | ||
|
|
||
| ### Added | ||
| - **First use of a model.** The first time a model shows up on this machine (at | ||
| session start or after `/model`), Claude is asked to offer, in one question, | ||
| to check your CLAUDE.md and memory files for instructions about model | ||
| selection and usage and update them for the new model; it does nothing | ||
| unless you agree. You see `DevScope: first time on <model>.` and a | ||
| `model.first_use` event (`model`, `trigger`, `previousModel`) is sent, which | ||
| the dashboard shows in the live feed. Seen models are kept in | ||
| `~/.cache/devscope/models-seen`; the context-window suffix (`[1m]`) does not | ||
| make a model new, and the model in use when the plugin first runs is the | ||
| silent baseline. New synchronous hook `scripts/model-first-use.sh` on | ||
| `SessionStart` and `PostModelSwitch`: local only, the event is sent in the | ||
| background. `DEVSCOPE_HINTS=off` turns the message off; the event is still | ||
| sent. Older servers reject the new event type with a 400 and the plugin | ||
| drops it, so nothing else is affected. | ||
|
Comment on lines
+22
to
+23
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: set -u
printf '%s\n' '--- changed files ---'
git diff --stat 73b278feef483dcbb8848221f76f811af99053e7 f7efda322cdc229e9882f3d3386e607a17ef6d4e
printf '%s\n' '--- changelog ---'
nl -ba CHANGELOG.md | sed -n '1,45p'
printf '%s\n' '--- hook and related references ---'
rg -n -F --glob '!node_modules' --glob '!dist' -- 'model.first_use' . || test "$?" -eq 1
rg -n -F --glob '!node_modules' --glob '!dist' -- 'models-seen' . || test "$?" -eq 1
rg -n -F --glob '!node_modules' --glob '!dist' -- 'DEVSCOPE_HINTS' . || test "$?" -eq 1
printf '%s\n' '--- changed shell files ---'
git diff --name-only 73b278feef483dcbb8848221f76f811af99053e7 f7efda322cdc229e9882f3d3386e607a17ef6d4e -- '*.sh'
for f in $(git diff --name-only 73b278feef483dcbb8848221f76f811af99053e7 f7efda322cdc229e9882f3d3386e607a17ef6d4e -- '*.sh'); do
printf '\n--- %s ---\n' "$f"
nl -ba "$f"
done
printf '%s\n' '--- endpoint/event configuration candidates ---'
rg -n -i --glob '!node_modules' --glob '!dist' -- '(endpoint|server|ingest|event|telemetry|devscope-live|api_url|url)' README.md docs .github scripts tests 2>/dev/null | sed -n '1,240p' || true
printf '%s\n' '--- release/publish guidance ---'
rg -n -i --glob '!node_modules' --glob '!dist' -- '(release|publish|deploy|backend|server|compatibility|prerequisite)' .github docs README.md CHANGELOG.md 2>/dev/null | sed -n '1,240p' || trueRepository: DowLucas/devscope-plugin Length of output: 41813 🏁 Script executed: printf '%s\n' '--- sender ---'
nl -ba scripts/send-event.sh | sed -n '1,240p'
printf '%s\n' '--- sender bindings and configuration ---'
rg -n -F --glob '*.sh' -- 'send-event.sh' scripts tests | sed -n '1,160p'
rg -n -F --glob '*.sh' -- 'DEVSCOPE_URL' scripts | sed -n '1,160p'
printf '%s\n' '--- release contract ---'
nl -ba CHANGELOG.md | sed -n '450,485p'
printf '%s\n' '--- backend compatibility workflow ---'
nl -ba .github/workflows/ci.yml | sed -n '108,180p'
printf '%s\n' '--- configured server documentation ---'
nl -ba README.md | sed -n '1,75p'Repository: DowLucas/devscope-plugin Length of output: 28472 Verify backend support before publishing 0.35.0. If the configured server does not support 🤖 Prompt for AI Agents |
||
|
|
||
| ## [0.34.0] - 2026-10-08 | ||
|
|
||
| ### Added | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -148,6 +148,10 @@ DEVSCOPE_HINTS=off # no hints | |
| DEVSCOPE_HINT_AFTER_PR=/review-pr # suggest a different command (default: /code-review) | ||
| ``` | ||
|
|
||
| ## New model check | ||
|
|
||
| The first time you use a model (at session start or after `/model`), DevScope shows `DevScope: first time on <model>.` and asks Claude to offer, in one question, to check your CLAUDE.md and memory files for instructions about model selection and usage, such as rules that name an older model, and update them. Nothing is read or changed unless you say yes. Models already seen are listed in `~/.cache/devscope/models-seen`; the model you are on when the plugin first runs counts as seen. A `model.first_use` event appears in the dashboard's live feed. `DEVSCOPE_HINTS=off` turns the question off. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win Document the new synchronous-hook exception. Line 276 says the next-step hint is the only synchronous hook. This first-use hook also runs synchronously to return 🤖 Prompt for AI Agents |
||
|
|
||
| ## Token usage and cost | ||
|
|
||
| DevScope shows each session's tokens and its **API-equivalent cost**: what those tokens would cost at Anthropic's API list prices, which is not what a Claude subscription plan charges you. The Stop and SessionEnd hooks sum every API call in the session transcript and its subagent transcripts, per model, and send the totals. Only token counts and model ids are sent, never transcript content. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| #!/usr/bin/env bash | ||
| # First use of a model: offer to review model instructions in CLAUDE.md and memory. | ||
| # | ||
| # SessionStart (`model`) and PostModelSwitch (`to_model`). A CLAUDE.md or memory | ||
| # file often says which model to use for what, or carries advice written for an | ||
| # older model; a new model is the moment those go stale. The first time a model | ||
| # shows up on this machine, Claude is asked to offer a review of them (never to | ||
| # do it unasked), the user sees a one-line notice, and a `model.first_use` event | ||
| # is sent. | ||
| # | ||
| # Seen models live in ~/.cache/devscope/models-seen, one id per line, without | ||
| # the context-window suffix (`claude-opus-5-5[1m]` is `claude-opus-5-5`). When | ||
| # that file does not exist yet it is seeded silently with the model in use (on | ||
| # a switch, the model switched from), so installing the plugin or clearing the | ||
| # cache does not make every model look new. | ||
| # | ||
| # Synchronous by necessity: Claude Code ignores the output of async hooks. It | ||
| # is local, the event is sent in the background, and it is silent on any error. | ||
| # DEVSCOPE_HINTS=off drops the notice and the message to Claude; the event is | ||
| # still sent. | ||
| set -uo pipefail | ||
| SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" | ||
| # shellcheck disable=SC1091 | ||
| . "$SCRIPT_DIR/_helpers.sh" 2>/dev/null || exit 0 | ||
|
|
||
| INPUT=$(cat) | ||
| HOOK=$(printf '%s' "$INPUT" | jq -r '.hook_event_name // ""' 2>/dev/null) || exit 0 | ||
|
|
||
| # Lowercased, context-window suffix dropped; empty unless it looks like a model id. | ||
| normalize() { | ||
| local m | ||
| m=$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]' | sed -E 's/\[[^]]*\]$//') | ||
| printf '%s' "$m" | grep -Eq '^[a-z0-9][a-z0-9._:@/-]{0,99}$' && printf '%s' "$m" | ||
| } | ||
|
|
||
| case "$HOOK" in | ||
| SessionStart) | ||
| MODEL=$(normalize "$(printf '%s' "$INPUT" | jq -r '.model // ""' 2>/dev/null)") | ||
| PREVIOUS="" | ||
| TRIGGER="session_start" | ||
| ;; | ||
| PostModelSwitch) | ||
| MODEL=$(normalize "$(printf '%s' "$INPUT" | jq -r '.to_model // ""' 2>/dev/null)") | ||
| PREVIOUS=$(normalize "$(printf '%s' "$INPUT" | jq -r '.from_model // ""' 2>/dev/null)") | ||
| TRIGGER="model_switch" | ||
| ;; | ||
| *) exit 0 ;; | ||
| esac | ||
| [ -n "$MODEL" ] || exit 0 | ||
|
|
||
| DIR="${HOME}/.cache/devscope" | ||
| SEEN="$DIR/models-seen" | ||
| mkdir -p -m 0700 "$DIR" 2>/dev/null || exit 0 | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: sed -n '1,74p' scripts/model-first-use.sh
rg -n 'devscope|XDG_CACHE_HOME|mkdir -p' scripts | head -50Repository: DowLucas/devscope-plugin Length of output: 8613 🏁 Script executed: set -u
printf '%s\n' '--- model-first-use.sh ---'
nl -ba scripts/model-first-use.sh | sed -n '45,67p'
printf '%s\n' '--- exact path and models-seen references ---'
rg -n -F --glob 'scripts/**' -- 'models-seen' . || test "$?" -eq 1
rg -n -F --glob 'scripts/**' -- 'DIR="${HOME}/.cache/devscope"' . || test "$?" -eq 1
printf '%s\n' '--- cache directory creation patterns ---'
rg -n -E --glob 'scripts/**' '(mkdir|install)[^#]*\.cache/devscope|\.cache/devscope[^"]*' . || test "$?" -eq 1
printf '%s\n' '--- relevant complete blocks ---'
nl -ba scripts/tool-use.sh | sed -n '48,64p'
nl -ba scripts/elicitation.sh | sed -n '12,28p'
nl -ba scripts/error-recall.sh | sed -n '20,35p'
nl -ba scripts/skill-hint.sh | sed -n '26,40p'Repository: DowLucas/devscope-plugin Length of output: 4937 🏁 Script executed: set -u
printf '%s\n' '--- all cache-path references and creation commands ---'
rg -n --glob 'scripts/**' -e '(\.cache/devscope|mkdir[[:space:]]+(-p[[:space:]]+)?[^#]*devscope)' . || test "$?" -eq 1
printf '%s\n' '--- parent-directory creation candidates ---'
rg -n --glob 'scripts/**' -e 'mkdir[^#\n]*\.cache/devscope([[:space:]"$]|/)' . || test "$?" -eq 1Repository: DowLucas/devscope-plugin Length of output: 2728 Reject a shared or non-private cache directory before writing.
🤖 Prompt for AI Agents |
||
| if [ ! -f "$SEEN" ]; then | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win Serialize the seen-model check and update. If two sessions start together, both hooks can observe a missing 🤖 Prompt for AI Agents |
||
| # Baseline: what was in use before the plugin could see it is not new. | ||
| printf '%s\n' "${PREVIOUS:-$MODEL}" > "$SEEN" 2>/dev/null || exit 0 | ||
| [ -z "$PREVIOUS" ] && exit 0 | ||
| fi | ||
| grep -Fxq "$MODEL" "$SEEN" 2>/dev/null && exit 0 | ||
| printf '%s\n' "$MODEL" >> "$SEEN" 2>/dev/null || exit 0 | ||
|
|
||
| PAYLOAD=$(jq -n --arg m "$MODEL" --arg t "$TRIGGER" --arg p "$PREVIOUS" \ | ||
| '{model: $m, trigger: $t} | if $p != "" then . + {previousModel: $p} else . end') || exit 0 | ||
| # Detached with every stream closed, so Claude Code does not wait on the network. | ||
| ( printf '%s' "$INPUT" | "$SCRIPT_DIR/send-event.sh" "model.first_use" "$PAYLOAD" ) \ | ||
| </dev/null >/dev/null 2>&1 & | ||
|
|
||
| [ "${DEVSCOPE_HINTS:-on}" = "off" ] && exit 0 | ||
|
|
||
| jq -n --arg h "$HOOK" \ | ||
| --arg msg "DevScope: first time on ${MODEL}." \ | ||
| --arg ctx "DevScope: this is the first time the user has used the model ${MODEL}. Before starting on their request (or right away if there is none yet), ask the user in one short question whether they want you to check their CLAUDE.md files and memory files for instructions about model selection or model usage (for example rules that name a specific model, which model to use for subagents, or advice written for an older model) and update them for ${MODEL}. Do not read or change those files unless the user agrees." \ | ||
| '{systemMessage: $msg, hookSpecificOutput: {hookEventName: $h, additionalContext: $ctx}}' | ||
| exit 0 | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| #!/usr/bin/env bash | ||
| # Tests for scripts/model-first-use.sh (first use of a model). Stub server, no backend. | ||
| # | ||
| # bash tests/models/run.sh | ||
| set -euo pipefail | ||
| ROOT="$(cd "$(dirname "$0")/../.." && pwd)" | ||
| HOOK="$ROOT/scripts/model-first-use.sh" | ||
| TMP=$(mktemp -d) | ||
| export HOME="$TMP" XDG_CONFIG_HOME="$TMP/config" | ||
| unset DEVSCOPE_HINTS DEVSCOPE_PRIVACY | ||
| # shellcheck disable=SC1091 | ||
| . "$ROOT/tests/lib/stub.sh" | ||
| SEEN="$HOME/.cache/devscope/models-seen" | ||
| pass=0; fail=0 | ||
|
|
||
| start() { # model [session] | ||
| jq -n --arg m "$1" --arg s "${2:-s1}" \ | ||
| '{session_id: $s, hook_event_name: "SessionStart", source: "startup", cwd: "/nonexistent", model: $m}' | ||
| } | ||
| switch() { # from to | ||
| jq -n --arg f "$1" --arg t "$2" \ | ||
| '{session_id: "s1", hook_event_name: "PostModelSwitch", cwd: "/nonexistent", | ||
| from_model: $f, to_model: $t, requested_model: null, source: "command", | ||
| context_tokens: 1000, prompt_cache_warm: true}' | ||
| } | ||
| check() { # name expected-substring-or-EMPTY actual | ||
| if { [ "$2" = "EMPTY" ] && [ -z "$3" ]; } || { [ "$2" != "EMPTY" ] && printf '%s' "$3" | grep -Fq "$2"; }; then | ||
| pass=$((pass + 1)); echo "ok $1" | ||
| else | ||
| fail=$((fail + 1)); echo "FAIL $1: expected '$2', got '$3'" | ||
| fi | ||
| } | ||
| # The event is sent in the background: wait for a POST to /api/events. | ||
| wait_event() { | ||
| for _ in $(seq 1 50); do paths | grep -q '/api/events' && return 0; sleep 0.1; done | ||
| return 1 | ||
| } | ||
| event() { last_at /api/events ".body$1"; } | ||
|
|
||
| respond '{"ok":true}' | ||
| reset_hits | ||
| check "first model ever is the silent baseline" EMPTY "$(start claude-opus-5-5 | "$HOOK")" | ||
| check "baseline is recorded" "claude-opus-5-5" "$(cat "$SEEN")" | ||
| sleep 0.5 | ||
| check "baseline sends no event" EMPTY "$(paths)" | ||
| check "a seen model is silent" EMPTY "$(start claude-opus-5-5 s2 | "$HOOK")" | ||
| check "context-window suffix is the same model" EMPTY "$(start 'claude-opus-5-5[1m]' s3 | "$HOOK")" | ||
| check "case is ignored" EMPTY "$(start Claude-Opus-5-5 s4 | "$HOOK")" | ||
|
|
||
| out=$(start claude-fable-5-1 s5 | "$HOOK") | ||
| check "new model tells the user" "DevScope: first time on claude-fable-5-1." "$(printf '%s' "$out" | jq -r .systemMessage)" | ||
| ctx=$(printf '%s' "$out" | jq -r .hookSpecificOutput.additionalContext) | ||
| check "new model asks Claude to offer the review" "ask the user" "$ctx" | ||
| check "it names CLAUDE.md and memory" "CLAUDE.md files and memory files" "$ctx" | ||
| check "it is about model selection and usage" "model selection or model usage" "$ctx" | ||
| check "it forbids doing it unasked" "unless the user agrees" "$ctx" | ||
| check "hook event name is SessionStart" "SessionStart" "$(printf '%s' "$out" | jq -r .hookSpecificOutput.hookEventName)" | ||
| wait_event || true | ||
| check "event type" "model.first_use" "$(event .eventType)" | ||
| check "event model" "claude-fable-5-1" "$(event .payload.model)" | ||
| check "event trigger" "session_start" "$(event .payload.trigger)" | ||
| check "no previousModel on session start" "null" "$(event .payload.previousModel)" | ||
| check "new model is recorded" "claude-fable-5-1" "$(cat "$SEEN")" | ||
| check "second time is silent" EMPTY "$(start claude-fable-5-1 s6 | "$HOOK")" | ||
|
|
||
| reset_hits | ||
| out=$(switch claude-opus-5-5 'claude-sonnet-5-5[1m]' | "$HOOK") | ||
| check "switch to a new model tells Claude" "claude-sonnet-5-5" "$(printf '%s' "$out" | jq -r .hookSpecificOutput.additionalContext)" | ||
| check "hook event name is PostModelSwitch" "PostModelSwitch" "$(printf '%s' "$out" | jq -r .hookSpecificOutput.hookEventName)" | ||
| wait_event || true | ||
| check "switch event trigger" "model_switch" "$(event .payload.trigger)" | ||
| check "switch event previous model" "claude-opus-5-5" "$(event .payload.previousModel)" | ||
| check "switch to a seen model is silent" EMPTY "$(switch claude-sonnet-5-5 claude-opus-5-5 | "$HOOK")" | ||
|
|
||
| # No seen file yet and the first thing seen is a switch: the model switched | ||
| # from is the baseline, the one switched to is new. | ||
| rm -f "$SEEN" | ||
| out=$(switch claude-opus-5-5 claude-haiku-5-5 | "$HOOK") | ||
| check "switch with no baseline still announces the new model" "claude-haiku-5-5" "$(printf '%s' "$out" | jq -r .systemMessage)" | ||
| check "switch with no baseline records both" "claude-opus-5-5 claude-haiku-5-5" "$(tr '\n' ' ' < "$SEEN")" | ||
|
|
||
| reset_hits | ||
| out=$(start claude-new-1 s7 | DEVSCOPE_HINTS=off "$HOOK") | ||
| check "hints off is silent" EMPTY "$out" | ||
| wait_event || true | ||
| check "hints off still sends the event" "claude-new-1" "$(event .payload.model)" | ||
| check "hints off still records the model" "claude-new-1" "$(cat "$SEEN")" | ||
|
|
||
| check "missing model is silent" EMPTY "$(start '' s8 | "$HOOK")" | ||
| check "unsafe model id is ignored" EMPTY "$(start 'x"; rm -rf ~' s9 | "$HOOK")" | ||
| check "other hook events are ignored" EMPTY "$(jq -n '{hook_event_name: "Stop", model: "claude-z"}' | "$HOOK")" | ||
| check "garbage stdin is silent" EMPTY "$(printf 'not json' | "$HOOK")" | ||
| check "empty stdin is silent" EMPTY "$(printf '' | "$HOOK")" | ||
|
|
||
| # Runs synchronously at every session start: the seen path must be fast. | ||
| t0=$(date +%s%N); for _ in 1 2 3 4 5 6 7 8 9 10; do start claude-opus-5-5 | "$HOOK" >/dev/null; done; t1=$(date +%s%N) | ||
| avg_ms=$(( (t1 - t0) / 10000000 )) | ||
| if [ "$avg_ms" -lt 150 ]; then pass=$((pass + 1)); echo "ok seen path averages ${avg_ms}ms (< 150ms)"; else fail=$((fail + 1)); echo "FAIL seen path averages ${avg_ms}ms"; fi | ||
|
|
||
| # A slow backend must not hold up the hook (the event is detached). | ||
| slow | ||
| t0=$(date +%s%N); start claude-slow-1 s10 | "$HOOK" >/dev/null; t1=$(date +%s%N) | ||
| ms=$(( (t1 - t0) / 1000000 )) | ||
| if [ "$ms" -lt 1000 ]; then pass=$((pass + 1)); echo "ok slow backend does not block (${ms}ms)"; else fail=$((fail + 1)); echo "FAIL slow backend blocked for ${ms}ms"; fi | ||
|
|
||
| echo "---"; echo "$pass passed, $fail failed" | ||
| [ "$fail" -eq 0 ] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "session_id": "smoke-session-0001", | ||
| "cwd": "__SMOKE_CWD__", | ||
| "hook_event_name": "PostModelSwitch", | ||
| "permission_mode": "default", | ||
| "from_model": "claude-opus-5-5", | ||
| "to_model": "claude-fable-5-1", | ||
| "requested_model": "fable", | ||
| "source": "command", | ||
| "context_tokens": 41234, | ||
| "prompt_cache_warm": true | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Mark
previousModelas optional.SessionStartsendsmodelandtriggerwithoutpreviousModel, as documented inCLAUDE.md, Lines 291-293. Update this field list so it does not imply that all three fields are always present.🤖 Prompt for AI Agents