Repository navigation
Open
Labels
Description
Tracking issue: #47
Depends on: #48 (closed)
Feature PR: #94 (supersedes #61)
Design record: docs/auto-advisor/findings.md
Baseline: OpenCode 2.2.0. 2.2.0 removed the experimental Evaluation/System One API that #61 used and replaced it with Decision (ctx.experimental.generate.decisions) and session instruction entries.
Desired outcome
Automatic Advisor reviews triggered by an OpenCode Decision model, with a safe observe mode and an active mode that delivers reviews without breaking the prompt cache.
This issue absorbs the former active-delivery scope from #51.
Scope
Decision routing
- One host Decision call per primary request, with two questions:
moment(choice):approach|stuck|done|routine;consequence(score 0–4) with fixed semantic anchors.
- Deterministic policy: consult when 1 − P(routine) ≥
routing.momentThreshold(0.7) and the raw score ≥routing.consequenceThreshold(1.5). The non-routine moment with the highest probability labels the review. - The Decision input is a small projection: the latest real user prompt, recent tool calls and truncated results, step and error counts (≤ 32k chars). No system prompt or tool definitions.
- Turns are keyed on prompt ids recorded by the
prompthook, so synthetic, skill, shell and compaction messages don't start a new turn.
Ordered model fallback
Default routing.models:
["opencode/jev-1.13-free"]Users may list 1–8 catalog decision models (provider/model[#variant]). Behavior:
- every routing opportunity walks the chain from the first model;
- the host already retries rate limits, 503/529 and unsent requests; the plugin never retries;
- next model on availability, auth or credential failures (
ModelNotFound,UnsupportedModel,Unavailable, non-requestProviderRequestError) and per-attempt timeouts (5 s; 10 s for the whole chain); - stop on
invalid_request/content_filterand unknown errors; - a model that was rate-limited, overloaded or timed out is skipped for 60 s;
- whole-chain failure fails open for that opportunity only.
Observe mode and telemetry
observe: run the Decision and policy, never consult the Advisor or change the request. One Decision call in flight per session.- JSONL telemetry at
~/.local/share/opencode/auto-advisor/telemetry.jsonl(5 MB, one rotation, queued writes, failures swallowed), with models tried, answers, verdict, timings and Advisor outcome. Never transcript or advice text.
Active mode and delivery
- On an accepted moment with budget left, consult the Advisor before the request continues.
- Deliver the review twice:
- as a
role: "system"message at the tail of the current request; - as the
auto-advisorinstruction entry, which the host admits from the next step on.
- as a
- Never edit
event.system. The prompt prefix stays byte-stable; only the tail changes. - Each new review replaces the entry; it stays until superseded, including through session compaction.
- A retried step reuses its memoized outcome; a review marker prevents duplicates.
routing.maxConsultationsPerTurn(default 2) counts consults started per user turn. Explicitadvisor()calls don't count.- The run's end (
session.execution.{succeeded,failed,interrupted}) aborts an in-flight consult; a stale consult delivers nothing. - Auxiliary
generate.*calls do not re-enter the plugin's hooks (verified).
Failure behavior
- Any Decision, Advisor, delivery or telemetry failure fails open; neither hook throws into the user's request.
- Failures are visible in telemetry and the TUI footer ("Advisor unavailable").
- A failure never disables later opportunities.
Explicit exclusions
- no plugin-side retries or retry knobs (the host retries);
- no edits to the system prompt; no
session.syntheticfor advice; no fabricatedadvisor()call; - no separate retained-review store (the instruction entry is the retention);
- no persistent circuit breaker beyond the 60 s skip;
- no mid-generation interruption;
- no adaptive or learned thresholds in v1.
Acceptance criteria
- Works on OpenCode 2.2.0; host error shapes captured as fixtures.
- The router walks the chain per opportunity and classifies host errors into next-model vs stop.
- Whole-chain failure fails open without disabling later attempts.
-
observewrites telemetry and never consults the Advisor or changes the request. -
activedelivers the review at the tail of the current step and through the entry from the next step; system prompt and tools byte-identical across requests. - Budget enforced per user turn; explicit calls excluded.
- Retry reuses the memo; no duplicate reviews.
- Interrupt during a consult: aborted, nothing delivered,
stalein telemetry. - Auxiliary inference cannot recurse into routing.
- Runtime matrix covers
off,observe,active(mock executor + real Jev). - Repository gates and package-root smoke pass.
- USER live test in the real TUI:
observetelemetry,activefooter/panel, advice uptake. - Live: session compaction keeping the entry; fallback to a second real decision provider.
Final integration responsibilities
Consolidated from #52, tracked on #47:
- package README and package-local contributor guidance;
- root documentation for implemented behavior;
- release-channel policy/guard and tests;
- changeset decision;
- full repository gates and runtime smoke.
History (labeled)
- Remediation (2026-09-30): canonical projections; active-only budget short-circuit; 5s Zen deadline; retry/fallback/terminal split; paid-on-public session ineligibility; suite 286/0; live 2.0.19 matrix 6/6.
- Simplification (2026-10-02): baseline 2.0.21; custom permission engine removed (host tools already permission-filtered); multi-record history/compaction removed in favor of one bounded retained review; TUI shared store; suite 478/0; live active runs accepted/delivered with retained real advice.
- Final remediation (2026-10-02): retention off the hook critical path; fresh-supersedes-old; oversize tombstone; write-failure keeps new; parser hardening; Gate 2 review WITH FIXES remediated; suite 488/0.
- Rebuild on 2.2.0 (2026-10-28): feat(auto-advisor): add OpenCode Auto Advisor #61 abandoned (2.2.0 removed the Evaluation API it used). Lean rebuild in feat(auto-advisor): rebuild Auto Advisor on OpenCode 2.2.5 #94: Decision API with moment + consequence, delivery through a tail message plus an instruction entry, JSONL telemetry, budget 2, threshold 1.5, no retained-review store, ~1,250 source lines. Spikes and live checks in
findings.md. - Advisor context (2026-10-29):
#variantmodel refs;advisor.compactContext(host catalogs replaced with markers in the Advisor's copy) andadvisor.maxTokens;advisor()returns config errors instead of silently using the agent's model. Live in the real TUI: variant reaches the provider, ~14k → ~6.8k tokens with compaction, cap holds. Suite 99/0.
Activity
[-][Auto Advisor] Add Jev adapters and observe-mode telemetry[/-][+][Auto Advisor] Add experimental Zen automatic routing and delivery[/+]capyBearista commentedon Sep 29, 2026
#50 status — implemented on 2.0.19, pending final integration
Baseline is now OpenCode 2.0.19 (
@opencode/ai@2.0.19,@opencode/plugin@2.0.19; body acceptance line updated). The Evaluation/System One adapter is source-grounded: official V2 docs expose no Evaluation API, so the implementation follows the 2.0.19@opencode/aisource (Evaluation + SystemOne route, boolean→probability, 5-level consequence rubric).Implemented on
feat/auto-advisor(uncommitted, PR #61 draft):AdvisorRouteradapter behind the existing seam; chainjev-1.13-free→jev-1.13with restart-at-top; bounded retry on retryable classes only; quota→advance, terminal→fail-open. Auth via hostopencodeintegration connection with public-bearer fallback (no secrets stored).observetelemetry: 5000-event cap with oldest-first eviction, digest-only payloads (no transcripts/secrets), read-onlytelemetry.query/telemetry.eventRPC.activedelivery as system-role context into the waiting primary dispatch (same-dispatch injection, reinjection on continuations, expiry on new turns, supersession, reentrancy protection). Nosession.synthetic, no fabricated tool calls, no persistent Advisor state.offinert;observeevaluates + records without consulting;activeconsults, delivers (delivered:true), budget-denied continuation. Reviews closed (final re-review: 0 critical, transcript bound deferred — see below).Deferred decision: transcript truncation policy is still unimplemented (pre-existing known limit).
activetherefore ships opt-in experimental on theoffdefault; truncation is required before recommending it broadly.Remaining here: final integration responsibilities (README/root-docs reconciliation, changeset when releasable, merge). Release-channel registration already exists.
capyBearista commentedon Sep 30, 2026
Remediation complete on
feat/auto-advisor(uncommitted): 23 items implemented, gate findings remediated, suite 286/0, root gates fresh green, live 6/6 matrix green. Details appended to the issue body. Remaining: commit/push on your order, then Changeset + merge.capyBearista commentedon Oct 2, 2026
Simplification pass complete on
feat/auto-advisor(PR #61, manual-test-ready, unmerged). Since this contract was written: multi-record history/compaction-absorption replaced by one bounded latest retained review (best-effort, never gates delivery); custom permission engine removed (2.0.21 host filtering suffices — root check + advertised tool); TUI reduced to one shared store; smoke 1337→317 LOC. Settled behavior kept (zero-arg advisor, off/observe/active, Jev-only, budget, suppression, 300s soft timeout, model-aware projection, privileged system delivery, fail-open). Fresh pkg + root gates green; live 2.0.21 probes show accepted/delivered/retained automatic reviews. Details in PR #61 body anddocs/plans/opencode-auto-advisor-manual-test.md(19 cases). No merge/publish/dogfood — awaiting manual testing.[-][Auto Advisor] Add experimental Zen automatic routing and delivery[/-][+][Auto Advisor] Add Decision-model automatic routing and delivery[/+]