Skip to content

[Auto Advisor] Add Decision-model automatic routing and delivery #50

Open

Description

@capyBearista

Tracking issue: #47
Depends on: #48 (closed)
Feature PR: #94 (supersedes #61)
Design record: docs/auto-advisor/findings.md

Baseline: OpenCode 2.2.0. 2.2.0 removed the experimental Evaluation/System One API that #61 used and replaced it with Decision (ctx.experimental.generate.decisions) and session instruction entries.

Desired outcome

Automatic Advisor reviews triggered by an OpenCode Decision model, with a safe observe mode and an active mode that delivers reviews without breaking the prompt cache.

This issue absorbs the former active-delivery scope from #51.

Scope

Decision routing

  • One host Decision call per primary request, with two questions:
    • moment (choice): approach | stuck | done | routine;
    • consequence (score 0–4) with fixed semantic anchors.
  • Deterministic policy: consult when 1 − P(routine) ≥ routing.momentThreshold (0.7) and the raw score ≥ routing.consequenceThreshold (1.5). The non-routine moment with the highest probability labels the review.
  • The Decision input is a small projection: the latest real user prompt, recent tool calls and truncated results, step and error counts (≤ 32k chars). No system prompt or tool definitions.
  • Turns are keyed on prompt ids recorded by the prompt hook, so synthetic, skill, shell and compaction messages don't start a new turn.

Ordered model fallback

Default routing.models:

["opencode/jev-1.13-free"]

Users may list 1–8 catalog decision models (provider/model[#variant]). Behavior:

  • every routing opportunity walks the chain from the first model;
  • the host already retries rate limits, 503/529 and unsent requests; the plugin never retries;
  • next model on availability, auth or credential failures (ModelNotFound, UnsupportedModel, Unavailable, non-request ProviderRequestError) and per-attempt timeouts (5 s; 10 s for the whole chain);
  • stop on invalid_request / content_filter and unknown errors;
  • a model that was rate-limited, overloaded or timed out is skipped for 60 s;
  • whole-chain failure fails open for that opportunity only.

Observe mode and telemetry

  • observe: run the Decision and policy, never consult the Advisor or change the request. One Decision call in flight per session.
  • JSONL telemetry at ~/.local/share/opencode/auto-advisor/telemetry.jsonl (5 MB, one rotation, queued writes, failures swallowed), with models tried, answers, verdict, timings and Advisor outcome. Never transcript or advice text.

Active mode and delivery

  • On an accepted moment with budget left, consult the Advisor before the request continues.
  • Deliver the review twice:
    • as a role: "system" message at the tail of the current request;
    • as the auto-advisor instruction entry, which the host admits from the next step on.
  • Never edit event.system. The prompt prefix stays byte-stable; only the tail changes.
  • Each new review replaces the entry; it stays until superseded, including through session compaction.
  • A retried step reuses its memoized outcome; a review marker prevents duplicates.
  • routing.maxConsultationsPerTurn (default 2) counts consults started per user turn. Explicit advisor() calls don't count.
  • The run's end (session.execution.{succeeded,failed,interrupted}) aborts an in-flight consult; a stale consult delivers nothing.
  • Auxiliary generate.* calls do not re-enter the plugin's hooks (verified).

Failure behavior

  • Any Decision, Advisor, delivery or telemetry failure fails open; neither hook throws into the user's request.
  • Failures are visible in telemetry and the TUI footer ("Advisor unavailable").
  • A failure never disables later opportunities.

Explicit exclusions

  • no plugin-side retries or retry knobs (the host retries);
  • no edits to the system prompt; no session.synthetic for advice; no fabricated advisor() call;
  • no separate retained-review store (the instruction entry is the retention);
  • no persistent circuit breaker beyond the 60 s skip;
  • no mid-generation interruption;
  • no adaptive or learned thresholds in v1.

Acceptance criteria

  • Works on OpenCode 2.2.0; host error shapes captured as fixtures.
  • The router walks the chain per opportunity and classifies host errors into next-model vs stop.
  • Whole-chain failure fails open without disabling later attempts.
  • observe writes telemetry and never consults the Advisor or changes the request.
  • active delivers the review at the tail of the current step and through the entry from the next step; system prompt and tools byte-identical across requests.
  • Budget enforced per user turn; explicit calls excluded.
  • Retry reuses the memo; no duplicate reviews.
  • Interrupt during a consult: aborted, nothing delivered, stale in telemetry.
  • Auxiliary inference cannot recurse into routing.
  • Runtime matrix covers off, observe, active (mock executor + real Jev).
  • Repository gates and package-root smoke pass.
  • USER live test in the real TUI: observe telemetry, active footer/panel, advice uptake.
  • Live: session compaction keeping the entry; fallback to a second real decision provider.

Final integration responsibilities

Consolidated from #52, tracked on #47:

  • package README and package-local contributor guidance;
  • root documentation for implemented behavior;
  • release-channel policy/guard and tests;
  • changeset decision;
  • full repository gates and runtime smoke.

History (labeled)

  • Remediation (2026-09-30): canonical projections; active-only budget short-circuit; 5s Zen deadline; retry/fallback/terminal split; paid-on-public session ineligibility; suite 286/0; live 2.0.19 matrix 6/6.
  • Simplification (2026-10-02): baseline 2.0.21; custom permission engine removed (host tools already permission-filtered); multi-record history/compaction removed in favor of one bounded retained review; TUI shared store; suite 478/0; live active runs accepted/delivered with retained real advice.
  • Final remediation (2026-10-02): retention off the hook critical path; fresh-supersedes-old; oversize tombstone; write-failure keeps new; parser hardening; Gate 2 review WITH FIXES remediated; suite 488/0.
  • Rebuild on 2.2.0 (2026-10-28): feat(auto-advisor): add OpenCode Auto Advisor #61 abandoned (2.2.0 removed the Evaluation API it used). Lean rebuild in feat(auto-advisor): rebuild Auto Advisor on OpenCode 2.2.5 #94: Decision API with moment + consequence, delivery through a tail message plus an instruction entry, JSONL telemetry, budget 2, threshold 1.5, no retained-review store, ~1,250 source lines. Spikes and live checks in findings.md.
  • Advisor context (2026-10-29): #variant model refs; advisor.compactContext (host catalogs replaced with markers in the Advisor's copy) and advisor.maxTokens; advisor() returns config errors instead of silently using the agent's model. Live in the real TUI: variant reaches the provider, ~14k → ~6.8k tokens with compaction, cap holds. Suite 99/0.

Activity

  1. changed the title [-][Auto Advisor] Add Jev adapters and observe-mode telemetry[/-] [+][Auto Advisor] Add experimental Zen automatic routing and delivery[/+] on Sep 28, 2026
  2. capyBearista commented on Sep 29, 2026

    @capyBearista
    OwnerAuthor

    #50 status — implemented on 2.0.19, pending final integration

    Baseline is now OpenCode 2.0.19 (@opencode/ai@2.0.19, @opencode/plugin@2.0.19; body acceptance line updated). The Evaluation/System One adapter is source-grounded: official V2 docs expose no Evaluation API, so the implementation follows the 2.0.19 @opencode/ai source (Evaluation + SystemOne route, boolean→probability, 5-level consequence rubric).

    Implemented on feat/auto-advisor (uncommitted, PR #61 draft):

    • Production Zen AdvisorRouter adapter behind the existing seam; chain jev-1.13-free → jev-1.13 with restart-at-top; bounded retry on retryable classes only; quota→advance, terminal→fail-open. Auth via host opencode integration connection with public-bearer fallback (no secrets stored).
    • observe telemetry: 5000-event cap with oldest-first eviction, digest-only payloads (no transcripts/secrets), read-only telemetry.query / telemetry.event RPC.
    • active delivery as system-role context into the waiting primary dispatch (same-dispatch injection, reinjection on continuations, expiry on new turns, supersession, reentrancy protection). No session.synthetic, no fabricated tool calls, no persistent Advisor state.
    • Live 2.0.19 runtime matrix green: off inert; observe evaluates + records without consulting; active consults, delivers (delivered:true), budget-denied continuation. Reviews closed (final re-review: 0 critical, transcript bound deferred — see below).

    Deferred decision: transcript truncation policy is still unimplemented (pre-existing known limit). active therefore ships opt-in experimental on the off default; truncation is required before recommending it broadly.

    Remaining here: final integration responsibilities (README/root-docs reconciliation, changeset when releasable, merge). Release-channel registration already exists.

  3. capyBearista commented on Sep 30, 2026

    @capyBearista
    OwnerAuthor

    Remediation complete on feat/auto-advisor (uncommitted): 23 items implemented, gate findings remediated, suite 286/0, root gates fresh green, live 6/6 matrix green. Details appended to the issue body. Remaining: commit/push on your order, then Changeset + merge.

  4. capyBearista commented on Oct 2, 2026

    @capyBearista
    OwnerAuthor

    Simplification pass complete on feat/auto-advisor (PR #61, manual-test-ready, unmerged). Since this contract was written: multi-record history/compaction-absorption replaced by one bounded latest retained review (best-effort, never gates delivery); custom permission engine removed (2.0.21 host filtering suffices — root check + advertised tool); TUI reduced to one shared store; smoke 1337→317 LOC. Settled behavior kept (zero-arg advisor, off/observe/active, Jev-only, budget, suppression, 300s soft timeout, model-aware projection, privileged system delivery, fail-open). Fresh pkg + root gates green; live 2.0.21 probes show accepted/delivered/retained automatic reviews. Details in PR #61 body and docs/plans/opencode-auto-advisor-manual-test.md (19 cases). No merge/publish/dogfood — awaiting manual testing.

  5. changed the title [-][Auto Advisor] Add experimental Zen automatic routing and delivery[/-] [+][Auto Advisor] Add Decision-model automatic routing and delivery[/+] on Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions