Skip to content

feat: optional AI explanations and agent hand-off - #6

Merged
piekstra merged 3 commits into
mainfrom
piekstra/ai
Oct 7, 2026
Merged

piekstra merged 3 commits into
mainfrom
piekstra/ai

Conversation

@piekstra

@piekstra piekstra commented Oct 7, 2026

Copy link
Copy Markdown
Member

Summary

Optional AI, described in docs/ai.md and following the rules in docs/design.md. AI is off by default, nothing is sent anywhere until a provider is chosen, and turning it on changes no layout.

  • Provider (crates/lest-core/src/ai.rs): one headless call. The prompt goes on stdin with no tools, no MCP servers and no user customizations, in an empty temporary directory, killed after 90s. Choices are none, auto, a supported agent CLI, or command (any program that reads stdin and prints text). Answers are cached by input, calls are counted per day, and dailyLimit caps them.
  • Explanations: the failure card's slot shows the model's text with a "Written by " marker, and the deterministic headline moves to the second line. Without a provider, or when the provider fails, the headline stays as the answer, with a note in the fallback case. Explanations are requested only for finished runs.
  • Hand-off: Copy context exists today. When an agent is configured, the card's menu adds Open in agent and Copy agent command. Open in agent starts the user's own agent CLI in a terminal, in the project, with the task. The server builds the command; it never comes from the request. Lest has no chat.
  • CLI: lest explain <run>, lest context <run>, lest docs agent (the authoring guide plus the format, browser and demo references, as installed), and lest config show|set.
  • Settings: an AI section with provider, command, model, agent command and daily limit, and a fixed status line.

Evidence

make check passes, including 37 UI tests (8 new ones cover each state of the explanation slot) and 5 provider tests: the deterministic fallback, a command provider with caching, a failing provider, the context contents, and the agent command. I dogfooded the CLI against the failing example with a real agent CLI. The explanation came back and was cached: a second call took 17ms. Screenshots use a custom command provider (local-model), so no vendor appears.

AI on: a model explanation in the slot, the headline beside it, and Open in agent in the menu.

Failure card with AI

AI off: the same card, for comparison.

Failure card without AI

Settings, AI section

AI settings

AI is off by default and changes no layout. With a provider (a detected
agent CLI or any stdin-to-stdout command), a failed step's card shows a
model-written explanation beside its deterministic headline, and the menu
offers Open in agent, which starts the user's own agent CLI in the project
with the failure's context. The CLI adds `lest explain`, `lest context`,
`lest docs agent` and `lest config show|set`; calls are cached and capped
per day.
@piekstra

piekstra commented Oct 7, 2026

Copy link
Copy Markdown
Member Author

A review found 12 issues. Dispositions:

  1. The timeout did not cover writing the prompt. The prompt is now written on its own thread, so the 90s timeout bounds the whole call. Prompts are cut at 48 KB. New test a_provider_that_never_reads_is_stopped_on_time: a 400 KB prompt to a provider that never reads stdin stops in about 1s.
  2. The docs overstated the second CLI's isolation. It now also runs with --ignore-user-config --ignore-rules. The docs say plainly that it runs in a read-only sandbox and can still read files.
  3. Env stripping broke CLI sign-in. Only the session-marker variables are removed now. The CLIs' config and credential variables stay.
  4. Secrets under a custom env prefix leaked. The provider no longer gets any variable the configured env secret backends read, whatever the prefix. The CLI and server pass these.
  5. A quoted agent template ran commands. The task now reaches the agent through LEST_AGENT_PROMPT, so a quoted {prompt} in the template is safe. The prompt has no backticks. This is tested.
  6. Open in agent showed and then failed. It is offered only when a runnable agent command exists. The Settings hint says a custom command provider has no agent.
  7. Cache keys collided. The cache key now includes the full command and the model.
  8. Timeouts left children running. The provider runs in its own process group, and the whole group is killed on timeout (verified: no leftover sleep).
  9. AI on changed the card layout. The marker no longer reserves width when AI is on.
  10. The trust model was undocumented. docs/ai.md now has a Trust section: the UI token is equivalent to running commands as the user.
  11. The daily limit could be exceeded. Check-and-record now takes a lock, and the docs say calls are counted per UTC day.
  12. Copy was stale. Fixed in docs/ui.md, docs/ai.md and the Settings copy: the agent command field, when menu items appear, what the model receives, the cache key, and Copy context being a button.

make check passes.

@piekstra
piekstra marked this pull request as ready for review October 7, 2026 14:32
@piekstra
piekstra merged commit 6e20577 into main Oct 7, 2026
5 checks passed
@piekstra
piekstra deleted the piekstra/ai branch October 7, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant