Repository navigation
feat: add multimodal Skills for keyed Providers - #252
Merged
Merged
Conversation
A command that exited 0 could still be reported as stalled: a process it left behind kept the output pipes open, so Wait blocked until the stall watchdog fired. Run now starts the command in its own process tree, stops reading the pipes a few seconds after the command exits, and treats a clean exit as success whatever the watchdog did meanwhile. npm prints nothing while it downloads, which the watchdog also read as a stall. Install and update now pass npm's cache and install directories as activity paths, and growth under them counts as progress. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A model the provider lists can still be refused when an Agent calls it, and a Profile edit rewrites every Agent bound to it at once, so a single model change could break Claude Code and Codex with nothing said. Profile cards and the Agent's Profile page now have a 检测模型 button that sends one live request per protocol the Agents use and reports the verdict: available, unavailable (refused model or rejected key), or unverified (network, rate limit, server error). A failure offers the next step that fits its cause: switching back to the model the page just replaced, fixing the key, or checking again. Saving and applying a Profile never check on their own: each check is a billed request, so the user decides when it goes out. Inline model saves on the Agent's Profile page now also report Agents they could not rewrite, and 应用 waits for an in-flight model save instead of applying the stale model. The probe result carries the provider's own error message, cleaned, bounded and with the API key scrubbed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Opening the editor on the Providers, Profiles and Agent Profile pages left the full list and the last save's notices rendered below the form, which read as the previous page appended to the editor. The editor now replaces the list until it is saved or cancelled, and opens scrolled to the top so a card's editor is not left above the fold. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add the multimodal Skills design, ADR-011 for delivering Provider keys to generated Skills as Agent environment variables, and trimmed public /models snapshots of Novita, PPIO and JieKou.AI for replay tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add modalities.lock.json with the ID rules and native routes, and a modality package that resolves each listed model's capabilities in the order probe verdict, declared metadata, ID rule. ListModels now keeps the declared modality lists, and the capability cache moves to schema 2 with per-model modality verdicts while still reading version 1. A MultimodalService exposes Detect, an unbilled model listing, and Probe, a one-token image request sent only from a button. The Skills page gets a Multimodal capabilities section that shows each finding's source and whether it can be installed yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Generate a Skill per capability (SKILL.md, config.json and a dependency-free run.mjs) for image understanding, image generation, speech-to-text and text-to-speech over the OpenAI-compatible routes, store it as a generated variant, and publish it to the chosen Agents. The Provider key reaches the script as BOOTAGENT_<PROVIDER>_API_KEY in Claude Code's env block and Codex's shell_environment_policy.set, with a private fallback file for other Agents (ADR-011). One reconcile keeps those variables matching the generated Skills after install, uninstall, Skills page changes and Provider edits; only variants rendered here earn a key. config.toml is written as a secret once it holds one. Add probes for transcription, speech and image generation; the image probe needs an explicit confirmation. Models can be entered by hand and install once their probe passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add the routes Novita, PPIO and JieKou.AI document for one shared native API family: synchronous MiniMax Speech (novita-audio) and async submit, poll and download (novita-async). run.mjs gains a native speak and a video command that paces polling by the provider's eta, streams the result to disk without the key, and exits 7 with the task ID when a run outlasts its timeout, so video-result can collect it without paying again. The catalog now lists only models whose request was read from vendor docs: MiniMax Speech on all three, GPT Image and two text-to-video models on JieKou.AI, and ming-image on Novita. A catalog entry outranks a declaration for the capabilities it names. PPIO and Novita no longer document video generation, so it is marked unsupported for both and the ppio-video adapter stays reserved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Answer questions about a local video by sampling frames with ffmpeg and sending them to an image-understanding model, which lends the capability its own evidence. ffmpeg runs with an argv array, a timeout and PATH alone. Native video content parts stay probe-only: Moonshot takes video only as an uploaded file, not a data URL. Offer OpenCode as an install target on the private fallback key file, since neither OpenCode nor Hermes documents a way to set its shell tool's environment. Hermes is held back pending the decision on Node. Agents without a Skills directory can get the same tree as a guide in ~/.bootagent/multimodal/<capability>/ with a prompt pointing at it. The guide is found by its hash rather than registered as a target, and follows the Skill through key changes, Provider deletes and uninstall. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When Get-AppxPackage returns nothing or fails, look the package family up through PackageManager.FindPackagesForUser before reporting the version as unavailable. The family is passed through the environment, never interpolated into the script. If metadata is still missing, the card says launch can still be attempted and that version detection is unrelated to model API support. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A model check refused over the Responses API now explains that Codex needs a Responses model and offers a button to the API protocol adapter. Other reasons and protocols keep their existing remedies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Below a long Skill library the section was out of sight. It now has a sidebar entry at /multimodal, and the Skills page links to it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Install reads the vendor's electron-updater feed, resolving its relative URLs against code.kimi.com and keeping downloads on code.kimi.com or cdn.kimi.com. On macOS it verifies the zip's SHA-512 and checks the bundle ID com.kimi.code.desktop, the Developer ID team 2J9472RW75 and notarization before copying the app into Applications. On Windows it verifies the installer's digest and its Authenticode publisher, NOVASCENT PRIVATE LIMITED (the signer of the vendor's published installer, pinned deliberately), then starts the NSIS installer. The app shares ~/.kimi-code/config.toml with the Kimi Code CLI, so applying a Profile to it activates the kimi-code Agent. The card reuses the Kimi mark. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Generated images, audio and video now default to ~/.bootagent/output instead of bootagent-output in the working directory. The Skill templates told Agents to run from the Skill directory, so the first generation changed the managed tree's hash and blocked later reinstall, model switches, endpoint refreshes and uninstall. The script also: - redacts successful text responses (vision, transcription, video understanding) before printing them - sets process.exitCode instead of calling process.exit, so long transcripts written to a pipe are not truncated - writes each result to its own temporary directory and publishes it with a hard link, so concurrent downloads in the same second never share a .part file or replace each other's output Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A 200 response whose body could not be read, was not JSON, lacked base_resp.status_code or carried no decodable audio was reported as a supported speech route and cached. These now classify as a transient failure, so the capability stays unverified. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- InstallMultimodal re-reads the Provider after taking writeMu and refuses with a retryable error if it changed while models were being listed, so a Skill never pairs an old endpoint with a new key. - Key writes from Profile save, Agent activation and install now refresh generated Skills like SaveProvider does, and report refresh failures. - v2 transfer imports refresh generated Skills for every Provider they change, and roll those Skill trees and credential files back if the refresh fails. - Key reconciliation never hands a new key to a Skill that still points at a previous endpoint, and reports registry and catalog read failures instead of silently succeeding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A model check that resolved after a later save could still offer to switch back, and reverting overwrote the newer model. Every save now invalidates in-flight checks, and reverting first confirms the Profile still holds the checked model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- The install button opens the dialog even with no install targets, so users can still generate the guide files for other Agents. - A manual probe's result or error is shown unless the same model's result is already shown in its capability row. - The install dialog cannot be closed while an installation is running, so its result and the follow-up refresh are never lost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds multimodal capabilities: BootAgent detects which models on keyed Providers can understand images, video or audio, or generate images, speech and video, and installs a generated Skill for each into Claude Code, Codex, OpenCode and other Agents. Agents without a Skills directory get guide files under
~/.bootagent/multimodal/. Design and credential delivery are recorded indocs/multimodal-skills-design.md.The branch also carries a few independent changes made along the way: on-demand Profile model checks, Kimi Code Desktop as an installable Agent, ChatGPT desktop version recovery from AppX, and a fix for npm commands falsely reported as stalled.
Review fixes (last 5 commits)
The full-branch review found 3 P1 and 8 P2 issues; each is fixed with a regression test.
~/.bootagent/output.process.exit..partfile.Tests
go vet ./...,go test ./...,staticcheck ./...pass;go test -racepasses forinternal/app,internal/skillgen,internal/providerinternal/app,internal/skillgen,internal/providertests on its ownpnpm run buildsucceedsMarketplace detail back restores the search and filter sessionfailed once and passed on 4 reruns; unrelated to this changeLocally,
pnpm run test:e2etimes out unless the Go caches are passed through, becausee2e/wails-server.mjsreplacesHOME:🤖 Generated with Claude Code