feat(#3996): strip unsupported audio/video media parts - #4016
Draft
aheritier wants to merge 3 commits into
Draft
Conversation
Extend ModelCapabilities with SupportsAudio/SupportsVideo, parse the models.dev "audio"/"video" input modalities in LoadCaps, and thread the new fields through CapsWith, CapsOverride and ResolveCaps so every call site keeps compiling. Unknown/uncatalogued models stay conservative (no audio/video inferred), matching the existing image/pdf posture. This is foundation-only: no config schema, override wiring, or provider behavior changes — config-level capability overrides and provider gating build on it in later commits.
Extend CapabilitiesConfig with Audio/Video, completing the override introduced by the previous commit for models the models.dev catalogue does not describe correctly (custom OpenAI-compatible providers, local models, dropped catalogue entries) — without it such models silently lose audio/video attachments the same way image/PDF did before capabilities existed (issue #2741). Update schema, docs, and example; CapsOverride now threads all four flags through to modelinfo.
Generalize the strip_unsupported_modalities transform from image-only to all three media families (image/audio/video), classified by MIME family from Document/File parts (legacy ImageURL parts stay image). The transform now consumes the already-resolved capability set instead of querying models.dev itself: the loop resolves capabilities for the model it actually chose (per-tool override + alloy pick) via the new modelinfo.ResolveCapsFromModel — explicit capabilities: config overrides win, unknown models resolve to the conservative text-only default — and threads the result through hooks.Input.ModelCapabilities (json:"-", in-process only). This fixes the override bug where a model declared capabilities.audio/image: true still had its media stripped based on the catalogue, and aligns unknown-model behavior with the attachment pipeline's conservative posture. Each stripped part emits a debug log naming its media kind and reason. Harness labels carry no resolvable capabilities, so that path passes nil and capability-gated transforms skip, preserving harness behavior.
aheritier
added
area/config
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds capability-aware filtering that strips unsupported audio/video media parts before provider requests, with schema, model metadata, runtime transforms, provider coverage, examples, and tests.
Why
Prevents incompatible media from reaching gateways while preserving supported modalities and making capability overrides explicit.
Validation
task test; targeted attachment, capability, modelinfo, runtime, and provider tests.Test instructions
Run:
task testThen run an agent with a mixed text/image/audio/video prompt against a model that does not support every modality. Expected: unsupported audio/video parts are omitted before the provider request, while text and supported image content remain.