diff --git a/docs/configuration/models/index.md b/docs/configuration/models/index.md index e9be86cb4c..1a5356096b 100644 --- a/docs/configuration/models/index.md +++ b/docs/configuration/models/index.md @@ -40,8 +40,8 @@ models: pdf: boolean # Optional: whether the model accepts PDF attachments audio: boolean # Optional: whether the model accepts audio attachments video: boolean # Optional: whether the model accepts video attachments - output_capabilities: # Optional: owner-declared generative output capabilities (never inferred) - image: boolean # Optional: whether the model is declared able to generate image output + output_capabilities: # Optional: override generative output capabilities (otherwise detected from models.dev) + image: boolean # Optional: whether the model can generate image output cost: # Optional: explicit token pricing (USD per 1M tokens) input: float # Optional: price per 1M input tokens output: float # Optional: price per 1M output tokens @@ -76,7 +76,7 @@ models: | `track_usage` | boolean | ✗ | Track and report token usage for this model | | `routing` | array | ✗ | Rule-based routing to different models. See [Model Routing](../routing/index.md). | | `capabilities` | object | ✗ | Override attachment (input) capabilities for this model. See [Attachment Capability Overrides](#attachment-capability-overrides). | -| `output_capabilities` | object | ✗ | Owner-declared generative output capabilities for this model, e.g. image generation. Never inferred. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). | +| `output_capabilities` | object | ✗ | Override generative output capabilities for this model, e.g. image generation. Omitted flags are detected from models.dev; explicit values take precedence. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). | | `cost` | object | ✗ | Explicit token pricing in USD per 1M tokens, overriding the built-in catalogue. See [Custom Token Pricing](#custom-token-pricing). | | `provider_opts` | object | ✗ | Provider-specific options (see provider pages) | | `title_model` | string | ✗ | Model used for session-title generation. Can be a named model from the `models:` section or an inline `provider/model` string. When omitted, the agent's primary model generates titles. Cannot be combined with `first_available`. | @@ -151,14 +151,12 @@ See [`examples/capability-overrides.yaml`](https://github.com/docker/docker-agen [`examples/strip-unsupported-media.yaml`](https://github.com/docker/docker-agent/blob/main/examples/strip-unsupported-media.yaml) for a fixture demonstrating the stripping behaviour with and without an override. -## Output Capabilities +### Output capabilities -`output_capabilities` declares what a model can generate, as opposed to -`capabilities`, which declares what it accepts as input. There is no -automatic detection for output capabilities: no catalogue of -output-capable models exists, and matching on the model name string is -deliberately avoided as unreliable. A model's output capabilities are -therefore always unknown/off unless the owner declares them. +`output_capabilities` overrides what a model can generate, as opposed to +`capabilities`, which overrides what it accepts as input. When omitted, Docker +Agent reads output modalities from the models.dev catalogue. Use an explicit +value for custom models or to correct catalogue metadata. ```yaml models: @@ -173,18 +171,15 @@ models: | --------------------------- | ------- | -------------------------------------------------------------| | `output_capabilities.image` | boolean | Whether the model is declared able to generate image output | -Omitting `output_capabilities`, or leaving `image` unset or `false`, always -preserves existing behavior. Setting it to `true` only opts the model into -behavior that specifically keys off a declared image-output capability (for -example, a provider-specific request-shape guard); it does not by itself -change what Docker Agent sends to or renders from the model. - -One side effect of declaring `image: true`: the model is skipped as a -session-title candidate, because titles are generated by a plain text-only -completion that image-output routes can reject. Title generation uses the -first non-image-output candidate (dedicated `title_model`, then the agent's -model, then its fallbacks); when every candidate declares image output, the -automatic title is skipped and the session keeps its default title. +Omitting `output_capabilities` uses models.dev metadata when available. +Setting `image` explicitly overrides the catalogue; an explicit `false` +preserves ordinary text-only behavior even when the catalogue lists image +output. Enabling image output only opts the model into behavior that keys off +that capability (for example, a provider-specific image-output request +contract); it does not guarantee that a provider will return an image. + +Session-title requests remain text-only even for image-output-capable models, +so those models remain eligible as title candidates. > [!WARNING] > **Constraint** diff --git a/docs/features/sessions/index.md b/docs/features/sessions/index.md index dd264814b4..dcdc16902c 100644 --- a/docs/features/sessions/index.md +++ b/docs/features/sessions/index.md @@ -29,6 +29,22 @@ Override the location with `-s`/`--session-db`, or by overriding the data direct $ docker agent run agent.yaml --session-db ./sessions.db ``` +## Generated Media Files + +Some models (e.g. Gemini image-output models) can generate an image as part +of a reply. Docker Agent saves each generated image into the session's +workspace — the directory the session was started in — as an ordinary, +untracked file, not into `session.db` or the data directory. The session +keeps only a reference to that file: reopening the session in the TUI +re-renders the image from the workspace file, so a valid image file +replaced at the same path renders its current bytes. If the file has +since been deleted, moved, replaced by a symlink, become an invalid +image, or grown too large, the transcript shows a short "unavailable" +note instead. The files themselves are yours — edit, commit, +or delete them like any other workspace file. See +[Generated Media](../tui/index.md#generated-media) in the Terminal UI docs +for naming, collision handling, and rendering details. + ## Resuming a Session Pass `--session ` to continue a previous conversation instead of starting a new one: diff --git a/docs/features/tui/index.md b/docs/features/tui/index.md index bfca71d5c9..24ccc868d9 100644 --- a/docs/features/tui/index.md +++ b/docs/features/tui/index.md @@ -292,11 +292,68 @@ Attached files are also recorded on the session so sub-agents spawned by task tr ### Generated Media -Some models (e.g. Gemini image-output models) can generate binary media — typically an image — as part of their reply. When that happens, docker-agent writes the generated bytes into the session's workspace (the directory the session was started in) as an ordinary, visible file, and the assistant message keeps only a relative reference to that file plus its MIME type, display name, and size — never the raw bytes. - -This keeps session JSON/database rows lightweight regardless of how many images a conversation accumulates, and the generated file is a regular workspace deliverable — visible to every tool, and yours to edit, commit, move, or delete — the same way generated code or text lands there. - -Generated media is **not** automatically resent to the model on later turns: only the surrounding text is replayed in the outgoing history, the same way a large tool result would be summarized rather than repeated. This avoids silently ballooning the context window with image bytes on every follow-up message. A future step will add TUI rendering for these files (e.g. displaying the generated image inline); today this slice covers the domain, persistence, and safety mechanics only. +Some models (e.g. Gemini image-output models like `gemini-2.5-flash-image`) +are designed to generate an image directly as part of their reply, not just +describe one. When models.dev reports that a model can generate images, or +[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities) +explicitly enables it, Docker Agent asks it for text *and* image output on the +models gateway, direct Gemini API, and Vertex AI. An explicit `false` disables +this behavior. See +[Google Gemini: Generated Images](../../providers/google/index.md#generated-images) +for exact configuration and limitations — requests with custom or built-in +tools, or structured output, are rejected locally before any request is sent +on every supported Google surface. Gateway probing confirms these combinations +are rejected; direct Gemini API and Vertex AI must pass live acceptance checks +before that conservative restriction is relaxed. + +**Where images land.** Each generated image is saved into the session's +workspace — the directory the session was started in — as an ordinary +file, the same way generated code or text lands there. Generated files are +regular untracked files: visible to every tool, and yours to edit, commit, +move, or delete. On a remote runtime the files are written into that +runtime's workspace, so there is no local file for the TUI to display. + +**Naming.** A filename named explicitly in your prompt ("Generate an image +of a red panda as `assets/red-panda.jpg`", "… and save it as `logo.png`") +is honored, including subdirectories, which are created inside the +workspace as needed. Without one, the model chooses a short, meaningful +name, with a generic `generated-1`, `generated-2`, … name as the last +resort. Two rules always apply: + +- **The extension matches the data.** The image format is decided by the + provider (typically PNG) — asking for `sunshine.gif` or `diagram.svg` + does not transcode anything. If the model returns PNG data, the file is + saved as `sunshine.png` and a notice tells you so. +- **Existing files are never overwritten.** A name collision gets a dash + suffix instead: a second `red-panda.jpg` is saved as `red-panda-1.jpg`. + +**Paths outside the workspace.** A prompt-directed target that is absolute, +`~`-rooted, or climbs above the workspace with `..` is never written +silently: you are asked to confirm the exact resolved path first, and the +safe "keep it in the workspace" choice is the default. If you decline — or +nothing can answer, as in a non-interactive run — the image is saved into +the workspace root under its plain filename instead, with a warning. The +generated bytes are never discarded. + +**Rendering.** The image appears inline in the same assistant turn, using +the same Kitty-graphics support and `render_images` setting as +[Markdown Images](#markdown-images). On a terminal without graphics support +the TUI prints the saved workspace path instead. Reopening a session +re-renders its generated images from the workspace files, so a file +replaced by another valid image at the same path shows its current +bytes. If a file has since been deleted, moved, replaced by a symlink, +become an invalid image, or grown too large, the transcript shows a +short "unavailable" note rather than wrong content. + +If a save fails (unwritable directory, full disk, …), only that image is +dropped, with a concise warning — the reply text and any sibling images in +the same turn are kept. Note also that an image-capable model can answer +with text only and generate no image at all; that is provider behavior, so +reword or repeat the prompt. + +Inline image rendering in the TUI also covers a tool/MCP result that +returns an image, or a Markdown image reference to a file a tool actually +writes to disk — see [Markdown Images](#markdown-images) above. ### Team Context Budgets and Targeted Compaction diff --git a/docs/providers/google/index.md b/docs/providers/google/index.md index e03dbe3ef8..c1fa7c69da 100644 --- a/docs/providers/google/index.md +++ b/docs/providers/google/index.md @@ -64,29 +64,43 @@ models: Some Gemini models (e.g. `gemini-2.5-flash-image`) are designed to generate an image directly as part of their reply, not just describe one. Docker -Agent's Gemini request path doesn't yet ask for that image output — that -support is still being completed — so today a request like this gets a -text-only reply. See -[Generated Media](../../features/tui/index.md#generated-media) for the -current, verified state. +Agent requests that image output on supported Google surfaces — the models +gateway, direct Gemini API, and Vertex AI — when models.dev reports that the +model can generate images or `output_capabilities.image` explicitly enables +it. An explicit `false` overrides the catalogue and keeps the model text-only. +Each ordinary chat request then asks for text *and* image output, and every +image the model returns is saved into the session's workspace and rendered +inline in the TUI — see [Generated Media](../../features/tui/index.md#generated-media) +for file naming, collision handling, and rendering details. ```yaml -agents: - root: - model: google/gemini-2.5-flash-image +models: + gemini-image: + provider: google + model: gemini-2.5-flash-image + output_capabilities: + image: true ``` -When the model is accessed through a Docker AI Gateway and explicitly -declared image-output-capable with -[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities), -Docker Agent has verified that request combined with custom function tools, -a built-in tool (e.g. `google_search`), or structured output gets rejected -by the gateway with an opaque, empty-body HTTP 400. To avoid that, Docker -Agent rejects such a combination itself, before any request is sent, with a -clear error naming which feature is incompatible. Plain text requests to -that model (no tools, no structured output) are unaffected, as is every -other route: direct Gemini API/Vertex AI calls, and gateway calls to a model -without the declaration. +When `output_capabilities.image` is omitted, Docker Agent uses models.dev +output modalities. Set it explicitly for custom models or to override +incorrect catalogue data; capability is never guessed from the model name. +Title generation and compaction remain text-only. + +Image-output requests with custom or built-in tools, or structured output, +are rejected locally before any request is sent on every supported Google +surface. Gateway probing confirms these combinations are rejected; direct +Gemini API and Vertex AI must pass live acceptance checks before that +conservative restriction is relaxed. + +A few provider-side behaviors to know: + +- **The provider decides the image format** (typically PNG). Asking for a + `.gif` or `.svg` filename does not transcode anything — the saved file's + extension is corrected to match the data actually returned. +- **An image is not guaranteed.** Even a correctly configured image model + can answer with text only and generate no image; reword or repeat the + prompt. ## Thinking Budget diff --git a/examples/README.md b/examples/README.md index 759c7f603d..d7f2255f87 100644 --- a/examples/README.md +++ b/examples/README.md @@ -216,7 +216,7 @@ remote MCP endpoints. | [`rule_based_routing.yaml`](rule_based_routing.yaml) | Cheap router model dispatches the user message to fast or capable models. | | [`structured-output.yaml`](structured-output.yaml) | Forces the model to return JSON matching a schema. | | [`google_search_grounding.yaml`](google_search_grounding.yaml) | Enables Google Search grounding on Gemini models. | -| [`gemini_image_output.yaml`](gemini_image_output.yaml) | Gemini image-output model (generated images are saved into the workspace, not inlined as base64). | +| [`gemini_image_output.yaml`](gemini_image_output.yaml) | Gemini image-output model: generated images are saved into the workspace and rendered inline in the TUI. | | [`sampling-opts.yaml`](sampling-opts.yaml) | Provider-specific sampling parameters (`top_k`, `repetition_penalty`, …). | | [`thinking_budget.yaml`](thinking_budget.yaml) | Reasoning/thinking budgets across OpenAI, Anthropic and Google. | | [`task_budget.yaml`](task_budget.yaml) | Anthropic `task_budget`: cap total tokens spent across a multi-step agentic task. | diff --git a/examples/gemini_image_output.yaml b/examples/gemini_image_output.yaml index e8490a4f62..623c8b6c6e 100644 --- a/examples/gemini_image_output.yaml +++ b/examples/gemini_image_output.yaml @@ -1,19 +1,38 @@ # Gemini image-output model: the model can generate an image directly as # part of its reply, instead of only describing one. # -# The `gemini-image` model below declares `output_capabilities.image: true` — -# an explicit, owner-provided statement that this model can generate image -# output. It is never inferred from the model name or any catalogue; omit -# it, or leave it false, and behavior is unchanged. +# The `gemini-image` model below explicitly overrides +# `output_capabilities.image: true`. When this flag is omitted, Docker Agent +# uses models.dev output modalities; explicit true or false takes precedence. +# Capability is never guessed from the model name. # -# Native generated images aren't currently presented inline in the -# terminal UI, so the model's text reply is what you'll see today. +# On the models gateway, direct Gemini API, and Vertex AI, Docker Agent +# requests text AND image output for this image-capable model. Each +# generated image is saved into the session's workspace as an ordinary file +# and rendered inline in the TUI (terminals without graphics support print +# the saved path instead). +# +# Notes: +# - Requests with custom or built-in tools, or structured output, are +# rejected locally before any request is sent on every supported Google +# surface. Gateway probing confirms these combinations are rejected; +# direct Gemini API and Vertex AI must pass live acceptance checks before +# that conservative restriction is relaxed. +# - Naming: an explicit filename in the prompt is honored (subdirectories +# are created inside the workspace); otherwise the model picks a +# meaningful name. Extensions are corrected to match the returned data +# (the provider decides the format, typically PNG), and existing files +# are never overwritten — collisions get a dash suffix (logo-1.png). +# - The model may reply with text only and no image — that's provider +# behavior; reword or repeat the prompt. # # Try it out: # docker agent run examples/gemini_image_output.yaml \ # "Generate an image of a red panda working at a terminal" # docker agent run examples/gemini_image_output.yaml \ -# "Generate an image of a lighthouse at sunset, and describe the color palette you used" +# "Generate an image of a lighthouse at sunset as assets/lighthouse.png" +# docker agent run examples/gemini_image_output.yaml \ +# "Generate an image of a whale and save it as whale.jpg" models: gemini-image: provider: google