Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 17 additions & 22 deletions docs/configuration/models/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ models:
pdf: boolean # Optional: whether the model accepts PDF attachments
audio: boolean # Optional: whether the model accepts audio attachments
video: boolean # Optional: whether the model accepts video attachments
output_capabilities: # Optional: owner-declared generative output capabilities (never inferred)
image: boolean # Optional: whether the model is declared able to generate image output
output_capabilities: # Optional: override generative output capabilities (otherwise detected from models.dev)
image: boolean # Optional: whether the model can generate image output
cost: # Optional: explicit token pricing (USD per 1M tokens)
input: float # Optional: price per 1M input tokens
output: float # Optional: price per 1M output tokens
Expand Down Expand Up @@ -76,7 +76,7 @@ models:
| `track_usage` | boolean | ✗ | Track and report token usage for this model |
| `routing` | array | ✗ | Rule-based routing to different models. See [Model Routing](../routing/index.md). |
| `capabilities` | object | ✗ | Override attachment (input) capabilities for this model. See [Attachment Capability Overrides](#attachment-capability-overrides). |
| `output_capabilities` | object | ✗ | Owner-declared generative output capabilities for this model, e.g. image generation. Never inferred. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). |
| `output_capabilities` | object | ✗ | Override generative output capabilities for this model, e.g. image generation. Omitted flags are detected from models.dev; explicit values take precedence. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). |
| `cost` | object | ✗ | Explicit token pricing in USD per 1M tokens, overriding the built-in catalogue. See [Custom Token Pricing](#custom-token-pricing). |
| `provider_opts` | object | ✗ | Provider-specific options (see provider pages) |
| `title_model` | string | ✗ | Model used for session-title generation. Can be a named model from the `models:` section or an inline `provider/model` string. When omitted, the agent's primary model generates titles. Cannot be combined with `first_available`. |
Expand Down Expand Up @@ -151,14 +151,12 @@ See [`examples/capability-overrides.yaml`](https://github.com/docker/docker-agen
[`examples/strip-unsupported-media.yaml`](https://github.com/docker/docker-agent/blob/main/examples/strip-unsupported-media.yaml) for a fixture demonstrating the
stripping behaviour with and without an override.

## Output Capabilities
### Output capabilities

`output_capabilities` declares what a model can generate, as opposed to
`capabilities`, which declares what it accepts as input. There is no
automatic detection for output capabilities: no catalogue of
output-capable models exists, and matching on the model name string is
deliberately avoided as unreliable. A model's output capabilities are
therefore always unknown/off unless the owner declares them.
`output_capabilities` overrides what a model can generate, as opposed to
`capabilities`, which overrides what it accepts as input. When omitted, Docker
Agent reads output modalities from the models.dev catalogue. Use an explicit
value for custom models or to correct catalogue metadata.

```yaml
models:
Expand All @@ -173,18 +171,15 @@ models:
| --------------------------- | ------- | -------------------------------------------------------------|
| `output_capabilities.image` | boolean | Whether the model is declared able to generate image output |

Omitting `output_capabilities`, or leaving `image` unset or `false`, always
preserves existing behavior. Setting it to `true` only opts the model into
behavior that specifically keys off a declared image-output capability (for
example, a provider-specific request-shape guard); it does not by itself
change what Docker Agent sends to or renders from the model.

One side effect of declaring `image: true`: the model is skipped as a
session-title candidate, because titles are generated by a plain text-only
completion that image-output routes can reject. Title generation uses the
first non-image-output candidate (dedicated `title_model`, then the agent's
model, then its fallbacks); when every candidate declares image output, the
automatic title is skipped and the session keeps its default title.
Omitting `output_capabilities` uses models.dev metadata when available.
Setting `image` explicitly overrides the catalogue; an explicit `false`
preserves ordinary text-only behavior even when the catalogue lists image
output. Enabling image output only opts the model into behavior that keys off
that capability (for example, a provider-specific image-output request
contract); it does not guarantee that a provider will return an image.

Session-title requests remain text-only even for image-output-capable models,
so those models remain eligible as title candidates.

> [!WARNING]
> **Constraint**
Expand Down
16 changes: 16 additions & 0 deletions docs/features/sessions/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,22 @@ Override the location with `-s`/`--session-db`, or by overriding the data direct
$ docker agent run agent.yaml --session-db ./sessions.db
```

## Generated Media Files

Some models (e.g. Gemini image-output models) can generate an image as part
of a reply. Docker Agent saves each generated image into the session's
workspace — the directory the session was started in — as an ordinary,
untracked file, not into `session.db` or the data directory. The session
keeps only a reference to that file: reopening the session in the TUI
re-renders the image from the workspace file, so a valid image file
replaced at the same path renders its current bytes. If the file has
since been deleted, moved, replaced by a symlink, become an invalid
image, or grown too large, the transcript shows a short "unavailable"
note instead. The files themselves are yours — edit, commit,
or delete them like any other workspace file. See
[Generated Media](../tui/index.md#generated-media) in the Terminal UI docs
for naming, collision handling, and rendering details.

## Resuming a Session

Pass `--session <id>` to continue a previous conversation instead of starting a new one:
Expand Down
67 changes: 62 additions & 5 deletions docs/features/tui/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -292,11 +292,68 @@ Attached files are also recorded on the session so sub-agents spawned by task tr

### Generated Media

Some models (e.g. Gemini image-output models) can generate binary media — typically an image — as part of their reply. When that happens, docker-agent writes the generated bytes into the session's workspace (the directory the session was started in) as an ordinary, visible file, and the assistant message keeps only a relative reference to that file plus its MIME type, display name, and size — never the raw bytes.

This keeps session JSON/database rows lightweight regardless of how many images a conversation accumulates, and the generated file is a regular workspace deliverable — visible to every tool, and yours to edit, commit, move, or delete — the same way generated code or text lands there.

Generated media is **not** automatically resent to the model on later turns: only the surrounding text is replayed in the outgoing history, the same way a large tool result would be summarized rather than repeated. This avoids silently ballooning the context window with image bytes on every follow-up message. A future step will add TUI rendering for these files (e.g. displaying the generated image inline); today this slice covers the domain, persistence, and safety mechanics only.
Some models (e.g. Gemini image-output models like `gemini-2.5-flash-image`)
are designed to generate an image directly as part of their reply, not just
describe one. When models.dev reports that a model can generate images, or
[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities)
explicitly enables it, Docker Agent asks it for text *and* image output on the
models gateway, direct Gemini API, and Vertex AI. An explicit `false` disables
this behavior. See
[Google Gemini: Generated Images](../../providers/google/index.md#generated-images)
for exact configuration and limitations — requests with custom or built-in
tools, or structured output, are rejected locally before any request is sent
on every supported Google surface. Gateway probing confirms these combinations
are rejected; direct Gemini API and Vertex AI must pass live acceptance checks
before that conservative restriction is relaxed.

**Where images land.** Each generated image is saved into the session's
workspace — the directory the session was started in — as an ordinary
file, the same way generated code or text lands there. Generated files are
regular untracked files: visible to every tool, and yours to edit, commit,
move, or delete. On a remote runtime the files are written into that
runtime's workspace, so there is no local file for the TUI to display.

**Naming.** A filename named explicitly in your prompt ("Generate an image
of a red panda as `assets/red-panda.jpg`", "… and save it as `logo.png`")
is honored, including subdirectories, which are created inside the
workspace as needed. Without one, the model chooses a short, meaningful
name, with a generic `generated-1`, `generated-2`, … name as the last
resort. Two rules always apply:

- **The extension matches the data.** The image format is decided by the
provider (typically PNG) — asking for `sunshine.gif` or `diagram.svg`
does not transcode anything. If the model returns PNG data, the file is
saved as `sunshine.png` and a notice tells you so.
- **Existing files are never overwritten.** A name collision gets a dash
suffix instead: a second `red-panda.jpg` is saved as `red-panda-1.jpg`.

**Paths outside the workspace.** A prompt-directed target that is absolute,
`~`-rooted, or climbs above the workspace with `..` is never written
silently: you are asked to confirm the exact resolved path first, and the
safe "keep it in the workspace" choice is the default. If you decline — or
nothing can answer, as in a non-interactive run — the image is saved into
the workspace root under its plain filename instead, with a warning. The
generated bytes are never discarded.

**Rendering.** The image appears inline in the same assistant turn, using
the same Kitty-graphics support and `render_images` setting as
[Markdown Images](#markdown-images). On a terminal without graphics support
the TUI prints the saved workspace path instead. Reopening a session
re-renders its generated images from the workspace files, so a file
replaced by another valid image at the same path shows its current
bytes. If a file has since been deleted, moved, replaced by a symlink,
become an invalid image, or grown too large, the transcript shows a
short "unavailable" note rather than wrong content.

If a save fails (unwritable directory, full disk, …), only that image is
dropped, with a concise warning — the reply text and any sibling images in
the same turn are kept. Note also that an image-capable model can answer
with text only and generate no image at all; that is provider behavior, so
reword or repeat the prompt.

Inline image rendering in the TUI also covers a tool/MCP result that
returns an image, or a Markdown image reference to a file a tool actually
writes to disk — see [Markdown Images](#markdown-images) above.

### Team Context Budgets and Targeted Compaction

Expand Down
52 changes: 33 additions & 19 deletions docs/providers/google/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,29 +64,43 @@ models:

Some Gemini models (e.g. `gemini-2.5-flash-image`) are designed to generate
an image directly as part of their reply, not just describe one. Docker
Agent's Gemini request path doesn't yet ask for that image output — that
support is still being completed — so today a request like this gets a
text-only reply. See
[Generated Media](../../features/tui/index.md#generated-media) for the
current, verified state.
Agent requests that image output on supported Google surfaces — the models
gateway, direct Gemini API, and Vertex AI — when models.dev reports that the
model can generate images or `output_capabilities.image` explicitly enables
it. An explicit `false` overrides the catalogue and keeps the model text-only.
Each ordinary chat request then asks for text *and* image output, and every
image the model returns is saved into the session's workspace and rendered
inline in the TUI — see [Generated Media](../../features/tui/index.md#generated-media)
for file naming, collision handling, and rendering details.

```yaml
agents:
root:
model: google/gemini-2.5-flash-image
models:
gemini-image:
provider: google
model: gemini-2.5-flash-image
output_capabilities:
image: true
```

When the model is accessed through a Docker AI Gateway and explicitly
declared image-output-capable with
[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities),
Docker Agent has verified that request combined with custom function tools,
a built-in tool (e.g. `google_search`), or structured output gets rejected
by the gateway with an opaque, empty-body HTTP 400. To avoid that, Docker
Agent rejects such a combination itself, before any request is sent, with a
clear error naming which feature is incompatible. Plain text requests to
that model (no tools, no structured output) are unaffected, as is every
other route: direct Gemini API/Vertex AI calls, and gateway calls to a model
without the declaration.
When `output_capabilities.image` is omitted, Docker Agent uses models.dev
output modalities. Set it explicitly for custom models or to override
incorrect catalogue data; capability is never guessed from the model name.
Title generation and compaction remain text-only.

Image-output requests with custom or built-in tools, or structured output,
are rejected locally before any request is sent on every supported Google
surface. Gateway probing confirms these combinations are rejected; direct
Gemini API and Vertex AI must pass live acceptance checks before that
conservative restriction is relaxed.

A few provider-side behaviors to know:

- **The provider decides the image format** (typically PNG). Asking for a
`.gif` or `.svg` filename does not transcode anything — the saved file's
extension is corrected to match the data actually returned.
- **An image is not guaranteed.** Even a correctly configured image model
can answer with text only and generate no image; reword or repeat the
prompt.

## Thinking Budget

Expand Down
2 changes: 1 addition & 1 deletion examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ remote MCP endpoints.
| [`rule_based_routing.yaml`](rule_based_routing.yaml) | Cheap router model dispatches the user message to fast or capable models. |
| [`structured-output.yaml`](structured-output.yaml) | Forces the model to return JSON matching a schema. |
| [`google_search_grounding.yaml`](google_search_grounding.yaml) | Enables Google Search grounding on Gemini models. |
| [`gemini_image_output.yaml`](gemini_image_output.yaml) | Gemini image-output model (generated images are saved into the workspace, not inlined as base64). |
| [`gemini_image_output.yaml`](gemini_image_output.yaml) | Gemini image-output model: generated images are saved into the workspace and rendered inline in the TUI. |
| [`sampling-opts.yaml`](sampling-opts.yaml) | Provider-specific sampling parameters (`top_k`, `repetition_penalty`, …). |
| [`thinking_budget.yaml`](thinking_budget.yaml) | Reasoning/thinking budgets across OpenAI, Anthropic and Google. |
| [`task_budget.yaml`](task_budget.yaml) | Anthropic `task_budget`: cap total tokens spent across a multi-step agentic task. |
Expand Down
33 changes: 26 additions & 7 deletions examples/gemini_image_output.yaml
Original file line number Diff line number Diff line change
@@ -1,19 +1,38 @@
# Gemini image-output model: the model can generate an image directly as
# part of its reply, instead of only describing one.
#
# The `gemini-image` model below declares `output_capabilities.image: true` —
# an explicit, owner-provided statement that this model can generate image
# output. It is never inferred from the model name or any catalogue; omit
# it, or leave it false, and behavior is unchanged.
# The `gemini-image` model below explicitly overrides
# `output_capabilities.image: true`. When this flag is omitted, Docker Agent
# uses models.dev output modalities; explicit true or false takes precedence.
# Capability is never guessed from the model name.
#
# Native generated images aren't currently presented inline in the
# terminal UI, so the model's text reply is what you'll see today.
# On the models gateway, direct Gemini API, and Vertex AI, Docker Agent
# requests text AND image output for this image-capable model. Each
# generated image is saved into the session's workspace as an ordinary file
# and rendered inline in the TUI (terminals without graphics support print
# the saved path instead).
#
# Notes:
# - Requests with custom or built-in tools, or structured output, are
# rejected locally before any request is sent on every supported Google
# surface. Gateway probing confirms these combinations are rejected;
# direct Gemini API and Vertex AI must pass live acceptance checks before
# that conservative restriction is relaxed.
# - Naming: an explicit filename in the prompt is honored (subdirectories
# are created inside the workspace); otherwise the model picks a
# meaningful name. Extensions are corrected to match the returned data
# (the provider decides the format, typically PNG), and existing files
# are never overwritten — collisions get a dash suffix (logo-1.png).
# - The model may reply with text only and no image — that's provider
# behavior; reword or repeat the prompt.
#
# Try it out:
# docker agent run examples/gemini_image_output.yaml \
# "Generate an image of a red panda working at a terminal"
# docker agent run examples/gemini_image_output.yaml \
# "Generate an image of a lighthouse at sunset, and describe the color palette you used"
# "Generate an image of a lighthouse at sunset as assets/lighthouse.png"
# docker agent run examples/gemini_image_output.yaml \
# "Generate an image of a whale and save it as whale.jpg"
models:
gemini-image:
provider: google
Expand Down
Loading