From 499db930a949a01c310df89194c98fe29d75a594 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Arnaud=20He=CC=81ritier?= Date: Mon, 27 Jul 2026 22:13:34 +0200 Subject: [PATCH 1/2] docs(#3996): document generated-media workspace storage, lifecycle, and validation walkthrough User-facing documentation for the generated-media feature across docs/features/tui/index.md (Generated Media section: storage location, naming and collision behavior, no-resend placeholders, bounded/sanitized warning and placeholder wording), docs/features/sessions/index.md (generated-media lifecycle), and docs/providers/google/index.md (image-output model guidance), plus a step-by-step manual validation walkthrough for contributors. --- docs/features/sessions/index.md | 29 +++++ docs/features/tui/index.md | 189 +++++++++++++++++++++++++++++++- 2 files changed, 215 insertions(+), 3 deletions(-) diff --git a/docs/features/sessions/index.md b/docs/features/sessions/index.md index dd264814b4..74493f0d3c 100644 --- a/docs/features/sessions/index.md +++ b/docs/features/sessions/index.md @@ -29,6 +29,35 @@ Override the location with `-s`/`--session-db`, or by overriding the data direct $ docker agent run agent.yaml --session-db ./sessions.db ``` +## Generated-Media Artifacts + +Some models (e.g. Gemini image-output models) can generate binary media — typically an image — as part of a reply. Docker Agent writes that data to a managed **artifact root** kept outside `session.db`: `/session_artifacts//` (`~/.cagent/session_artifacts//` by default). The session message itself keeps only a relative reference plus safe display metadata (MIME type, sanitized name, size) — never the raw bytes. See [Generated Media](../tui/index.md#generated-media) in the Terminal UI docs for the full placeholder, no-resend, and sanitization mechanics; this section covers only where and how the artifacts themselves are stored, referenced, and cleaned up. + +### Owner-Qualified References + +`` in the artifact path is always the **owning** session — the session that was active when the media was generated — not necessarily whichever session currently holds the message. That owning session ID is recorded on the message itself and never changes, so branching or forking a session (see [Browsing Sessions in the TUI](#browsing-sessions-in-the-tui) below) keeps every generated-media reference resolving under the *original* artifact directory instead of a nonexistent one under the new session's ID. + +References materialized before this owner-qualified scheme existed carry no owner and cannot be resolved at all: there is no session ID left to safely resolve them against, so they surface as an unresolvable attachment rather than risk silently reading the wrong (or no) file under a guessed session. + +### Independent of `session.db` + +The artifact root and the session database are two independent stores that happen to share the same default parent directory: + +- `--data-dir` moves both by default — `session.db` and `session_artifacts/` relocate together under the new data directory. +- `-s`/`--session-db` moves *only* the database. Pointing it at a path outside the data directory does **not** relocate `session_artifacts/`, which always stays under `/session_artifacts/` regardless of where the database file itself ends up. +- Copying, moving, or backing up one of the two without the other silently leaves generated-media references unresolvable on whichever side didn't move. + +### No Automatic Cleanup + +Deleting a session — from `/sessions` in the TUI, or by removing its row from `session.db` directly — does **not** delete its generated artifacts, and no automatic garbage collection exists today: the artifact files and the session record are independent stores that are never reconciled for you. + +To reclaim space or fully remove a session's generated media, delete its subdirectory under `session_artifacts/` yourself once you no longer need it. This is safe: each session's artifacts live in their own subdirectory, so removing one never touches another session's files or the session database itself. + +```bash +# Remove one session's generated-media artifacts. +$ rm -rf ~/.cagent/session_artifacts// +``` + ## Resuming a Session Pass `--session ` to continue a previous conversation instead of starting a new one: diff --git a/docs/features/tui/index.md b/docs/features/tui/index.md index bfca71d5c9..88a04fcf2c 100644 --- a/docs/features/tui/index.md +++ b/docs/features/tui/index.md @@ -292,11 +292,194 @@ Attached files are also recorded on the session so sub-agents spawned by task tr ### Generated Media -Some models (e.g. Gemini image-output models) can generate binary media — typically an image — as part of their reply. When that happens, docker-agent writes the generated bytes into the session's workspace (the directory the session was started in) as an ordinary, visible file, and the assistant message keeps only a relative reference to that file plus its MIME type, display name, and size — never the raw bytes. +Some models (e.g. Gemini image-output models) can generate binary media — typically an image — as part of their reply. When that happens, docker-agent writes the generated bytes to a managed artifact directory under `/session_artifacts//` (defaults to `~/.cagent/session_artifacts//`; override the data directory with the `--data-dir` flag) and the assistant message keeps only a relative reference to that file plus its MIME type, display name, and size — never the raw bytes. `` here is always the **owning** session — the session that was active when the media was generated — recorded alongside the reference so it keeps resolving correctly even after the message is copied into a branched or forked session (see below). -This keeps session JSON/database rows lightweight regardless of how many images a conversation accumulates, and the generated file is a regular workspace deliverable — visible to every tool, and yours to edit, commit, move, or delete — the same way generated code or text lands there. +This keeps session JSON/database rows lightweight regardless of how many images a conversation accumulates. If writing a generated item to the artifact directory fails (e.g. a full or read-only disk), that one item is dropped and a notification reports it by its exact position/count in the batch plus its sanitized MIME type and display name — falling back to the same `generated media` / `application/octet-stream` defaults as the placeholder text below when the name or MIME type is empty, whitespace-only, or missing, and never simply omitting the name — never the artifact directory's absolute path or the underlying OS error, which are logged to the debug log (`--debug`) instead; any other item in the same reply that saved successfully is unaffected. -Generated media is **not** automatically resent to the model on later turns: only the surrounding text is replayed in the outgoing history, the same way a large tool result would be summarized rather than repeated. This avoids silently ballooning the context window with image bytes on every follow-up message. A future step will add TUI rendering for these files (e.g. displaying the generated image inline); today this slice covers the domain, persistence, and safety mechanics only. +> [!WARNING] +> **Artifact cleanup is currently manual.** Deleting a session (`/sessions` delete, or removing its row from the session database) does **not** delete its generated artifacts — the artifact files and the session record are independent today, and no automatic garbage collection exists yet. Likewise, copying or moving `` (or just the session database file) does **not** bring artifacts with it unless you also copy `session_artifacts/` alongside it. If you point `--session-db` at a path outside the data directory (or use `--data-dir` to relocate storage), remember that generated artifacts always live under `/session_artifacts/`, independent of wherever the session database file itself ends up — the two are not automatically kept together. To reclaim space or fully remove a session's generated images, delete its subdirectory under `session_artifacts/` yourself once you no longer need them (see the Manual Validation walkthrough below for exact commands). + +Generated media is **not** automatically resent to the model on later turns: only the surrounding text is replayed in the outgoing history, the same way a large tool result would be summarized rather than repeated. This avoids silently ballooning the context window with image bytes on every follow-up message. In its place, every stripped item gets a short, stable placeholder (its position/count, sanitized display name, and MIME type — e.g. `[Generated media omitted from history 1/1: cat.png (image/png)]`) appended after any surrounding text; a media-only reply keeps its placeholder as the entire turn rather than becoming empty, so the conversation stays valid and coherent for providers that reject or misparse an empty assistant turn. The provider-supplied display name and MIME type are both sanitized and length-bounded (control characters, path separators, angle brackets, and traversal-like sequences are neutralized) before either is ever stored or shown, since both are untrusted input from the model; an empty, whitespace-only, or invalid MIME type always falls back to `application/octet-stream`, wherever it is shown or persisted. An empty or whitespace-only display name is handled per surface instead of with one shared value: on successful materialization the *persisted* [chat.Document] name deterministically becomes `generated-` (e.g. `generated-1.png`) rather than an empty string, while the placeholder text and the materialization-failure warning — which never store anything — both fall back to the literal `generated media` rather than ever omitting the name. A future step will add TUI rendering for these artifacts (e.g. displaying the generated image inline); today this slice covers the domain, persistence, and safety mechanics only. + +#### Manual Validation + +This walkthrough exercises the actual Step 4 mechanics end to end using a real Gemini image-output model, then falls back to a focused Go test for anything a live run can't easily assert on (e.g. the branch/fork artifact-resolution contract). + +Commands are split into two groups so it's unambiguous which ones were actually exercised without a live model call: **credential-free** commands (environment/directory setup, filesystem and SQLite inspection, cleanup) versus the **two commands that need a real `GOOGLE_API_KEY`** (the two `docker-agent run` invocations). Everything credential-free in this section has been executed as part of this change; the credential-dependent commands have not, and their expected output is described but not claimed as observed. + +```bash +#!/usr/bin/env bash +# Credential-free: environment and isolated storage setup. +set -euo pipefail + +# 1. Use an isolated data directory and session database so this walkthrough +# never touches your real sessions or artifacts. +export DATA_DIR="$(mktemp -d)" +export SESSION_DB="$DATA_DIR/session.db" + +# 2. A fixed, valid session ID (matches the artifact package's session-ID +# validation: starts with an alphanumeric, then letters/digits/._-). +# Using a fixed ID rather than an auto-generated UUID makes every +# following inspection command copy/pasteable verbatim. +export SESSION_ID=demo-image-4 + +# 3. Show the three variables that every following command and the final +# cleanup step depend on, so nothing below is acting on an unexpected +# path. +echo "DATA_DIR=$DATA_DIR" +echo "SESSION_DB=$SESSION_DB" +echo "SESSION_ID=$SESSION_ID" +``` + +```bash +# Requires a real Gemini credential — see docs/providers/google/index.md. +# NOT executed as part of this change; no live Gemini credential was +# available. Everything below this point through the debug-log check +# describes expected behavior, not an observed result. +set -euo pipefail +export GOOGLE_API_KEY="..." + +# 4. Run turn 1: ask for an image, non-interactively, with debug logging +# enabled so step 8 can grep it for no-resend evidence. +export DEBUG_LOG="$DATA_DIR/turn1.debug.log" +./bin/docker-agent run --exec --data-dir "$DATA_DIR" --session-db "$SESSION_DB" \ + --session "$SESSION_ID" --debug --log-file "$DEBUG_LOG" \ + examples/gemini_image_output.yaml "draw a small red circle" +``` + +```bash +# Credential-free: inspecting the results of turn 1 above. These commands +# only produce meaningful output once turn 1 has actually run; the +# commands themselves are still credential-free (they never talk to a +# model), so this is exactly what you'd run against a real walkthrough's +# output. +set -euo pipefail + +# 5. Inspect the artifact directory: one file should exist, named by a +# generated UUID with an extension derived from the MIME type (e.g. .png). +find "$DATA_DIR/session_artifacts/$SESSION_ID" -type f -exec ls -la {} \; + +# 6. Inspect the persisted record: the assistant message's document part +# must carry artifact_path + artifact_owner_session_id (matching +# $SESSION_ID) and MUST NOT carry inline_data (no raw bytes in the DB). +# session_items can hold multiple message rows (at least one user turn +# and one assistant turn) — piping every row's message_json through a +# single `python3 -m json.tool` call fails once there is more than one +# row, since concatenated JSON documents aren't valid JSON on their +# own. Read and pretty-print one row at a time instead; a row with no +# match (the user turn) legitimately produces no grep output. +sqlite3 "$SESSION_DB" \ + "SELECT message_json FROM session_items WHERE session_id = '$SESSION_ID' AND item_type = 'message';" \ + | while IFS= read -r row; do + printf '%s\n' "$row" | python3 -m json.tool | grep -E "artifact_path|artifact_owner_session_id|inline_data" || true + done +``` + +```bash +# Requires the same live Gemini credential as turn 1. NOT executed as +# part of this change. +set -euo pipefail +export GOOGLE_API_KEY="..." + +# 7. Run turn 2, continuing the same session — this is the no-resend check. +export DEBUG_LOG_TURN2="$DATA_DIR/turn2.debug.log" +./bin/docker-agent run --exec --data-dir "$DATA_DIR" --session-db "$SESSION_DB" \ + --session "$SESSION_ID" --debug --log-file "$DEBUG_LOG_TURN2" \ + examples/gemini_image_output.yaml "thanks, what colour was it?" +``` + +```bash +# Credential-free: inspecting the results of turn 2 above. +set -euo pipefail + +# 8. Confirm the model still answered correctly from memory ("red") without +# the image ever being resent. The reply itself is one signal; the +# debug log is the stronger, wire-level one: strip_generated_media logs +# every artifact it removes from outgoing history (see +# pkg/runtime/strip_generated_media.go) — turn 2's debug log must show +# it stripping turn 1's artifact right before the model call, which is +# the actual no-resend evidence, not just an inference from the reply. +grep "strip_generated_media: stripped generated artifact from outgoing history" "$DEBUG_LOG_TURN2" + +# 9. Before cleanup, show exactly what is about to be deleted and confirm +# it is owned by this walkthrough's session: the artifact file's own +# path, and the owner/ref recorded in the session DB (same query as +# step 6, narrowed to the ownership fields only). +find "$DATA_DIR/session_artifacts/$SESSION_ID" -type f +sqlite3 "$SESSION_DB" \ + "SELECT message_json FROM session_items WHERE session_id = '$SESSION_ID' AND item_type = 'message';" \ + | while IFS= read -r row; do + printf '%s\n' "$row" | python3 -m json.tool | grep -E "artifact_path|artifact_owner_session_id" || true + done + +# 10. Cleanup: artifacts are NOT removed automatically (see warning above), +# so remove the isolated directory this walkthrough created. +rm -rf "$DATA_DIR" +``` + +The following properties are impractical to assert from a live CLI run (they require inspecting in-memory state or constructing an out-of-band scenario like a stale artifact reference) and are instead covered by focused Go tests. These are credential-free and were run as part of this change: + +```bash +# No-resend: the generated artifact is absent from the next turn's outgoing +# provider history, while the assistant's text from that turn still is; a +# media-only reply is replaced by a placeholder part (never just Content +# with an empty MultiContent) rather than an empty turn, even for a +# capability-less or unknown model — that last case is exercised through +# the actual registered transform chain (RunStream end to end), not the +# transform function in isolation, since transform ORDER is what makes it +# correct: see strip_generated_media.go's package doc for why it must run +# before strip_unsupported_modalities. +go test ./pkg/runtime/... -run 'TestRunStream_GeneratedMediaAbsentFromNextTurnHistory|TestRunStream_MediaOnlyAssistantHistoryRemainsCoherent$|TestRunStream_MediaOnlyAssistantHistoryRemainsCoherent_UnknownModel|TestStripGeneratedMediaTransform' + +# Placeholder contract: every stripped artifact gets its own placeholder +# part with sanitized display metadata, mirrored into both Content and +# MultiContent so Content-only converters (Anthropic) and +# MultiContent-authoritative converters (Gemini, OpenAI/oaistream) both +# see it. +go test ./pkg/model/provider/anthropic/... -run GeneratedMediaPlaceholder +go test ./pkg/model/provider/gemini/... -run GeneratedMediaPlaceholder +go test ./pkg/model/provider/oaistream/... -run GeneratedMediaPlaceholder + +# Ordering: strip_generated_media must run before strip_unsupported_modalities +# in the actual registered transform chain (asserted directly on +# NewLocalRuntime's registered transforms), or a capability-less/unknown +# model could strip a media-only turn to nothing before the placeholder +# logic ever sees it — the end-to-end regression for that failure mode is +# TestRunStream_MediaOnlyAssistantHistoryRemainsCoherent_UnknownModel above. +go test ./pkg/runtime/... -run TestWithMessageTransform_RejectsEmptyAndNil + +# Materialization warnings: one failure, multiple failures, and partial +# success all go through the real EventSink, and the warning payload never +# carries an absolute artifact/data-dir path or raw OS error text. +go test ./pkg/runtime/... -run TestMaterializeGeneratedMedia + +# Display-name sanitization: control characters, path separators, and +# traversal-like sequences are neutralized before a provider-supplied name +# ever reaches Document or warning metadata. +go test ./pkg/chat/... -run TestSanitizeDisplayName + +# Accounting: Content mirrored into a MultiContent text part (the +# runtime-generated assistant shape) is not double-counted by the local +# token estimator. +go test ./pkg/compaction/... -run TestEstimateMessageTokens + +# Multi-media streaming correctness: every inline blob in a single chunk is +# retained, and media sharing a chunk with a terminal finish reason +# (stop/length/refusal) is not dropped. +go test ./pkg/runtime/... -run 'TestHandleStream_MultipleMediaBlobsInOneChunk|TestHandleStream_MediaInTerminalChunkIsAccumulated' +go test ./pkg/model/provider/gemini/... -run TestStreamAdapter_GeneratedImage + +# Artifact identity across branch/fork: the artifact remains resolvable +# from a branched or forked child session because references are +# owner-qualified rather than tied to "whichever session is asking"; a +# legacy (owner-less) reference returns a stable, typed "unresolvable" +# error instead of a silent wrong lookup. +go test ./pkg/session/... -run 'TestBranchSession_GeneratedMediaRemainsResolvableFromChild|TestForkSession_GeneratedMediaRemainsResolvableFromChild|TestBranchSession_PersistedThenResolvedFromChild|TestResolveRef_LegacyGeneratedMediaIsUnresolvableNotWrong' + +# Security: artifact resolution rejects absolute paths, ".." traversal, and +# symlink escapes, confined to the owning session's directory via os.Root. +go test ./pkg/artifact/... -run 'TestRead_RejectsAbsolutePath|TestRead_RejectsDotDotTraversal|TestRead_RejectsSymlinkEscape' +``` ### Team Context Budgets and Targeted Compaction From a5107cd4aa24533532587a8c98bd90eca97ad007 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Arnaud=20He=CC=81ritier?= Date: Tue, 28 Jul 2026 00:35:48 +0200 Subject: [PATCH 2/2] docs(#3996): document verified generated-image behavior Document that image-output-capable models request TEXT+IMAGE on supported Google surfaces, with explicit output_capabilities values overriding automatic models.dev detection. Explain local rejection of unsupported tool and structured-output combinations, provider-controlled formats and text-only responses, generated-file naming and collision handling, workspace escape confirmation, and inline TUI rendering with safe fallbacks. Update the example and remove stale claims that image output is not requested or rendered, while keeping developer-only validation details out of user docs. --- docs/configuration/models/index.md | 39 ++--- docs/features/sessions/index.md | 43 ++--- docs/features/tui/index.md | 250 +++++++---------------------- docs/providers/google/index.md | 52 +++--- examples/README.md | 2 +- examples/gemini_image_output.yaml | 33 +++- 6 files changed, 154 insertions(+), 265 deletions(-) diff --git a/docs/configuration/models/index.md b/docs/configuration/models/index.md index e9be86cb4c..1a5356096b 100644 --- a/docs/configuration/models/index.md +++ b/docs/configuration/models/index.md @@ -40,8 +40,8 @@ models: pdf: boolean # Optional: whether the model accepts PDF attachments audio: boolean # Optional: whether the model accepts audio attachments video: boolean # Optional: whether the model accepts video attachments - output_capabilities: # Optional: owner-declared generative output capabilities (never inferred) - image: boolean # Optional: whether the model is declared able to generate image output + output_capabilities: # Optional: override generative output capabilities (otherwise detected from models.dev) + image: boolean # Optional: whether the model can generate image output cost: # Optional: explicit token pricing (USD per 1M tokens) input: float # Optional: price per 1M input tokens output: float # Optional: price per 1M output tokens @@ -76,7 +76,7 @@ models: | `track_usage` | boolean | ✗ | Track and report token usage for this model | | `routing` | array | ✗ | Rule-based routing to different models. See [Model Routing](../routing/index.md). | | `capabilities` | object | ✗ | Override attachment (input) capabilities for this model. See [Attachment Capability Overrides](#attachment-capability-overrides). | -| `output_capabilities` | object | ✗ | Owner-declared generative output capabilities for this model, e.g. image generation. Never inferred. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). | +| `output_capabilities` | object | ✗ | Override generative output capabilities for this model, e.g. image generation. Omitted flags are detected from models.dev; explicit values take precedence. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). | | `cost` | object | ✗ | Explicit token pricing in USD per 1M tokens, overriding the built-in catalogue. See [Custom Token Pricing](#custom-token-pricing). | | `provider_opts` | object | ✗ | Provider-specific options (see provider pages) | | `title_model` | string | ✗ | Model used for session-title generation. Can be a named model from the `models:` section or an inline `provider/model` string. When omitted, the agent's primary model generates titles. Cannot be combined with `first_available`. | @@ -151,14 +151,12 @@ See [`examples/capability-overrides.yaml`](https://github.com/docker/docker-agen [`examples/strip-unsupported-media.yaml`](https://github.com/docker/docker-agent/blob/main/examples/strip-unsupported-media.yaml) for a fixture demonstrating the stripping behaviour with and without an override. -## Output Capabilities +### Output capabilities -`output_capabilities` declares what a model can generate, as opposed to -`capabilities`, which declares what it accepts as input. There is no -automatic detection for output capabilities: no catalogue of -output-capable models exists, and matching on the model name string is -deliberately avoided as unreliable. A model's output capabilities are -therefore always unknown/off unless the owner declares them. +`output_capabilities` overrides what a model can generate, as opposed to +`capabilities`, which overrides what it accepts as input. When omitted, Docker +Agent reads output modalities from the models.dev catalogue. Use an explicit +value for custom models or to correct catalogue metadata. ```yaml models: @@ -173,18 +171,15 @@ models: | --------------------------- | ------- | -------------------------------------------------------------| | `output_capabilities.image` | boolean | Whether the model is declared able to generate image output | -Omitting `output_capabilities`, or leaving `image` unset or `false`, always -preserves existing behavior. Setting it to `true` only opts the model into -behavior that specifically keys off a declared image-output capability (for -example, a provider-specific request-shape guard); it does not by itself -change what Docker Agent sends to or renders from the model. - -One side effect of declaring `image: true`: the model is skipped as a -session-title candidate, because titles are generated by a plain text-only -completion that image-output routes can reject. Title generation uses the -first non-image-output candidate (dedicated `title_model`, then the agent's -model, then its fallbacks); when every candidate declares image output, the -automatic title is skipped and the session keeps its default title. +Omitting `output_capabilities` uses models.dev metadata when available. +Setting `image` explicitly overrides the catalogue; an explicit `false` +preserves ordinary text-only behavior even when the catalogue lists image +output. Enabling image output only opts the model into behavior that keys off +that capability (for example, a provider-specific image-output request +contract); it does not guarantee that a provider will return an image. + +Session-title requests remain text-only even for image-output-capable models, +so those models remain eligible as title candidates. > [!WARNING] > **Constraint** diff --git a/docs/features/sessions/index.md b/docs/features/sessions/index.md index 74493f0d3c..dcdc16902c 100644 --- a/docs/features/sessions/index.md +++ b/docs/features/sessions/index.md @@ -29,34 +29,21 @@ Override the location with `-s`/`--session-db`, or by overriding the data direct $ docker agent run agent.yaml --session-db ./sessions.db ``` -## Generated-Media Artifacts - -Some models (e.g. Gemini image-output models) can generate binary media — typically an image — as part of a reply. Docker Agent writes that data to a managed **artifact root** kept outside `session.db`: `/session_artifacts//` (`~/.cagent/session_artifacts//` by default). The session message itself keeps only a relative reference plus safe display metadata (MIME type, sanitized name, size) — never the raw bytes. See [Generated Media](../tui/index.md#generated-media) in the Terminal UI docs for the full placeholder, no-resend, and sanitization mechanics; this section covers only where and how the artifacts themselves are stored, referenced, and cleaned up. - -### Owner-Qualified References - -`` in the artifact path is always the **owning** session — the session that was active when the media was generated — not necessarily whichever session currently holds the message. That owning session ID is recorded on the message itself and never changes, so branching or forking a session (see [Browsing Sessions in the TUI](#browsing-sessions-in-the-tui) below) keeps every generated-media reference resolving under the *original* artifact directory instead of a nonexistent one under the new session's ID. - -References materialized before this owner-qualified scheme existed carry no owner and cannot be resolved at all: there is no session ID left to safely resolve them against, so they surface as an unresolvable attachment rather than risk silently reading the wrong (or no) file under a guessed session. - -### Independent of `session.db` - -The artifact root and the session database are two independent stores that happen to share the same default parent directory: - -- `--data-dir` moves both by default — `session.db` and `session_artifacts/` relocate together under the new data directory. -- `-s`/`--session-db` moves *only* the database. Pointing it at a path outside the data directory does **not** relocate `session_artifacts/`, which always stays under `/session_artifacts/` regardless of where the database file itself ends up. -- Copying, moving, or backing up one of the two without the other silently leaves generated-media references unresolvable on whichever side didn't move. - -### No Automatic Cleanup - -Deleting a session — from `/sessions` in the TUI, or by removing its row from `session.db` directly — does **not** delete its generated artifacts, and no automatic garbage collection exists today: the artifact files and the session record are independent stores that are never reconciled for you. - -To reclaim space or fully remove a session's generated media, delete its subdirectory under `session_artifacts/` yourself once you no longer need it. This is safe: each session's artifacts live in their own subdirectory, so removing one never touches another session's files or the session database itself. - -```bash -# Remove one session's generated-media artifacts. -$ rm -rf ~/.cagent/session_artifacts// -``` +## Generated Media Files + +Some models (e.g. Gemini image-output models) can generate an image as part +of a reply. Docker Agent saves each generated image into the session's +workspace — the directory the session was started in — as an ordinary, +untracked file, not into `session.db` or the data directory. The session +keeps only a reference to that file: reopening the session in the TUI +re-renders the image from the workspace file, so a valid image file +replaced at the same path renders its current bytes. If the file has +since been deleted, moved, replaced by a symlink, become an invalid +image, or grown too large, the transcript shows a short "unavailable" +note instead. The files themselves are yours — edit, commit, +or delete them like any other workspace file. See +[Generated Media](../tui/index.md#generated-media) in the Terminal UI docs +for naming, collision handling, and rendering details. ## Resuming a Session diff --git a/docs/features/tui/index.md b/docs/features/tui/index.md index 88a04fcf2c..24ccc868d9 100644 --- a/docs/features/tui/index.md +++ b/docs/features/tui/index.md @@ -292,194 +292,68 @@ Attached files are also recorded on the session so sub-agents spawned by task tr ### Generated Media -Some models (e.g. Gemini image-output models) can generate binary media — typically an image — as part of their reply. When that happens, docker-agent writes the generated bytes to a managed artifact directory under `/session_artifacts//` (defaults to `~/.cagent/session_artifacts//`; override the data directory with the `--data-dir` flag) and the assistant message keeps only a relative reference to that file plus its MIME type, display name, and size — never the raw bytes. `` here is always the **owning** session — the session that was active when the media was generated — recorded alongside the reference so it keeps resolving correctly even after the message is copied into a branched or forked session (see below). - -This keeps session JSON/database rows lightweight regardless of how many images a conversation accumulates. If writing a generated item to the artifact directory fails (e.g. a full or read-only disk), that one item is dropped and a notification reports it by its exact position/count in the batch plus its sanitized MIME type and display name — falling back to the same `generated media` / `application/octet-stream` defaults as the placeholder text below when the name or MIME type is empty, whitespace-only, or missing, and never simply omitting the name — never the artifact directory's absolute path or the underlying OS error, which are logged to the debug log (`--debug`) instead; any other item in the same reply that saved successfully is unaffected. - -> [!WARNING] -> **Artifact cleanup is currently manual.** Deleting a session (`/sessions` delete, or removing its row from the session database) does **not** delete its generated artifacts — the artifact files and the session record are independent today, and no automatic garbage collection exists yet. Likewise, copying or moving `` (or just the session database file) does **not** bring artifacts with it unless you also copy `session_artifacts/` alongside it. If you point `--session-db` at a path outside the data directory (or use `--data-dir` to relocate storage), remember that generated artifacts always live under `/session_artifacts/`, independent of wherever the session database file itself ends up — the two are not automatically kept together. To reclaim space or fully remove a session's generated images, delete its subdirectory under `session_artifacts/` yourself once you no longer need them (see the Manual Validation walkthrough below for exact commands). - -Generated media is **not** automatically resent to the model on later turns: only the surrounding text is replayed in the outgoing history, the same way a large tool result would be summarized rather than repeated. This avoids silently ballooning the context window with image bytes on every follow-up message. In its place, every stripped item gets a short, stable placeholder (its position/count, sanitized display name, and MIME type — e.g. `[Generated media omitted from history 1/1: cat.png (image/png)]`) appended after any surrounding text; a media-only reply keeps its placeholder as the entire turn rather than becoming empty, so the conversation stays valid and coherent for providers that reject or misparse an empty assistant turn. The provider-supplied display name and MIME type are both sanitized and length-bounded (control characters, path separators, angle brackets, and traversal-like sequences are neutralized) before either is ever stored or shown, since both are untrusted input from the model; an empty, whitespace-only, or invalid MIME type always falls back to `application/octet-stream`, wherever it is shown or persisted. An empty or whitespace-only display name is handled per surface instead of with one shared value: on successful materialization the *persisted* [chat.Document] name deterministically becomes `generated-` (e.g. `generated-1.png`) rather than an empty string, while the placeholder text and the materialization-failure warning — which never store anything — both fall back to the literal `generated media` rather than ever omitting the name. A future step will add TUI rendering for these artifacts (e.g. displaying the generated image inline); today this slice covers the domain, persistence, and safety mechanics only. - -#### Manual Validation - -This walkthrough exercises the actual Step 4 mechanics end to end using a real Gemini image-output model, then falls back to a focused Go test for anything a live run can't easily assert on (e.g. the branch/fork artifact-resolution contract). - -Commands are split into two groups so it's unambiguous which ones were actually exercised without a live model call: **credential-free** commands (environment/directory setup, filesystem and SQLite inspection, cleanup) versus the **two commands that need a real `GOOGLE_API_KEY`** (the two `docker-agent run` invocations). Everything credential-free in this section has been executed as part of this change; the credential-dependent commands have not, and their expected output is described but not claimed as observed. - -```bash -#!/usr/bin/env bash -# Credential-free: environment and isolated storage setup. -set -euo pipefail - -# 1. Use an isolated data directory and session database so this walkthrough -# never touches your real sessions or artifacts. -export DATA_DIR="$(mktemp -d)" -export SESSION_DB="$DATA_DIR/session.db" - -# 2. A fixed, valid session ID (matches the artifact package's session-ID -# validation: starts with an alphanumeric, then letters/digits/._-). -# Using a fixed ID rather than an auto-generated UUID makes every -# following inspection command copy/pasteable verbatim. -export SESSION_ID=demo-image-4 - -# 3. Show the three variables that every following command and the final -# cleanup step depend on, so nothing below is acting on an unexpected -# path. -echo "DATA_DIR=$DATA_DIR" -echo "SESSION_DB=$SESSION_DB" -echo "SESSION_ID=$SESSION_ID" -``` - -```bash -# Requires a real Gemini credential — see docs/providers/google/index.md. -# NOT executed as part of this change; no live Gemini credential was -# available. Everything below this point through the debug-log check -# describes expected behavior, not an observed result. -set -euo pipefail -export GOOGLE_API_KEY="..." - -# 4. Run turn 1: ask for an image, non-interactively, with debug logging -# enabled so step 8 can grep it for no-resend evidence. -export DEBUG_LOG="$DATA_DIR/turn1.debug.log" -./bin/docker-agent run --exec --data-dir "$DATA_DIR" --session-db "$SESSION_DB" \ - --session "$SESSION_ID" --debug --log-file "$DEBUG_LOG" \ - examples/gemini_image_output.yaml "draw a small red circle" -``` - -```bash -# Credential-free: inspecting the results of turn 1 above. These commands -# only produce meaningful output once turn 1 has actually run; the -# commands themselves are still credential-free (they never talk to a -# model), so this is exactly what you'd run against a real walkthrough's -# output. -set -euo pipefail - -# 5. Inspect the artifact directory: one file should exist, named by a -# generated UUID with an extension derived from the MIME type (e.g. .png). -find "$DATA_DIR/session_artifacts/$SESSION_ID" -type f -exec ls -la {} \; - -# 6. Inspect the persisted record: the assistant message's document part -# must carry artifact_path + artifact_owner_session_id (matching -# $SESSION_ID) and MUST NOT carry inline_data (no raw bytes in the DB). -# session_items can hold multiple message rows (at least one user turn -# and one assistant turn) — piping every row's message_json through a -# single `python3 -m json.tool` call fails once there is more than one -# row, since concatenated JSON documents aren't valid JSON on their -# own. Read and pretty-print one row at a time instead; a row with no -# match (the user turn) legitimately produces no grep output. -sqlite3 "$SESSION_DB" \ - "SELECT message_json FROM session_items WHERE session_id = '$SESSION_ID' AND item_type = 'message';" \ - | while IFS= read -r row; do - printf '%s\n' "$row" | python3 -m json.tool | grep -E "artifact_path|artifact_owner_session_id|inline_data" || true - done -``` - -```bash -# Requires the same live Gemini credential as turn 1. NOT executed as -# part of this change. -set -euo pipefail -export GOOGLE_API_KEY="..." - -# 7. Run turn 2, continuing the same session — this is the no-resend check. -export DEBUG_LOG_TURN2="$DATA_DIR/turn2.debug.log" -./bin/docker-agent run --exec --data-dir "$DATA_DIR" --session-db "$SESSION_DB" \ - --session "$SESSION_ID" --debug --log-file "$DEBUG_LOG_TURN2" \ - examples/gemini_image_output.yaml "thanks, what colour was it?" -``` - -```bash -# Credential-free: inspecting the results of turn 2 above. -set -euo pipefail - -# 8. Confirm the model still answered correctly from memory ("red") without -# the image ever being resent. The reply itself is one signal; the -# debug log is the stronger, wire-level one: strip_generated_media logs -# every artifact it removes from outgoing history (see -# pkg/runtime/strip_generated_media.go) — turn 2's debug log must show -# it stripping turn 1's artifact right before the model call, which is -# the actual no-resend evidence, not just an inference from the reply. -grep "strip_generated_media: stripped generated artifact from outgoing history" "$DEBUG_LOG_TURN2" - -# 9. Before cleanup, show exactly what is about to be deleted and confirm -# it is owned by this walkthrough's session: the artifact file's own -# path, and the owner/ref recorded in the session DB (same query as -# step 6, narrowed to the ownership fields only). -find "$DATA_DIR/session_artifacts/$SESSION_ID" -type f -sqlite3 "$SESSION_DB" \ - "SELECT message_json FROM session_items WHERE session_id = '$SESSION_ID' AND item_type = 'message';" \ - | while IFS= read -r row; do - printf '%s\n' "$row" | python3 -m json.tool | grep -E "artifact_path|artifact_owner_session_id" || true - done - -# 10. Cleanup: artifacts are NOT removed automatically (see warning above), -# so remove the isolated directory this walkthrough created. -rm -rf "$DATA_DIR" -``` - -The following properties are impractical to assert from a live CLI run (they require inspecting in-memory state or constructing an out-of-band scenario like a stale artifact reference) and are instead covered by focused Go tests. These are credential-free and were run as part of this change: - -```bash -# No-resend: the generated artifact is absent from the next turn's outgoing -# provider history, while the assistant's text from that turn still is; a -# media-only reply is replaced by a placeholder part (never just Content -# with an empty MultiContent) rather than an empty turn, even for a -# capability-less or unknown model — that last case is exercised through -# the actual registered transform chain (RunStream end to end), not the -# transform function in isolation, since transform ORDER is what makes it -# correct: see strip_generated_media.go's package doc for why it must run -# before strip_unsupported_modalities. -go test ./pkg/runtime/... -run 'TestRunStream_GeneratedMediaAbsentFromNextTurnHistory|TestRunStream_MediaOnlyAssistantHistoryRemainsCoherent$|TestRunStream_MediaOnlyAssistantHistoryRemainsCoherent_UnknownModel|TestStripGeneratedMediaTransform' - -# Placeholder contract: every stripped artifact gets its own placeholder -# part with sanitized display metadata, mirrored into both Content and -# MultiContent so Content-only converters (Anthropic) and -# MultiContent-authoritative converters (Gemini, OpenAI/oaistream) both -# see it. -go test ./pkg/model/provider/anthropic/... -run GeneratedMediaPlaceholder -go test ./pkg/model/provider/gemini/... -run GeneratedMediaPlaceholder -go test ./pkg/model/provider/oaistream/... -run GeneratedMediaPlaceholder - -# Ordering: strip_generated_media must run before strip_unsupported_modalities -# in the actual registered transform chain (asserted directly on -# NewLocalRuntime's registered transforms), or a capability-less/unknown -# model could strip a media-only turn to nothing before the placeholder -# logic ever sees it — the end-to-end regression for that failure mode is -# TestRunStream_MediaOnlyAssistantHistoryRemainsCoherent_UnknownModel above. -go test ./pkg/runtime/... -run TestWithMessageTransform_RejectsEmptyAndNil - -# Materialization warnings: one failure, multiple failures, and partial -# success all go through the real EventSink, and the warning payload never -# carries an absolute artifact/data-dir path or raw OS error text. -go test ./pkg/runtime/... -run TestMaterializeGeneratedMedia - -# Display-name sanitization: control characters, path separators, and -# traversal-like sequences are neutralized before a provider-supplied name -# ever reaches Document or warning metadata. -go test ./pkg/chat/... -run TestSanitizeDisplayName - -# Accounting: Content mirrored into a MultiContent text part (the -# runtime-generated assistant shape) is not double-counted by the local -# token estimator. -go test ./pkg/compaction/... -run TestEstimateMessageTokens - -# Multi-media streaming correctness: every inline blob in a single chunk is -# retained, and media sharing a chunk with a terminal finish reason -# (stop/length/refusal) is not dropped. -go test ./pkg/runtime/... -run 'TestHandleStream_MultipleMediaBlobsInOneChunk|TestHandleStream_MediaInTerminalChunkIsAccumulated' -go test ./pkg/model/provider/gemini/... -run TestStreamAdapter_GeneratedImage - -# Artifact identity across branch/fork: the artifact remains resolvable -# from a branched or forked child session because references are -# owner-qualified rather than tied to "whichever session is asking"; a -# legacy (owner-less) reference returns a stable, typed "unresolvable" -# error instead of a silent wrong lookup. -go test ./pkg/session/... -run 'TestBranchSession_GeneratedMediaRemainsResolvableFromChild|TestForkSession_GeneratedMediaRemainsResolvableFromChild|TestBranchSession_PersistedThenResolvedFromChild|TestResolveRef_LegacyGeneratedMediaIsUnresolvableNotWrong' - -# Security: artifact resolution rejects absolute paths, ".." traversal, and -# symlink escapes, confined to the owning session's directory via os.Root. -go test ./pkg/artifact/... -run 'TestRead_RejectsAbsolutePath|TestRead_RejectsDotDotTraversal|TestRead_RejectsSymlinkEscape' -``` +Some models (e.g. Gemini image-output models like `gemini-2.5-flash-image`) +are designed to generate an image directly as part of their reply, not just +describe one. When models.dev reports that a model can generate images, or +[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities) +explicitly enables it, Docker Agent asks it for text *and* image output on the +models gateway, direct Gemini API, and Vertex AI. An explicit `false` disables +this behavior. See +[Google Gemini: Generated Images](../../providers/google/index.md#generated-images) +for exact configuration and limitations — requests with custom or built-in +tools, or structured output, are rejected locally before any request is sent +on every supported Google surface. Gateway probing confirms these combinations +are rejected; direct Gemini API and Vertex AI must pass live acceptance checks +before that conservative restriction is relaxed. + +**Where images land.** Each generated image is saved into the session's +workspace — the directory the session was started in — as an ordinary +file, the same way generated code or text lands there. Generated files are +regular untracked files: visible to every tool, and yours to edit, commit, +move, or delete. On a remote runtime the files are written into that +runtime's workspace, so there is no local file for the TUI to display. + +**Naming.** A filename named explicitly in your prompt ("Generate an image +of a red panda as `assets/red-panda.jpg`", "… and save it as `logo.png`") +is honored, including subdirectories, which are created inside the +workspace as needed. Without one, the model chooses a short, meaningful +name, with a generic `generated-1`, `generated-2`, … name as the last +resort. Two rules always apply: + +- **The extension matches the data.** The image format is decided by the + provider (typically PNG) — asking for `sunshine.gif` or `diagram.svg` + does not transcode anything. If the model returns PNG data, the file is + saved as `sunshine.png` and a notice tells you so. +- **Existing files are never overwritten.** A name collision gets a dash + suffix instead: a second `red-panda.jpg` is saved as `red-panda-1.jpg`. + +**Paths outside the workspace.** A prompt-directed target that is absolute, +`~`-rooted, or climbs above the workspace with `..` is never written +silently: you are asked to confirm the exact resolved path first, and the +safe "keep it in the workspace" choice is the default. If you decline — or +nothing can answer, as in a non-interactive run — the image is saved into +the workspace root under its plain filename instead, with a warning. The +generated bytes are never discarded. + +**Rendering.** The image appears inline in the same assistant turn, using +the same Kitty-graphics support and `render_images` setting as +[Markdown Images](#markdown-images). On a terminal without graphics support +the TUI prints the saved workspace path instead. Reopening a session +re-renders its generated images from the workspace files, so a file +replaced by another valid image at the same path shows its current +bytes. If a file has since been deleted, moved, replaced by a symlink, +become an invalid image, or grown too large, the transcript shows a +short "unavailable" note rather than wrong content. + +If a save fails (unwritable directory, full disk, …), only that image is +dropped, with a concise warning — the reply text and any sibling images in +the same turn are kept. Note also that an image-capable model can answer +with text only and generate no image at all; that is provider behavior, so +reword or repeat the prompt. + +Inline image rendering in the TUI also covers a tool/MCP result that +returns an image, or a Markdown image reference to a file a tool actually +writes to disk — see [Markdown Images](#markdown-images) above. ### Team Context Budgets and Targeted Compaction diff --git a/docs/providers/google/index.md b/docs/providers/google/index.md index e03dbe3ef8..c1fa7c69da 100644 --- a/docs/providers/google/index.md +++ b/docs/providers/google/index.md @@ -64,29 +64,43 @@ models: Some Gemini models (e.g. `gemini-2.5-flash-image`) are designed to generate an image directly as part of their reply, not just describe one. Docker -Agent's Gemini request path doesn't yet ask for that image output — that -support is still being completed — so today a request like this gets a -text-only reply. See -[Generated Media](../../features/tui/index.md#generated-media) for the -current, verified state. +Agent requests that image output on supported Google surfaces — the models +gateway, direct Gemini API, and Vertex AI — when models.dev reports that the +model can generate images or `output_capabilities.image` explicitly enables +it. An explicit `false` overrides the catalogue and keeps the model text-only. +Each ordinary chat request then asks for text *and* image output, and every +image the model returns is saved into the session's workspace and rendered +inline in the TUI — see [Generated Media](../../features/tui/index.md#generated-media) +for file naming, collision handling, and rendering details. ```yaml -agents: - root: - model: google/gemini-2.5-flash-image +models: + gemini-image: + provider: google + model: gemini-2.5-flash-image + output_capabilities: + image: true ``` -When the model is accessed through a Docker AI Gateway and explicitly -declared image-output-capable with -[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities), -Docker Agent has verified that request combined with custom function tools, -a built-in tool (e.g. `google_search`), or structured output gets rejected -by the gateway with an opaque, empty-body HTTP 400. To avoid that, Docker -Agent rejects such a combination itself, before any request is sent, with a -clear error naming which feature is incompatible. Plain text requests to -that model (no tools, no structured output) are unaffected, as is every -other route: direct Gemini API/Vertex AI calls, and gateway calls to a model -without the declaration. +When `output_capabilities.image` is omitted, Docker Agent uses models.dev +output modalities. Set it explicitly for custom models or to override +incorrect catalogue data; capability is never guessed from the model name. +Title generation and compaction remain text-only. + +Image-output requests with custom or built-in tools, or structured output, +are rejected locally before any request is sent on every supported Google +surface. Gateway probing confirms these combinations are rejected; direct +Gemini API and Vertex AI must pass live acceptance checks before that +conservative restriction is relaxed. + +A few provider-side behaviors to know: + +- **The provider decides the image format** (typically PNG). Asking for a + `.gif` or `.svg` filename does not transcode anything — the saved file's + extension is corrected to match the data actually returned. +- **An image is not guaranteed.** Even a correctly configured image model + can answer with text only and generate no image; reword or repeat the + prompt. ## Thinking Budget diff --git a/examples/README.md b/examples/README.md index 759c7f603d..d7f2255f87 100644 --- a/examples/README.md +++ b/examples/README.md @@ -216,7 +216,7 @@ remote MCP endpoints. | [`rule_based_routing.yaml`](rule_based_routing.yaml) | Cheap router model dispatches the user message to fast or capable models. | | [`structured-output.yaml`](structured-output.yaml) | Forces the model to return JSON matching a schema. | | [`google_search_grounding.yaml`](google_search_grounding.yaml) | Enables Google Search grounding on Gemini models. | -| [`gemini_image_output.yaml`](gemini_image_output.yaml) | Gemini image-output model (generated images are saved into the workspace, not inlined as base64). | +| [`gemini_image_output.yaml`](gemini_image_output.yaml) | Gemini image-output model: generated images are saved into the workspace and rendered inline in the TUI. | | [`sampling-opts.yaml`](sampling-opts.yaml) | Provider-specific sampling parameters (`top_k`, `repetition_penalty`, …). | | [`thinking_budget.yaml`](thinking_budget.yaml) | Reasoning/thinking budgets across OpenAI, Anthropic and Google. | | [`task_budget.yaml`](task_budget.yaml) | Anthropic `task_budget`: cap total tokens spent across a multi-step agentic task. | diff --git a/examples/gemini_image_output.yaml b/examples/gemini_image_output.yaml index e8490a4f62..623c8b6c6e 100644 --- a/examples/gemini_image_output.yaml +++ b/examples/gemini_image_output.yaml @@ -1,19 +1,38 @@ # Gemini image-output model: the model can generate an image directly as # part of its reply, instead of only describing one. # -# The `gemini-image` model below declares `output_capabilities.image: true` — -# an explicit, owner-provided statement that this model can generate image -# output. It is never inferred from the model name or any catalogue; omit -# it, or leave it false, and behavior is unchanged. +# The `gemini-image` model below explicitly overrides +# `output_capabilities.image: true`. When this flag is omitted, Docker Agent +# uses models.dev output modalities; explicit true or false takes precedence. +# Capability is never guessed from the model name. # -# Native generated images aren't currently presented inline in the -# terminal UI, so the model's text reply is what you'll see today. +# On the models gateway, direct Gemini API, and Vertex AI, Docker Agent +# requests text AND image output for this image-capable model. Each +# generated image is saved into the session's workspace as an ordinary file +# and rendered inline in the TUI (terminals without graphics support print +# the saved path instead). +# +# Notes: +# - Requests with custom or built-in tools, or structured output, are +# rejected locally before any request is sent on every supported Google +# surface. Gateway probing confirms these combinations are rejected; +# direct Gemini API and Vertex AI must pass live acceptance checks before +# that conservative restriction is relaxed. +# - Naming: an explicit filename in the prompt is honored (subdirectories +# are created inside the workspace); otherwise the model picks a +# meaningful name. Extensions are corrected to match the returned data +# (the provider decides the format, typically PNG), and existing files +# are never overwritten — collisions get a dash suffix (logo-1.png). +# - The model may reply with text only and no image — that's provider +# behavior; reword or repeat the prompt. # # Try it out: # docker agent run examples/gemini_image_output.yaml \ # "Generate an image of a red panda working at a terminal" # docker agent run examples/gemini_image_output.yaml \ -# "Generate an image of a lighthouse at sunset, and describe the color palette you used" +# "Generate an image of a lighthouse at sunset as assets/lighthouse.png" +# docker agent run examples/gemini_image_output.yaml \ +# "Generate an image of a whale and save it as whale.jpg" models: gemini-image: provider: google