diff --git a/README.md b/README.md index 2405f8b9..d08d54ba 100644 --- a/README.md +++ b/README.md @@ -62,6 +62,7 @@ - **视频生成**: 支持 Veo 视频生成和图片转视频，目录中没有 Veo 时视频长任务由 Gemini Omni 生成；Gemini Omni 通过四套生成接口接收文本、图片与视频输入，输出文本与 MP4 视频 - **Deep Research**: Pro 与 Ultra 账户可通过四套生成接口与 Interactions 的 `agent` 字段调用 Deep Research，输出研究过程、带引用的报告与图表 - **YouTube 输入**: YouTube 视频 URL 作为外部视频附件读取；开启 `YOUTUBE_LINKS` 后，消息文本中的 YouTube 链接同样作为视频读取 +- **视频理解设置**: Gemini 与 Interactions 支持逐视频的起止时间、0–24 FPS 范围内的正数采样率和静态/Agentic 处理；Agentic 返回服务端处理步骤并支持历史续接 - **智能模型切换**: 从 AI Studio 实时发现模型并按 `model` 字段路由 - **Google 工具**: 支持 Search、Image Search、URL Context、Code Execution 和 Maps - **Files 与 Transcribe**: 支持文件上传、查询、内容读取、删除和音频转录 diff --git a/README_en.md b/README_en.md index 88a519d3..2206ed96 100644 --- a/README_en.md +++ b/README_en.md @@ -62,6 +62,7 @@ - **Video Generation**: Veo video generation and image-to-video, with video tasks generated by Gemini Omni when the catalog has no Veo model; Gemini Omni accepts text, image, and video input and returns text and MP4 video through the four generation APIs - **Deep Research**: Pro and Ultra accounts can run Deep Research through the four generation APIs and the Interactions `agent` field, returning the research process, a cited report, and charts - **YouTube Input**: YouTube video URLs are read as external video attachments; with `YOUTUBE_LINKS` enabled, YouTube links in message text are read as video too +- **Video Understanding Controls**: Gemini and Interactions support per-video start/end offsets, sampling rates greater than 0 and up to 24 FPS, and static/agentic processing; agentic results preserve server-side processing steps and conversation history - **Smart Model Switching**: Discover models from AI Studio and route through the `model` field - **Google Tools**: Search, Image Search, URL Context, Code Execution, and Maps - **Files and Transcribe**: File upload, metadata, content, deletion, and audio transcription diff --git a/docs/protocol.md b/docs/protocol.md index dd4f3b8d..54efdeb7 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -308,6 +308,7 @@ field 1 是 `request_key`，field 2 是 `botguard_response`。正确 proof、损 | 52 | thinking level | 53 | music route | | 54 | image search | 58 | Google Maps | | 59 | private Interaction route | 85 | TTS 分段说话人 | +| 69 | Agentic 视频理解 | | | | 46 | Live 实时翻译 | 48 | Build 写代码助手可选 | 未知能力码按原值保留为 `capability_code_` 或 `secondary_capability_code_`。 @@ -326,6 +327,7 @@ field 1 是 `request_key`，field 2 是 `botguard_response`。正确 proof、损 | 52 | `thinking_level` | 53 | `music_route` | | 54 | `image_search` | 58 | `google_maps` | | 59 | `interaction_route` | 74 | `transcription_word_timestamps` | +| 69 | `agentic_video` | | | | 76 | `transcription_language_codes` | 77 | `transcription_output` | | 80 | `transcription_speaker_labels` | 81 | `transcription_custom_vocabulary` | | 84 | `transcription_smart` | 85 | `speech_metadata` | @@ -429,8 +431,12 @@ Content 形状： | 11 | 12 | function result `[name, Struct, callId?]` | | 12 | 13 | thought boolean | | 14 | 15 | thought signature | +| 15 | 16 | 视频静态处理元数据 `[startOffset?, endOffset?, fps?]`；offset 为 `[seconds,nanos?]` | | 17 | 18 | 上传附件 `[mime, 上传路径, 文件名, 字节数]`，字节数为十进制字符串；用于 Build 写代码助手，见 [Build 通道](build.md) 第 8 节 | | 22 | 23 | transcription metadata `[text, speaker?, timestamp spans?]` | +| 25 | 26 | 服务端媒体处理调用 `[13, name?, Struct?, callId?]` | +| 26 | 27 | 服务端媒体处理结果 `[13, name?, Struct?, callId?]` | +| 39 | 40 | 视频处理模式：`1` STATIC、`2` AGENTIC | system instruction： @@ -1969,6 +1975,7 @@ delta 联合类型为 `text_delta{text}`、`thinking_delta{thinking}`、`signatu | --- | --- | | `input` | 字符串、单个内容块、内容块数组或步骤数组；内容类型为 `text`、`image`、`audio`、`video`、`document` | | 媒体内容 | `data`（Base64）或 `uri`，可选 `mime_type` | +| 视频处理 | `processing:"static"`、`"agentic"` 或 `{type:"static",start_offset?,end_offset?,fps?}`；offset 为带 `s` 的 Duration 字符串，例如 `"2.5s"` | | 输入步骤 | `user_input`、`model_output`、`thought`、`function_call`、`function_result`、`code_execution_call`、`code_execution_result` 与工具 call/result 步骤 | | `system_instruction` | 当前请求的系统指令 | | `generation_config` | `temperature`、`top_p`、`top_k`、`max_output_tokens`、`seed`、`stop_sequences`、`thinking_level`、`thinking_summaries`、`speech_config`、`tool_choice`、`image_config:{aspect_ratio?,image_size?}` | @@ -1983,7 +1990,8 @@ delta 联合类型为 `text_delta{text}`、`thinking_delta{thinking}`、`signatu | 续接 | `previous_interaction_id`、`store` | - 媒体内容的 `data` 与 `uri` 二选一，复用 Gemini 文件与内联媒体解析：省略 `mime_type` 时内联数据按内容识别类型，无法识别时返回 `400 INVALID_ARGUMENT`；`uri` 为外部 http(s) URL 时由服务下载为内联媒体，规则与 GenerateContent 的 `fileData` 相同；本服务 Gemini 文件接口返回的 `uri` 作为文件引用 -- 函数结果通过 `call_id` 匹配历史调用，没有对应调用时使用自带的 `name`；`is_error:true` 的结果以 `{"error":}` 发送上游；`code_execution_call` 与 `code_execution_result` 以代码块文本并入模型轮次；`google_search`、`url_context`、`google_maps`、`file_search`、`mcp_server_tool`、`retrieval`、`processing` 的 call 与 result 步骤不发送上游 +- 函数结果通过 `call_id` 匹配历史调用，没有对应调用时使用自带的 `name`；`is_error:true` 的结果以 `{"error":}` 发送上游；`code_execution_call` 与 `code_execution_result` 以代码块文本并入模型轮次；`google_search`、`url_context`、`google_maps`、`file_search`、`mcp_server_tool`、`retrieval` 的 call 与 result 步骤不发送上游 +- `processing_call` 和 `processing_result` 保留服务端媒体处理调用的 `id`、对应的 `call_id` 与各自的 `signature`，作为模型历史续接；流式响应按 `step.start`、`step.delta`、`step.stop` 发布，签名位于 delta - 文本 `response_format` 带 `schema` 时请求结构化输出，`mime_type` 默认 `application/json`；不带 `type` 或 `type` 为 JSON Schema 类型的对象直接作为结构化输出 Schema；图片指定 `mime_type:"image/jpeg"` 时其他格式的图片输出转为 JPEG - `response_format` 中的图片比例、尺寸与文本 `mime_type` 优先于 `generation_config.image_config` 与 `response_mime_type`；`response_modalities` 中的其他取值返回 `400 INVALID_ARGUMENT` - `speech_config` 的 `language` 不发送上游，多说话人 `mode` 为 `verbatim` 或 `conversational`；`speech_metadata` 保留说话人与风格 @@ -2032,11 +2040,27 @@ Content 字段为 `role` 与 `parts`。Part oneof： | file data | `fileData:{mimeType?,fileUri,displayName}` | | function call | `functionCall:{id,name,args}` | | function response | `functionResponse:{id,name,response}` | +| server media processing call | `toolCall:{id?,toolType:"MEDIA_PROCESSING",toolName?,args?}` | +| server media processing response | `toolResponse:{id?,toolType:"MEDIA_PROCESSING",response?}` | | executable code | `executableCode:{language,code}` | | code result | `codeExecutionResult:{outcome,output,error}` | 不含数据字段且没有 `thoughtSignature` 的 Part 不发送；同一 Part 含多个数据字段时返回 `400 INVALID_ARGUMENT`。`fileData.fileUri` 为 YouTube 链接时作为外部视频，为本服务 `/upload/v1beta/files` 返回的文件 URI 或 `/v1/files` 文件 ID 时作为文件引用，为其他 http(s) URL 时由服务下载为内联媒体；其他 Gemini Files API URI 不下载。外部 URL 在同一请求内只下载一次，上限 512 MiB，MIME 依次取响应 `Content-Type`、`mimeType` 与内容识别；主机或任一重定向目标解析到本机、私网、链路本地、组播或未指定地址时不连接，按下载失败处理；下载失败返回带 URL 与原因的 `400 INVALID_ARGUMENT`。`:countTokens` 同样先下载外部 URL 再计数。 +视频 Part 可设置 `mediaProcessing` 为 `STATIC`、`AGENTIC` 或 `MEDIA_PROCESSING_UNSPECIFIED`。省略时沿用模型默认处理方式。`AGENTIC` 使用模型目录的 `agentic_video` 能力，并返回 `toolType:"MEDIA_PROCESSING"` 的服务端 `toolCall` 与 `toolResponse`；调用标识与 `thoughtSignature` 随完整模型 Part 一起用于下一轮历史。 + +静态视频的 `videoMetadata` 可设置 `startOffset`、`endOffset` 与 `fps`，与该 Part 的视频附件一起参与生成及 token 计数。offset 为非负秒数 Duration 字符串，支持最多九位小数，例如 `"2.5s"`；显式 `"0s"` 保留起点，终点须晚于起点。FPS 范围为 `(0,24]`，静态参数与 `AGENTIC` 互斥。例如： + +```json +{ + "fileData": {"mimeType":"video/*","fileUri":"https://www.youtube.com/watch?v="}, + "mediaProcessing": "STATIC", + "videoMetadata": {"startOffset":"2s","endOffset":"4s","fps":0.5} +} +``` + +这些参数与媒体处理历史适用于上游使用 GenerateContent 的模型；使用原生 CreateInteractionStream 或 CreateInteraction 的模型收到这些参数时返回参数错误。 + `generationConfig` 全字段： | 类别 | 字段 | diff --git a/internal/aistudio/build.go b/internal/aistudio/build.go index 9d0bb7d7..13327b68 100644 --- a/internal/aistudio/build.go +++ b/internal/aistudio/build.go @@ -240,6 +240,10 @@ func encodeBuildPart(part Part) (map[string]any, error) { result["output"] = output } wire["codeExecutionResult"] = result + case part.ProcessingCall != nil: + wire["toolCall"] = buildMediaProcessingStep(part.ProcessingCall, "args") + case part.ProcessingResult != nil: + wire["toolResponse"] = buildMediaProcessingStep(part.ProcessingResult, "response") case signature != "": wire["text"] = "" default: @@ -248,9 +252,37 @@ func encodeBuildPart(part Part) (map[string]any, error) { if signature != "" { wire["thoughtSignature"] = signature } + if video := part.VideoMetadata; video != nil { + metadata := map[string]any{} + if video.StartOffset != "" { + metadata["startOffset"] = video.StartOffset + } + if video.EndOffset != "" { + metadata["endOffset"] = video.EndOffset + } + if video.FPS != nil { + metadata["fps"] = *video.FPS + } + wire["videoMetadata"] = metadata + } + if part.MediaProcessing != "" { + wire["mediaProcessing"] = part.MediaProcessing + } return wire, nil } +// buildMediaProcessingStep 编码视频理解处理的 toolCall 或 toolResponse，payload 为 args 或 response +func buildMediaProcessingStep(step *MediaProcessingStep, payload string) map[string]any { + value := map[string]any{"toolType": "MEDIA_PROCESSING", "id": step.ID} + if step.Name != "" { + value["toolName"] = step.Name + } + if len(step.Data) != 0 { + value[payload] = step.Data + } + return value +} + // buildJSONObject 把 JSON 值转成 Gemini API Struct，非对象值放在 result 字段 func buildJSONObject(raw json.RawMessage) (any, error) { trimmed := bytes.TrimSpace(raw) @@ -491,6 +523,17 @@ type buildResponsePart struct { Name string `json:"name"` Args json.RawMessage `json:"args"` } `json:"functionCall"` + ToolCall *buildToolStep `json:"toolCall"` + ToolResponse *buildToolStep `json:"toolResponse"` +} + +// buildToolStep 表示 Gemini API 服务端工具的 toolCall 或 toolResponse +type buildToolStep struct { + ID string `json:"id"` + ToolType string `json:"toolType"` + ToolName string `json:"toolName"` + Args json.RawMessage `json:"args"` + Response json.RawMessage `json:"response"` } type buildGrounding struct { @@ -664,6 +707,12 @@ func decodeBuildPart(part buildResponsePart, index int, raw []byte) ([]Event, er call := FunctionCall{ID: part.FunctionCall.ID, Name: part.FunctionCall.Name, Arguments: arguments, ThoughtSignature: signature} events = append(events, Event{Kind: EventToolCall, ToolCall: &call, ThoughtSignature: signature}) } + if step := part.ToolCall; step != nil && step.ToolType == "MEDIA_PROCESSING" { + events = append(events, Event{Kind: EventMediaProcessingCall, MediaProcessing: &MediaProcessingStep{ID: step.ID, Name: step.ToolName, Data: step.Args}, ThoughtSignature: signature}) + } + if step := part.ToolResponse; step != nil && step.ToolType == "MEDIA_PROCESSING" { + events = append(events, Event{Kind: EventMediaProcessingResult, MediaProcessing: &MediaProcessingStep{ID: step.ID, Name: step.ToolName, Data: step.Response}, ThoughtSignature: signature}) + } if len(events) == 0 && signature != "" { events = append(events, Event{Kind: EventThoughtSignature, ThoughtSignature: signature}) } diff --git a/internal/aistudio/event.go b/internal/aistudio/event.go index ec4406e5..1b0dd8df 100644 --- a/internal/aistudio/event.go +++ b/internal/aistudio/event.go @@ -329,6 +329,20 @@ func (d *FrameDecoder) decodePart(raw json.RawMessage, path string, evidence jso }) } } + for _, field := range []struct { + index int + kind EventKind + }{{25, EventMediaProcessingCall}, {26, EventMediaProcessingResult}} { + if value := rawAt(part, field.index); !isJSONNull(value) { + step, err := decodeMediaProcessingStep(value, fmt.Sprintf("%s[%d]", path, field.index), raw) + if err != nil { + return nil, err + } + if step != nil { + events = append(events, Event{Kind: field.kind, MediaProcessing: step, ThoughtSignature: signature}) + } + } + } if len(events) == 0 { if signature != "" { return []Event{{Kind: EventThoughtSignature, ThoughtSignature: signature}}, nil diff --git a/internal/aistudio/generate.go b/internal/aistudio/generate.go index 1157ee24..9f856ba8 100644 --- a/internal/aistudio/generate.go +++ b/internal/aistudio/generate.go @@ -645,6 +645,9 @@ func (c *Client) validateGenerateRequest(request GenerateRequest) error { // validateGenerateEntry 按 Generate 的顺序校验工具、转写、生成参数与安全设置 func validateGenerateEntry(request GenerateRequest, entry modelEntry) error { + if err := validateVideoOffsets(request.Contents); err != nil { + return fmt.Errorf("%w: %v", ErrInvalidArgument, err) + } if _, err := supportedTools(request.Tools, entry.model); err != nil { return fmt.Errorf("%w: %v", ErrInvalidArgument, err) } @@ -676,6 +679,9 @@ func (c *Client) Generate(ctx context.Context, request GenerateRequest) (<-chan if err != nil { return nil, err } + if err := validateVideoOffsets(request.Contents); err != nil { + return nil, fmt.Errorf("%w: %v", ErrInvalidArgument, err) + } request.Tools, err = supportedTools(request.Tools, entry.model) if err != nil { return nil, fmt.Errorf("%w: %v", ErrInvalidArgument, err) diff --git a/internal/aistudio/interaction.go b/internal/aistudio/interaction.go index 68dacc09..fdb4dbb5 100644 --- a/internal/aistudio/interaction.go +++ b/internal/aistudio/interaction.go @@ -159,6 +159,8 @@ func encodeInteractionSteps(contents []Content) ([]any, []Content, bool, error) texts := make([]Part, 0, len(content.Parts)) for _, part := range content.Parts { switch { + case part.VideoMetadata != nil || part.MediaProcessing != "" || part.ProcessingCall != nil || part.ProcessingResult != nil: + return nil, nil, false, fmt.Errorf("Interaction 模型的 contents[%d] 不接受视频截取、采样、处理方式与处理步骤", index) case part.Thought: continue case part.ExecutableCode != nil || part.CodeExecutionResult != nil: diff --git a/internal/aistudio/models.go b/internal/aistudio/models.go index e957d688..31b483e1 100644 --- a/internal/aistudio/models.go +++ b/internal/aistudio/models.go @@ -31,6 +31,7 @@ var capabilityNames = map[int64]string{ 54: "image_search", 58: "google_maps", 59: "interaction_route", + 69: "agentic_video", 74: "transcription_word_timestamps", 76: "transcription_language_codes", 77: "transcription_output", diff --git a/internal/aistudio/request.go b/internal/aistudio/request.go index e48d689d..af57a314 100644 --- a/internal/aistudio/request.go +++ b/internal/aistudio/request.go @@ -212,6 +212,9 @@ func encodePart(part Part) ([]any, error) { if part.CodeExecutionResult != nil { variants++ } + if part.ProcessingCall != nil || part.ProcessingResult != nil { + variants++ + } if variants == 0 && part.ThoughtSignature != "" { return setPartThoughtSignature([]any{}, part.ThoughtSignature), nil } @@ -247,7 +250,7 @@ func encodePart(part Part) ([]any, error) { return nil, fmt.Errorf("inline data 缺少 MIME 或数据") } wire := []any{nil, nil, []any{part.InlineData.MIME, base64.StdEncoding.EncodeToString(part.InlineData.Data)}} - return setPartThoughtSignature(wire, part.ThoughtSignature), nil + return encodeVideoPart(wire, part) } if part.ExternalMedia != nil { if part.ExternalMedia.MIME == "" || part.ExternalMedia.URL == "" { @@ -255,20 +258,20 @@ func encodePart(part Part) ([]any, error) { } wire := make([]any, 7) wire[6] = []any{part.ExternalMedia.MIME, part.ExternalMedia.URL} - return setPartThoughtSignature(wire, part.ThoughtSignature), nil + return encodeVideoPart(wire, part) } if part.File != nil { wire, err := encodeFilePart(part.File) if err != nil { return nil, err } - return setPartThoughtSignature(wire, part.ThoughtSignature), nil + return encodeVideoPart(wire, part) } if upload := part.Upload; upload != nil { // Part field 18 上传附件：MIME、上传路径、文件名、字节数 wire := make([]any, 18) wire[17] = []any{upload.MIME, upload.Path, upload.Name, strconv.FormatInt(upload.Size, 10)} - return setPartThoughtSignature(wire, part.ThoughtSignature), nil + return encodeVideoPart(wire, part) } if part.FunctionCall != nil { if err := validateFunctionCall(part.FunctionCall); err != nil { @@ -306,6 +309,19 @@ func encodePart(part Part) ([]any, error) { wire[11] = result return setPartThoughtSignature(wire, part.ThoughtSignature), nil } + if part.ProcessingCall != nil || part.ProcessingResult != nil { + step, index := part.ProcessingCall, 25 + if step == nil { + step, index = part.ProcessingResult, 26 + } + value, err := encodeMediaProcessingStep(step) + if err != nil { + return nil, err + } + wire := make([]any, index+1) + wire[index] = value + return setPartThoughtSignature(wire, part.ThoughtSignature), nil + } if part.ExecutableCode != nil { language, ok := map[string]int64{"LANGUAGE_UNSPECIFIED": 0, "PYTHON": 1}[part.ExecutableCode.Language] if !ok { @@ -352,7 +368,7 @@ func countTokensNeedsGenerateRequest(contents []Content) bool { for _, content := range contents { for _, part := range content.Parts { if part.InlineData != nil || part.ExternalMedia != nil || part.File != nil || part.FunctionCall != nil || part.FunctionResult != nil || - part.ExecutableCode != nil || part.CodeExecutionResult != nil { + part.ExecutableCode != nil || part.CodeExecutionResult != nil || part.ProcessingCall != nil || part.ProcessingResult != nil { return true } } diff --git a/internal/aistudio/types.go b/internal/aistudio/types.go index 92e22e36..36e059d1 100644 --- a/internal/aistudio/types.go +++ b/internal/aistudio/types.go @@ -45,6 +45,20 @@ type UploadRef struct { Size int64 `json:"size"` } +// VideoMetadata 是视频 Part 的截取起止与采样帧率，起止为 Duration 字符串 +type VideoMetadata struct { + StartOffset string `json:"start_offset,omitempty"` + EndOffset string `json:"end_offset,omitempty"` + FPS *float64 `json:"fps,omitempty"` +} + +// MediaProcessingStep 是上游视频理解发起的一次服务端处理调用或其结果 +type MediaProcessingStep struct { + ID string `json:"id,omitempty"` + Name string `json:"name,omitempty"` + Data json.RawMessage `json:"data,omitempty"` +} + // FunctionCall 表示模型发起的函数调用 type FunctionCall struct { ID string `json:"id"` @@ -136,6 +150,11 @@ type Part struct { Thought bool `json:"thought,omitempty"` ThoughtSignature string `json:"thought_signature,omitempty"` SpeechMetadata *SpeechMetadata `json:"speech_metadata,omitempty"` + VideoMetadata *VideoMetadata `json:"video_metadata,omitempty"` + // MediaProcessing 为视频处理方式 STATIC 或 AGENTIC + MediaProcessing string `json:"media_processing,omitempty"` + ProcessingCall *MediaProcessingStep `json:"processing_call,omitempty"` + ProcessingResult *MediaProcessingStep `json:"processing_result,omitempty"` } // SpeechMetadata 表示语音文本分段的说话人与风格 @@ -449,6 +468,10 @@ const ( EventReasoning EventKind = "reasoning" // EventToolCall 表示函数调用 EventToolCall EventKind = "tool_call" + // EventMediaProcessingCall 表示视频理解的服务端处理调用 + EventMediaProcessingCall EventKind = "media_processing_call" + // EventMediaProcessingResult 表示视频理解的服务端处理结果 + EventMediaProcessingResult EventKind = "media_processing_result" // EventExecutableCode 表示内置代码执行器生成代码 EventExecutableCode EventKind = "executable_code" // EventCodeExecutionResult 表示内置代码执行结果 @@ -474,6 +497,7 @@ type Event struct { Kind EventKind `json:"kind"` Text string `json:"text,omitempty"` ToolCall *FunctionCall `json:"tool_call,omitempty"` + MediaProcessing *MediaProcessingStep `json:"media_processing,omitempty"` ExecutableCode *ExecutableCode `json:"executable_code,omitempty"` CodeExecutionResult *CodeExecutionResult `json:"code_execution_result,omitempty"` Grounding *GroundingMetadata `json:"grounding,omitempty"` diff --git a/internal/aistudio/video_input.go b/internal/aistudio/video_input.go new file mode 100644 index 00000000..5babeb6a --- /dev/null +++ b/internal/aistudio/video_input.go @@ -0,0 +1,152 @@ +package aistudio + +import ( + "encoding/json" + "fmt" + "regexp" + "strconv" + "strings" +) + +// videoOffsetPattern 匹配秒数后带 s 的 Duration 字符串，小数最多九位 +var videoOffsetPattern = regexp.MustCompile(`^([0-9]+)(?:\.([0-9]{1,9}))?s$`) + +// mediaProcessingCodes 是 Part field 40 视频处理方式的枚举值 +var mediaProcessingCodes = map[string]int64{"MEDIA_PROCESSING_UNSPECIFIED": 0, "STATIC": 1, "AGENTIC": 2} + +// mediaProcessingToolType 是 ToolCall 与 ToolResponse field 1 中视频理解处理的工具类型 +const mediaProcessingToolType = 13 + +// videoOffset 把 Duration 字符串解析为秒与纳秒 +func videoOffset(value string) (int64, int64, error) { + match := videoOffsetPattern.FindStringSubmatch(value) + if match == nil { + return 0, 0, fmt.Errorf("video offset %q 必须是秒数加 s 的 Duration，例如 \"2.5s\"", value) + } + seconds, err := strconv.ParseInt(match[1], 10, 64) + if err != nil { + return 0, 0, fmt.Errorf("video offset %q: %w", value, err) + } + var nanos int64 + if match[2] != "" { + nanos, _ = strconv.ParseInt(match[2]+strings.Repeat("0", 9-len(match[2])), 10, 64) + } + return seconds, nanos, nil +} + +// videoOffsetWire 把 Duration 字符串编码为 google.protobuf.Duration 的 [seconds, nanos] +func videoOffsetWire(value string) ([]any, error) { + seconds, nanos, err := videoOffset(value) + if err != nil { + return nil, err + } + if nanos == 0 { + return []any{seconds}, nil + } + return []any{seconds, nanos}, nil +} + +// validateVideoOffsets 校验视频截取同时设置起止时终点晚于起点 +func validateVideoOffsets(contents []Content) error { + for _, content := range contents { + for _, part := range content.Parts { + video := part.VideoMetadata + if video == nil || video.StartOffset == "" || video.EndOffset == "" { + continue + } + startSeconds, startNanos, err := videoOffset(video.StartOffset) + if err != nil { + return err + } + endSeconds, endNanos, err := videoOffset(video.EndOffset) + if err != nil { + return err + } + if endSeconds < startSeconds || endSeconds == startSeconds && endNanos <= startNanos { + return fmt.Errorf("video endOffset %s 必须晚于 startOffset %s", video.EndOffset, video.StartOffset) + } + } + } + return nil +} + +// encodeVideoPart 为媒体 Part 写入 field 16 视频元数据与 field 40 处理方式，再写入思考签名 +func encodeVideoPart(wire []any, part Part) ([]any, error) { + if video := part.VideoMetadata; video != nil { + var metadata []any + for index, offset := range []string{video.StartOffset, video.EndOffset} { + if offset == "" { + continue + } + value, err := videoOffsetWire(offset) + if err != nil { + return nil, err + } + metadata = setWireField(metadata, index, value) + } + if video.FPS != nil { + metadata = setWireField(metadata, 2, *video.FPS) + } + if metadata != nil { + wire = setWireField(wire, 15, metadata) + } + } + if part.MediaProcessing != "" { + code, ok := mediaProcessingCodes[part.MediaProcessing] + if !ok { + return nil, fmt.Errorf("未知 mediaProcessing %q", part.MediaProcessing) + } + if code != 0 { + wire = setWireField(wire, 39, code) + } + } + return setPartThoughtSignature(wire, part.ThoughtSignature), nil +} + +// encodeMediaProcessingStep 编码 ToolCall 或 ToolResponse：工具类型、名称、Struct 与调用 ID +func encodeMediaProcessingStep(step *MediaProcessingStep) ([]any, error) { + wire := []any{int64(mediaProcessingToolType), nil, nil, step.ID} + if step.Name != "" { + wire[1] = step.Name + } + if len(step.Data) != 0 { + value, err := encodeWireStructJSON(step.Data) + if err != nil { + return nil, fmt.Errorf("media processing data: %w", err) + } + wire[2] = value + } + return wire, nil +} + +// decodeMediaProcessingStep 解码视频理解处理类型的 ToolCall 或 ToolResponse，其他工具类型返回 nil +func decodeMediaProcessingStep(raw json.RawMessage, path string, evidence json.RawMessage) (*MediaProcessingStep, error) { + fields, err := rawArray(raw, path, evidence) + if err != nil { + return nil, withMethod(err, "GenerateContent") + } + toolType, err := rawInt64(rawAt(fields, 0), path+"[0]", evidence) + if err != nil { + return nil, withMethod(err, "GenerateContent") + } + if toolType != mediaProcessingToolType { + return nil, nil + } + step := &MediaProcessingStep{} + if value := rawAt(fields, 1); !isJSONNull(value) { + if step.Name, err = rawString(value, path+"[1]", evidence); err != nil { + return nil, withMethod(err, "GenerateContent") + } + } + if value := rawAt(fields, 2); !isJSONNull(value) { + if step.Data, err = decodeWireStruct(value, path+"[2]", evidence); err != nil { + return nil, err + } + } + if value := rawAt(fields, 3); !isJSONNull(value) { + if step.ID, err = rawString(value, path+"[3]", evidence); err != nil { + return nil, withMethod(err, "GenerateContent") + } + } + return step, nil +} diff --git a/internal/api/gemini.go b/internal/api/gemini.go index 51bcfd39..5712b079 100644 --- a/internal/api/gemini.go +++ b/internal/api/gemini.go @@ -114,6 +114,34 @@ type geminiPart struct { } `json:"codeExecutionResult"` SpeechMetadata *aistudio.SpeechMetadata `json:"speechMetadata"` SpeechMetadataSnake *aistudio.SpeechMetadata `json:"speech_metadata"` + VideoMetadata *geminiVideoMetadata `json:"videoMetadata"` + MediaProcessing string `json:"mediaProcessing"` + ToolCall *geminiToolStep `json:"toolCall"` + ToolResponse *geminiToolStep `json:"toolResponse"` +} + +// geminiVideoMetadata 为 Part.videoMetadata 的截取起止与采样帧率 +type geminiVideoMetadata struct { + StartOffset string `json:"startOffset"` + EndOffset string `json:"endOffset"` + FPS *float64 `json:"fps"` +} + +// geminiToolStep 为 Part.toolCall 与 Part.toolResponse 的服务端工具步骤 +type geminiToolStep struct { + ID string `json:"id"` + ToolType string `json:"toolType"` + ToolName string `json:"toolName"` + Args json.RawMessage `json:"args"` + Response json.RawMessage `json:"response"` +} + +// mediaProcessing 把视频理解处理类型的服务端工具步骤转为规范步骤，payload 为 args 或 response +func (step geminiToolStep) mediaProcessing(payload json.RawMessage) (*aistudio.MediaProcessingStep, error) { + if step.ToolType != "MEDIA_PROCESSING" { + return nil, fmt.Errorf("unsupported server tool type %q", step.ToolType) + } + return &aistudio.MediaProcessingStep{ID: step.ID, Name: step.ToolName, Data: payload}, nil } func (p geminiPart) speechMetadata() *aistudio.SpeechMetadata { @@ -770,6 +798,13 @@ func mapGeminiParts(input []geminiPart) ([]aistudio.Part, bool, error) { if part.CodeExecutionResult != nil { variants++ } + if part.ToolCall != nil { + variants++ + } + if part.ToolResponse != nil { + variants++ + } + video, processing := (*aistudio.VideoMetadata)(part.VideoMetadata), part.MediaProcessing if variants == 0 { if part.ThoughtSignature != "" { parts = append(parts, aistudio.Part{ThoughtSignature: part.ThoughtSignature}) @@ -793,6 +828,7 @@ func mapGeminiParts(input []geminiPart) ([]aistudio.Part, bool, error) { parts = append(parts, aistudio.Part{ InlineData: &aistudio.Blob{MIME: mimeType, Data: data}, ThoughtSignature: part.ThoughtSignature, + VideoMetadata: video, MediaProcessing: processing, }) case file != nil: uri := file.URI() @@ -804,13 +840,14 @@ func mapGeminiParts(input []geminiPart) ([]aistudio.Part, bool, error) { uri = driveID } if media, ok := aistudio.ExternalMediaForURL(uri); ok { - parts = append(parts, aistudio.Part{ExternalMedia: media, ThoughtSignature: part.ThoughtSignature}) + parts = append(parts, aistudio.Part{ExternalMedia: media, ThoughtSignature: part.ThoughtSignature, VideoMetadata: video, MediaProcessing: processing}) } else { parts = append(parts, aistudio.Part{ File: &aistudio.FileRef{ ID: uri, Name: file.DisplayName, MIME: mime, }, ThoughtSignature: part.ThoughtSignature, + VideoMetadata: video, MediaProcessing: processing, }) } case part.FunctionCall != nil: @@ -847,6 +884,18 @@ func mapGeminiParts(input []geminiPart) ([]aistudio.Part, bool, error) { }) parts = append(parts, media...) hasResult = true + case part.ToolCall != nil: + step, err := part.ToolCall.mediaProcessing(part.ToolCall.Args) + if err != nil { + return nil, false, fmt.Errorf("parts[%d]: %w", index, err) + } + parts = append(parts, aistudio.Part{ProcessingCall: step, ThoughtSignature: part.ThoughtSignature}) + case part.ToolResponse != nil: + step, err := part.ToolResponse.mediaProcessing(part.ToolResponse.Response) + if err != nil { + return nil, false, fmt.Errorf("parts[%d]: %w", index, err) + } + parts = append(parts, aistudio.Part{ProcessingResult: step, ThoughtSignature: part.ThoughtSignature}) case part.ExecutableCode != nil: parts = append(parts, aistudio.Part{ ExecutableCode: &aistudio.ExecutableCode{ @@ -1145,6 +1194,8 @@ func geminiOutputParts(result generationResult) []map[string]any { if event.ToolCall != nil { appendPart(geminiFunctionCallPart(*event.ToolCall), event.ThoughtSignature) } + case aistudio.EventMediaProcessingCall, aistudio.EventMediaProcessingResult: + appendPart(geminiProcessingPart(event), event.ThoughtSignature) case aistudio.EventExecutableCode: if event.ExecutableCode != nil { appendPart(map[string]any{"executableCode": map[string]any{ @@ -1545,6 +1596,8 @@ func geminiStreamCandidate(event aistudio.Event, index int) map[string]any { return nil } part = geminiSignedPart(geminiFunctionCallPart(*event.ToolCall), event.ThoughtSignature) + case aistudio.EventMediaProcessingCall, aistudio.EventMediaProcessingResult: + part = geminiSignedPart(geminiProcessingPart(event), event.ThoughtSignature) case aistudio.EventExecutableCode: if event.ExecutableCode == nil { return nil diff --git a/internal/api/interaction_input.go b/internal/api/interaction_input.go index 36f3aab0..bc153ad5 100644 --- a/internal/api/interaction_input.go +++ b/internal/api/interaction_input.go @@ -74,6 +74,7 @@ type interactionInput struct { Data string `json:"data"` URI string `json:"uri"` MIME string `json:"mime_type"` + Processing json.RawMessage `json:"processing"` Content []interactionInput `json:"content"` Summary []interactionInput `json:"summary"` ID string `json:"id"` @@ -360,6 +361,22 @@ func interactionContents(raw json.RawMessage) ([]aistudio.Content, error) { if item.Signature != "" { pendingSignature = item.Signature } + case "processing_call", "processing_result": + role = aistudio.RoleAssistant + if item.Type == "processing_call" && item.ID == "" || item.Type == "processing_result" && item.CallID == "" { + return nil, fmt.Errorf("processing step requires its call identifier") + } + mapped := aistudio.Part{ThoughtSignature: item.Signature} + if item.Type == "processing_call" { + mapped.ProcessingCall = &aistudio.MediaProcessingStep{ID: item.ID} + } else { + mapped.ProcessingResult = &aistudio.MediaProcessingStep{ID: item.CallID} + } + if pendingSignature != "" { + parts = append(parts, aistudio.Part{ThoughtSignature: pendingSignature}) + pendingSignature = "" + } + parts = append(parts, mapped) case "code_execution_call", "code_execution_result": role = aistudio.RoleAssistant if text := item.codeExecutionText(); text != "" { @@ -367,8 +384,7 @@ func interactionContents(raw json.RawMessage) ([]aistudio.Content, error) { } case "google_search_call", "google_search_result", "url_context_call", "url_context_result", "google_maps_call", "google_maps_result", "file_search_call", "file_search_result", - "mcp_server_tool_call", "mcp_server_tool_result", "retrieval_call", "retrieval_result", - "processing_call", "processing_result": + "mcp_server_tool_call", "mcp_server_tool_result", "retrieval_call", "retrieval_result": continue default: parts, err = item.parts() @@ -422,6 +438,12 @@ func (item interactionInput) codeExecutionText() string { // parts 复用 Gemini 的文本、媒体与文件输入映射 func (content interactionInput) parts() ([]aistudio.Part, error) { var part geminiPart + if rawJSONConfigured(content.Processing) { + var err error + if part.VideoMetadata, part.MediaProcessing, err = interactionVideoProcessing(content.Processing); err != nil { + return nil, err + } + } switch content.Type { case "text": if content.Text == nil { diff --git a/internal/api/interaction_output.go b/internal/api/interaction_output.go index ecfabcc8..6a97b8e8 100644 --- a/internal/api/interaction_output.go +++ b/internal/api/interaction_output.go @@ -3,6 +3,7 @@ package api import ( "encoding/base64" "fmt" + "maps" "mime" "net/http" "strconv" @@ -82,6 +83,16 @@ func interactionStep(event aistudio.Event, audio interactionAudioFormat) (map[st } call := event.ToolCall return map[string]any{"type": "function_call", "id": call.ID, "name": call.Name, "arguments": call.Arguments}, nil, nil + case aistudio.EventMediaProcessingCall, aistudio.EventMediaProcessingResult: + step := map[string]any{"type": "processing_call", "id": event.MediaProcessing.ID} + if event.Kind == aistudio.EventMediaProcessingResult { + step = map[string]any{"type": "processing_result", "call_id": event.MediaProcessing.ID} + } + delta := map[string]any{"type": step["type"]} + if event.ThoughtSignature != "" { + step["signature"], delta["signature"] = event.ThoughtSignature, event.ThoughtSignature + } + return step, delta, nil case aistudio.EventMedia: if event.Media == nil { return nil, nil, nil @@ -333,7 +344,8 @@ func (s *server) streamInteraction(w http.ResponseWriter, r *http.Request, reque } textLength += length } - if active != kind || kind == "function_call" { + standalone := kind == "function_call" || kind == "processing_call" || kind == "processing_result" + if active != kind || standalone { if err := closeStep(); err != nil { return err } @@ -346,8 +358,9 @@ func (s *server) streamInteraction(w http.ResponseWriter, r *http.Request, reque if kind == "thought" { start["summary"] = []any{} } - if kind == "function_call" { - start = step + if standalone { + start = maps.Clone(step) + delete(start, "signature") } if err := send("step.start", map[string]any{"index": index, "step": start}); err != nil { return err diff --git a/internal/api/responses.go b/internal/api/responses.go index 071e3e30..d03906c0 100644 --- a/internal/api/responses.go +++ b/internal/api/responses.go @@ -579,7 +579,20 @@ func responseHistoryOutput(result generationResult) aistudio.Content { if event.Media != nil && len(event.Media.Data) > 0 { blob := &aistudio.Blob{MIME: event.Media.MIME, Data: append([]byte(nil), event.Media.Data...)} output.Parts = append(output.Parts, aistudio.Part{InlineData: blob, ThoughtSignature: event.ThoughtSignature}) + } else if event.Media != nil && event.Media.URL != "" { + file := &aistudio.FileRef{ID: event.Media.URL, Name: event.Media.Name, MIME: event.Media.MIME} + output.Parts = append(output.Parts, aistudio.Part{File: file, ThoughtSignature: event.ThoughtSignature}) } + case aistudio.EventExecutableCode: + code := *event.ExecutableCode + output.Parts = append(output.Parts, aistudio.Part{ExecutableCode: &code, ThoughtSignature: event.ThoughtSignature}) + case aistudio.EventCodeExecutionResult: + result := *event.CodeExecutionResult + output.Parts = append(output.Parts, aistudio.Part{CodeExecutionResult: &result, ThoughtSignature: event.ThoughtSignature}) + case aistudio.EventMediaProcessingCall: + output.Parts = append(output.Parts, aistudio.Part{ProcessingCall: event.MediaProcessing, ThoughtSignature: event.ThoughtSignature}) + case aistudio.EventMediaProcessingResult: + output.Parts = append(output.Parts, aistudio.Part{ProcessingResult: event.MediaProcessing, ThoughtSignature: event.ThoughtSignature}) case aistudio.EventReasoning, aistudio.EventThoughtSignature: if event.ThoughtSignature != "" { output.Parts = append(output.Parts, aistudio.Part{ThoughtSignature: event.ThoughtSignature}) diff --git a/internal/api/video_input.go b/internal/api/video_input.go new file mode 100644 index 00000000..003fc97e --- /dev/null +++ b/internal/api/video_input.go @@ -0,0 +1,44 @@ +package api + +import ( + "encoding/json" + "fmt" + "strings" + + "github.com/Mag1cFall/AIStudio2API/internal/aistudio" +) + +// geminiProcessingPart 把视频理解处理事件投影为 Gemini 的 toolCall 或 toolResponse Part +func geminiProcessingPart(event aistudio.Event) map[string]any { + step := event.MediaProcessing + value := map[string]any{"toolType": "MEDIA_PROCESSING", "id": step.ID} + key, payload := "toolResponse", "response" + if event.Kind == aistudio.EventMediaProcessingCall { + key, payload = "toolCall", "args" + if step.Name != "" { + value["toolName"] = step.Name + } + } + if len(step.Data) != 0 { + value[payload] = step.Data + } + return map[string]any{key: value} +} + +// interactionVideoProcessing 把 video 内容的 processing 映射为截取采样设置与处理方式，取值为 static、agentic 或 static 设置对象 +func interactionVideoProcessing(raw json.RawMessage) (*geminiVideoMetadata, string, error) { + var mode string + if json.Unmarshal(raw, &mode) == nil { + return nil, strings.ToUpper(mode), nil + } + var config struct { + Type string `json:"type"` + StartOffset string `json:"start_offset"` + EndOffset string `json:"end_offset"` + FPS *float64 `json:"fps"` + } + if err := json.Unmarshal(raw, &config); err != nil { + return nil, "", fmt.Errorf("video processing: %w", err) + } + return &geminiVideoMetadata{StartOffset: config.StartOffset, EndOffset: config.EndOffset, FPS: config.FPS}, strings.ToUpper(config.Type), nil +} diff --git a/internal/api/video_input_test.go b/internal/api/video_input_test.go new file mode 100644 index 00000000..2f525168 --- /dev/null +++ b/internal/api/video_input_test.go @@ -0,0 +1,50 @@ +package api + +import ( + "encoding/json" + "testing" + + "github.com/Mag1cFall/AIStudio2API/internal/aistudio" +) + +// TestGeminiVideoProcessingRoundTrip 保留视频截取、采样与处理方式，服务端处理步骤按 Gemini 形状输出后可作为下一轮历史读回 +func TestGeminiVideoProcessingRoundTrip(t *testing.T) { + var request geminiRequest + if err := json.Unmarshal([]byte(`{"contents":[{"role":"user","parts":[ + {"fileData":{"mimeType":"video/*","fileUri":"https://www.youtube.com/watch?v=rOVtjJkqtiA"},"videoMetadata":{"startOffset":"2s","endOffset":"4.5s","fps":0.5},"mediaProcessing":"STATIC"}, + {"text":"describe"}]}]}`), &request); err != nil { + t.Fatal(err) + } + generate, err := request.toGenerateRequest("id", "m") + if err != nil { + t.Fatal(err) + } + video := generate.Contents[0].Parts[0] + if video.ExternalMedia == nil || video.MediaProcessing != "STATIC" || video.VideoMetadata == nil || + video.VideoMetadata.StartOffset != "2s" || video.VideoMetadata.EndOffset != "4.5s" || video.VideoMetadata.FPS == nil || *video.VideoMetadata.FPS != 0.5 { + t.Fatalf("video part = %#v", video) + } + + events := []aistudio.Event{ + {Kind: aistudio.EventMediaProcessingCall, MediaProcessing: &aistudio.MediaProcessingStep{ID: "call_1"}, ThoughtSignature: "call-signature"}, + {Kind: aistudio.EventMediaProcessingResult, MediaProcessing: &aistudio.MediaProcessingStep{ID: "call_1"}, ThoughtSignature: "result-signature"}, + {Kind: aistudio.EventText, Text: "answer"}, + } + output, err := json.Marshal(map[string]any{"contents": []any{map[string]any{"role": "model", "parts": geminiOutputParts(generationResult{events: events})}}}) + if err != nil { + t.Fatal(err) + } + var history geminiRequest + if err := json.Unmarshal(output, &history); err != nil { + t.Fatal(err) + } + replayed, err := history.toGenerateRequest("id", "m") + if err != nil { + t.Fatal(err) + } + parts := replayed.Contents[0].Parts + if len(parts) != 3 || parts[0].ProcessingCall == nil || parts[0].ProcessingCall.ID != "call_1" || parts[0].ThoughtSignature != "call-signature" || + parts[1].ProcessingResult == nil || parts[1].ProcessingResult.ID != "call_1" || parts[1].ThoughtSignature != "result-signature" || parts[2].Text != "answer" { + t.Fatalf("history parts = %s", output) + } +}