diff --git a/Cargo.lock b/Cargo.lock index 9cb659bd..cc24e18b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1325,6 +1325,7 @@ dependencies = [ "base64 0.22.1", "bytes", "futures", + "http", "reqwest", "serde", "serde_json", diff --git a/README.md b/README.md index e23fc8ef..a3df93a5 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,9 @@ The workspace provides: - OpenAI Chat Completions, OpenAI Responses, and OpenAI-compatible provider adapters for Anthropic, Ollama, DeepSeek, Groq, xAI, OpenRouter, Together, and Mistral; +- native Perplexity Agent API model/preset selection, cited and hosted-tool + output, incremental streaming, signed function continuation, background + recovery, and generated-file retrieval; - OpenAI, Cohere, Ollama, Voyage, cloud, no-op, and deterministic mock embeddings; - standalone document reranking through Voyage, with original-position results, @@ -64,6 +67,43 @@ standards and OpenRouter media transport, download, resume), `tinyinference-decisions` for typed Jev and Sage decisions, and `tinyinference-core` only for shared infrastructure. +### Perplexity Agent API + +Use one Perplexity key with an explicit model or server-managed preset: + +```rust,no_run +use tinyinference_llm::{ChatModel, Message, ModelRequest, PerplexityModel, PerplexitySelection}; + +# async fn example(key: String) -> tinyinference_llm::Result<()> { +let model = PerplexityModel::new(key, PerplexitySelection::preset("low"))?; +let response = model.invoke(&(), ModelRequest::new(vec![ + Message::user("Explain the latest battery research with sources."), +])).await?; +println!("{}", response.text()); +// response.output contains typed citations, hosted results and file notices. +# Ok(()) +# } +``` + +`PerplexitySelection::Model("provider/model".into())` selects a specific model; +`Models` configures a provider-managed fallback list. Presets retain server +defaults unless you explicitly override them. Tool settings merge with preset +tools on the provider. Explicit Anthropic models require an output-token cap. + +`ModelResponse.output` preserves ordered rich output; `message` is the familiar +text/reasoning/local-function projection. Hosted MCP, search and sandbox activity +is never dispatched as a local function call. Stream-only progress survives in +`execution.progress`; incomplete and cancelled states remain explicit. + +Use `submit_background`, `retrieve_response`, `resume_background`, and +`cancel_background` for long jobs. List and explicitly download generated files +by response/file ID; filenames are metadata, not local paths. A disconnect never +silently starts a replacement run. Remote MCP credentials are bound to their +configured HTTPS server and kept out of serializable model requests. + +See [Perplexity contracts and migration](docs/migrations/perplexity-agent.md) +for configuration, replay, storage semantics, limits, and public Rust type changes. + ### Voyage reranking `tinyinference-embeddings` exports `Reranker`, `RerankRequest`, diff --git a/crates/tinyinference-llm/Cargo.toml b/crates/tinyinference-llm/Cargo.toml index 21cfa895..592ee3b4 100644 --- a/crates/tinyinference-llm/Cargo.toml +++ b/crates/tinyinference-llm/Cargo.toml @@ -18,7 +18,7 @@ bytes = { workspace = true } futures = { workspace = true } reqwest = { workspace = true } serde = { workspace = true } -serde_json = { workspace = true } +serde_json = { workspace = true, features = ["raw_value"] } sha2 = { workspace = true } thiserror = { workspace = true } tinyinference-core = { version = "0.3.0", path = "../tinyinference-core" } @@ -30,6 +30,8 @@ tokio = { workspace = true } tracing = { workspace = true } [dev-dependencies] +http = "1" +tokio = { workspace = true, features = ["test-util"] } tempfile = { workspace = true } [lints] diff --git a/crates/tinyinference-llm/examples/perplexity_function_replay.rs b/crates/tinyinference-llm/examples/perplexity_function_replay.rs new file mode 100644 index 00000000..beb22211 --- /dev/null +++ b/crates/tinyinference-llm/examples/perplexity_function_replay.rs @@ -0,0 +1,66 @@ +//! Run a Google function call and replay its result through Perplexity. +//! +//! Set `PERPLEXITY_API_KEY`, then run: +//! `cargo run -p tinyinference-llm --example perplexity_function_replay`. +//! This makes two billable requests and runs one local demonstration function. + +use serde_json::json; +use tinyinference_llm::{ + ChatModel, ContentBlock, ExecutionStatus, Message, ModelRequest, PerplexityModel, + PerplexitySelection, ToolSchema, message::ToolMessage, model::ToolChoice, +}; + +#[tokio::main] +async fn main() -> Result<(), Box> { + let key = std::env::var("PERPLEXITY_API_KEY")?; + let model = PerplexityModel::new( + key, + PerplexitySelection::Model("google/gemini-3.1-flash-lite".into()), + )?; + let tool = ToolSchema::new( + "lookup_code", + "Return the verification code for a record", + json!({ + "type": "object", + "properties": {"id": {"type": "integer"}}, + "required": ["id"], + "additionalProperties": false + }), + ); + let mut request = ModelRequest::new(vec![Message::user( + "Use lookup_code for record 7, then tell me its verification code.", + )]) + .with_tools(vec![tool]) + .with_tool_choice(ToolChoice::Tool("lookup_code".into())) + .with_max_tokens(512); + request.timeout_ms = Some(60_000); + let first = model.invoke(&(), request.clone()).await?; + let execution = first.execution.as_ref().ok_or("missing model identity")?; + if execution.status != ExecutionStatus::Completed || first.message.tool_calls.is_empty() { + return Err("expected a completed function call".into()); + } + + // Replay the original assistant call so its argument bytes and signature survive. + request + .messages + .push(Message::Assistant(first.message.clone())); + for call in &first.message.tool_calls { + if call.invalid.is_some() + || call.name != "lookup_code" + || call.arguments.get("id").and_then(serde_json::Value::as_u64) != Some(7) + { + return Err("unexpected function or invalid arguments".into()); + } + let result = ToolMessage::for_call(call, vec![ContentBlock::Text("CODE_7391".into())]); + request.messages.push(Message::Tool(result)); + } + request.model = Some(execution.model.clone()); + request.continuation_id = None; + request.tool_choice = ToolChoice::None; + let response = model.invoke(&(), request).await?; + println!("{}", response.text()); + if !response.text().contains("CODE_7391") { + return Err("the model did not return the demonstration function result".into()); + } + Ok(()) +} diff --git a/crates/tinyinference-llm/src/failure_tests.rs b/crates/tinyinference-llm/src/failure_tests.rs index 871d6bf6..951ae874 100644 --- a/crates/tinyinference-llm/src/failure_tests.rs +++ b/crates/tinyinference-llm/src/failure_tests.rs @@ -53,6 +53,7 @@ fn structured_status_takes_precedence_over_message_heuristics() { #[test] fn structured_provider_error_uses_the_same_classifier() { let error = ProviderError { + partial_response: None, status: Some(429), code: Some("insufficient_quota".into()), message: "quota exhausted".into(), @@ -68,6 +69,7 @@ fn structured_provider_error_uses_the_same_classifier() { #[test] fn normalized_provider_retryability_is_authoritative() { let error = ProviderError { + partial_response: None, message: "malformed streaming payload".into(), retryable: false, ..ProviderError::default() diff --git a/crates/tinyinference-llm/src/lib.rs b/crates/tinyinference-llm/src/lib.rs index c515068c..ba6430bd 100644 --- a/crates/tinyinference-llm/src/lib.rs +++ b/crates/tinyinference-llm/src/lib.rs @@ -25,11 +25,23 @@ pub use failure::{ provider_error_is_retryable, structured_http_status, }; pub use message::{AssistantMessage, ContentBlock, Message, MessageDelta}; +pub use model::{ + AnnotatedContent, ExecutionStatus, FetchedContent, FinanceResult, HostedToolUsage, + ImageSearchResult, ModelExecution, ModelFile, ModelFileContent, ModelOutputEvent, + ModelOutputItem, ModelOutputKind, ModelProgress, OutputAnnotation, RemoteToolDefinition, + RemoteToolNamespace, ReportedCost, SearchSource, +}; pub use model::{ ChatModel, InputModality, InputSource, ModelRequest, ModelResponse, ModelStream, ModelStreamItem, context_window_for_model_id, model_id_supports_vision, }; pub use network_guard::{allow_network_models, deny_network_models, network_models_denied}; +pub use providers::perplexity::{ + PerplexityConfig, PerplexityImageFilters, PerplexityLocation, PerplexityModel, + PerplexityOptions, PerplexityRemoteCredentials, PerplexitySearchFilters, PerplexitySelection, + PerplexityTool, PerplexityToolChoice, PerplexityWebSearch, build_perplexity_model, +}; pub use providers::{MockModel, ProviderKind, ProviderSpec}; pub use tool::{ToolCall, ToolDelta, ToolFormat, ToolSchema}; +pub use tool::{ToolCallReplay, ToolResultContext}; pub use usage::{Usage, UsageTotals}; diff --git a/crates/tinyinference-llm/src/message/mod.rs b/crates/tinyinference-llm/src/message/mod.rs index 23a7e4c0..4300cab1 100644 --- a/crates/tinyinference-llm/src/message/mod.rs +++ b/crates/tinyinference-llm/src/message/mod.rs @@ -133,6 +133,7 @@ impl Message { /// Creates a tool result message for the given tool call id. pub fn tool(tool_call_id: impl Into, content: impl Into) -> Self { Message::Tool(ToolMessage { + call_context: None, tool_call_id: tool_call_id.into(), content: vec![ContentBlock::Text(content.into())], trusted_verbatim: false, diff --git a/crates/tinyinference-llm/src/message/types.rs b/crates/tinyinference-llm/src/message/types.rs index beed646a..1b296cd5 100644 --- a/crates/tinyinference-llm/src/message/types.rs +++ b/crates/tinyinference-llm/src/message/types.rs @@ -343,6 +343,9 @@ pub struct MessageOrigin { /// A tool result message correlated to a prior tool call. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] pub struct ToolMessage { + /// Name and signed-call context for stateful native function results. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub call_context: Option, /// Id of the tool call this message answers. pub tool_call_id: String, /// Ordered content blocks. @@ -355,6 +358,22 @@ pub struct ToolMessage { pub artifact: Option, } +impl ToolMessage { + /// Creates a native function result retaining the originating call context. + pub fn for_call(call: &ToolCall, content: Vec) -> Self { + Self { + call_context: Some(crate::tool::ToolResultContext { + name: call.name.clone(), + replay: call.replay.clone(), + }), + tool_call_id: call.id.clone(), + content, + trusted_verbatim: false, + artifact: None, + } + } +} + fn is_false(value: &bool) -> bool { !*value } diff --git a/crates/tinyinference-llm/src/model/migration_tests.rs b/crates/tinyinference-llm/src/model/migration_tests.rs index c20ceb6c..70d78363 100644 --- a/crates/tinyinference-llm/src/model/migration_tests.rs +++ b/crates/tinyinference-llm/src/model/migration_tests.rs @@ -327,6 +327,7 @@ async fn every_terminal_stream_item_disarms_its_abort_guard() { ModelStreamItem::Completed(ModelResponse::assistant("done")), ModelStreamItem::Failed("failed".to_string()), ModelStreamItem::ProviderFailed(ProviderError { + partial_response: None, provider: "mock".to_string(), message: "provider failed".to_string(), ..ProviderError::default() @@ -359,6 +360,8 @@ async fn observer_reports_each_terminal_outcome_once() { let cached = ObservingModel::new( Arc::new(RecordingModel::new(ModelResponse { + output: Vec::new(), + execution: None, served_from_cache: true, ..ModelResponse::assistant("cached") })), @@ -449,6 +452,8 @@ async fn observer_streams_report_one_terminal_outcome_with_stream_metadata() { let cached = ObservingModel::new( Arc::new(ScriptedStreamModel::new( vec![ModelStreamItem::Completed(ModelResponse { + output: Vec::new(), + execution: None, served_from_cache: true, ..ModelResponse::assistant("cached") })], @@ -466,6 +471,7 @@ async fn observer_streams_report_one_terminal_outcome_with_stream_metadata() { let provider_failed = ObservingModel::new( Arc::new(ScriptedStreamModel::new( vec![ModelStreamItem::ProviderFailed(ProviderError { + partial_response: None, provider: "mock".to_string(), message: "provider failed".to_string(), ..ProviderError::default() diff --git a/crates/tinyinference-llm/src/model/mod.rs b/crates/tinyinference-llm/src/model/mod.rs index fe5090dd..772818e6 100644 --- a/crates/tinyinference-llm/src/model/mod.rs +++ b/crates/tinyinference-llm/src/model/mod.rs @@ -15,6 +15,7 @@ mod decorators; pub mod discover; +mod output; mod types; #[cfg(test)] @@ -29,6 +30,7 @@ use crate::tool::{ToolCall, ToolSchema}; use crate::usage::Usage; pub use decorators::*; +pub use output::*; pub use types::*; /// How a context-window pattern is matched against a model id. @@ -584,6 +586,8 @@ impl ModelResponse { /// Creates a response wrapping a plain assistant text message. pub fn assistant(content: impl Into) -> Self { Self { + output: Vec::new(), + execution: None, message: AssistantMessage { id: None, content: vec![ContentBlock::Text(content.into())], @@ -680,6 +684,9 @@ impl ModelResponse { /// usage. #[derive(Clone, Debug, Default)] pub struct StreamAccumulator { + output: std::collections::BTreeMap, + execution: Option, + progress: Vec, /// Concatenated text fragments. text: String, /// Accumulated reasoning/thinking fragments (side channel; not merged into @@ -714,6 +721,16 @@ impl StreamAccumulator { /// Folds one stream item into the running state. pub fn push(&mut self, item: &ModelStreamItem) { match item { + ModelStreamItem::OutputEvent(event) => match event { + ModelOutputEvent::Execution(execution) => { + self.execution = Some((**execution).clone()) + } + ModelOutputEvent::Item(item) => { + self.output.insert(item.index, (**item).clone()); + } + ModelOutputEvent::Progress(progress) => self.progress.push(progress.clone()), + _ => {} + }, ModelStreamItem::Started => {} ModelStreamItem::MessageDelta(delta) => { self.text.push_str(&delta.text); @@ -824,6 +841,17 @@ impl StreamAccumulator { } if let Some(mut response) = self.completed { + if response.execution.is_none() { + response.execution = self.execution; + } + if let Some(execution) = &mut response.execution + && execution.progress.is_empty() + { + execution.progress = self.progress; + } + if response.output.is_empty() { + response.output = self.output.into_values().collect(); + } // Reconcile the response and message usage with any streamed // `UsageDelta`, preferring an already-present value and never // overwriting a known usage with `None` (which previously clobbered a @@ -870,6 +898,13 @@ impl StreamAccumulator { origin: None, }; Ok(ModelResponse { + output: self.output.into_values().collect(), + execution: self.execution.map(|mut execution| { + if execution.progress.is_empty() { + execution.progress = self.progress; + } + execution + }), message, usage: self.usage, finish_reason: None, diff --git a/crates/tinyinference-llm/src/model/mod_tests.rs b/crates/tinyinference-llm/src/model/mod_tests.rs index 247ca459..acb95dfb 100644 --- a/crates/tinyinference-llm/src/model/mod_tests.rs +++ b/crates/tinyinference-llm/src/model/mod_tests.rs @@ -400,6 +400,7 @@ fn finish_preserves_provider_error_classification_from_provider_failed() { let mut acc = StreamAccumulator::new(); acc.push(&ModelStreamItem::Started); acc.push(&ModelStreamItem::ProviderFailed(ProviderError { + partial_response: None, provider: "openai".into(), status: Some(401), code: Some("invalid_api_key".into()), @@ -535,11 +536,13 @@ fn model_stream_item_roundtrips_every_variant() { // The scalar-carrying variant an internally tagged enum could not encode. roundtrip_stream_item(ModelStreamItem::Failed("boom".to_string())); roundtrip_stream_item(ModelStreamItem::ProviderFailed(ProviderError { + partial_response: None, provider: "openai".into(), message: "nope".into(), ..ProviderError::default() })); roundtrip_stream_item(ModelStreamItem::ProviderFailed(ProviderError { + partial_response: None, provider: "anthropic".into(), message: "overloaded".into(), stop_reason: Some("pause_turn".into()), diff --git a/crates/tinyinference-llm/src/model/output.rs b/crates/tinyinference-llm/src/model/output.rs new file mode 100644 index 00000000..6fa2d2bb --- /dev/null +++ b/crates/tinyinference-llm/src/model/output.rs @@ -0,0 +1,467 @@ +//! Ordered model output and execution metadata. + +use std::collections::BTreeMap; + +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +use crate::message::ContentBlock; +use crate::tool::ToolCall; + +/// A generated file owned by a provider response, not a local filesystem path. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct ModelFile { + /// Owning response identifier. + pub response_id: String, + /// File identifier, distinct from response ID and filename. + pub id: String, + /// Provider filename, treated as untrusted display metadata. + pub filename: String, + /// Reported byte length. + pub bytes: u64, + /// Provider creation timestamp in Unix seconds. + pub created_at: u64, +} + +/// Downloaded artifact content. The library never writes it to disk. +#[derive(Clone, PartialEq, Eq)] +pub struct ModelFileContent { + /// Owning response identifier. + pub response_id: String, + /// Downloaded file identifier. + pub file_id: String, + /// Content type reported by the provider. + pub media_type: Option, + /// Raw file bytes, bounded by the client's configured download limit. + pub bytes: Vec, +} + +impl std::fmt::Debug for ModelFileContent { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("ModelFileContent") + .field("byte_length", &self.bytes.len()) + .finish_non_exhaustive() + } +} + +/// Provider execution state, preserving future states without guessing success. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(from = "String", into = "String")] +pub enum ExecutionStatus { + /// The operation completed. + Completed, + /// The operation failed. + Failed, + /// The operation stopped with partial output. + Incomplete, + /// Accepted but not yet running. + Queued, + /// Work is in progress. + InProgress, + /// Cancellation was requested but not yet confirmed. + Cancelling, + /// Cancellation is confirmed. + Cancelled, + /// A provider state not yet understood by this version. + Other(String), +} + +impl From for ExecutionStatus { + fn from(value: String) -> Self { + match value.as_str() { + "completed" => Self::Completed, + "failed" => Self::Failed, + "incomplete" => Self::Incomplete, + "queued" => Self::Queued, + "in_progress" => Self::InProgress, + "cancelling" => Self::Cancelling, + "cancelled" => Self::Cancelled, + _ => Self::Other(value), + } + } +} + +impl From for String { + fn from(value: ExecutionStatus) -> Self { + match value { + ExecutionStatus::Completed => "completed".into(), + ExecutionStatus::Failed => "failed".into(), + ExecutionStatus::Incomplete => "incomplete".into(), + ExecutionStatus::Queued => "queued".into(), + ExecutionStatus::InProgress => "in_progress".into(), + ExecutionStatus::Cancelling => "cancelling".into(), + ExecutionStatus::Cancelled => "cancelled".into(), + ExecutionStatus::Other(value) => value, + } + } +} + +impl ExecutionStatus { + /// Whether the provider has reported a known terminal state. + pub fn is_terminal(&self) -> bool { + matches!( + self, + Self::Completed | Self::Failed | Self::Incomplete | Self::Cancelled + ) + } +} + +/// Structured execution metadata, separate from the assistant message identity. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct ModelExecution { + /// Provider response/run identifier used for retrieval and continuation. + pub id: String, + /// Actual model returned by the provider, not a preset name. + /// May be empty in a provisional stream snapshot before model selection. + pub model: String, + /// Current run status. + pub status: ExecutionStatus, + /// Explanation for an incomplete response, when supplied. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub incomplete_reason: Option, + /// Last provider event sequence number for explicit reconnect. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub sequence_number: Option, + /// Provider-reported costs, without locally computed prices. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub cost: Option, + /// Invocation counts and optional measured charges per hosted tool. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub tool_usage: BTreeMap, + /// Stream-only progress, retained even when absent from the final provider snapshot. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub progress: Vec, +} + +/// Reported decimal cost amounts. Strings preserve sub-micro-unit precision. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct ReportedCost { + /// Provider currency identifier. + pub currency: String, + /// Reported components: input, output, cache_read, cache_creation, tools, total. + pub components: BTreeMap, + /// Reported charges attributed to individual tools. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub tools: BTreeMap, +} + +/// Measured usage for one provider-executed tool. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct HostedToolUsage { + /// Invocation count, when supplied. + pub invocations: Option, + /// Provider-reported decimal USD charge, when supplied. + pub cost_usd: Option, +} + +/// A source annotation attached to its original content part. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(tag = "type", rename_all = "snake_case")] +pub enum OutputAnnotation { + /// A provider-supplied URL citation. Offsets are provider character units, + /// not UTF-8 bytes; callers must not use them to slice Rust strings. + UrlCitation { + /// `provider_characters` until the provider specifies an exact offset encoding. + offset_unit: String, + /// Source URL. + url: String, + /// Source title. + title: Option, + /// Original start offset, if supplied. + start: Option, + /// Original end offset, if supplied. + end: Option, + }, + /// An annotation this adapter cannot normalize without losing information. + Extension { + /// Provider identifier. + provider: String, + /// Original annotation data. + data: Value, + }, +} + +/// Content and its annotations, without merging citation coordinate spaces. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct AnnotatedContent { + /// Normalized text, reasoning, media, or extension block. + pub block: ContentBlock, + /// Annotations on this content part only. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub annotations: Vec, +} + +/// Source returned by web or people search. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct SearchSource { + /// Provider source identifier, scoped to its output item. + pub id: String, + /// Source URL. + pub url: String, + /// Source title. + pub title: String, + /// Retrieved snippet. + pub snippet: String, + /// Source category, when supplied. + pub source: Option, + /// Publication date in the provider's original representation. + pub date: Option, + /// Last-update date in the provider's original representation. + pub last_updated: Option, +} + +/// A hosted image-search match, not a downloaded image. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct ImageSearchResult { + /// Image URL. + pub image_url: String, + /// Page the image came from. + pub origin_url: String, + /// Reported pixel width. + pub width: u64, + /// Reported pixel height. + pub height: u64, + /// Image title, when supplied. + pub title: Option, +} + +/// Content fetched by a hosted URL tool. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct FetchedContent { + /// Requested URL. + pub url: String, + /// Page title. + pub title: String, + /// Retrieved content. + pub snippet: String, +} + +/// A finance-search result; original tables/content remain unmodified. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct FinanceResult { + /// Requested data category. + pub category: String, + /// Provider-rendered table or snippet. + pub content: String, + /// Associated ticker symbols. + #[serde(default)] + pub tickers: Vec, + /// Supporting source URLs. + #[serde(default)] + pub sources: Vec, +} + +/// A tool definition discovered remotely, never a request for local execution. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct RemoteToolDefinition { + /// Tool name. + pub name: String, + /// Tool description when supplied. + pub description: Option, + /// JSON Schema for the tool arguments; null means the provider omitted it. + pub parameters: Value, +} + +/// Hosted tool namespace returned by tool discovery/search. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct RemoteToolNamespace { + /// Namespace name. + pub name: String, + /// Namespace description. + pub description: Option, + /// Available tools. + pub tools: Vec, +} + +/// A provider output item, preserving ordering and item identity. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct ModelOutputItem { + /// Original zero-based output position. + pub index: usize, + /// Provider item ID, distinct from run and function call IDs. + pub id: Option, + /// Provider item status, when present. + pub status: Option, + /// Normalized item content. + pub kind: ModelOutputKind, +} + +/// Normalized output families. Hosted calls never become local function calls. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(tag = "type", rename_all = "snake_case")] +pub enum ModelOutputKind { + /// Assistant or provider message. + Message { + /// Original role. + role: String, + /// Ordered, individually annotated content parts. + content: Vec, + }, + /// Supplied reasoning content and opaque replay signatures. + Reasoning { + /// Ordered reasoning content. + content: Vec, + }, + /// Function request for the consuming application to execute. + FunctionCall { + /// Valid or explicitly invalid model call. + call: ToolCall, + }, + /// Web or people search results, grouped by invocation. + Search { + /// Search tool identity. + tool: String, + /// Queries used by the provider. + queries: Vec, + /// Sources from this invocation. + results: Vec, + }, + /// Hosted image search. + ImageSearch { + /// Search queries. + queries: Vec, + /// Returned images. + results: Vec, + /// Tool-local error, independent of overall response success. + error: Option, + }, + /// Hosted URL retrieval. + Fetch { + /// Retrieved pages. + contents: Vec, + }, + /// Hosted finance retrieval. + Finance { + /// Requested categories. + categories: Vec, + /// Requested symbols. + tickers: Vec, + /// Returned financial content and sources. + results: Vec, + }, + /// Execution inside the provider's sandbox. + Sandbox { + /// Code that was run, when supplied. + code: Option, + /// Standard output. + stdout: Option, + /// Standard error. + stderr: Option, + /// Process exit status. + exit_code: Option, + /// Provider-reported runtime. + duration_ms: Option, + /// Tool execution state, preserving unrecognized values. + status: String, + }, + /// Tools discovered on a remote server. + ToolDiscovery { + /// Remote server label. + server: String, + /// Managed connector ID, when applicable. + connector_id: Option, + /// Discovered definitions. + tools: Vec, + /// Discovery failure when supplied. + error: Option, + }, + /// A tool already executed by the provider on a remote server. + RemoteCall { + /// Remote server label. + server: String, + /// Managed connector ID, when applicable. + connector_id: Option, + /// Tool name. + name: String, + /// Original argument text, including malformed text if supplied. + arguments: String, + /// Returned tool content. + output: Option, + /// Tool-local failure. + error: Option, + }, + /// Hosted search for tool definitions. + ToolSearch { + /// Execution location as reported by the provider. + execution: String, + /// Call identifier when supplied. + call_id: Option, + /// Original search arguments. + arguments: Option, + /// Discovered namespaces. + tools: Vec, + }, + /// Notice of generated files. List files by response ID for authoritative descriptors. + GeneratedFiles { + /// Owning response ID. + response_id: String, + /// Unspecified provider notice fields, preserved without guessing meaning. + metadata: Value, + }, + /// Future provider output that cannot yet be normalized. + Extension { + /// Provider identifier. + provider: String, + /// Original item type. + item_type: String, + /// Original item data. + data: Value, + }, +} + +/// Rich stream updates, parallel to the existing text/tool/usage delta channels. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(tag = "type", content = "content", rename_all = "snake_case")] +pub enum ModelOutputEvent { + /// Progress that is not an indexed response item. + Progress(ModelProgress), + /// An output item opened before its final required fields were available. + ItemStarted { + /// Original output index. + index: usize, + /// Provider item ID, when supplied. + id: Option, + /// Original item type. + item_type: String, + }, + /// Current execution snapshot, including response ID and reconnect cursor. + Execution(Box), + /// Current item snapshot; replaces the same output index, never appends twice. + Item(Box), +} + +/// Non-item progress retained by stream collection and partial failures. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(tag = "type", content = "content", rename_all = "snake_case")] +pub enum ModelProgress { + /// Reasoning note supplied alongside a hosted-tool progress event. + ReasoningText { + /// Supplied text; never inferred from hidden reasoning. + text: String, + }, + /// Hosted results reported without an output index to upsert. + HostedActivity(Box), + /// Queries issued by a hosted tool before results arrive. + Queries { + /// Hosted tool name. + tool: String, + /// Queries in this event. + queries: Vec, + }, + /// Provider reasoning phase; no hidden reasoning text is inferred. + ReasoningPhase { + /// Whether reasoning started or stopped. + active: bool, + }, + /// Unrecognized provider event, preserved for observers. + Extension { + /// Provider event discriminator. + event_type: String, + /// Original event data. + data: Value, + }, +} + +#[cfg(test)] +#[path = "output_tests.rs"] +mod tests; diff --git a/crates/tinyinference-llm/src/model/output_tests.rs b/crates/tinyinference-llm/src/model/output_tests.rs new file mode 100644 index 00000000..84a61d9a --- /dev/null +++ b/crates/tinyinference-llm/src/model/output_tests.rs @@ -0,0 +1,81 @@ +use super::*; + +use crate::model::ModelResponse; +use serde_json::json; + +#[test] +fn old_responses_read_without_inventing_execution_or_rich_output() { + let response: ModelResponse = serde_json::from_value(json!({ + "message":{"content":[{"text":"hello"}],"tool_calls":[]} + })) + .unwrap(); + assert!(response.output.is_empty()); + assert!(response.execution.is_none()); + assert_eq!(response.text(), "hello"); + let encoded = serde_json::to_value(response).unwrap(); + assert!(encoded.get("execution").is_none()); + assert!(encoded.get("output").is_none()); +} + +#[test] +fn future_statuses_survive_serialization_without_becoming_success() { + let status: ExecutionStatus = serde_json::from_value(json!("waiting_for_review")).unwrap(); + assert!(!status.is_terminal()); + assert_eq!( + serde_json::to_value(status).unwrap(), + json!("waiting_for_review") + ); +} + +#[test] +fn deferred_cancellation_serializes_with_or_without_a_snapshot() { + use crate::model::DeferredStatus; + for response in [None, Some(Box::new(ModelResponse::assistant("partial")))] { + let status = DeferredStatus::Cancelled { response }; + let value = serde_json::to_value(status).unwrap(); + assert_eq!(value["status"], "cancelled"); + assert!(matches!( + serde_json::from_value::(value).unwrap(), + DeferredStatus::Cancelled { .. } + )); + } +} + +#[test] +fn reconstruction_preserves_progress_supplied_in_execution_snapshots() { + use crate::model::{ModelStreamItem, StreamAccumulator}; + let reported = ModelProgress::ReasoningPhase { active: true }; + let later = ModelProgress::ReasoningText { + text: "working".into(), + }; + for snapshot_progress in [vec![reported.clone()], Vec::new()] { + for separate_progress in [vec![later.clone()], Vec::new()] { + let execution = ModelExecution { + id: "resp_1".into(), + model: "provider/model".into(), + status: ExecutionStatus::InProgress, + incomplete_reason: None, + sequence_number: Some(3), + cost: None, + tool_usage: Default::default(), + progress: snapshot_progress.clone(), + }; + let mut accumulator = StreamAccumulator::new(); + accumulator.push(&ModelStreamItem::OutputEvent(ModelOutputEvent::Execution( + Box::new(execution), + ))); + for progress in &separate_progress { + accumulator.push(&ModelStreamItem::OutputEvent(ModelOutputEvent::Progress( + progress.clone(), + ))); + } + let response = accumulator.finish().unwrap(); + let expected = if snapshot_progress.is_empty() { + separate_progress + } else { + snapshot_progress.clone() + }; + assert_eq!(response.execution.unwrap().progress, expected); + } + } +} diff --git a/crates/tinyinference-llm/src/model/types.rs b/crates/tinyinference-llm/src/model/types.rs index 29bbd5ff..9ab5f335 100644 --- a/crates/tinyinference-llm/src/model/types.rs +++ b/crates/tinyinference-llm/src/model/types.rs @@ -813,8 +813,14 @@ pub struct ModelRequest { } /// A provider-neutral chat model response. -#[derive(Clone, Debug, Serialize, Deserialize)] +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] pub struct ModelResponse { + /// Complete ordered provider output. Empty for legacy providers. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub output: Vec, + /// Execution identity, status and detailed provider usage. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub execution: Option, /// The assistant message produced by the model. pub message: AssistantMessage, /// Token usage, when reported. @@ -883,6 +889,9 @@ pub struct ModelDelta { /// in [`ProviderError::raw`]. #[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] pub struct ProviderError { + /// Full partial response, including hosted output and recovery identifiers. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub partial_response: Option>, /// Provider family identifier, for example `openai` or `ollama`. pub provider: String, /// Provider model id, when known. @@ -986,6 +995,8 @@ pub enum BlockDelta { reason = "the serialized stream contract keeps Completed inline; boxing it would be a needless allocation on every terminal response" )] pub enum ModelStreamItem { + /// Rich execution, hosted-tool, citation or source output update. + OutputEvent(super::ModelOutputEvent), /// The stream has opened; no content has arrived yet. Started, /// An incremental message fragment (text and/or a tool-call fragment). @@ -1082,6 +1093,14 @@ impl DeferredHandle { #[derive(Clone, Debug, Serialize, Deserialize)] #[serde(rename_all = "snake_case", tag = "status")] pub enum DeferredStatus { + /// A structured terminal provider failure, retaining partial output. + ProviderFailed(Box), + /// Confirmed cancellation, with the final snapshot if available. + Cancelled { + /// Final response snapshot, when the provider supplies one. + #[serde(default, skip_serializing_if = "Option::is_none")] + response: Option>, + }, /// Still queued or in progress; not yet ready. Pending, /// Finished successfully. diff --git a/crates/tinyinference-llm/src/providers/anthropic/mod.rs b/crates/tinyinference-llm/src/providers/anthropic/mod.rs index 6b6e2fc8..1f15cb6f 100644 --- a/crates/tinyinference-llm/src/providers/anthropic/mod.rs +++ b/crates/tinyinference-llm/src/providers/anthropic/mod.rs @@ -383,6 +383,7 @@ impl AnthropicModel { crate::failure::classify_provider_failure(status, code.as_deref(), &message) .is_retryable(); ProviderError { + partial_response: None, provider: PROVIDER.to_string(), model: Some(self.request_model(request).to_string()), status, diff --git a/crates/tinyinference-llm/src/providers/anthropic/mod_tests.rs b/crates/tinyinference-llm/src/providers/anthropic/mod_tests.rs index 145a8f47..f4f6f267 100644 --- a/crates/tinyinference-llm/src/providers/anthropic/mod_tests.rs +++ b/crates/tinyinference-llm/src/providers/anthropic/mod_tests.rs @@ -196,6 +196,7 @@ fn assistant_tool_calls_become_tool_use_blocks() { #[test] fn tool_results_use_anthropic_tool_result_blocks() { let request = ModelRequest::new(vec![Message::Tool(ToolMessage { + call_context: None, tool_call_id: "tool_1".into(), content: vec![ContentBlock::Text("42".into())], trusted_verbatim: false, @@ -235,6 +236,7 @@ fn custom_messages_are_never_sent_to_the_provider() { fn consecutive_tool_results_merge_into_one_user_message() { let tool = |id: &str| { Message::Tool(ToolMessage { + call_context: None, tool_call_id: id.into(), content: vec![ContentBlock::Text("ok".into())], trusted_verbatim: false, @@ -261,6 +263,7 @@ fn consecutive_tool_results_merge_into_one_user_message() { fn empty_tool_result_sends_an_empty_string() { let body = request_body( &ModelRequest::new(vec![Message::Tool(ToolMessage { + call_context: None, tool_call_id: "t".into(), content: vec![], trusted_verbatim: false, diff --git a/crates/tinyinference-llm/src/providers/anthropic/response.rs b/crates/tinyinference-llm/src/providers/anthropic/response.rs index 27e4ed0d..651d242a 100644 --- a/crates/tinyinference-llm/src/providers/anthropic/response.rs +++ b/crates/tinyinference-llm/src/providers/anthropic/response.rs @@ -74,6 +74,8 @@ pub(crate) fn parse_response(body: Value) -> Result { } } Ok(ModelResponse { + output: Vec::new(), + execution: None, message: AssistantMessage { id: Some(id), content, diff --git a/crates/tinyinference-llm/src/providers/anthropic/stream.rs b/crates/tinyinference-llm/src/providers/anthropic/stream.rs index 4c0037cc..abee2afb 100644 --- a/crates/tinyinference-llm/src/providers/anthropic/stream.rs +++ b/crates/tinyinference-llm/src/providers/anthropic/stream.rs @@ -347,6 +347,7 @@ impl AnthropicStreamAcc { crate::failure::classify_provider_failure(None, code.as_deref(), &message) .is_retryable(); return Err(Error::Provider(Box::new(ProviderError { + partial_response: None, provider: PROVIDER.to_string(), code, message, @@ -424,6 +425,8 @@ impl AnthropicStreamAcc { }); let usage = self.usage; ModelResponse { + output: Vec::new(), + execution: None, message: AssistantMessage { id: self.id, content, @@ -486,6 +489,7 @@ impl SseState { let retryable = crate::failure::classify_provider_failure(None, None, &message).is_retryable(); ProviderError { + partial_response: None, provider: PROVIDER.to_string(), message, retryable, diff --git a/crates/tinyinference-llm/src/providers/mock.rs b/crates/tinyinference-llm/src/providers/mock.rs index ecfd30d2..ba124755 100644 --- a/crates/tinyinference-llm/src/providers/mock.rs +++ b/crates/tinyinference-llm/src/providers/mock.rs @@ -228,6 +228,7 @@ impl ChatModel for MockModel { MockBehavior::ToolCall { name, arguments } => { let tool_call = ToolCall { + replay: None, id: format!("mock-tool-{call_id}"), name: name.clone(), arguments: arguments.clone(), @@ -242,6 +243,8 @@ impl ChatModel for MockModel { origin: None, }; ModelResponse { + output: Vec::new(), + execution: None, message, usage: Some(usage), finish_reason: Some("tool_calls".to_string()), @@ -350,6 +353,8 @@ impl MockModel { let s = text.into(); let output_tokens = estimate_output_tokens(&s); ModelResponse { + output: Vec::new(), + execution: None, message: AssistantMessage { id: None, content: vec![ContentBlock::Text(s)], diff --git a/crates/tinyinference-llm/src/providers/mod.rs b/crates/tinyinference-llm/src/providers/mod.rs index 56ce46dd..4f7839ea 100644 --- a/crates/tinyinference-llm/src/providers/mod.rs +++ b/crates/tinyinference-llm/src/providers/mod.rs @@ -15,6 +15,7 @@ //! | [`MockModel`] | Implemented — deterministic, no network | //! | [`openai`] (and OpenAI-compatible endpoints) | Implemented | //! | [`anthropic`] (Messages API, including prompt caching) | Implemented | +//! | [`perplexity`] (Agent API with model/preset selection) | Implemented | //! //! [`MockModel`] is always compiled and needs no network, keeping the default //! build offline and deterministic. The [`openai`] module is always compiled @@ -25,8 +26,8 @@ //! adapter only touches the network when invoked, and the live tests //! early-return without `OPENAI_API_KEY`. //! -//! To add a provider with a different wire protocol, gate it behind a new -//! Cargo feature and add the corresponding module declaration: +//! Native providers using existing dependencies are compiled alongside the +//! compatible transport. Construction performs no network calls: //! //! ```text //! pub mod openai; // always compiled @@ -39,11 +40,11 @@ mod types; // --- real provider integrations --- // The OpenAI Chat Completions adapter is always compiled; it also serves every -// OpenAI-compatible endpoint. Providers with a different wire protocol would be -// added behind their own Cargo feature. +// OpenAI-compatible endpoint. Native providers own their protocol translation. pub mod anthropic; pub mod omission; pub mod openai; +pub mod perplexity; // #[cfg(feature = "ollama")] pub mod ollama; pub use types::*; diff --git a/crates/tinyinference-llm/src/providers/openai/convert.rs b/crates/tinyinference-llm/src/providers/openai/convert.rs index 6d89bb59..4e57e61c 100644 --- a/crates/tinyinference-llm/src/providers/openai/convert.rs +++ b/crates/tinyinference-llm/src/providers/openai/convert.rs @@ -441,6 +441,8 @@ pub(super) fn parse_chat_response( }; Ok(ModelResponse { + output: Vec::new(), + execution: None, message, usage, finish_reason: choice.finish_reason, @@ -496,6 +498,7 @@ pub(super) fn tool_call_from_wire( let call_id = tool_call_id(slot, id); match parse_tool_arguments(raw) { Ok(arguments) => ToolCall { + replay: None, id: call_id, name: name.to_string(), arguments, diff --git a/crates/tinyinference-llm/src/providers/openai/mod_tests.rs b/crates/tinyinference-llm/src/providers/openai/mod_tests.rs index ea01616f..a60a65e6 100644 --- a/crates/tinyinference-llm/src/providers/openai/mod_tests.rs +++ b/crates/tinyinference-llm/src/providers/openai/mod_tests.rs @@ -219,6 +219,7 @@ fn translates_assistant_tool_calls_to_stringified_arguments() { id: Some("msg-1".to_string()), content: Vec::new(), tool_calls: vec![ToolCall { + replay: None, id: "call-1".to_string(), name: "get_weather".to_string(), arguments: json!({ "city": "Paris" }), @@ -258,6 +259,7 @@ fn translates_assistant_tool_calls_to_stringified_arguments() { #[test] fn translates_structured_tool_result_content() { let request = ModelRequest::new(vec![Message::Tool(crate::message::ToolMessage { + call_context: None, tool_call_id: "call-1".into(), content: vec![ContentBlock::Json(json!({"temperature": 21}))], trusted_verbatim: false, @@ -721,6 +723,7 @@ fn provider_failed_stream_item_finishes_as_provider_error() { // stringified it into `Model` and always retried it as transient. let mut accumulator = StreamAccumulator::new(); accumulator.push(&ModelStreamItem::ProviderFailed(ProviderError { + partial_response: None, provider: "groq".to_string(), model: Some("llama-3.3-70b-versatile".to_string()), status: Some(429), @@ -3166,6 +3169,8 @@ fn stamp_origin_records_provider_api_and_effective_model() { let model = OpenAiModel::new("key").with_model("gpt-4.1"); let request = ModelRequest::default(); let mut response = ModelResponse { + output: Vec::new(), + execution: None, message: crate::message::AssistantMessage { id: None, content: Vec::new(), @@ -3197,6 +3202,8 @@ fn stamp_origin_prefers_a_per_request_model_override() { ..Default::default() }; let mut response = ModelResponse { + output: Vec::new(), + execution: None, message: crate::message::AssistantMessage { id: None, content: Vec::new(), diff --git a/crates/tinyinference-llm/src/providers/openai/responses.rs b/crates/tinyinference-llm/src/providers/openai/responses.rs index 92eed216..ee4af66a 100644 --- a/crates/tinyinference-llm/src/providers/openai/responses.rs +++ b/crates/tinyinference-llm/src/providers/openai/responses.rs @@ -636,6 +636,8 @@ pub(super) fn parse_responses_response(value: Value) -> ModelResponse { _ => Some("stop".to_string()), }; ModelResponse { + output: Vec::new(), + execution: None, message: AssistantMessage { id: None, content, diff --git a/crates/tinyinference-llm/src/providers/openai/sse.rs b/crates/tinyinference-llm/src/providers/openai/sse.rs index aba4e3b6..e2a1a70e 100644 --- a/crates/tinyinference-llm/src/providers/openai/sse.rs +++ b/crates/tinyinference-llm/src/providers/openai/sse.rs @@ -502,6 +502,8 @@ impl OpenAiStreamAcc { origin: None, }; ModelResponse { + output: Vec::new(), + execution: None, message, usage: self.usage, finish_reason: self.finish_reason, @@ -617,6 +619,7 @@ impl SseState { let Ok(value) = serde_json::from_str::(payload) else { if payload.starts_with('{') || payload.starts_with('[') { let item = self.provider_failure(ProviderError { + partial_response: None, message: "provider returned malformed SSE JSON".into(), retryable: false, raw: Some(Value::String(payload.into())), @@ -675,6 +678,7 @@ impl SseState { crate::failure::classify_provider_failure(status, code.as_deref(), &message) .is_retryable(); ProviderError { + partial_response: None, provider: self.provider.clone(), model: Some(self.model.clone()), status, @@ -740,6 +744,7 @@ pub(super) async fn sse_next(mut state: SseState) -> Option<(ModelStreamItem, Ss } Some(Err(error)) => { let item = state.provider_failure(ProviderError { + partial_response: None, message: error.to_string(), retryable: true, ..ProviderError::default() @@ -754,6 +759,7 @@ pub(super) async fn sse_next(mut state: SseState) -> Option<(ModelStreamItem, Ss state.finished = true; } else { let item = state.provider_failure(ProviderError { + partial_response: None, message: "provider stream ended before a completion signal".into(), retryable: true, ..ProviderError::default() diff --git a/crates/tinyinference-llm/src/providers/openai/transport.rs b/crates/tinyinference-llm/src/providers/openai/transport.rs index bec34d51..0a068c20 100644 --- a/crates/tinyinference-llm/src/providers/openai/transport.rs +++ b/crates/tinyinference-llm/src/providers/openai/transport.rs @@ -733,6 +733,11 @@ impl OpenAiModel { /// Builds an OpenAI-compatible model from a provider spec and explicit API /// key. pub fn from_spec(spec: ProviderSpec, api_key: impl Into) -> Result { + if spec.kind == crate::providers::ProviderKind::Perplexity { + return Err(Error::Unsupported( + "use PerplexityModel for the native Agent API".into(), + )); + } if spec.model.trim().is_empty() { return Err(Error::Validation( "provider spec model must not be empty".to_string(), @@ -1870,6 +1875,7 @@ impl OpenAiModel { crate::failure::classify_provider_failure(status, code.as_deref(), &message) .is_retryable(); ProviderError { + partial_response: None, provider: self.provider.clone(), model: Some(self.model.clone()), status, diff --git a/crates/tinyinference-llm/src/providers/perplexity/config.rs b/crates/tinyinference-llm/src/providers/perplexity/config.rs new file mode 100644 index 00000000..d92a18ba --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/config.rs @@ -0,0 +1,370 @@ +//! Perplexity selection and request configuration. + +use crate::{Error, Result, model::ModelRequest}; +use serde::{Deserialize, Serialize}; +use std::{collections::BTreeMap, fmt, time::Duration}; + +/// Explicit model or server-managed preset selection. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde( + tag = "type", + content = "value", + rename_all = "snake_case", + deny_unknown_fields +)] +pub enum PerplexitySelection { + /// Provider-qualified model ID. + Model(String), + /// Provider-managed fallback chain, in priority order (one to five IDs). + Models(Vec), + /// A dynamic preset with optional model override. + Preset { + /// Preset name; new provider presets do not require a library update. + name: String, + /// Optional explicit model override. + model: Option, + /// Optional fallback override, mutually exclusive with `model`. + models: Option>, + }, +} + +impl PerplexitySelection { + /// Selects an unmodified server-managed preset. + pub fn preset(name: impl Into) -> Self { + Self::Preset { + name: name.into(), + model: None, + models: None, + } + } + + pub(super) fn validate(&self) -> Result<()> { + let valid_model = |model: &str| { + !model.trim().is_empty() + && model + .split_once('/') + .is_some_and(|(a, b)| !a.is_empty() && !b.is_empty()) + }; + let valid_models = |models: &[String]| { + (1..=5).contains(&models.len()) && models.iter().all(|model| valid_model(model)) + }; + let valid = match self { + Self::Model(model) => valid_model(model), + Self::Models(models) => valid_models(models), + Self::Preset { + name, + model, + models, + } => { + !name.trim().is_empty() + && !(model.is_some() && models.is_some()) + && model.as_deref().is_none_or(valid_model) + && models.as_deref().is_none_or(valid_models) + } + }; + if valid { + Ok(()) + } else { + Err(Error::Validation( + "invalid Perplexity model/preset selection".into(), + )) + } + } +} + +/// Search filtering; include and exclude domain modes cannot be mixed. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PerplexitySearchFilters { + /// At most twenty domains or URLs, with `-` prefixes for exclusion. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub search_domain_filter: Option>, + /// Publication start date in provider MM/DD/YYYY format. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub search_after_date_filter: Option, + /// Publication end date. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub search_before_date_filter: Option, + /// Last-update start date. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_updated_after_filter: Option, + /// Last-update end date. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_updated_before_filter: Option, + /// Relative age: day, week, month, or year. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub search_recency_filter: Option, +} + +/// Geographic context for hosted web search. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PerplexityLocation { + /// Two-letter country code; required with coordinates. + pub country: String, + /// Optional region. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub region: Option, + /// Optional city. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub city: Option, + /// Latitude, paired with longitude. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub latitude: Option, + /// Longitude, paired with latitude. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub longitude: Option, +} + +/// Web-search settings; unset fields inherit the provider/preset defaults. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PerplexityWebSearch { + /// Standard `web` search or lower-latency `fast` search. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub search_type: Option, + /// Named context budget: low, medium, high. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub search_context_size: Option, + /// Maximum results per invocation, from 1 to 50. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_results: Option, + /// Total search context token budget. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_tokens: Option, + /// Token budget per page. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_tokens_per_page: Option, + /// Domain and date filters. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub filters: Option, + /// Geographic context. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub user_location: Option, +} + +/// Image-search filters. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PerplexityImageFilters { + /// Domain allowlist or exclusion list. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub domain_filter: Option>, + /// Permitted image formats. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub format_filter: Option>, + /// Safe-search setting. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub safe_search: Option, +} + +/// Provider-executed tools. Custom local functions use `ModelRequest.tools`. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(tag = "type", rename_all = "snake_case", deny_unknown_fields)] +pub enum PerplexityTool { + /// Hosted web search. + WebSearch { + /// Optional search settings. + #[serde(flatten)] + options: Box, + }, + /// Hosted image search. + ImageSearch { + /// Maximum results, from 1 to 30. + #[serde(default, skip_serializing_if = "Option::is_none")] + max_results: Option, + /// Optional filters. + #[serde(default, skip_serializing_if = "Option::is_none")] + filters: Option, + }, + /// Hosted URL content retrieval. + FetchUrl { + /// Maximum URLs per invocation, from 1 to 10. + #[serde(default, skip_serializing_if = "Option::is_none")] + max_urls: Option, + }, + /// Financial data retrieval. + FinanceSearch, + /// People search. + PeopleSearch, + /// Hosted code execution, never executed locally by this crate. + Sandbox, + /// Remote MCP execution. Credentials are resolved by server label from config. + Mcp { + /// Unique label, 1–64 ASCII letters/digits/underscores/hyphens. + server_label: String, + /// HTTPS Streamable HTTP server URL. + server_url: String, + /// Explicit allowed tools; empty exposes all discovered tools. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + allowed_tools: Vec, + /// Ask the provider to load tool definitions on demand. + #[serde(default, skip_serializing_if = "Option::is_none")] + defer_loading: Option, + }, + /// An existing provider-managed connector; this does not create one. + Connector { + /// Existing connector ID. + id: String, + /// Unique tool namespace label. + server_label: String, + /// Optional namespace description. + #[serde(default, skip_serializing_if = "Option::is_none")] + server_description: Option, + /// Explicit allowed tools; empty exposes all live tools. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + allowed_tools: Vec, + }, +} + +/// Explicit override of a preset's tool-choice default. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "type", content = "name", rename_all = "snake_case")] +pub enum PerplexityToolChoice { + /// Provider chooses. + Auto, + /// Disable tool calls. + None, + /// Require a tool call. + Required, + /// Require a named client-executed function. + Function(String), + /// Require a hosted tool type, such as image_search. + Hosted(String), +} + +/// Typed per-call overrides. Credentials never belong in this serializable type. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PerplexityOptions { + /// Optional per-call model/preset selection. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub selection: Option, + /// Hosted-tool overrides, merged by the provider with preset tools. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tools: Option>, + /// Explicit tool-choice override; absence preserves preset defaults. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tool_choice: Option, + /// Maximum agent steps, from 1 to 100. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_steps: Option, + /// Storage visibility. False does not disable persistence or continuation. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub store: Option, + /// Background execution; use submit_background or streaming for this mode. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub background: Option, + /// Optional output cap; the common request's max_tokens wins. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_output_tokens: Option, + /// ISO 639-1 language preference. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub language_preference: Option, + /// Provider reasoning effort, including its additional `max` level. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub reasoning_effort: Option, +} + +impl PerplexityOptions { + /// Sets the provider options on a common model request. + /// + /// # Errors + /// Returns validation errors for invalid hosted tools, or a serialization + /// error if options cannot be encoded. Leaves the request unchanged on error. + pub fn apply_to(&self, request: &mut ModelRequest) -> Result<()> { + super::request::validate_options_before_serialization(self)?; + request.provider_options = serde_json::json!({"perplexity": serde_json::to_value(self)?}); + Ok(()) + } +} + +impl PerplexityTool { + /// Creates a hosted web-search tool with explicit optional settings. + pub fn web_search(options: PerplexityWebSearch) -> Self { + Self::WebSearch { + options: Box::new(options), + } + } +} + +/// Remote-server secrets, kept out of model request serialization and Debug. +#[derive(Clone, Default)] +pub struct PerplexityRemoteCredentials { + /// HTTPS server URL these credentials are authorized for. + pub server_url: String, + /// Raw remote access token, not a prefixed bearer header. + pub authorization: Option, + /// Extra remote request headers. + pub headers: BTreeMap, +} + +impl fmt::Debug for PerplexityRemoteCredentials { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("PerplexityRemoteCredentials([REDACTED])") + } +} + +/// Immutable settings for a Perplexity Agent client. +#[derive(Clone)] +pub struct PerplexityConfig { + /// Required model or preset selection. + pub selection: PerplexitySelection, + /// API base URL, without the create route. + pub base_url: String, + /// Use `/responses` for creation instead of the canonical `/agent` route. + pub responses_alias: bool, + /// Default options; typed per-call values override them. + pub options: PerplexityOptions, + /// Total request/stream budget, including retries. + pub timeout: Duration, + /// Maximum inactivity while waiting for stream bytes. + pub idle_timeout: Duration, + /// Connection-establishment budget. + pub connect_timeout: Duration, + /// Bounded retry count; zero disables retries. + pub max_retries: u32, + /// Maximum request, JSON response, or reconstructed output bytes. + pub max_body_bytes: usize, + /// Maximum bytes in one SSE event. + pub max_event_bytes: usize, + /// Maximum bytes downloaded for one generated file. + pub max_file_bytes: usize, + /// Remote secrets by server label, injected only into outgoing requests. + pub remote_credentials: BTreeMap, + /// Optional last-mile payload callback. The result is validated before sending. + pub on_payload: Option, + /// Optional observer of sanitized provider replies. + pub on_response: Option, +} + +impl PerplexityConfig { + /// Creates configuration with bounded defaults and explicit selection. + pub fn new(selection: PerplexitySelection) -> Self { + Self { + selection, + base_url: "https://api.perplexity.ai/v1".into(), + responses_alias: false, + options: PerplexityOptions::default(), + timeout: Duration::from_secs(600), + idle_timeout: Duration::from_secs(60), + connect_timeout: Duration::from_secs(30), + max_retries: 3, + max_body_bytes: 32 * 1024 * 1024, + max_event_bytes: 4 * 1024 * 1024, + max_file_bytes: 64 * 1024 * 1024, + remote_credentials: BTreeMap::new(), + on_payload: None, + on_response: None, + } + } +} + +impl fmt::Debug for PerplexityConfig { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("PerplexityConfig") + .field("responses_alias", &self.responses_alias) + .field("timeout", &self.timeout) + .finish_non_exhaustive() + } +} diff --git a/crates/tinyinference-llm/src/providers/perplexity/lifecycle.rs b/crates/tinyinference-llm/src/providers/perplexity/lifecycle.rs new file mode 100644 index 00000000..7a09f3b2 --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/lifecycle.rs @@ -0,0 +1,354 @@ +use super::{ + PerplexityModel, response, stream, + transport::{self, Operation}, + types::FileList, +}; +use crate::model::{ + DeferredHandle, ExecutionStatus, ModelFile, ModelFileContent, ModelRequest, ModelResponse, + ModelStream, ModelStreamItem, +}; +use crate::{Error, Result}; +use serde_json::json; +use std::collections::HashSet; +use tokio::time::Instant; + +impl PerplexityModel { + /// Submits one durable run without waiting for its result. + /// + /// Persist the returned handle in the caller. Submission is billable; + /// interrupted acknowledgement is not automatically resubmitted. + /// + /// # Errors + /// Returns validation, network, timeout, or provider errors. Explicit + /// `store: false` is rejected because durable recovery requires retrieval. + pub async fn submit_background(&self, request: ModelRequest) -> Result { + crate::network_guard::ensure_network_models_allowed()?; + let deadline = self.inner.deadline(request.timeout_ms)?; + let body = self.body(&request, false, true)?; + let outgoing = + self.inner + .request(reqwest::Method::POST, self.create_path(), Some(&body))?; + let incoming = self + .inner + .send(outgoing, Operation::Create, deadline) + .await?; + let snapshot = response::parse(self.inner.json(incoming, deadline).await?)? + .inherit_correlation(request.correlation); + match snapshot.execution.as_ref().map(|e| &e.status) { + Some( + ExecutionStatus::Queued + | ExecutionStatus::InProgress + | ExecutionStatus::Completed + | ExecutionStatus::Incomplete, + ) => { + let mut handle = self.response_handle(&snapshot)?; + handle.kind = Some("background".into()); + Ok(handle) + } + _ => response::completed(snapshot) + .and_then(|_| Err(response::malformed("invalid background acknowledgement"))), + } + } + + /// Creates a serializable, endpoint-scoped handle for an existing response. + /// + /// # Errors + /// Rejects responses without Perplexity origin or a nonblank execution ID. + pub fn response_handle(&self, response: &ModelResponse) -> Result { + let execution = response + .execution + .as_ref() + .ok_or_else(|| Error::Validation("response has no execution identity".into()))?; + if !response + .message + .origin + .as_ref() + .is_some_and(|o| o.provider == "perplexity" && o.api == "agent") + { + return Err(Error::Validation( + "response belongs to another provider".into(), + )); + } + validate_id(&execution.id)?; + let mut handle = DeferredHandle::new("perplexity", &execution.id); + handle.kind = Some("response".into()); + handle + .metadata + .insert("base_url".into(), json!(self.inner.config.base_url)); + if let Some(correlation) = &response.correlation { + handle + .metadata + .insert("correlation".into(), serde_json::to_value(correlation)?); + } + Ok(handle) + } + + /// Retrieves one snapshot without polling or creating another run. + /// + /// # Errors + /// Rejects foreign handles and propagates retrieval/decoding failures. + /// A failed or incomplete run remains visible as an explicit snapshot status. + pub async fn retrieve_response(&self, handle: &DeferredHandle) -> Result { + self.retrieve_at(handle, self.inner.deadline(None)?).await + } + + async fn retrieve_at( + &self, + handle: &DeferredHandle, + deadline: Instant, + ) -> Result { + self.validate_handle(handle)?; + let outgoing = + self.inner + .request(reqwest::Method::GET, &path(&["agent", &handle.id])?, None)?; + let incoming = self.inner.send(outgoing, Operation::Read, deadline).await?; + let mut snapshot = response::parse(self.inner.json(incoming, deadline).await?)?; + if snapshot + .execution + .as_ref() + .is_none_or(|execution| execution.id != handle.id) + { + return Err(response::with_partial( + response::malformed("retrieval returned a different response id"), + snapshot, + )); + } + if let Some(value) = handle.metadata.get("correlation") { + snapshot.correlation = Some( + serde_json::from_value(value.clone()) + .map_err(|_| Error::Validation("invalid handle correlation".into()))?, + ); + } + Ok(snapshot) + } + + /// Reconnects to a durable stream after a previously observed sequence number. + /// + /// Only new events are emitted; the final response snapshot is complete. + /// An already-terminal run is returned directly. An expired reconnect window + /// produces an error; call [`Self::retrieve_response`] to inspect the run. + /// + /// # Errors + /// Rejects non-background/foreign handles and propagates retrieval/reconnect + /// errors. Never submits a replacement run. + pub async fn resume_background( + &self, + handle: &DeferredHandle, + after: Option, + ) -> Result { + self.require_background(handle)?; + let deadline = self.inner.deadline(None)?; + let snapshot = self.retrieve_at(handle, deadline).await?; + let execution = snapshot + .execution + .clone() + .ok_or_else(|| response::malformed("response has no execution metadata"))?; + if execution.status.is_terminal() { + let terminal = match response::completed(snapshot) { + Ok(result) => ModelStreamItem::Completed(result), + Err(Error::Provider(error)) => ModelStreamItem::ProviderFailed(*error), + Err(error) => return Err(error), + }; + return Ok(ModelStream::new(Box::pin(futures::stream::iter([ + ModelStreamItem::Started, + terminal, + ])))); + } + if !matches!( + execution.status, + ExecutionStatus::Queued | ExecutionStatus::InProgress | ExecutionStatus::Cancelling + ) { + return Err(response::with_partial( + response::malformed("unknown background state"), + snapshot, + )); + } + let mut outgoing = + self.inner + .request(reqwest::Method::GET, &path(&["agent", &handle.id])?, None)?; + outgoing + .url_mut() + .query_pairs_mut() + .append_pair("stream", "true"); + if let Some(after) = after { + outgoing + .url_mut() + .query_pairs_mut() + .append_pair("starting_after", &after.to_string()); + } + let incoming = match self.inner.send(outgoing, Operation::Read, deadline).await { + Ok(incoming) => incoming, + Err(Error::Provider(mut error)) if error.status == Some(400) => { + error.code = Some("reconnect_unavailable".into()); + error.message = + "reconnect unavailable; retrieve the existing response instead of resubmitting" + .into(); + return Err(response::with_partial(Error::Provider(error), snapshot)); + } + Err(error) => return Err(response::with_partial(error, snapshot)), + }; + let mut result = stream::open( + self.inner.clone(), + incoming, + deadline, + Some(execution), + after, + )?; + if let Some(correlation) = snapshot.correlation { + result = result.with_correlation(correlation); + } + Ok(result) + } + + /// Requests remote cancellation; `Cancelling` is acknowledgement, not completion. + /// + /// Poll afterward to observe `Cancelled`. Dropping a local stream does not + /// implicitly call this endpoint. + /// + /// # Errors + /// Rejects foreign/non-background handles and propagates cancel errors, + /// including already-terminal (400) and unknown (404) responses. No auto retry. + pub async fn cancel_background(&self, handle: &DeferredHandle) -> Result { + self.require_background(handle)?; + let deadline = self.inner.deadline(None)?; + let outgoing = self.inner.request( + reqwest::Method::POST, + &path(&["agent", &handle.id, "cancel"])?, + None, + )?; + let incoming = self + .inner + .send(outgoing, Operation::Cancel, deadline) + .await?; + let value = self.inner.json(incoming, deadline).await?.value; + if value.get("response_id").and_then(serde_json::Value::as_str) != Some(&handle.id) + || value.get("status").and_then(serde_json::Value::as_str) != Some("cancelling") + { + return Err(response::malformed("invalid cancellation acknowledgement")); + } + Ok(ExecutionStatus::Cancelling) + } + + /// Lists authoritative file descriptors for this response without downloading them. + /// + /// # Errors + /// Rejects foreign handles, invalid descriptors, duplicate IDs or provider errors. + pub async fn list_response_files(&self, handle: &DeferredHandle) -> Result> { + self.validate_handle(handle)?; + let deadline = self.inner.deadline(None)?; + let outgoing = self.inner.request( + reqwest::Method::GET, + &path(&["agent", &handle.id, "files"])?, + None, + )?; + let incoming = self.inner.send(outgoing, Operation::Read, deadline).await?; + let list: FileList = + serde_json::from_value(self.inner.json(incoming, deadline).await?.value) + .map_err(|_| response::malformed("invalid file list"))?; + let mut ids = HashSet::new(); + list.data + .into_iter() + .map(|file| { + validate_id(&file.id)?; + if !ids.insert(file.id.clone()) { + return Err(response::malformed("duplicate file id")); + } + Ok(ModelFile { + response_id: handle.id.clone(), + id: file.id, + filename: file.filename, + bytes: file.bytes, + created_at: file.created_at, + }) + }) + .collect() + } + + /// Downloads one existing file into bounded memory. Filenames are never used as paths. + /// + /// # Errors + /// Rejects foreign handles or invalid file IDs; returns provider, size, or + /// deadline errors without regenerating the original response. + pub async fn download_response_file( + &self, + handle: &DeferredHandle, + file_id: &str, + ) -> Result { + self.validate_handle(handle)?; + validate_id(file_id)?; + let deadline = self.inner.deadline(None)?; + let outgoing = self.inner.request( + reqwest::Method::GET, + &path(&["agent", &handle.id, "files", file_id, "content"])?, + None, + )?; + let incoming = self.inner.send(outgoing, Operation::Read, deadline).await?; + let media_type = incoming + .headers() + .get(reqwest::header::CONTENT_TYPE) + .and_then(|v| v.to_str().ok()) + .map(str::to_owned); + let bytes = tokio::time::timeout_at( + deadline, + transport::read_limited(incoming, self.inner.config.max_file_bytes), + ) + .await + .map_err(|_| transport::timeout())??; + Ok(ModelFileContent { + response_id: handle.id.clone(), + file_id: file_id.into(), + media_type, + bytes, + }) + } + + fn validate_handle(&self, handle: &DeferredHandle) -> Result<()> { + if handle.provider != "perplexity" + || !matches!(handle.kind.as_deref(), Some("response" | "background")) + || handle + .metadata + .get("base_url") + .and_then(serde_json::Value::as_str) + != Some(&self.inner.config.base_url) + { + return Err(Error::Validation( + "handle belongs to a different provider or endpoint".into(), + )); + } + validate_id(&handle.id) + } + + fn require_background(&self, handle: &DeferredHandle) -> Result<()> { + self.validate_handle(handle)?; + if handle.kind.as_deref() != Some("background") { + return Err(Error::Unsupported( + "operation requires a background handle".into(), + )); + } + Ok(()) + } +} + +fn validate_id(id: &str) -> Result<()> { + if id.trim().is_empty() || id.len() > 512 || matches!(id, "." | "..") { + Err(Error::Validation("invalid response/file identifier".into())) + } else { + Ok(()) + } +} + +fn path(segments: &[&str]) -> Result { + let mut url = reqwest::Url::parse("https://path.invalid/") + .map_err(|_| Error::Validation("invalid path base".into()))?; + { + let mut path = url + .path_segments_mut() + .map_err(|_| Error::Validation("invalid path base".into()))?; + path.clear(); + for segment in segments { + validate_id(segment)?; + path.push(segment); + } + } + Ok(url.path().into()) +} diff --git a/crates/tinyinference-llm/src/providers/perplexity/lifecycle_tests.rs b/crates/tinyinference-llm/src/providers/perplexity/lifecycle_tests.rs new file mode 100644 index 00000000..445e8605 --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/lifecycle_tests.rs @@ -0,0 +1,252 @@ +use super::*; + +use super::tests::{complete, fixture, json_reply}; +use crate::{ + message::Message, + model::{DeferredStatus, ExecutionStatus}, +}; +use serde_json::json; + +#[tokio::test] +async fn responses_alias_submits_once_and_uses_canonical_agent_recovery_routes() { + let mut queued = complete(); + queued["status"] = json!("queued"); + queued["output"] = json!([]); + let mut cancelled = queued.clone(); + cancelled["status"] = json!("cancelled"); + let (mut model, scripted) = fixture(vec![ + json_reply(200, queued.clone()), + json_reply(200, queued), + json_reply(200, json!({"response_id":"resp_1","status":"cancelling"})), + json_reply(200, cancelled), + ]); + Arc::get_mut(&mut model.inner) + .unwrap() + .config + .responses_alias = true; + let handle = model + .submit_background(ModelRequest::new(vec![Message::user("question")])) + .await + .unwrap(); + assert_eq!(handle.id, "resp_1"); + assert!(matches!( + >::fetch_deferred(&model, &handle) + .await + .unwrap(), + DeferredStatus::Pending + )); + assert_eq!( + model.cancel_background(&handle).await.unwrap(), + ExecutionStatus::Cancelling + ); + assert!(matches!( + >::fetch_deferred(&model, &handle) + .await + .unwrap(), + DeferredStatus::Cancelled { .. } + )); + let requests = scripted.requests.lock().unwrap(); + assert_eq!( + requests + .iter() + .filter(|r| r.method() == reqwest::Method::POST && r.url().path() == "/v1/responses") + .count(), + 1 + ); + assert_eq!(requests[0].url().path(), "/v1/responses"); + assert_eq!(requests[1].url().path(), "/v1/agent/resp_1"); + assert_eq!(requests[2].url().path(), "/v1/agent/resp_1/cancel"); +} + +fn handle(model: &PerplexityModel) -> crate::model::DeferredHandle { + let response = super::response::parse(complete()).unwrap(); + let mut handle = model.response_handle(&response).unwrap(); + handle.kind = Some("background".into()); + handle +} + +#[tokio::test] +async fn file_listing_and_download_use_file_ids_without_treating_filenames_as_paths() { + let bytes = http::Response::builder() + .header("content-type", "text/csv") + .body("a,b\n1,2\n") + .unwrap() + .into(); + let (model, scripted) = fixture(vec![ + json_reply( + 200, + json!({"data":[{"id":"file/1","filename":"../../report.csv","bytes":8,"created_at":7}]}), + ), + Ok(bytes), + ]); + let handle = handle(&model); + let files = model.list_response_files(&handle).await.unwrap(); + assert_eq!(files[0].filename, "../../report.csv"); + assert_eq!(files[0].response_id, "resp_1"); + let content = model + .download_response_file(&handle, &files[0].id) + .await + .unwrap(); + assert_eq!(content.bytes, b"a,b\n1,2\n"); + assert_eq!(content.media_type.as_deref(), Some("text/csv")); + let requests = scripted.requests.lock().unwrap(); + assert_eq!( + requests[1].url().path(), + "/v1/agent/resp_1/files/file%2F1/content" + ); + assert!(requests.iter().all(|r| r.method() == reqwest::Method::GET)); +} + +#[tokio::test] +async fn foreign_handles_invalid_ids_and_duplicate_files_fail_clearly() { + let (model, scripted) = fixture(vec![]); + let mut foreign = handle(&model); + foreign + .metadata + .insert("base_url".into(), json!("https://attacker.example")); + assert!(matches!( + model.retrieve_response(&foreign).await, + Err(Error::Validation(_)) + )); + let mut invalid = handle(&model); + invalid.id = "..".into(); + assert!(model.list_response_files(&invalid).await.is_err()); + assert!(scripted.requests.lock().unwrap().is_empty()); + let (model, _) = fixture(vec![json_reply( + 200, + json!({"data":[ + {"id":"f","filename":"one","bytes":1,"created_at":1}, + {"id":"f","filename":"two","bytes":1,"created_at":1} + ]}), + )]); + assert!(model.list_response_files(&handle(&model)).await.is_err()); +} + +#[tokio::test] +async fn oversized_file_and_failed_download_never_regenerate_the_run() { + let (mut model, scripted) = fixture(vec![ + Ok(http::Response::new("123456").into()), + json_reply(404, json!({"error":{"message":"missing file"}})), + ]); + Arc::get_mut(&mut model.inner) + .unwrap() + .config + .max_file_bytes = 3; + let handle = handle(&model); + let first = model + .download_response_file(&handle, "f") + .await + .unwrap_err(); + assert!( + matches!(first,Error::Provider(error) if error.code.as_deref()==Some("response_too_large")) + ); + assert!( + model + .download_response_file(&handle, "missing") + .await + .is_err() + ); + assert!( + scripted + .requests + .lock() + .unwrap() + .iter() + .all(|r| r.method() == reqwest::Method::GET) + ); +} + +#[tokio::test] +async fn reconnect_uses_saved_cursor_and_keeps_the_authoritative_full_response() { + use futures::StreamExt; + let mut progress = complete(); + progress["status"] = json!("in_progress"); + progress["output"] = json!([]); + let body=[ + json!({"type":"response.output_text.delta","sequence_number":3,"output_index":0,"content_index":0,"item_id":"msg_1","delta":"old"}), + json!({"type":"response.output_text.delta","sequence_number":4,"output_index":0,"content_index":0,"item_id":"msg_1","delta":"swer"}), + json!({"type":"response.completed","sequence_number":5,"response":complete()}), + ].into_iter().map(|event|format!("data: {event}\n\n")).collect::(); + let incoming = http::Response::builder() + .header("content-type", "text/event-stream") + .body(body) + .unwrap() + .into(); + let (model, scripted) = fixture(vec![json_reply(200, progress), Ok(incoming)]); + let items = model + .resume_background(&handle(&model), Some(3)) + .await + .unwrap() + .collect::>() + .await; + let deltas = items + .iter() + .filter_map(|i| { + if let crate::model::ModelStreamItem::MessageDelta(d) = i { + Some(d.text.as_str()) + } else { + None + } + }) + .collect::(); + assert_eq!(deltas, "swer"); + let Some(crate::model::ModelStreamItem::Completed(response)) = items.last() else { + panic!("missing completion {items:?}"); + }; + assert_eq!(response.text(), "answer"); + assert_eq!( + scripted.requests.lock().unwrap()[1].url().query(), + Some("stream=true&starting_after=3") + ); +} + +#[tokio::test] +async fn expired_reconnect_and_cancel_errors_are_not_replayed() { + let mut progress = complete(); + progress["status"] = json!("in_progress"); + progress["output"] = json!([]); + let (model, scripted) = fixture(vec![ + json_reply(200, progress), + json_reply(400, json!({"error":{"message":"expired window"}})), + json_reply(503, json!({})), + ]); + let handle = handle(&model); + assert!( + matches!(model.resume_background(&handle,Some(3)).await,Err(Error::Provider(error)) if error.code.as_deref()==Some("reconnect_unavailable")) + ); + assert!(model.cancel_background(&handle).await.is_err()); + let requests = scripted.requests.lock().unwrap(); + assert_eq!(requests.len(), 3); + assert!( + requests + .iter() + .filter(|r| r.method() == reqwest::Method::POST) + .all(|r| r.url().path().ends_with("/cancel")) + ); +} + +#[tokio::test] +async fn failed_and_incomplete_background_runs_keep_their_terminal_state() { + let mut incomplete = complete(); + incomplete["status"] = json!("incomplete"); + incomplete["incomplete_details"] = json!({"reason":"max_output_tokens"}); + let mut failed = complete(); + failed["status"] = json!("failed"); + failed["error"] = json!({"message":"failed","code":"provider_error"}); + let (model, _) = fixture(vec![json_reply(200, incomplete), json_reply(200, failed)]); + let handle = handle(&model); + let DeferredStatus::Completed(response) = + >::fetch_deferred(&model, &handle) + .await + .unwrap() + else { + panic!("missing incomplete snapshot"); + }; + assert_eq!(response.finish_reason.as_deref(), Some("max_output_tokens")); + assert!(matches!( + >::fetch_deferred(&model, &handle) + .await + .unwrap(), + DeferredStatus::ProviderFailed(_) + )); +} diff --git a/crates/tinyinference-llm/src/providers/perplexity/mod.rs b/crates/tinyinference-llm/src/providers/perplexity/mod.rs new file mode 100644 index 00000000..cee488bb --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/mod.rs @@ -0,0 +1,198 @@ +//! Native Perplexity Agent API provider. +//! +//! Select a provider-qualified model or a dynamic preset with one explicit key. +//! Perplexity executes hosted tools; this crate returns local function requests +//! to the caller. Full ordered output lives in [`ModelResponse::output`], while +//! [`ModelResponse::message`] remains the text/reasoning/function-call projection. +//! Stream progress and unknown events survive in execution metadata. +//! +//! ```no_run +//! use tinyinference_llm::{ChatModel, Message, ModelRequest, PerplexityModel, PerplexitySelection}; +//! # async fn example(key: String) -> tinyinference_llm::Result<()> { +//! let model = PerplexityModel::new(key, PerplexitySelection::preset("low"))?; +//! let response = model.invoke(&(), ModelRequest::new(vec![Message::user( +//! "Explain the latest changes to this project with sources." +//! )])).await?; +//! println!("{}", response.text()); +//! for item in &response.output { +//! // Inspect typed citations, hosted results, or generated-file notices. +//! let _ = &item.kind; +//! } +//! # Ok(()) +//! # } +//! ``` +//! +//! To replay a signed custom call, use [`crate::message::ToolMessage::for_call`] +//! and pin the returned model. Include the preceding conversation, original +//! assistant call and matching result; leave [`ModelRequest::continuation_id`] +//! unset. This is the verified Google function-continuation path. For ordinary +//! chat follow-up, set the continuation ID and send only the new user turn. `store: false` +//! hides retrieval; it does not promise that Perplexity avoids persistence. +//! +//! [`PerplexityModel::submit_background`] returns a scoped handle. Retrieve or +//! resume it instead of retrying creation after a disconnect. Remote cancellation +//! is explicit; dropping a local stream only stops local work. Creation retries +//! cover explicit rate limits only, never ambiguous transport/5xx failures. +//! +//! Defaults: 600-second total budget, 30-second connection budget, 60-second +//! stream inactivity budget, 32 MiB input/body/stream traffic, 4 MiB per event, +//! and 64 MiB per explicit file download. All are configurable. No local file +//! reads, file writes, tool execution, or automatic response caching are performed. + +use crate::{ + Error, Result, + model::{ChatModel, ModelRequest, ModelResponse}, +}; +use async_trait::async_trait; +use std::sync::Arc; + +mod config; +mod lifecycle; +mod request; +mod response; +mod stream; +mod transport; +mod types; + +pub use config::*; + +/// Perplexity Agent model. The key is supplied explicitly and never read from the environment. +#[derive(Clone, Debug)] +pub struct PerplexityModel { + inner: Arc, +} + +impl PerplexityModel { + /// Constructs a model/preset client with bounded defaults. + /// + /// # Errors + /// Returns validation errors for invalid credentials, selection, or limits. + pub fn new(key: impl Into, selection: PerplexitySelection) -> Result { + Self::with_config(key, PerplexityConfig::new(selection)) + } + + /// Constructs a native provider using validated immutable configuration. + /// + /// # Errors + /// Returns validation or HTTP-client setup errors; makes no provider call. + pub fn with_config(key: impl Into, config: PerplexityConfig) -> Result { + Ok(Self { + inner: Arc::new(transport::Transport::new(key.into(), config)?), + }) + } + + fn body( + &self, + request: &ModelRequest, + streaming: bool, + submit: bool, + ) -> Result { + self.inner.check_input_size(request)?; + let mut body = request::build(&self.inner.config, request, streaming, submit)?; + if let Some(hook) = &self.inner.config.on_payload { + let original = body.clone(); + hook(&mut body); + request::validate_hook(&original, &body)?; + } + Ok(body) + } + + fn create_path(&self) -> &'static str { + if self.inner.config.responses_alias { + "/responses" + } else { + "/agent" + } + } +} + +/// Constructs the provider behind the existing object-safe chat interface. +/// +/// # Errors +/// Returns the same configuration errors as [`PerplexityModel::with_config`]. +pub fn build_perplexity_model( + key: impl Into, + config: PerplexityConfig, +) -> Result>> { + Ok(Arc::new(PerplexityModel::with_config(key, config)?)) +} + +#[async_trait] +impl ChatModel for PerplexityModel { + async fn invoke(&self, _state: &State, request: ModelRequest) -> Result { + crate::network_guard::ensure_network_models_allowed()?; + let deadline = self.inner.deadline(request.timeout_ms)?; + let body = self.body(&request, false, false)?; + let outgoing = + self.inner + .request(reqwest::Method::POST, self.create_path(), Some(&body))?; + let incoming = self + .inner + .send(outgoing, transport::Operation::Create, deadline) + .await?; + let result = + response::completed(response::parse(self.inner.json(incoming, deadline).await?)?)?; + if tokio::time::Instant::now() >= deadline { + return Err(response::with_partial(transport::timeout(), result)); + } + Ok(result.inherit_correlation(request.correlation)) + } + + async fn stream( + &self, + _state: &State, + request: ModelRequest, + ) -> Result { + crate::network_guard::ensure_network_models_allowed()?; + let deadline = self.inner.deadline(request.timeout_ms)?; + let body = self.body(&request, true, false)?; + let outgoing = + self.inner + .request(reqwest::Method::POST, self.create_path(), Some(&body))?; + let incoming = self + .inner + .send(outgoing, transport::Operation::Create, deadline) + .await?; + let mut stream = stream::open(self.inner.clone(), incoming, deadline, None, None)?; + if let Some(correlation) = request.correlation { + stream = stream.with_correlation(correlation); + } + Ok(stream) + } + + async fn fetch_deferred( + &self, + handle: &crate::model::DeferredHandle, + ) -> Result { + use crate::model::{DeferredStatus, ExecutionStatus}; + let snapshot = self.retrieve_response(handle).await?; + match snapshot.execution.as_ref().map(|e| &e.status) { + Some( + ExecutionStatus::Queued | ExecutionStatus::InProgress | ExecutionStatus::Cancelling, + ) => Ok(DeferredStatus::Pending), + Some(ExecutionStatus::Cancelled) => Ok(DeferredStatus::Cancelled { + response: Some(Box::new(snapshot)), + }), + Some(ExecutionStatus::Completed | ExecutionStatus::Incomplete) => Ok( + DeferredStatus::Completed(Box::new(response::completed(snapshot)?)), + ), + _ => match response::completed(snapshot) { + Err(Error::Provider(error)) => Ok(DeferredStatus::ProviderFailed(error)), + Err(error) => Err(error), + Ok(_) => Err(response::malformed("unknown deferred state")), + }, + } + } +} + +#[cfg(test)] +#[path = "mod_tests.rs"] +mod tests; + +#[cfg(test)] +#[path = "stream_tests.rs"] +mod stream_test; + +#[cfg(test)] +#[path = "lifecycle_tests.rs"] +mod lifecycle_test; diff --git a/crates/tinyinference-llm/src/providers/perplexity/mod_tests.rs b/crates/tinyinference-llm/src/providers/perplexity/mod_tests.rs new file mode 100644 index 00000000..86b3a054 --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/mod_tests.rs @@ -0,0 +1,247 @@ +use super::*; +use crate::{ + message::Message, + model::{ChatModel, ModelRequest}, +}; +use async_trait::async_trait; +use std::{ + collections::VecDeque, + sync::{Arc, Mutex}, +}; + +pub(super) struct Scripted { + pub requests: Mutex>, + replies: Mutex>>, +} + +#[async_trait] +impl transport::HttpTransport for Scripted { + async fn send(&self, request: reqwest::Request) -> crate::Result { + self.requests.lock().unwrap().push(request); + self.replies + .lock() + .unwrap() + .pop_front() + .expect("unexpected network operation") + } +} + +pub(super) fn fixture( + replies: Vec>, +) -> (PerplexityModel, Arc) { + let scripted = Arc::new(Scripted { + requests: Mutex::new(vec![]), + replies: Mutex::new(replies.into()), + }); + let mut model = PerplexityModel::new("test-key", PerplexitySelection::preset("low")).unwrap(); + Arc::get_mut(&mut model.inner).unwrap().sender = scripted.clone(); + (model, scripted) +} + +pub(super) fn json_reply(status: u16, body: serde_json::Value) -> crate::Result { + Ok(http::Response::builder() + .status(status) + .header("content-type", "application/json") + .body(body.to_string()) + .unwrap() + .into()) +} + +pub(super) fn complete() -> serde_json::Value { + serde_json::json!({"id":"resp_1","model":"openai/test","status":"completed", + "output":[{"id":"msg_1","type":"message","role":"assistant","content":[{"type":"output_text","text":"answer"}]}]}) +} + +#[tokio::test] +async fn public_invoke_uses_native_route_and_preserves_response_identity() { + let (model, scripted) = fixture(vec![json_reply(200, complete())]); + let result = model + .invoke(&(), ModelRequest::new(vec![Message::user("question")])) + .await + .unwrap(); + assert_eq!(result.text(), "answer"); + assert_eq!(result.execution.unwrap().id, "resp_1"); + let requests = scripted.requests.lock().unwrap(); + assert_eq!(requests.len(), 1); + assert_eq!( + requests[0].url().as_str(), + "https://api.perplexity.ai/v1/agent" + ); + assert_eq!(requests[0].headers()["authorization"], "Bearer test-key"); +} + +#[tokio::test(start_paused = true)] +async fn creation_retries_only_explicit_rate_limits_and_bounds_the_attempts() { + let throttled = http::Response::builder() + .status(429) + .header("retry-after", "2") + .body("") + .unwrap() + .into(); + let (model, scripted) = fixture(vec![Ok(throttled), json_reply(200, complete())]); + let start = tokio::time::Instant::now(); + assert_eq!( + model + .invoke(&(), ModelRequest::new(vec![Message::user("q")])) + .await + .unwrap() + .text(), + "answer" + ); + assert_eq!(start.elapsed(), std::time::Duration::from_secs(2)); + assert_eq!(scripted.requests.lock().unwrap().len(), 2); + let (model, scripted) = fixture( + (0..4) + .map(|_| json_reply(429, serde_json::json!({}))) + .collect(), + ); + assert!( + matches!(model.invoke(&(),ModelRequest::new(vec![Message::user("q")])).await,Err(Error::Provider(error)) if error.status==Some(429)) + ); + assert_eq!(scripted.requests.lock().unwrap().len(), 4); + for status in [400, 401, 403, 500, 502, 503, 504] { + let (model, scripted) = fixture(vec![json_reply( + status, + serde_json::json!({"error":{"message":"failed test-key"}}), + )]); + let error = model + .invoke(&(), ModelRequest::new(vec![Message::user("q")])) + .await + .unwrap_err(); + assert!( + matches!(&error,Error::Provider(error) if error.status==Some(status)&&!error.retryable) + ); + assert!(!format!("{error:?} {error}").contains("test-key")); + assert_eq!(scripted.requests.lock().unwrap().len(), 1); + } +} + +#[tokio::test(start_paused = true)] +async fn total_deadline_covers_header_and_body_waits() { + let incoming = http::Response::new(reqwest::Body::wrap_stream(futures::stream::pending::< + std::result::Result, + >())) + .into(); + let (model, scripted) = fixture(vec![Ok(incoming)]); + let mut request = ModelRequest::new(vec![Message::user("q")]); + request.timeout_ms = Some(10); + assert!( + matches!(model.invoke(&(),request).await,Err(Error::Provider(error)) if error.code.as_deref()==Some("timeout")) + ); + assert_eq!(scripted.requests.lock().unwrap().len(), 1); +} + +#[tokio::test(start_paused = true)] +async fn total_deadline_cancels_a_pending_transport_send() { + struct DelayedTransport; + #[async_trait] + impl transport::HttpTransport for DelayedTransport { + async fn send(&self, _request: reqwest::Request) -> crate::Result { + tokio::time::sleep(std::time::Duration::from_secs(1)).await; + json_reply(200, complete()) + } + } + let (mut model, _) = fixture(vec![]); + Arc::get_mut(&mut model.inner).unwrap().sender = Arc::new(DelayedTransport); + let mut request = ModelRequest::new(vec![Message::user("question")]); + request.timeout_ms = Some(10); + let start = tokio::time::Instant::now(); + assert!(matches!(model.invoke(&(), request).await, + Err(Error::Provider(error)) if error.code.as_deref() == Some("timeout"))); + assert_eq!(start.elapsed(), std::time::Duration::from_millis(10)); +} + +#[tokio::test] +async fn aliases_and_payload_hooks_are_validated_before_sending() { + let (mut model, scripted) = fixture(vec![json_reply(200, complete())]); + let config = &mut Arc::get_mut(&mut model.inner).unwrap().config; + config.responses_alias = true; + config.on_payload = Some(Arc::new(|body| body["max_steps"] = serde_json::json!(3))); + model + .invoke(&(), ModelRequest::new(vec![Message::user("q")])) + .await + .unwrap(); + assert_eq!( + scripted.requests.lock().unwrap()[0].url().path(), + "/v1/responses" + ); + let (mut model, scripted) = fixture(vec![]); + Arc::get_mut(&mut model.inner).unwrap().config.on_payload = Some(Arc::new(|body| { + body["tools"] = serde_json::json!([{"type":"sandbox"}]) + })); + assert!(matches!( + model + .invoke(&(), ModelRequest::new(vec![Message::user("q")])) + .await, + Err(Error::Unsupported(_)) + )); + assert!(scripted.requests.lock().unwrap().is_empty()); +} + +#[tokio::test] +async fn invoke_preserves_unknown_fields_but_redacts_client_credentials() { + let mut value = complete(); + value["future_metadata"] = serde_json::json!({"credential":"test-key","ordinary":"retained"}); + let (model, _) = fixture(vec![json_reply(200, value)]); + let response = model + .invoke(&(), ModelRequest::new(vec![Message::user("q")])) + .await + .unwrap(); + assert_eq!( + response.raw.as_ref().unwrap()["future_metadata"]["ordinary"], + "retained" + ); + assert_eq!( + response.raw.as_ref().unwrap()["future_metadata"]["credential"], + "[REDACTED]" + ); + assert!(!format!("{model:?}").contains("test-key")); +} + +#[tokio::test] +async fn public_function_replay_keeps_history_arguments_signature_and_call_id() { + use crate::{ + ToolSchema, + message::{ContentBlock, ToolMessage}, + }; + let first = serde_json::json!({"id":"resp_1","model":"google/test","status":"completed","output":[ + {"type":"function_call","id":"fc_1","call_id":"call_1","name":"lookup","arguments":"{ \"id\": 7 }","thought_signature":"opaque-signature"} + ]}); + let (model, scripted) = fixture(vec![json_reply(200, first), json_reply(200, complete())]); + let tool = ToolSchema::new( + "lookup", + "look up a record", + serde_json::json!({"type":"object"}), + ); + let mut request = ModelRequest::new(vec![ + Message::system("Use the supplied record."), + Message::user("lookup"), + ]) + .with_tools(vec![tool]); + let first = model.invoke(&(), request.clone()).await.unwrap(); + let result = ToolMessage::for_call( + &first.message.tool_calls[0], + vec![ContentBlock::Text("found".into())], + ); + request.messages.push(Message::Assistant(first.message)); + request.messages.push(Message::Tool(result)); + request.model = Some(first.execution.unwrap().model); + request.continuation_id = None; + assert_eq!(model.invoke(&(), request).await.unwrap().text(), "answer"); + let requests = scripted.requests.lock().unwrap(); + assert_eq!(requests.len(), 2); + let second: serde_json::Value = + serde_json::from_slice(requests[1].body().unwrap().as_bytes().unwrap()).unwrap(); + assert!(second.get("previous_response_id").is_none()); + assert_eq!(second["model"], "google/test"); + assert_eq!(second["instructions"], "Use the supplied record."); + assert_eq!( + second["input"], + serde_json::json!([ + {"type":"message","role":"user","content":[{"type":"input_text","text":"lookup"}]}, + {"type":"function_call","call_id":"call_1","name":"lookup","arguments":"{ \"id\": 7 }","thought_signature":"opaque-signature"}, + {"type":"function_call_output","call_id":"call_1","name":"lookup","output":[{"type":"input_text","text":"found"}],"thought_signature":"opaque-signature"} + ]) + ); + assert_eq!(second["tools"][0]["name"], "lookup"); +} diff --git a/crates/tinyinference-llm/src/providers/perplexity/request.rs b/crates/tinyinference-llm/src/providers/perplexity/request.rs new file mode 100644 index 00000000..63091967 --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/request.rs @@ -0,0 +1,685 @@ +use std::collections::{BTreeMap, HashSet}; + +use super::config::*; +use crate::message::{ContentBlock, MediaRef, Message, replay_system_state}; +use crate::model::{ModelRequest, ResponseFormat, ToolChoice}; +use crate::tool::{ToolCall, ToolResultContext}; +use crate::{Error, Result}; +use base64::Engine; +use serde_json::{Map, Value, json}; + +pub(super) fn effective_options( + config: &PerplexityConfig, + request: &ModelRequest, +) -> Result { + validate_options_before_serialization(&config.options)?; + let mut value = serde_json::to_value(&config.options)?; + if !request.provider_options.is_null() { + let root = request + .provider_options + .as_object() + .ok_or_else(|| invalid("provider options must be an object"))?; + if root.keys().any(|key| key != "perplexity") { + return Err(Error::Unsupported( + "use typed PerplexityOptions for provider settings".into(), + )); + } + if let Some(override_value) = root.get("perplexity") { + if override_value.get("profile").is_some() + || override_value.get("skills").is_some() + || override_value + .get("tools") + .and_then(Value::as_array) + .is_some_and(|tools| { + tools + .iter() + .any(|tool| tool.get("require_approval").is_some()) + }) + { + return Err(Error::Unsupported("profiles, skills and remote approval controls are not supported by this adapter".into())); + } + let overrides: PerplexityOptions = serde_json::from_value(override_value.clone()) + .map_err(|_| invalid("unknown or invalid Perplexity option"))?; + if let (Some(base), Some(overrides)) = ( + value.as_object_mut(), + serde_json::to_value(overrides)?.as_object(), + ) { + base.extend(overrides.clone()); + } + } + } + serde_json::from_value(value).map_err(|_| invalid("invalid Perplexity options")) +} + +pub(super) fn build( + config: &PerplexityConfig, + request: &ModelRequest, + streaming: bool, + submit: bool, +) -> Result { + if request.seed.is_some() || !request.stop_sequences.is_empty() { + return Err(Error::Unsupported( + "Perplexity does not support seed or stop sequences".into(), + )); + } + if request + .reasoning + .as_ref() + .is_some_and(|r| r.budget_tokens.is_some() || r.summary.is_some()) + { + return Err(Error::Unsupported( + "Perplexity does not support reasoning budgets or summaries in requests".into(), + )); + } + if request.required_capabilities.is_some() { + return Err(Error::Unsupported( + "Perplexity model capabilities have not been resolved authoritatively".into(), + )); + } + let options = effective_options(config, request)?; + let mut selection = options + .selection + .clone() + .unwrap_or_else(|| config.selection.clone()); + if let Some(model) = &request.model { + match &mut selection { + PerplexitySelection::Model(selected) => *selected = model.clone(), + PerplexitySelection::Preset { + model: selected, + models: None, + .. + } => *selected = Some(model.clone()), + _ => { + return Err(invalid( + "a model override is ambiguous with a fallback chain", + )); + } + } + } + selection.validate()?; + let target = selected_model(&selection); + let mut body = Map::new(); + match &selection { + PerplexitySelection::Model(model) => { + body.insert("model".into(), json!(model)); + } + PerplexitySelection::Models(models) => { + body.insert("models".into(), json!(models)); + } + PerplexitySelection::Preset { + name, + model, + models, + } => { + body.insert("preset".into(), json!(name)); + if let Some(model) = model { + body.insert("model".into(), json!(model)); + } + if let Some(models) = models { + body.insert("models".into(), json!(models)); + } + } + } + let cap = request.max_tokens.or(options.max_output_tokens); + let needs_cap = match &selection { + PerplexitySelection::Model(m) => m.starts_with("anthropic/"), + PerplexitySelection::Models(ms) => ms.iter().any(|m| m.starts_with("anthropic/")), + PerplexitySelection::Preset { model, models, .. } => { + model.as_ref().is_some_and(|m| m.starts_with("anthropic/")) + || models + .as_ref() + .is_some_and(|ms| ms.iter().any(|m| m.starts_with("anthropic/"))) + } + }; + if cap == Some(0) || (needs_cap && cap.is_none()) { + return Err(invalid( + "an explicit positive output cap is required for Anthropic models", + )); + } + if let Some(cap) = cap { + body.insert("max_output_tokens".into(), json!(cap)); + } + if let Some(steps) = options.max_steps { + if !(1..=100).contains(&steps) { + return Err(invalid("max_steps must be between 1 and 100")); + } + body.insert("max_steps".into(), json!(steps)); + } + let background = submit || options.background == Some(true); + if background && !streaming && !submit { + return Err(Error::Unsupported( + "use submit_background or stream for background runs".into(), + )); + } + if background && options.store == Some(false) { + return Err(invalid("background runs require retrievable storage")); + } + if background { + body.insert("background".into(), json!(true)); + } + body.insert("stream".into(), json!(streaming)); + if let Some(store) = options.store { + body.insert("store".into(), json!(store)); + } + if let Some(language) = &options.language_preference { + if language.len() != 2 || !language.bytes().all(|b| b.is_ascii_alphabetic()) { + return Err(invalid("language preference must be a two-letter code")); + } + body.insert("language_preference".into(), json!(language)); + } + let effort = options.reasoning_effort.as_deref().or_else(|| { + request + .reasoning + .as_ref() + .and_then(|r| r.effort.map(|e| e.as_str())) + }); + if let Some(effort) = effort { + if !["minimal", "low", "medium", "high", "xhigh", "max"].contains(&effort) { + return Err(Error::Unsupported( + "unsupported Perplexity reasoning effort".into(), + )); + } + body.insert("reasoning".into(), json!({"effort":effort})); + } + for (key, value, max) in [ + ("temperature", request.temperature, 2.0), + ("top_p", request.top_p, 1.0), + ] { + if let Some(value) = value { + if !value.is_finite() || !(0.0..=max).contains(&value) { + return Err(invalid("sampling setting is outside provider bounds")); + } + body.insert(key.into(), json!(value)); + } + } + if let Some(format) = &request.response_format { + let schema = match format { + ResponseFormat::Text => None, + ResponseFormat::JsonObject => Some(("response", json!({"type":"object"}))), + ResponseFormat::JsonSchema { name, schema } | ResponseFormat::Auto { name, schema } => { + Some((name.as_str(), schema.clone())) + } + }; + if let Some((name, schema)) = schema { + if name.is_empty() + || name.len() > 64 + || !name + .bytes() + .all(|b| b.is_ascii_alphanumeric() || b == b'_' || b == b'-') + || !schema.is_object() + { + return Err(invalid("invalid JSON schema name or shape")); + } + body.insert( + "response_format".into(), + json!({"type":"json_schema","json_schema":{"name":name,"schema":schema}}), + ); + } + } + let (instructions, system_tools) = replay_system_state(&request.messages); + if !instructions.is_empty() { + body.insert("instructions".into(), json!(instructions)); + } + let mut tools = match &options.tools { + Some(tools) => encode_hosted_tools(tools, &config.remote_credentials)?, + None => Vec::new(), + }; + let mut functions = BTreeMap::new(); + for tool in system_tools { + functions.insert(tool.name.clone(), tool); + } + let mut explicit_names = HashSet::new(); + for tool in &request.tools { + if tool.name.trim().is_empty() || !explicit_names.insert(&tool.name) { + return Err(invalid("duplicate or blank function name")); + } + functions.insert(tool.name.clone(), tool.clone()); + } + for tool in functions.values() { + if !tool.parameters.is_object() || !tool.format.is_json() { + return Err(Error::Unsupported( + "Perplexity functions require native JSON schemas".into(), + )); + } + tools.push(json!({"type":"function","name":tool.name,"description":tool.description,"parameters":tool.parameters})); + } + if options.tools.is_some() || !tools.is_empty() { + body.insert("tools".into(), json!(tools)); + } + let choice = options + .tool_choice + .as_ref() + .map(|choice| match choice { + PerplexityToolChoice::Auto => json!("auto"), + PerplexityToolChoice::None => json!("none"), + PerplexityToolChoice::Required => json!("required"), + PerplexityToolChoice::Function(name) => json!({"type":"function","name":name}), + PerplexityToolChoice::Hosted(kind) => json!({"type":kind}), + }) + .or_else(|| match &request.tool_choice { + ToolChoice::Auto => None, + ToolChoice::None => Some(json!("none")), + ToolChoice::Required => Some(json!("required")), + ToolChoice::Tool(name) => Some(json!({"type":"function","name":name})), + }); + if let Some(choice) = choice { + if let Some(name) = choice.get("name").and_then(Value::as_str) { + if !functions.contains_key(name) { + return Err(invalid("selected function is not declared")); + } + } else if let Some(kind) = choice.get("type").and_then(Value::as_str) { + if ![ + "web_search", + "image_search", + "fetch_url", + "finance_search", + "people_search", + "sandbox", + ] + .contains(&kind) + { + return Err(invalid("unsupported forced hosted tool")); + } + if !matches!(selection, PerplexitySelection::Preset { .. }) + && !body + .get("tools") + .and_then(Value::as_array) + .is_some_and(|tools| { + tools + .iter() + .any(|tool| tool.get("type").and_then(Value::as_str) == Some(kind)) + }) + { + return Err(invalid("selected hosted tool is not declared")); + } + } + body.insert("tool_choice".into(), choice); + } + if let Some(id) = &request.continuation_id { + if id.trim().is_empty() { + return Err(invalid("continuation id must not be blank")); + } + body.insert("previous_response_id".into(), json!(id)); + } + body.insert("input".into(), json!(input_items(request, target)?)); + Ok(Value::Object(body)) +} + +fn selected_model(selection: &PerplexitySelection) -> Option<&str> { + match selection { + PerplexitySelection::Model(model) => Some(model), + PerplexitySelection::Preset { + model: Some(model), .. + } => Some(model), + _ => None, + } +} + +fn input_items(request: &ModelRequest, target: Option<&str>) -> Result> { + let mut input = Vec::new(); + let mut calls: BTreeMap<&str, &ToolCall> = BTreeMap::new(); + let mut results = HashSet::new(); + let mut documents = 0; + for message in &request.messages { + match message { + Message::System(_) | Message::Custom(_) => {}, + Message::User(user) => input.push(json!({"type":"message","role":"user","content":content(&user.content,true,&mut documents)?})), + Message::Assistant(assistant) => { + if request.continuation_id.is_some() { return Err(invalid("stateful continuation accepts new input, not replayed assistant history")); } + if !assistant.content.is_empty() { input.push(json!({"type":"message","role":"assistant","content":content(&assistant.content,false,&mut documents)?})); } + for call in &assistant.tool_calls { + if call.id.is_empty() || call.name.is_empty() || calls.insert(&call.id,call).is_some() { return Err(invalid("duplicate or missing function call identity")); } + let mut item = json!({"type":"function_call","call_id":call.id,"name":call.name,"arguments":call.arguments.to_string()}); + if let Some(replay) = &call.replay { + validate_replay(replay,target)?; + let unchanged = if call.invalid.is_some() { call.arguments.as_str()==Some(replay.arguments.as_str()) } + else { serde_json::from_str::(&replay.arguments).ok().as_ref()==Some(&call.arguments) }; + if !unchanged { return Err(invalid("replayed function arguments were changed")); } + item["arguments"] = json!(replay.arguments); + if let Some(signature) = &replay.thought_signature { item["thought_signature"]=json!(signature); } + } else if call.invalid.is_some() { + item["arguments"] = json!(call.arguments.as_str().ok_or_else(|| invalid("invalid function arguments lost their raw text"))?); + } + input.push(item); + } + }, + Message::Tool(result) => { + if result.tool_call_id.is_empty() || !results.insert(&result.tool_call_id) { return Err(invalid("duplicate or missing function result id")); } + let known = calls.get(result.tool_call_id.as_str()); + let inherited = known.map(|call| ToolResultContext {name:call.name.clone(),replay:call.replay.clone()}); + if let (Some(known),Some(context)) = (&inherited,&result.call_context) + && known!=context { return Err(invalid("function result context conflicts with its call")); } + let context = result.call_context.as_ref().or(inherited.as_ref()).ok_or_else(|| invalid("function result has no matching call context"))?; + if known.is_none() && request.continuation_id.is_none() { return Err(invalid("function result has no preceding call")); } + let mut item = json!({"type":"function_call_output","call_id":result.tool_call_id,"name":context.name, + "output":content(&result.content,false,&mut documents)?}); + if let Some(replay) = &context.replay { + validate_replay(replay,target)?; + if let Some(signature)=&replay.thought_signature { item["thought_signature"]=json!(signature); } + } + input.push(item); + }, + } + } + if input.is_empty() { + return Err(invalid("Perplexity requires input messages")); + } + if documents > 30 { + return Err(invalid("Perplexity accepts at most 30 input documents")); + } + Ok(input) +} + +fn validate_replay(replay: &crate::tool::ToolCallReplay, target: Option<&str>) -> Result<()> { + if replay.origin.provider != "perplexity" + || replay.origin.api != "agent" + || (replay.thought_signature.is_some() && target != Some(replay.origin.model.as_str())) + { + return Err(Error::Unsupported("signed function replay requires its originating Perplexity model; pin the returned model when using presets".into())); + } + Ok(()) +} + +fn content(blocks: &[ContentBlock], user: bool, documents: &mut usize) -> Result> { + blocks.iter().map(|block| match block { + ContentBlock::Text(text) => Ok(json!({"type":"input_text","text":text})), + ContentBlock::Json(value) => Ok(json!({"type":"input_text","text":value.to_string()})), + ContentBlock::Image(image) => { + validate_image(&image.url)?; + Ok(json!({"type":"input_image","image_url":image.url})) + }, + ContentBlock::Document(reference) if user => { + *documents+=1; + match reference { + MediaRef::Url {url,media_type} => { + require_https(url)?; + let mut part=json!({"type":"input_file","file_url":url}); + if let Some(mime)=media_type { part["filename"]=json!(document_filename(mime)?); } + Ok(part) + }, + MediaRef::Base64 {data,media_type} => { + base64::engine::general_purpose::STANDARD.decode(data).map_err(|_| invalid("invalid inline document base64"))?; + Ok(json!({"type":"input_file","file_data":data,"filename":document_filename(media_type)?})) + }, + MediaRef::Path {..} => Err(Error::Unsupported("resolve local documents before calling Perplexity".into())), + } + }, + _ => Err(Error::Unsupported("this content block cannot be replayed through Perplexity; use server continuation for hosted output".into())), + }).collect() +} + +fn document_filename(mime: &str) -> Result<&'static str> { + match mime { + "application/pdf" => Ok("document.pdf"), + "application/msword" => Ok("document.doc"), + "application/vnd.openxmlformats-officedocument.wordprocessingml.document" => { + Ok("document.docx") + } + "text/plain" => Ok("document.txt"), + "application/rtf" | "text/rtf" => Ok("document.rtf"), + _ => Err(Error::Unsupported("unsupported document media type".into())), + } +} + +fn validate_image(url: &str) -> Result<()> { + if let Some((prefix, data)) = url.split_once(",") + && prefix.starts_with("data:image/") + && prefix.ends_with(";base64") + { + base64::engine::general_purpose::STANDARD + .decode(data) + .map_err(|_| invalid("invalid inline image base64"))?; + return Ok(()); + } + require_https(url) +} + +fn require_https(url: &str) -> Result<()> { + if reqwest::Url::parse(url).is_ok_and(|u| { + u.scheme() == "https" + && u.host_str().is_some() + && u.username().is_empty() + && u.password().is_none() + }) { + Ok(()) + } else { + Err(invalid( + "remote media and MCP endpoints require HTTPS without embedded credentials", + )) + } +} + +fn encode_hosted_tools( + tools: &[PerplexityTool], + credentials: &BTreeMap, +) -> Result> { + let mut identities = HashSet::new(); + let mut labels = HashSet::new(); + tools + .iter() + .map(|tool| { + let mut value = serde_json::to_value(tool)?; + let kind = value["type"].as_str().unwrap_or_default().to_owned(); + let label = match tool { + PerplexityTool::Mcp { server_label, .. } + | PerplexityTool::Connector { server_label, .. } => Some(server_label.as_str()), + _ => None, + }; + if !identities.insert((kind.clone(), label)) { + return Err(invalid("duplicate hosted tool configuration")); + } + if let Some(label) = label + && (label.is_empty() + || label.len() > 64 + || !label + .bytes() + .all(|b| b.is_ascii_alphanumeric() || b == b'_' || b == b'-') + || !labels.insert(label)) + { + return Err(invalid("invalid or duplicate remote server label")); + } + match tool { + PerplexityTool::WebSearch { options } => validate_web_search(options)?, + PerplexityTool::ImageSearch { + max_results, + filters, + } => { + if max_results.is_some_and(|v| !(1..=30).contains(&v)) { + return Err(invalid("image search result limit must be 1..30")); + } + if let Some(filters) = filters { + validate_domains(filters.domain_filter.as_deref(), 10)?; + if filters.format_filter.as_ref().is_some_and(|formats| { + formats.len() > 10 + || formats.iter().any(|v| { + !["bmp", "gif", "jpeg", "png", "webp", "svg"] + .contains(&v.as_str()) + }) + }) { + return Err(invalid("invalid image format filter")); + } + } + } + PerplexityTool::FetchUrl { max_urls } + if max_urls.is_some_and(|v| !(1..=10).contains(&v)) => + { + return Err(invalid("fetch URL limit must be 1..10")); + } + PerplexityTool::Mcp { + server_url, + server_label, + .. + } => { + require_https(server_url)?; + if let Some(secret) = credentials.get(server_label) { + if reqwest::Url::parse(&secret.server_url).ok() + != reqwest::Url::parse(server_url).ok() + { + return Err(invalid( + "remote credentials belong to a different endpoint", + )); + } + if let Some(token) = &secret.authorization { + value["authorization"] = json!(token); + } + if !secret.headers.is_empty() { + value["headers"] = json!(secret.headers); + } + } + } + PerplexityTool::Connector { id, .. } if id.trim().is_empty() => { + return Err(invalid("connector id must not be blank")); + } + _ => {} + } + Ok(value) + }) + .collect() +} + +fn validate_web_search(options: &PerplexityWebSearch) -> Result<()> { + if options.max_results.is_some_and(|v| !(1..=50).contains(&v)) + || options.max_tokens == Some(0) + || options.max_tokens_per_page == Some(0) + || options + .search_type + .as_deref() + .is_some_and(|v| !["web", "fast"].contains(&v)) + || options + .search_context_size + .as_deref() + .is_some_and(|v| !["low", "medium", "high"].contains(&v)) + { + return Err(invalid("invalid web search budget or type")); + } + if let Some(location) = &options.user_location + && (location.country.len() != 2 + || !location.country.bytes().all(|b| b.is_ascii_alphabetic()) + || location.latitude.is_some() != location.longitude.is_some() + || location + .latitude + .is_some_and(|v| !v.is_finite() || !(-90.0..=90.0).contains(&v)) + || location + .longitude + .is_some_and(|v| !v.is_finite() || !(-180.0..=180.0).contains(&v))) + { + return Err(invalid("invalid search location")); + } + if let Some(filters) = &options.filters { + validate_domains(filters.search_domain_filter.as_deref(), 20)?; + if filters + .search_recency_filter + .as_deref() + .is_some_and(|v| !["hour", "day", "week", "month", "year"].contains(&v)) + { + return Err(invalid("invalid search recency")); + } + for date in [ + &filters.search_after_date_filter, + &filters.search_before_date_filter, + &filters.last_updated_after_filter, + &filters.last_updated_before_filter, + ] + .into_iter() + .flatten() + { + let parts = date + .split('/') + .map(str::parse::) + .collect::, _>>() + .map_err(|_| invalid("invalid search date"))?; + let valid = if let [month, day, year] = parts.as_slice() { + let leap = year % 4 == 0 && (year % 100 != 0 || year % 400 == 0); + let days = match month { + 1 | 3 | 5 | 7 | 8 | 10 | 12 => 31, + 4 | 6 | 9 | 11 => 30, + 2 => { + if leap { + 29 + } else { + 28 + } + } + _ => 0, + }; + *year >= 1 && *day >= 1 && *day <= days + } else { + false + }; + if !valid { + return Err(invalid("invalid search date")); + } + } + } + Ok(()) +} + +fn validate_domains(domains: Option<&[String]>, max: usize) -> Result<()> { + if let Some(domains) = domains { + let exclude = domains.first().is_some_and(|s| s.starts_with('-')); + if domains.len() > max + || domains.iter().any(|s| { + s.trim().is_empty() || s == "-" || s.len() > 253 || s.starts_with('-') != exclude + }) + { + return Err(invalid("invalid or mixed search domain filters")); + } + } + Ok(()) +} + +fn invalid(message: &str) -> Error { + Error::Validation(message.into()) +} + +pub(super) fn validate_options_before_serialization(options: &PerplexityOptions) -> Result<()> { + if let Some(tools) = &options.tools { + encode_hosted_tools(tools, &BTreeMap::new())?; + } + Ok(()) +} + +pub(super) fn validate_hook(original: &Value, changed: &Value) -> Result<()> { + let old = original + .as_object() + .ok_or_else(|| invalid("invalid original request"))?; + let new = changed + .as_object() + .ok_or_else(|| invalid("payload hook must leave an object"))?; + for key in old.keys().chain(new.keys()) { + if old.get(key) != new.get(key) + && !["max_steps", "temperature", "top_p", "max_output_tokens"].contains(&key.as_str()) + { + return Err(Error::Unsupported("payload hooks may adjust budgets/sampling only; use typed options for other settings".into())); + } + } + for key in ["max_steps", "max_output_tokens"] { + if let Some(value) = new.get(key) { + let max = if key == "max_steps" { + 100 + } else { + u32::MAX as u64 + }; + if value.as_u64().is_none_or(|v| v == 0 || v > max) { + return Err(invalid("payload hook produced an invalid budget")); + } + } + } + if old.contains_key("max_output_tokens") && !new.contains_key("max_output_tokens") { + return Err(invalid("payload hook removed the output budget")); + } + for (key, max) in [("temperature", 2.0), ("top_p", 1.0)] { + if let Some(value) = new.get(key) + && value + .as_f64() + .is_none_or(|v| !v.is_finite() || !(0.0..=max).contains(&v)) + { + return Err(invalid("payload hook produced invalid sampling settings")); + } + } + Ok(()) +} + +#[cfg(test)] +#[path = "request_tests.rs"] +mod tests; diff --git a/crates/tinyinference-llm/src/providers/perplexity/request_tests.rs b/crates/tinyinference-llm/src/providers/perplexity/request_tests.rs new file mode 100644 index 00000000..e73469cc --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/request_tests.rs @@ -0,0 +1,306 @@ +use super::*; + +use super::super::{PerplexityConfig, PerplexitySelection}; +use crate::message::Message; +use crate::model::ModelRequest; +use serde_json::json; + +use super::super::{ + PerplexityOptions, PerplexityRemoteCredentials, PerplexitySearchFilters, PerplexityTool, + PerplexityWebSearch, +}; +use crate::message::{ContentBlock, MediaRef, ToolMessage}; +use crate::model::{ReasoningConfig, ReasoningEffort, ResponseFormat}; +use crate::tool::{ToolCall, ToolCallReplay, ToolSchema}; + +#[test] +fn preset_request_preserves_server_defaults() { + let config = PerplexityConfig::new(PerplexitySelection::preset("low")); + let request = ModelRequest::new(vec![Message::user("hello")]); + assert_eq!( + build(&config, &request, false, false).unwrap(), + json!({ + "preset":"low", "stream":false, + "input":[{"type":"message","role":"user","content":[{"type":"input_text","text":"hello"}]}] + }) + ); +} + +#[test] +fn invalid_hosted_options_are_rejected_without_mutating_the_request() { + for tool in [ + PerplexityTool::ImageSearch { + max_results: Some(0), + filters: None, + }, + PerplexityTool::ImageSearch { + max_results: Some(31), + filters: None, + }, + PerplexityTool::FetchUrl { max_urls: Some(0) }, + PerplexityTool::FetchUrl { max_urls: Some(11) }, + ] { + let mut request = ModelRequest::new(vec![Message::user("question")]); + request.provider_options = json!({"perplexity":{"max_steps":2}}); + let original = request.provider_options.clone(); + let options = PerplexityOptions { + tools: Some(vec![tool]), + ..Default::default() + }; + assert!(options.apply_to(&mut request).is_err()); + assert_eq!(request.provider_options, original); + } +} + +#[test] +fn forced_hosted_tools_require_a_declaration_unless_a_preset_can_supply_it() { + let tool = PerplexityTool::web_search(PerplexityWebSearch::default()); + for selection in [ + PerplexitySelection::Model("openai/test".into()), + PerplexitySelection::Models(vec!["openai/test".into()]), + PerplexitySelection::preset("low"), + ] { + for declared in [false, true] { + let mut config = PerplexityConfig::new(selection.clone()); + config.options.tool_choice = Some(super::super::PerplexityToolChoice::Hosted( + "web_search".into(), + )); + if declared { + config.options.tools = Some(vec![tool.clone()]); + } + let request = ModelRequest::new(vec![Message::user("question")]); + let result = build(&config, &request, false, false); + if declared || matches!(selection, PerplexitySelection::Preset { .. }) { + assert_eq!(result.unwrap()["tool_choice"], json!({"type":"web_search"})); + } else { + assert!(matches!(result, Err(Error::Validation(_)))); + } + } + } +} + +#[test] +fn each_preset_remains_dynamic_and_explicit_model_overrides_are_preserved() { + for preset in ["fast", "low", "medium", "high", "xhigh", "future-preset"] { + let config = PerplexityConfig::new(PerplexitySelection::preset(preset)); + let mut request = ModelRequest::new(vec![ + Message::system("my instructions"), + Message::user("question"), + ]); + request.model = Some("google/model".into()); + request.max_tokens = Some(128); + let body = build(&config, &request, false, false).unwrap(); + assert_eq!(body["preset"], preset); + assert_eq!(body["model"], "google/model"); + assert_eq!(body["instructions"], "my instructions"); + assert_eq!(body["max_output_tokens"], 128); + assert!(body.get("tools").is_none()); + } +} + +#[test] +fn hosted_tool_options_survive_typed_serialization_and_preserve_omissions() { + let config = PerplexityConfig::new(PerplexitySelection::preset("low")); + let mut request = ModelRequest::new(vec![Message::user("question")]); + PerplexityOptions { + tools: Some(vec![PerplexityTool::WebSearch { + options: Box::new(PerplexityWebSearch { + max_results: Some(3), + filters: Some(PerplexitySearchFilters { + search_domain_filter: Some(vec!["example.com".into()]), + ..Default::default() + }), + ..Default::default() + }), + }]), + ..Default::default() + } + .apply_to(&mut request) + .unwrap(); + let body = build(&config, &request, false, false).unwrap(); + assert_eq!( + body["tools"], + json!([{"type":"web_search","max_results":3,"filters":{"search_domain_filter":["example.com"]}}]) + ); + assert!(body.get("max_steps").is_none()); + assert!(body.get("tool_choice").is_none()); +} + +#[test] +fn invalid_provider_options_are_rejected_without_silent_dropping() { + let config = PerplexityConfig::new(PerplexitySelection::Model("openai/model".into())); + for options in [ + json!({"profile":"id"}), + json!({"skills":[]}), + json!({"unexpected":true}), + json!({"max_steps":0}), + json!({"tools":[{"type":"mcp","server_label":"x","server_url":"https://example.com","require_approval":"always"}]}), + json!({"tools":[{"type":"web_search","max_results":51}]}), + json!({"tools":[{"type":"web_search","filters":{"search_domain_filter":["a","-b"]}}]}), + json!({"tools":[{"type":"web_search","filters":{"search_after_date_filter":"02/30/2026"}}]}), + ] { + let mut request = ModelRequest::new(vec![Message::user("question")]); + request.provider_options = json!({"perplexity":options}); + assert!(build(&config, &request, false, false).is_err(), "{options}"); + } + let mut request = ModelRequest::new(vec![Message::user("question")]); + request.seed = Some(1); + assert!(matches!( + build(&config, &request, false, false), + Err(Error::Unsupported(_)) + )); + request.seed = None; + request.reasoning = Some(ReasoningConfig::effort(ReasoningEffort::None)); + assert!(matches!( + build(&config, &request, false, false), + Err(Error::Unsupported(_)) + )); +} + +#[test] +fn anthropic_and_fallback_selections_require_an_explicit_output_cap() { + let mut request = ModelRequest::new(vec![Message::user("question")]); + for selection in [ + PerplexitySelection::Model("anthropic/model".into()), + PerplexitySelection::Models(vec!["openai/model".into(), "anthropic/model".into()]), + ] { + let config = PerplexityConfig::new(selection); + assert!(build(&config, &request, false, false).is_err()); + request.max_tokens = Some(32); + assert_eq!( + build(&config, &request, false, false).unwrap()["max_output_tokens"], + 32 + ); + request.max_tokens = None; + } + let config = PerplexityConfig::new(PerplexitySelection::Models(vec!["openai/model".into()])); + request.model = Some("google/model".into()); + assert!(build(&config, &request, false, false).is_err()); +} + +#[test] +fn structured_output_and_document_inputs_use_the_native_agent_shape() { + let config = PerplexityConfig::new(PerplexitySelection::Model("openai/model".into())); + let mut request = ModelRequest::new(vec![Message::User(crate::message::UserMessage { + content: vec![ + ContentBlock::Text("summarize".into()), + ContentBlock::Document(MediaRef::base64("aGVsbG8=", "text/plain")), + ], + })]); + request.response_format = Some(ResponseFormat::json_schema( + "answer", + json!({"type":"object"}), + )); + let body = build(&config, &request, false, false).unwrap(); + assert_eq!( + body["input"][0]["content"][1], + json!({"type":"input_file","file_data":"aGVsbG8=","filename":"document.txt"}) + ); + assert_eq!(body["response_format"]["type"], "json_schema"); + assert!(body.get("text").is_none()); + request.messages = vec![Message::User(crate::message::UserMessage { + content: vec![ContentBlock::Document(MediaRef::path("/tmp/private"))], + })]; + assert!(matches!( + build(&config, &request, false, false), + Err(Error::Unsupported(_)) + )); +} + +#[test] +fn remote_credentials_are_bound_to_their_server_and_never_serialized_into_requests() { + let mut config = PerplexityConfig::new(PerplexitySelection::Model("openai/model".into())); + config.remote_credentials.insert( + "source".into(), + PerplexityRemoteCredentials { + server_url: "https://trusted.example/mcp".into(), + authorization: Some("remote-secret".into()), + headers: Default::default(), + }, + ); + let options = PerplexityOptions { + tools: Some(vec![PerplexityTool::Mcp { + server_label: "source".into(), + server_url: "https://trusted.example/mcp".into(), + allowed_tools: vec!["read".into()], + defer_loading: Some(true), + }]), + ..Default::default() + }; + let mut request = ModelRequest::new(vec![Message::user("question")]); + options.apply_to(&mut request).unwrap(); + assert!( + !serde_json::to_string(&request) + .unwrap() + .contains("remote-secret") + ); + assert_eq!( + build(&config, &request, false, false).unwrap()["tools"][0]["authorization"], + "remote-secret" + ); + request.provider_options["perplexity"]["tools"][0]["server_url"] = + json!("https://other.example/mcp"); + assert!(build(&config, &request, false, false).is_err()); + assert!(!format!("{config:?}").contains("remote-secret")); +} + +#[test] +fn signed_function_calls_and_results_replay_natively_with_original_arguments() { + let config = PerplexityConfig::new(PerplexitySelection::Model("google/model".into())); + let mut call = ToolCall::new("call_1", "lookup", json!({"id":7})); + call.replay = Some(ToolCallReplay { + origin: crate::message::MessageOrigin { + provider: "perplexity".into(), + api: "agent".into(), + model: "google/model".into(), + }, + item_id: Some("fc_1".into()), + arguments: "{ \"id\": 7 }".into(), + thought_signature: Some("signed".into()), + }); + let result = ToolMessage::for_call(&call, vec![ContentBlock::Json(json!({"ok":true}))]); + let mut assistant = crate::model::ModelResponse::assistant("").message; + assistant.content.clear(); + assistant.tool_calls = vec![call.clone()]; + let mut request = ModelRequest::new(vec![ + Message::user("lookup"), + Message::Assistant(assistant), + Message::Tool(result.clone()), + ]); + request.tools = vec![ToolSchema::new( + "lookup", + "lookup", + json!({"type":"object"}), + )]; + let body = build(&config, &request, false, false).unwrap(); + assert_eq!(body["input"][1]["type"], "function_call"); + assert_eq!(body["input"][1]["arguments"], "{ \"id\": 7 }"); + assert_eq!(body["input"][2]["call_id"], "call_1"); + assert_eq!(body["input"][2]["thought_signature"], "signed"); + request.continuation_id = Some("resp_1".into()); + request.messages = vec![Message::Tool(result)]; + assert_eq!( + build(&config, &request, false, false).unwrap()["previous_response_id"], + "resp_1" + ); + request.model = Some("openai/model".into()); + assert!(matches!( + build(&config, &request, false, false), + Err(Error::Unsupported(_)) + )); +} + +#[test] +fn unmatched_or_duplicate_function_results_are_rejected() { + let config = PerplexityConfig::new(PerplexitySelection::Model("openai/model".into())); + let call = ToolCall::new("call_1", "lookup", json!({})); + let result = Message::Tool(ToolMessage::for_call( + &call, + vec![ContentBlock::Text("result".into())], + )); + let request = ModelRequest::new(vec![Message::user("question"), result.clone()]); + assert!(build(&config, &request, false, false).is_err()); + let mut request = ModelRequest::new(vec![result.clone(), result]); + request.continuation_id = Some("resp_1".into()); + assert!(build(&config, &request, false, false).is_err()); +} diff --git a/crates/tinyinference-llm/src/providers/perplexity/response.rs b/crates/tinyinference-llm/src/providers/perplexity/response.rs new file mode 100644 index 00000000..b818e732 --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/response.rs @@ -0,0 +1,695 @@ +use crate::message::{ContentBlock, MessageOrigin}; +use crate::model::*; +use crate::tool::{ToolCall, ToolCallReplay}; +use crate::usage::{ChargedAmount, Usage}; +use crate::{Error, Result}; +use serde::de::DeserializeOwned; +use serde_json::Value; +use std::collections::{BTreeMap, HashSet}; + +pub(super) struct DecodedJson { + pub value: Value, + pub usage: Option>, +} + +impl From for DecodedJson { + fn from(value: Value) -> Self { + Self { value, usage: None } + } +} + +pub(super) fn decode(bytes: &[u8]) -> Result { + let value: Value = + serde_json::from_slice(bytes).map_err(|_| malformed("response is not valid JSON"))?; + let path: &[&str] = if value.get("usage").is_some_and(Value::is_object) { + &["usage"] + } else if value + .get("response") + .and_then(|v| v.get("usage")) + .is_some_and(Value::is_object) + { + &["response", "usage"] + } else { + return Ok(DecodedJson { value, usage: None }); + }; + let raw: Box = + serde_json::from_slice(bytes).map_err(|_| malformed("response is not valid JSON"))?; + Ok(DecodedJson { + value, + usage: raw_at(Some(&raw), path), + }) +} + +pub(super) fn parse(decoded: impl Into) -> Result { + let DecodedJson { + value, + usage: exact_usage, + } = decoded.into(); + let id = identity(&value, "id")?; + let status = ExecutionStatus::from(identity(&value, "status")?); + let model = if matches!( + status, + ExecutionStatus::Queued | ExecutionStatus::InProgress | ExecutionStatus::Cancelling + ) { + string(&value, "model")? + } else { + identity(&value, "model")? + }; + let mut response = ModelResponse::assistant(""); + response.message.content.clear(); + response.raw = Some(value.clone()); + response.execution = Some(ModelExecution { + id: id.clone(), + model: model.clone(), + status, + incomplete_reason: value + .get("incomplete_details") + .and_then(|v| v.get("reason")) + .and_then(Value::as_str) + .map(str::to_owned), + sequence_number: optional(&value, "sequence_number")?, + progress: Vec::new(), + cost: None, + tool_usage: BTreeMap::new(), + }); + let decode = (|| -> Result<()> { + let items = value + .get("output") + .and_then(Value::as_array) + .ok_or_else(|| malformed("output must be an array"))?; + let mut calls = HashSet::new(); + for (index, item) in items.iter().enumerate() { + let item = parse_item(item, index, &id, &model)?; + if let ModelOutputKind::FunctionCall { call } = &item.kind + && !calls.insert(call.id.clone()) + { + return Err(malformed("duplicate function call id")); + } + response.output.push(item); + } + if let Some(usage) = value.get("usage").filter(|v| !v.is_null()) { + let (tokens, cost, tools) = parse_usage(usage, exact_usage.as_deref())?; + response.usage = Some(tokens); + if let Some(execution) = &mut response.execution { + execution.cost = cost; + execution.tool_usage = tools; + } + } + Ok(()) + })(); + project(&mut response); + if let Err(error) = decode { + return Err(with_partial(error, response)); + } + Ok(response) +} + +pub(super) fn completed(response: ModelResponse) -> Result { + let status = response.execution.as_ref().map(|v| &v.status); + if matches!( + status, + Some(ExecutionStatus::Completed | ExecutionStatus::Incomplete) + ) { + if response + .raw + .as_ref() + .and_then(|v| v.get("error")) + .is_some_and(|v| !v.is_null()) + { + return Err(failed_response(response, "provider_error")); + } + Ok(response) + } else { + let code = match status { + Some(ExecutionStatus::Failed) => "provider_failed", + Some(ExecutionStatus::Cancelled) => "cancelled", + Some( + ExecutionStatus::Queued | ExecutionStatus::InProgress | ExecutionStatus::Cancelling, + ) => "not_completed", + _ => "unknown_status", + }; + Err(failed_response(response, code)) + } +} + +fn failed_response(response: ModelResponse, code: &str) -> Error { + let raw = response.raw.as_ref().and_then(|v| v.get("error")); + let message = raw + .and_then(|v| v.get("message")) + .and_then(Value::as_str) + .unwrap_or("Perplexity run did not complete successfully"); + let error = Error::Provider(Box::new(ProviderError { + provider: "perplexity".into(), + code: Some( + raw.and_then(|v| v.get("code")) + .and_then(Value::as_str) + .unwrap_or(code) + .into(), + ), + message: tinyinference_core::sanitize::sanitize_api_error(message), + ..Default::default() + })); + with_partial(error, response) +} + +pub(super) fn with_partial(error: Error, response: ModelResponse) -> Error { + let mut error = match error { + Error::Provider(error) => *error, + _ => ProviderError { + provider: "perplexity".into(), + code: Some("invalid_response".into()), + message: "invalid Perplexity response".into(), + ..Default::default() + }, + }; + error.model = response.execution.as_ref().map(|e| e.model.clone()); + error.stop_reason = response.finish_reason.clone(); + error.partial_message = Some(response.message.clone()); + error.partial_response = Some(Box::new(response)); + Error::Provider(Box::new(error)) +} + +pub(super) fn project(response: &mut ModelResponse) { + response.message.content.clear(); + response.message.tool_calls.clear(); + response.message.id = None; + if let Some(execution) = &response.execution { + response.message.origin = Some(origin(&execution.model)); + response.finish_reason = Some(match execution.status { + ExecutionStatus::Incomplete => execution + .incomplete_reason + .clone() + .unwrap_or_else(|| "incomplete".into()), + ExecutionStatus::Completed => "stop".into(), + _ => String::from(execution.status.clone()), + }); + } + for item in &response.output { + match &item.kind { + ModelOutputKind::Message { role, content } if role == "assistant" => { + if response.message.id.is_none() { + response.message.id = item.id.clone(); + } + response + .message + .content + .extend(content.iter().map(|part| part.block.clone())); + } + ModelOutputKind::Reasoning { content } => { + response.message.content.extend(content.clone()) + } + ModelOutputKind::FunctionCall { call } => { + response.message.tool_calls.push(call.clone()) + } + _ => {} + } + } + if !response.message.tool_calls.is_empty() + && response + .execution + .as_ref() + .is_some_and(|e| e.status == ExecutionStatus::Completed) + { + response.finish_reason = Some("tool_calls".into()); + } + response.message.usage = response.usage; +} + +pub(super) fn parse_item( + value: &Value, + index: usize, + response_id: &str, + model: &str, +) -> Result { + let item_type = identity(value, "type")?; + let id = optional::(value, "id")?; + let status = optional::(value, "status")?.map(ExecutionStatus::from); + let kind = match item_type.as_str() { + "message" => { + let role = identity(value, "role")?; + let content = required_array(value, "content")? + .iter() + .map(parse_content) + .collect::>>()?; + ModelOutputKind::Message { role, content } + } + "reasoning" => { + let mut content = Vec::new(); + for part in optional_array(value, "summary")? + .iter() + .chain(optional_array(value, "content")?.iter()) + { + if let Some(text) = part.get("text").and_then(Value::as_str) { + content.push(ContentBlock::Thinking { + text: text.into(), + signature: None, + }); + } else { + content.push(ContentBlock::ProviderExtension(part.clone())); + } + } + if let Some(data) = optional::(value, "encrypted_content")? { + content.push(ContentBlock::RedactedThinking { data }); + } + ModelOutputKind::Reasoning { content } + } + "function_call" => { + let args = string(value, "arguments")?; + let call_id = identity(value, "call_id")?; + let name = identity(value, "name")?; + let mut call = match serde_json::from_str::(&args) { + Ok(arguments) => ToolCall::new(call_id, name, arguments), + Err(_) => ToolCall::invalid(call_id, name, &args, "malformed argument JSON"), + }; + call.replay = Some(ToolCallReplay { + origin: origin(model), + item_id: id.clone(), + arguments: args, + thought_signature: optional(value, "thought_signature")?, + }); + ModelOutputKind::FunctionCall { call } + } + "search_results" | "people_search_results" => ModelOutputKind::Search { + tool: if item_type == "search_results" { + "web_search" + } else { + "people_search" + } + .into(), + queries: strings(value, "queries")?, + results: required_array(value, "results")? + .iter() + .map(search_source) + .collect::>()?, + }, + "image_search_results" => ModelOutputKind::ImageSearch { + queries: strings(value, "queries")?, + results: array(value, "results")?, + error: optional(value, "error")?, + }, + "fetch_url_results" => ModelOutputKind::Fetch { + contents: array(value, "contents")?, + }, + "finance_results" => ModelOutputKind::Finance { + categories: strings(value, "categories")?, + tickers: strings(value, "tickers")?, + results: array(value, "results")?, + }, + "sandbox_results" => ModelOutputKind::Sandbox { + code: optional(value, "code")?, + stdout: optional(value, "stdout")?, + stderr: optional(value, "stderr")?, + exit_code: optional(value, "exit_code")?, + duration_ms: optional(value, "duration_ms")?, + status: identity(value, "status")?, + }, + "mcp_list_tools" => ModelOutputKind::ToolDiscovery { + server: identity(value, "server_label")?, + connector_id: optional(value, "connector_id")?, + error: optional(value, "error")?, + tools: required_array(value, "tools")? + .iter() + .map(|tool| remote_tool(tool, "input_schema")) + .collect::>()?, + }, + "mcp_call" => ModelOutputKind::RemoteCall { + server: identity(value, "server_label")?, + connector_id: optional(value, "connector_id")?, + name: identity(value, "name")?, + arguments: string(value, "arguments")?, + output: optional(value, "output")?, + error: optional(value, "error")?, + }, + "tool_search_output" => { + let tools = required_array(value, "tools")? + .iter() + .map(|namespace| { + Ok(RemoteToolNamespace { + name: identity(namespace, "name")?, + description: optional(namespace, "description")?, + tools: required_array(namespace, "tools")? + .iter() + .map(|tool| remote_tool(tool, "parameters")) + .collect::>()?, + }) + }) + .collect::>()?; + ModelOutputKind::ToolSearch { + execution: identity(value, "execution")?, + call_id: optional(value, "call_id")?, + arguments: optional(value, "arguments")?, + tools, + } + } + "share_file" => ModelOutputKind::GeneratedFiles { + response_id: response_id.into(), + metadata: value.clone(), + }, + _ => ModelOutputKind::Extension { + provider: "perplexity".into(), + item_type, + data: value.clone(), + }, + }; + Ok(ModelOutputItem { + index, + id, + status, + kind, + }) +} + +fn parse_content(value: &Value) -> Result { + let kind = identity(value, "type")?; + let block = match kind.as_str() { + "output_text" => ContentBlock::Text(string(value, "text")?), + "reasoning_text" | "summary_text" => ContentBlock::thinking(string(value, "text")?), + _ => ContentBlock::ProviderExtension(value.clone()), + }; + let annotations = optional_array(value, "annotations")? + .iter() + .map(|v| { + if v.get("type").and_then(Value::as_str) == Some("url_citation") { + let start = optional::(v, "start_index")?; + let end = optional::(v, "end_index")?; + if start.zip(end).is_some_and(|(a, b)| a > b) { + return Err(malformed("invalid citation range")); + } + Ok(OutputAnnotation::UrlCitation { + offset_unit: "provider_characters".into(), + url: identity(v, "url")?, + title: optional(v, "title")?, + start, + end, + }) + } else { + Ok(OutputAnnotation::Extension { + provider: "perplexity".into(), + data: v.clone(), + }) + } + }) + .collect::>()?; + Ok(AnnotatedContent { block, annotations }) +} + +fn search_source(value: &Value) -> Result { + let id = match value.get("id") { + Some(Value::String(id)) if !id.is_empty() => id.clone(), + Some(Value::Number(id)) if id.as_u64().is_some() => id.to_string(), + _ => { + return Err(malformed( + "search result id must be a string or nonnegative integer", + )); + } + }; + Ok(SearchSource { + id, + url: identity(value, "url")?, + title: string(value, "title")?, + snippet: string(value, "snippet")?, + source: optional(value, "source")?, + date: optional(value, "date")?, + last_updated: optional(value, "last_updated")?, + }) +} + +fn remote_tool(value: &Value, field: &str) -> Result { + let parameters = match value.get(field) { + Some(schema) if schema.is_object() => schema.clone(), + None if field == "parameters" => Value::Null, + _ => return Err(malformed("remote tool schema must be an object")), + }; + Ok(RemoteToolDefinition { + name: identity(value, "name")?, + description: optional(value, "description")?, + parameters, + }) +} + +type UsageDetails = ( + Usage, + Option, + BTreeMap, +); + +fn parse_usage(value: &Value, exact: Option<&serde_json::value::RawValue>) -> Result { + let mut usage = Usage { + input_tokens: integer(value, "input_tokens")?, + output_tokens: integer(value, "output_tokens")?, + total_tokens: integer(value, "total_tokens")?, + cache_read_tokens: optional::( + &value["input_tokens_details"], + "cache_read_input_tokens", + )? + .or(optional(&value["input_tokens_details"], "cached_tokens")?) + .unwrap_or(0), + cache_creation_tokens: optional::( + &value["input_tokens_details"], + "cache_creation_input_tokens", + )? + .unwrap_or(0), + reasoning_tokens: optional::(&value["output_tokens_details"], "reasoning_tokens")? + .unwrap_or(0), + ..Usage::default() + }; + let cost = if let Some(value) = value.get("cost").filter(|v| !v.is_null()) { + let currency = identity(value, "currency")?; + let mut components = BTreeMap::new(); + for (wire, name) in [ + ("input_cost", "input"), + ("output_cost", "output"), + ("cache_read_cost", "cache_read"), + ("cache_creation_cost", "cache_creation"), + ("tool_calls_cost", "tools"), + ("total_cost", "total"), + ] { + if let Some(amount) = value.get(wire).filter(|v| !v.is_null()) { + components.insert( + name.into(), + decimal(amount, raw_at(exact, &["cost", wire]).as_deref())?, + ); + } + } + if currency == "USD" + && let Some(total) = components.get("total") + { + usage.charged_amount = Some(ChargedAmount::usd_micros(usd_micros(total)?)); + } + let mut tools = BTreeMap::new(); + if let Some(object) = value + .get("tool_calls_cost_details") + .filter(|v| !v.is_null()) + { + for (name, amount) in object + .as_object() + .ok_or_else(|| malformed("invalid tool cost details"))? + { + tools.insert( + name.clone(), + decimal( + amount, + raw_at(exact, &["cost", "tool_calls_cost_details", name]).as_deref(), + )?, + ); + } + } + Some(ReportedCost { + currency, + components, + tools, + }) + } else { + None + }; + let mut tools = BTreeMap::new(); + if let Some(object) = value.get("tool_calls_details").filter(|v| !v.is_null()) { + for (name, detail) in object + .as_object() + .ok_or_else(|| malformed("invalid tool usage details"))? + { + if !detail.is_object() { + return Err(malformed("invalid tool usage entry")); + } + tools.insert( + name.clone(), + HostedToolUsage { + invocations: optional(detail, "invocation")?, + cost_usd: detail + .get("cost_usd") + .filter(|v| !v.is_null()) + .map(|amount| { + decimal( + amount, + raw_at(exact, &["tool_calls_details", name, "cost_usd"]).as_deref(), + ) + }) + .transpose()?, + }, + ); + } + } + Ok((usage, cost, tools)) +} + +fn raw_at( + raw: Option<&serde_json::value::RawValue>, + path: &[&str], +) -> Option> { + let (key, tail) = path.split_first()?; + let mut fields: BTreeMap> = + serde_json::from_str(raw?.get()).ok()?; + let value = fields.remove(*key)?; + if tail.is_empty() { + Some(value) + } else { + raw_at(Some(&value), tail) + } +} + +fn decimal(value: &Value, exact: Option<&serde_json::value::RawValue>) -> Result { + match value { + Value::Number(number) if number.as_f64().is_some_and(|n| n.is_finite() && n >= 0.0) => { + let text = exact.map_or_else(|| number.to_string(), |v| v.get().to_owned()); + let coefficient = text.split(['e', 'E']).next().unwrap_or_default(); + if text.starts_with('-') && coefficient.bytes().any(|b| (b'1'..=b'9').contains(&b)) { + return Err(malformed("reported cost must not be negative")); + } + Ok(text) + } + _ => Err(malformed( + "reported cost must be a nonnegative finite number", + )), + } +} + +fn usd_micros(decimal: &str) -> Result { + let unsigned = decimal.trim_start_matches('-'); + let (coefficient, exponent) = match unsigned.split_once(['e', 'E']) { + Some((coefficient, exponent)) => ( + coefficient, + exponent + .parse::() + .map_err(|_| malformed("invalid cost exponent"))?, + ), + None => (unsigned, 0), + }; + let fraction = coefficient + .split_once('.') + .map_or(0, |(_, digits)| digits.len()); + let digits = coefficient.replace('.', ""); + let digits = digits.trim_start_matches('0'); + if digits.is_empty() { + return Ok(0); + } + let scale = i64::from(exponent) + 6 - fraction as i64; + let cut = digits.len() as i64 + scale; + if cut < 0 { + return Ok(0); + } + if cut > 19 { + return Err(malformed("reported cost overflows micro-units")); + } + let cut = cut as usize; + let (whole, round_up) = if cut >= digits.len() { + let whole = digits + .parse::() + .ok() + .and_then(|v| v.checked_mul(10_i64.pow((cut - digits.len()) as u32))) + .ok_or_else(|| malformed("reported cost overflows micro-units"))?; + (whole, false) + } else { + let whole = if cut == 0 { + 0 + } else { + digits[..cut] + .parse::() + .map_err(|_| malformed("reported cost overflows micro-units"))? + }; + (whole, digits.as_bytes()[cut] >= b'5') + }; + whole + .checked_add(i64::from(round_up)) + .ok_or_else(|| malformed("reported cost overflows micro-units")) +} + +fn origin(model: &str) -> MessageOrigin { + MessageOrigin { + provider: "perplexity".into(), + api: "agent".into(), + model: model.into(), + } +} +fn integer(value: &Value, field: &str) -> Result { + value + .get(field) + .and_then(Value::as_u64) + .ok_or_else(|| malformed("token usage must contain nonnegative integers")) +} +fn identity(value: &Value, field: &str) -> Result { + let text = string(value, field)?; + if text.is_empty() { + Err(malformed("required identity must not be empty")) + } else { + Ok(text) + } +} +fn string(value: &Value, field: &str) -> Result { + value + .get(field) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| malformed("required string is missing or invalid")) +} +fn optional(value: &Value, field: &str) -> Result> { + value + .get(field) + .filter(|v| !v.is_null()) + .map(|v| serde_json::from_value(v.clone()).map_err(|_| malformed("invalid optional field"))) + .transpose() +} +fn required_array<'a>(value: &'a Value, field: &str) -> Result<&'a [Value]> { + value + .get(field) + .and_then(Value::as_array) + .map(Vec::as_slice) + .ok_or_else(|| malformed("required array is missing or invalid")) +} +fn optional_array<'a>(value: &'a Value, field: &str) -> Result<&'a [Value]> { + match value.get(field) { + None | Some(Value::Null) => Ok(&[]), + Some(Value::Array(v)) => Ok(v), + _ => Err(malformed("invalid optional array")), + } +} +fn array(value: &Value, field: &str) -> Result> { + required_array(value, field)? + .iter() + .map(|v| { + serde_json::from_value(v.clone()).map_err(|_| malformed("malformed known tool result")) + }) + .collect() +} +fn strings(value: &Value, field: &str) -> Result> { + optional_array(value, field)? + .iter() + .map(|v| { + v.as_str() + .map(str::to_owned) + .ok_or_else(|| malformed("expected string array")) + }) + .collect() +} + +pub(super) fn malformed(message: &str) -> Error { + Error::Provider(Box::new(ProviderError { + provider: "perplexity".into(), + code: Some("invalid_response".into()), + message: message.into(), + ..Default::default() + })) +} + +#[cfg(test)] +#[path = "response_tests.rs"] +mod tests; diff --git a/crates/tinyinference-llm/src/providers/perplexity/response_tests.rs b/crates/tinyinference-llm/src/providers/perplexity/response_tests.rs new file mode 100644 index 00000000..a9fb5ec9 --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/response_tests.rs @@ -0,0 +1,203 @@ +use super::*; +use serde_json::json; + +#[test] +fn response_preserves_sources_annotations_cost_and_distinct_ids() { + let response = parse(json!({ + "id":"resp_1","model":"openai/model","status":"completed", + "output":[ + {"type":"search_results","queries":["question"],"results":[ + {"id":1,"url":"https://example.com","title":"Source","snippet":"evidence"} + ]}, + {"id":"msg_1","type":"message","status":"completed","role":"assistant","content":[ + {"type":"output_text","text":"Café [1]","annotations":[{"type":"url_citation","url":"https://example.com","start_index":0,"end_index":4}]} + ]} + ], + "usage":{"input_tokens":20,"output_tokens":5,"total_tokens":25, + "cost":{"currency":"USD","total_cost":0.0000007,"tool_calls_cost":0.0025}, + "tool_calls_details":{"web_search":{"invocation":1}} + } + })).unwrap(); + assert_eq!(response.text(), "Café [1]"); + assert_eq!(response.message.id.as_deref(), Some("msg_1")); + assert_eq!(response.execution.as_ref().unwrap().id, "resp_1"); + assert_eq!(response.output.len(), 2); + assert!(response.message.tool_calls.is_empty()); + assert_eq!(response.usage.unwrap().charged_amount.unwrap().micros, 1); + match &response.output[1].kind { + ModelOutputKind::Message { content, .. } => assert_eq!(content[0].annotations.len(), 1), + _ => panic!("expected normalized message"), + } +} + +#[test] +fn all_documented_hosted_outputs_remain_typed_and_are_never_local_calls() { + let values = vec![ + json!({"type":"search_results","queries":["q"],"results":[{"id":1,"url":"u","title":"t","snippet":"s"}]}), + json!({"type":"people_search_results","results":[{"id":1,"url":"u","title":"t","snippet":"s","source":"people"}]}), + json!({"type":"image_search_results","queries":["q"],"results":[{"image_url":"i","origin_url":"o","width":10,"height":20}],"error":"partial"}), + json!({"type":"fetch_url_results","contents":[{"url":"u","title":"t","snippet":"s"}]}), + json!({"type":"finance_results","tickers":["ABC"],"categories":["quote"],"results":[{"category":"quote","content":"price table","sources":["u"]}]}), + json!({"type":"sandbox_results","status":"failed","code":"exit(1)","stdout":"out","stderr":"err","exit_code":1,"duration_ms":7}), + json!({"type":"mcp_list_tools","id":"m1","server_label":"source","connector_id":"connector1","tools":[{"name":"lookup","input_schema":{"type":"object"}}]}), + json!({"type":"mcp_call","id":"m2","server_label":"source","name":"lookup","arguments":"{}","output":"found","error":null}), + json!({"type":"tool_search_output","id":"t1","status":"future_state","execution":"server","call_id":null,"arguments":"{}","tools":[{"type":"namespace","name":"source","description":"tools","tools":[{"type":"function","name":"lookup","parameters":{"type":"object"}}]}]}), + json!({"type":"share_file","filename":"report.csv","new_provider_field":7}), + json!({"type":"future_output","payload":{"answer":42}}), + ]; + let response = completed( + parse(json!({"id":"r","model":"openai/m","status":"completed","output":values})).unwrap(), + ) + .unwrap(); + assert_eq!(response.output.len(), 11); + let round_trip: ModelResponse = + serde_json::from_slice(&serde_json::to_vec(&response).unwrap()).unwrap(); + assert_eq!(round_trip, response); + assert!(response.message.tool_calls.is_empty()); + assert!(response.text().is_empty()); + assert!( + matches!(&response.output[0].kind,ModelOutputKind::Search {tool,..} if tool=="web_search") + ); + assert!( + matches!(&response.output[1].kind,ModelOutputKind::Search {tool,..} if tool=="people_search") + ); + assert!( + matches!(&response.output[2].kind,ModelOutputKind::ImageSearch {error:Some(error),..} if error=="partial") + ); + assert!( + matches!(&response.output[3].kind,ModelOutputKind::Fetch {contents} if contents[0].snippet=="s") + ); + assert!( + matches!(&response.output[4].kind,ModelOutputKind::Finance {results,..} if results[0].content=="price table") + ); + assert!(matches!( + &response.output[5].kind, + ModelOutputKind::Sandbox { + exit_code: Some(1), + .. + } + )); + assert!( + matches!(&response.output[6].kind,ModelOutputKind::ToolDiscovery {tools,..} if tools[0].name=="lookup") + ); + assert!( + matches!(&response.output[7].kind,ModelOutputKind::RemoteCall {output:Some(output),..} if output=="found") + ); + assert!( + matches!(&response.output[8].kind,ModelOutputKind::ToolSearch {tools,..} if tools[0].tools[0].name=="lookup") + ); + assert_eq!( + response.output[8].status, + Some(ExecutionStatus::Other("future_state".into())) + ); + assert!( + matches!(&response.output[9].kind,ModelOutputKind::GeneratedFiles {metadata,..} if metadata["new_provider_field"]==7) + ); + assert!( + matches!(&response.output[10].kind,ModelOutputKind::Extension {data,..} if data["payload"]["answer"]==42) + ); +} + +#[test] +fn function_calls_preserve_raw_arguments_and_signatures_even_when_invalid() { + let response=parse(json!({"id":"r","model":"google/m","status":"completed","output":[ + {"type":"function_call","id":"item1","call_id":"call1","name":"lookup","arguments":"{ broken","thought_signature":"opaque"} + ]})).unwrap(); + assert_eq!(response.finish_reason.as_deref(), Some("tool_calls")); + let call = &response.message.tool_calls[0]; + assert!(call.is_invalid()); + assert_eq!(call.id, "call1"); + assert_eq!( + call.replay.as_ref().unwrap().item_id.as_deref(), + Some("item1") + ); + assert_eq!( + call.replay.as_ref().unwrap().thought_signature.as_deref(), + Some("opaque") + ); + assert_eq!(call.replay.as_ref().unwrap().arguments, "{ broken"); +} + +#[test] +fn malformed_known_items_fail_and_retain_preceding_rich_output() { + for bad in [ + json!({"type":"search_results","results":"wrong"}), + json!({"type":"message","role":"assistant","content":[{"type":"output_text"}]}), + json!({"type":"function_call","arguments":"{}","name":"x"}), + json!({"type":"sandbox_results","status":"completed","duration_ms":-1}), + ] { + let result = parse( + json!({"id":"r","model":"openai/m","status":"completed","output":[ + {"type":"share_file","file_id":"f"},bad + ]}), + ); + let Err(Error::Provider(error)) = result else { + panic!("malformed known output must fail"); + }; + let partial = error.partial_response.unwrap(); + assert_eq!(partial.output.len(), 1); + assert!(matches!( + partial.output[0].kind, + ModelOutputKind::GeneratedFiles { .. } + )); + } +} + +#[test] +fn incomplete_failed_cancelled_and_unknown_states_are_not_reported_as_normal_success() { + let incomplete=completed(parse(json!({"id":"r","model":"openai/m","status":"incomplete","incomplete_details":{"reason":"max_output_tokens"},"output":[]})).unwrap()).unwrap(); + assert_eq!( + incomplete.finish_reason.as_deref(), + Some("max_output_tokens") + ); + for status in [ + "failed", + "cancelled", + "in_progress", + "queued", + "future_state", + ] { + let result = completed( + parse(json!({"id":"r","model":"openai/m","status":status,"output":[]})).unwrap(), + ); + let Err(Error::Provider(error)) = result else { + panic!("state {status} is not completed"); + }; + assert_eq!(error.partial_response.unwrap().execution.unwrap().id, "r"); + } +} + +#[test] +fn missing_cost_is_unknown_and_invalid_usage_is_not_fabricated() { + let response = + parse(json!({"id":"r","model":"openai/m","status":"completed","output":[]})).unwrap(); + assert!(response.usage.is_none()); + for usage in [ + json!({"input_tokens":-1,"output_tokens":0,"total_tokens":0}), + json!({"input_tokens":1,"output_tokens":2,"total_tokens":3,"cost":{"currency":"USD","total_cost":-0.1}}), + json!({"input_tokens":1,"output_tokens":2,"total_tokens":3,"cost":{"currency":"USD","total_cost":1e30}}), + ] { + assert!( + parse( + json!({"id":"r","model":"openai/m","status":"completed","output":[],"usage":usage}) + ) + .is_err() + ); + } +} + +#[test] +fn reported_decimal_cost_keeps_precision_at_micro_unit_rounding_boundaries() { + let value = decode( + br#"{"id":"r","model":"openai/m","status":"completed","output":[], + "usage":{"input_tokens":1,"output_tokens":1,"total_tokens":2, + "cost":{"currency":"USD","total_cost":0.000000499999999999999999}}}"#, + ) + .unwrap(); + let response = parse(value).unwrap(); + assert_eq!(response.usage.unwrap().charged_amount.unwrap().micros, 0); + assert_eq!( + response.execution.unwrap().cost.unwrap().components["total"], + "0.000000499999999999999999" + ); +} diff --git a/crates/tinyinference-llm/src/providers/perplexity/stream.rs b/crates/tinyinference-llm/src/providers/perplexity/stream.rs new file mode 100644 index 00000000..cd3c52de --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/stream.rs @@ -0,0 +1,833 @@ +use super::{ + response, + transport::{self, Transport}, +}; +use crate::message::MessageDelta; +use crate::model::*; +use crate::tool::ToolDelta; +use crate::{Error, Result}; +use bytes::Bytes; +use futures::{Stream, StreamExt}; +use serde_json::{Value, json}; +use std::{ + collections::{BTreeMap, VecDeque}, + pin::Pin, + sync::Arc, +}; +use tokio::time::Instant; + +pub(super) fn open( + inner: Arc, + incoming: reqwest::Response, + deadline: Instant, + seed: Option, + after: Option, +) -> Result { + if !incoming + .headers() + .get(reqwest::header::CONTENT_TYPE) + .and_then(|v| v.to_str().ok()) + .is_some_and(|v| v.to_ascii_lowercase().starts_with("text/event-stream")) + { + return Err(transport::failure( + "invalid_stream", + "expected an event stream", + None, + )); + } + let mut snapshot = ModelResponse::assistant(""); + snapshot.message.content.clear(); + snapshot.execution = seed; + let state = State { + bytes: Box::pin(incoming.bytes_stream().map(|r| { + r.map_err(|_| { + transport::failure( + "stream_interrupted", + "stream connection was interrupted", + None, + ) + }) + })), + inner, + deadline, + buffer: Vec::new(), + data: Vec::new(), + event_name: None, + queue: VecDeque::from([ModelStreamItem::Started]), + snapshot, + items: BTreeMap::new(), + text: BTreeMap::new(), + arguments: BTreeMap::new(), + sequence: after, + resumed: after.is_some(), + done: false, + received: 0, + progress: Vec::new(), + }; + Ok(ModelStream::new(Box::pin(futures::stream::unfold( + state, next, + )))) +} + +struct State { + bytes: Pin> + Send>>, + inner: Arc, + deadline: Instant, + buffer: Vec, + data: Vec, + event_name: Option, + queue: VecDeque, + snapshot: ModelResponse, + items: BTreeMap, + text: BTreeMap<(usize, &'static str, usize), String>, + arguments: BTreeMap, + sequence: Option, + resumed: bool, + done: bool, + received: usize, + progress: Vec, +} + +impl State { + fn record_progress(&mut self, progress: ModelProgress) { + self.progress.push(progress.clone()); + self.emit(ModelOutputEvent::Progress(progress)); + } + fn emit(&mut self, event: ModelOutputEvent) { + self.queue.push_back(ModelStreamItem::OutputEvent(event)); + } + + fn partial(&self) -> ModelResponse { + let mut partial = self.snapshot.clone(); + if let Some(execution) = &mut partial.execution { + execution.progress = self.progress.clone(); + } + let run = partial.execution.as_ref(); + let id = run.map(|v| v.id.as_str()).unwrap_or_default(); + let model = run.map(|v| v.model.as_str()).unwrap_or_default(); + partial.output = self + .items + .iter() + .filter_map(|(index, item)| response::parse_item(item, *index, id, model).ok()) + .collect(); + let mut raw = partial.raw.take().unwrap_or_else(|| json!({})); + raw["output"] = Value::Array(self.items.values().cloned().collect()); + partial.raw = Some(raw); + response::project(&mut partial); + partial + } + + fn fail(&mut self, error: Error) { + let mut partial = self.partial(); + let error = match error { + Error::Provider(mut error) => { + if let Some(terminal) = error.partial_response.take() { + partial.execution = terminal.execution.or(partial.execution); + partial.usage = terminal.usage.or(partial.usage); + for item in terminal.output { + let compatible = terminal + .raw + .as_ref() + .and_then(|v| v.get("output")) + .and_then(|v| v.get(item.index)) + .is_some_and(|raw| self.check_item(item.index, raw).is_ok()); + if compatible { + if let Some(previous) = + partial.output.iter_mut().find(|v| v.index == item.index) + { + *previous = item; + } else { + partial.output.push(item); + } + } + } + partial.raw = terminal.raw.or(partial.raw); + } + Error::Provider(error) + } + error => error, + }; + if let Some(execution) = &mut partial.execution { + execution.progress = self.progress.clone(); + execution.sequence_number = self.sequence; + } + partial.output.sort_by_key(|item| item.index); + response::project(&mut partial); + let error = response::with_partial(error, partial); + if let Error::Provider(error) = error { + self.queue + .push_back(ModelStreamItem::ProviderFailed(*error)); + } + self.done = true; + self.buffer.clear(); + self.data.clear(); + } + + fn frame(&mut self) -> Result<()> { + if Instant::now() >= self.deadline { + return Err(transport::timeout()); + } + if self.data.is_empty() { + self.event_name = None; + return Ok(()); + } + let bytes = std::mem::take(&mut self.data); + if bytes.as_slice() == b"[DONE]" { + return Err(response::malformed( + "stream ended without an authoritative terminal response", + )); + } + let decoded = response::decode(&bytes)?; + let mut value = decoded.value; + if !value.is_object() { + return Err(response::malformed("stream event must be an object")); + } + if value.get("type").is_none() + && let Some(name) = self.event_name.take() + { + value["type"] = json!(name); + } + self.event_name = None; + self.event(self.inner.scrub_value(value), decoded.usage) + } + + fn lines(&mut self) -> Result<()> { + while let Some(end) = self.buffer.iter().position(|b| *b == b'\n') { + let mut line = self.buffer.drain(..=end).collect::>(); + line.pop(); + if line.last() == Some(&b'\r') { + line.pop(); + } + if line.is_empty() { + self.frame()?; + if self.done { + self.buffer.clear(); + break; + } + } else if let Some(rest) = line.strip_prefix(b"data:") { + let rest = rest.strip_prefix(b" ").unwrap_or(rest); + if !self.data.is_empty() { + self.data.push(b'\n'); + } + if rest.len() + > self + .inner + .config + .max_event_bytes + .saturating_sub(self.data.len()) + { + return Err(response::malformed("stream event exceeds byte limit")); + } + self.data.extend_from_slice(rest); + } else if let Some(rest) = line.strip_prefix(b"event:") { + self.event_name = Some( + std::str::from_utf8(rest) + .map_err(|_| response::malformed("invalid SSE event name"))? + .trim() + .into(), + ); + } + } + if self.buffer.len().saturating_add(self.data.len()) > self.inner.config.max_event_bytes { + return Err(response::malformed( + "unterminated stream event exceeds byte limit", + )); + } + Ok(()) + } + + fn event( + &mut self, + value: Value, + usage: Option>, + ) -> Result<()> { + let event_type = value + .get("type") + .and_then(Value::as_str) + .ok_or_else(|| response::malformed("stream event has no type"))?; + let run_id = value + .get("response") + .and_then(|v| v.get("id")) + .or_else(|| value.get("response_id")) + .and_then(Value::as_str); + if let (Some(expected), Some(actual)) = (self.snapshot.execution.as_ref(), run_id) + && expected.id != actual + { + return Err(response::malformed("stream response identity changed")); + } + if let Some(sequence) = value.get("sequence_number") { + let sequence = sequence + .as_u64() + .ok_or_else(|| response::malformed("invalid stream sequence number"))?; + if self.sequence.is_some_and(|previous| sequence <= previous) { + return Ok(()); + } + self.sequence = Some(sequence); + } + if let Some(thought) = value.get("thought").filter(|v| !v.is_null()) { + self.record_progress(ModelProgress::ReasoningText { + text: thought + .as_str() + .ok_or_else(|| response::malformed("reasoning note must be text"))? + .into(), + }); + } + match event_type { + "response.created" | "response.in_progress" => { + let snapshot = response::parse(response::DecodedJson { + value: value + .get("response") + .cloned() + .ok_or_else(|| response::malformed("missing response snapshot"))?, + usage, + })?; + if let Some(raw) = &snapshot.raw + && let Some(items) = raw["output"].as_array() + { + for (index, item) in items.iter().enumerate() { + self.items.insert(index, item.clone()); + } + } + self.snapshot = snapshot; + } + "response.output_item.added" | "response.output_item.done" => { + let index = self.index(&value)?; + let item = value + .get("item") + .ok_or_else(|| response::malformed("missing output item"))? + .clone(); + self.check_item(index, &item)?; + self.items.insert(index, item.clone()); + if event_type.ends_with(".added") { + self.emit(ModelOutputEvent::ItemStarted { + index, + id: item.get("id").and_then(Value::as_str).map(str::to_owned), + item_type: item + .get("type") + .and_then(Value::as_str) + .ok_or_else(|| response::malformed("missing item type"))? + .into(), + }); + } else { + self.complete_item(index, &item)?; + } + } + "response.content_part.added" | "response.content_part.done" => { + let index = self.index(&value)?; + let part_index = content_index(&value)?; + let part = value + .get("part") + .ok_or_else(|| response::malformed("missing content part"))? + .clone(); + self.part(index, part_index, &value)?; + let mut item = self.items[&index].clone(); + item[content_area(&value)][part_index] = part; + self.check_item(index, &item)?; + self.items.insert(index, item); + if event_type.ends_with(".done") { + self.complete_text(index, content_area(&value), part_index)?; + } + } + "response.output_text.delta" + | "response.reasoning_text.delta" + | "response.reasoning_summary_text.delta" => { + let index = self.index(&value)?; + let part_index = content_index(&value)?; + let delta = value + .get("delta") + .and_then(Value::as_str) + .ok_or_else(|| response::malformed("missing text delta"))?; + let reasoning = event_type != "response.output_text.delta"; + let part = self.part(index, part_index, &value)?; + if reasoning { + part["type"] = json!("reasoning_text"); + } + let text = part + .get("text") + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned() + + delta; + part["text"] = json!(text); + self.text + .entry((index, content_area(&value), part_index)) + .or_default() + .push_str(delta); + self.queue + .push_back(ModelStreamItem::MessageDelta(if reasoning { + MessageDelta::reasoning(delta) + } else { + MessageDelta::text(delta) + })); + } + "response.output_text.done" + | "response.reasoning_text.done" + | "response.reasoning_summary_text.done" => { + let index = self.index(&value)?; + let part_index = content_index(&value)?; + self.part(index, part_index, &value)?; + let mut item = self.items[&index].clone(); + let part = &mut item[content_area(&value)][part_index]; + if event_type != "response.output_text.done" { + part["type"] = json!("reasoning_text"); + } + part["text"] = json!( + value + .get("text") + .and_then(Value::as_str) + .ok_or_else(|| response::malformed("missing final text"))? + ); + self.check_item(index, &item)?; + self.items.insert(index, item); + self.complete_text(index, content_area(&value), part_index)?; + } + "response.output_text.annotation.added" => { + let index = self.index(&value)?; + let part_index = number(&value, "content_index")?; + let annotation = value + .get("annotation") + .ok_or_else(|| response::malformed("missing annotation"))? + .clone(); + let part = self.part(index, part_index, &value)?; + if part.get("annotations").is_none() { + part["annotations"] = json!([]); + } + part["annotations"] + .as_array_mut() + .ok_or_else(|| response::malformed("invalid annotations"))? + .push(annotation); + let item = self.items[&index].clone(); + self.emit_normalized_item(index, &item)?; + } + "response.function_call_arguments.delta" => { + let index = self.index(&value)?; + let delta = value + .get("delta") + .and_then(Value::as_str) + .ok_or_else(|| response::malformed("missing argument delta"))?; + let item = self.items.get_mut(&index).ok_or_else(|| { + response::malformed("function delta has no preceding call item") + })?; + item["arguments"] = json!( + item.get("arguments") + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned() + + delta + ); + self.arguments.entry(index).or_default().push_str(delta); + let call_id = item + .get("call_id") + .and_then(Value::as_str) + .ok_or_else(|| response::malformed("function delta has no call id"))?; + self.queue + .push_back(ModelStreamItem::ToolCallDelta(ToolDelta { + call_id: call_id.into(), + content: delta.into(), + tool_name: item.get("name").and_then(Value::as_str).map(str::to_owned), + content_index: Some(index), + })); + } + "response.function_call_arguments.done" => { + let index = self.index(&value)?; + let args = value + .get("arguments") + .and_then(Value::as_str) + .ok_or_else(|| response::malformed("missing final arguments"))?; + let mut item = self + .items + .get(&index) + .ok_or_else(|| response::malformed("function completion has no call item"))? + .clone(); + item["arguments"] = json!(args); + self.check_item(index, &item)?; + self.items.insert(index, item.clone()); + self.complete_item(index, &item)?; + } + "response.reasoning.started" | "response.reasoning.stopped" => { + self.record_progress(ModelProgress::ReasoningPhase { + active: event_type.ends_with(".started"), + }) + } + "response.reasoning.search_queries" + | "response.reasoning.image_search_queries" + | "response.reasoning.fetch_url_queries" => { + let queries = serde_json::from_value( + value + .get(if event_type.contains("fetch_url") { + "urls" + } else { + "queries" + }) + .cloned() + .ok_or_else(|| response::malformed("missing hosted queries"))?, + ) + .map_err(|_| response::malformed("invalid hosted queries"))?; + let tool = if event_type.contains("image_search") { + "image_search" + } else if event_type.contains("fetch_url") { + "fetch_url" + } else { + "web_search" + }; + self.record_progress(ModelProgress::Queries { + tool: tool.into(), + queries, + }); + } + "response.completed" + | "response.incomplete" + | "response.cancelled" + | "response.failed" => { + let raw = value.get("response").cloned().ok_or_else(|| { + response::malformed("terminal event has no response snapshot") + })?; + let mut result = response::parse(response::DecodedJson { + value: raw.clone(), + usage, + })?; + if let Some(execution) = &mut result.execution { + execution.sequence_number = self.sequence; + execution.progress = self.progress.clone(); + } + result = response::completed(result)?; + self.snapshot.execution = result.execution.clone().map(|mut execution| { + execution.progress.clear(); + execution + }); + let items = raw["output"] + .as_array() + .ok_or_else(|| response::malformed("terminal output is not an array"))?; + if !self.resumed && self.items.keys().any(|index| *index >= items.len()) { + return Err(response::malformed( + "terminal response dropped streamed items", + )); + } + for (index, item) in items.iter().enumerate() { + self.check_item(index, item)?; + self.items.insert(index, item.clone()); + self.complete_item(index, item)?; + } + if let Some(usage) = result.usage { + self.queue.push_back(ModelStreamItem::UsageDelta(usage)); + } + if let Some(observer) = &self.inner.config.on_response { + observer(&raw); + } + if Instant::now() >= self.deadline { + return Err(response::with_partial(transport::timeout(), result)); + } + self.snapshot = result.clone(); + self.queue.push_back(ModelStreamItem::Completed(result)); + self.done = true; + } + "error" => { + return Err(transport::failure( + "stream_failed", + &self.inner.scrub_text( + value + .get("message") + .and_then(Value::as_str) + .unwrap_or("provider stream error"), + ), + None, + )); + } + "response.reasoning.search_results" + | "response.reasoning.image_search_results" + | "response.reasoning.fetch_url_results" => { + let mut item = value.clone(); + item["type"] = json!(if event_type.contains("image_search") { + "image_search_results" + } else if event_type.contains("fetch_url") { + "fetch_url_results" + } else { + "search_results" + }); + let run = self.snapshot.execution.as_ref().ok_or_else(|| { + response::malformed("hosted results arrived without response identity") + })?; + let result = response::parse_item(&item, 0, &run.id, &run.model)?; + self.record_progress(ModelProgress::HostedActivity(Box::new(result.kind))); + } + _ => self.record_progress(ModelProgress::Extension { + event_type: event_type.into(), + data: value.clone(), + }), + } + if !self.done + && let Some(execution) = &mut self.snapshot.execution + { + execution.sequence_number = self.sequence; + let execution = execution.clone(); + self.emit(ModelOutputEvent::Execution(Box::new(execution))); + } + Ok(()) + } + + fn index(&self, event: &Value) -> Result { + if event.get("output_index").is_some() { + return number(event, "output_index"); + } + if let Some(id) = event.get("item_id").and_then(Value::as_str) + && let Some((index, _)) = self + .items + .iter() + .find(|(_, item)| item.get("id").and_then(Value::as_str) == Some(id)) + { + return Ok(*index); + } + Err(response::malformed("event has no resolvable output index")) + } + + fn check_item(&self, index: usize, item: &Value) -> Result<()> { + if !self.resumed { + for ((output, area, part), delivered) in &self.text { + if *output == index + && !delivered.is_empty() + && !item + .get(*area) + .and_then(|v| v.get(*part)) + .and_then(|v| v.get("text")) + .and_then(Value::as_str) + .is_some_and(|text| text.starts_with(delivered)) + { + return Err(response::malformed( + "final item conflicts with delivered text", + )); + } + } + if let Some(delivered) = self.arguments.get(&index) + && !delivered.is_empty() + && !item + .get("arguments") + .and_then(Value::as_str) + .is_some_and(|args| args.starts_with(delivered)) + { + return Err(response::malformed( + "final item conflicts with delivered arguments", + )); + } + } + if let Some(previous) = self.items.get(&index) { + for key in ["id", "call_id", "type"] { + if let (Some(a), Some(b)) = (previous.get(key), item.get(key)) + && a != b + { + return Err(response::malformed("output item identity changed")); + } + } + } + Ok(()) + } + + fn part(&mut self, index: usize, part_index: usize, event: &Value) -> Result<&mut Value> { + if part_index > 4096 { + return Err(response::malformed("content index exceeds bounds")); + } + let area = content_area(event); + let reasoning = event + .get("type") + .and_then(Value::as_str) + .is_some_and(|t| t.contains("reasoning")); + let item = self.items.entry(index).or_insert_with(|| { + if reasoning { + json!({"type":"reasoning","id":event.get("item_id"),"content":[],"summary":[]}) + } else { + json!({"type":"message","id":event.get("item_id"),"role":"assistant","content":[]}) + } + }); + if let (Some(a), Some(b)) = ( + item.get("id").and_then(Value::as_str), + event.get("item_id").and_then(Value::as_str), + ) && a != b + { + return Err(response::malformed("text item identity changed")); + } + if matches!( + item.get("type").and_then(Value::as_str), + Some("message" | "reasoning") + ) && item.get(area).is_none_or(Value::is_null) + { + item[area] = json!([]); + } + let content = item + .get_mut(area) + .and_then(Value::as_array_mut) + .ok_or_else(|| response::malformed("text delta addressed a non-message item"))?; + if part_index > content.len() && !self.resumed { + return Err(response::malformed("content part index has a gap")); + } + while content.len() <= part_index { + content.push(json!({"type":"output_text","text":"","annotations":[]})); + } + if !content[part_index].is_object() { + return Err(response::malformed("content part must be an object")); + } + if content[part_index] + .get("text") + .is_some_and(|v| !v.is_string()) + { + return Err(response::malformed("content text must be a string")); + } + Ok(&mut content[part_index]) + } + + fn complete_text(&mut self, index: usize, area: &'static str, part_index: usize) -> Result<()> { + if self.resumed { + return Ok(()); + } + let part = &self.items[&index][area][part_index]; + let Some(text) = part.get("text").and_then(Value::as_str) else { + return Ok(()); + }; + let kind = part.get("type").and_then(Value::as_str).unwrap_or_default(); + if !["output_text", "reasoning_text", "summary_text"].contains(&kind) { + return Ok(()); + } + let previous = self.text.entry((index, area, part_index)).or_default(); + let delta = text + .strip_prefix(previous.as_str()) + .ok_or_else(|| response::malformed("final text conflicts with delivered text"))?; + if !delta.is_empty() { + self.queue + .push_back(ModelStreamItem::MessageDelta(if kind == "output_text" { + MessageDelta::text(delta) + } else { + MessageDelta::reasoning(delta) + })); + } + *previous = text.into(); + Ok(()) + } + + fn emit_normalized_item(&mut self, index: usize, item: &Value) -> Result<()> { + let run = self + .snapshot + .execution + .as_ref() + .ok_or_else(|| response::malformed("item arrived without response identity"))?; + let item = response::parse_item(item, index, &run.id, &run.model)?; + self.emit(ModelOutputEvent::Item(Box::new(item))); + Ok(()) + } + + fn complete_item(&mut self, index: usize, item: &Value) -> Result<()> { + for area in ["content", "summary"] { + if let Some(parts) = item.get(area).and_then(Value::as_array) { + for part_index in 0..parts.len() { + self.complete_text(index, area, part_index)?; + } + } + } + if !self.resumed && item.get("type").and_then(Value::as_str) == Some("function_call") { + let args = item + .get("arguments") + .and_then(Value::as_str) + .ok_or_else(|| response::malformed("missing function arguments"))?; + let previous = self.arguments.entry(index).or_default(); + let delta = args.strip_prefix(previous.as_str()).ok_or_else(|| { + response::malformed("final arguments conflict with delivered arguments") + })?; + if !delta.is_empty() { + self.queue + .push_back(ModelStreamItem::ToolCallDelta(ToolDelta { + call_id: item + .get("call_id") + .and_then(Value::as_str) + .ok_or_else(|| response::malformed("missing function call id"))? + .into(), + tool_name: item.get("name").and_then(Value::as_str).map(str::to_owned), + content: delta.into(), + content_index: Some(index), + })); + } + *previous = args.into(); + } + self.emit_normalized_item(index, item) + } +} + +fn content_area(value: &Value) -> &'static str { + if value + .get("type") + .and_then(Value::as_str) + .is_some_and(|kind| kind.contains("reasoning_summary")) + { + "summary" + } else { + "content" + } +} + +fn content_index(value: &Value) -> Result { + number( + value, + if content_area(value) == "summary" { + "summary_index" + } else { + "content_index" + }, + ) +} + +fn number(value: &Value, key: &str) -> Result { + value + .get(key) + .and_then(Value::as_u64) + .and_then(|n| usize::try_from(n).ok()) + .ok_or_else(|| response::malformed("invalid output/content index")) +} + +async fn next(mut state: State) -> Option<(ModelStreamItem, State)> { + loop { + if let Some(item) = state.queue.pop_front() { + return Some((item, state)); + } + if state.done { + return None; + } + let idle = Instant::now() + .checked_add(state.inner.config.idle_timeout) + .unwrap_or(state.deadline) + .min(state.deadline); + match tokio::time::timeout_at(idle, state.bytes.next()).await { + Err(_) => state.fail(transport::timeout()), + Ok(Some(Err(error))) => state.fail(error), + Ok(Some(Ok(chunk))) => { + state.received = state.received.saturating_add(chunk.len()); + if state.received > state.inner.config.max_body_bytes { + state.fail(response::malformed("stream exceeds total byte limit")); + continue; + } + state.buffer.extend_from_slice(&chunk); + if let Err(error) = state.lines() { + state.fail(error); + } + } + Ok(None) => { + if !state.buffer.is_empty() { + state.buffer.push(b'\n'); + if let Err(error) = state.lines() { + state.fail(error); + continue; + } + } + if !state.done + && !state.data.is_empty() + && let Err(error) = state.frame() + { + state.fail(error); + continue; + } + if !state.done { + state.fail(transport::failure( + "truncated_stream", + "stream ended before completion", + None, + )); + } + } + } + } +} diff --git a/crates/tinyinference-llm/src/providers/perplexity/stream_tests.rs b/crates/tinyinference-llm/src/providers/perplexity/stream_tests.rs new file mode 100644 index 00000000..5ddb4048 --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/stream_tests.rs @@ -0,0 +1,420 @@ +use super::*; + +use super::tests::{complete, fixture}; +use crate::message::Message; +use crate::model::{ModelStreamItem, collect_model_stream}; +use bytes::Bytes; +use futures::{FutureExt, StreamExt}; +use serde_json::{Value, json}; + +fn event(value: Value) -> Bytes { + Bytes::from(format!("data: {value}\n\n")) +} + +fn created() -> Value { + json!({"type":"response.created","sequence_number":0, + "response":{"id":"resp_1","model":"openai/test","status":"in_progress","output":[]}}) +} + +#[tokio::test] +async fn live_provisional_model_and_null_message_content_complete_normally() { + let items = run_events(vec![ + json!({"type":"response.in_progress","sequence_number":1,"response":{"id":"resp_1","model":"","status":"in_progress","output":[]}}), + json!({"type":"response.output_item.added","sequence_number":4,"output_index":0,"item":{"content":null,"id":"msg_1","role":"assistant","status":"completed","type":"message"}}), + json!({"type":"response.output_text.delta","sequence_number":5,"output_index":0,"content_index":0,"item_id":"msg_1","delta":"answer"}), + json!({"type":"response.completed","sequence_number":9,"response":complete()}), + ], true).await; + let Some(ModelStreamItem::Completed(response)) = items.last() else { + panic!("live event sequence must complete: {items:?}"); + }; + assert_eq!(response.text(), "answer"); + assert_eq!(response.execution.as_ref().unwrap().model, "openai/test"); +} + +async fn run_events(events: Vec, split_bytes: bool) -> Vec { + let body = events + .into_iter() + .flat_map(|v| event(v).to_vec()) + .collect::>(); + let chunks = if split_bytes { + body.into_iter() + .map(|b| Bytes::from(vec![b])) + .collect::>() + } else { + vec![Bytes::from(body)] + }; + let incoming = http::Response::builder() + .header("content-type", "text/event-stream") + .body(reqwest::Body::wrap_stream(futures::stream::iter( + chunks.into_iter().map(Ok::<_, std::io::Error>), + ))) + .unwrap() + .into(); + let (model, _) = fixture(vec![Ok(incoming)]); + model + .stream(&(), ModelRequest::new(vec![Message::user("q")])) + .await + .unwrap() + .collect() + .await +} + +#[tokio::test] +async fn split_utf8_and_final_snapshots_do_not_repeat_text_or_charges() { + let mut final_response = complete(); + final_response["output"][0]["content"][0]["text"] = json!("Café"); + final_response["usage"] = json!({"input_tokens":4,"output_tokens":2,"total_tokens":6}); + let item = final_response["output"][0].clone(); + let items=run_events(vec![created(), + json!({"type":"response.output_item.added","sequence_number":1,"output_index":0, + "item":{"type":"message","id":"msg_1","role":"assistant","content":[]}}), + json!({"type":"response.output_text.delta","sequence_number":2,"output_index":0,"content_index":0,"delta":"Caf"}), + json!({"type":"response.output_text.delta","sequence_number":3,"output_index":0,"content_index":0,"delta":"é"}), + json!({"type":"response.output_text.done","sequence_number":4,"output_index":0,"content_index":0,"text":"Café"}), + json!({"type":"response.output_item.done","sequence_number":5,"output_index":0,"item":item}), + json!({"type":"response.completed","sequence_number":6,"response":final_response}), + json!({"type":"response.completed","sequence_number":6,"response":final_response}), + ],true).await; + let text = items + .iter() + .filter_map(|item| { + if let ModelStreamItem::MessageDelta(delta) = item { + Some(delta.text.as_str()) + } else { + None + } + }) + .collect::(); + assert_eq!(text, "Café"); + assert_eq!( + items + .iter() + .filter(|i| matches!(i, ModelStreamItem::Completed(_))) + .count(), + 1 + ); + let Some(ModelStreamItem::Completed(result)) = items.last() else { + panic!("missing completion: {items:?}"); + }; + assert_eq!(result.usage.unwrap().total_tokens, 6); + assert_eq!( + result.output, + super::response::parse(final_response).unwrap().output + ); +} + +#[tokio::test] +async fn function_argument_fragments_preserve_call_ids_and_signatures() { + let call = json!({"type":"function_call","id":"fc1","call_id":"call1","name":"lookup","arguments":"{\"id\":7}","thought_signature":"signature"}); + let items=run_events(vec![created(), + json!({"type":"response.output_item.added","sequence_number":1,"output_index":0, + "item":{"type":"function_call","id":"fc1","call_id":"call1","name":"lookup","arguments":""}}), + json!({"type":"response.function_call_arguments.delta","sequence_number":2,"item_id":"fc1","delta":"{\"id\":"}), + json!({"type":"response.function_call_arguments.delta","sequence_number":3,"item_id":"fc1","delta":"7}"}), + json!({"type":"response.output_item.done","sequence_number":4,"output_index":0,"item":call}), + json!({"type":"response.completed","sequence_number":5,"response":{"id":"resp_1","model":"openai/test","status":"completed","output":[call]}}), + ],true).await; + let args = items + .iter() + .filter_map(|item| { + if let ModelStreamItem::ToolCallDelta(delta) = item { + assert_eq!(delta.call_id, "call1"); + Some(delta.content.as_str()) + } else { + None + } + }) + .collect::(); + assert_eq!(args, "{\"id\":7}"); + let Some(ModelStreamItem::Completed(response)) = items.last() else { + panic!("missing completion: {items:?}"); + }; + assert_eq!( + response.message.tool_calls[0] + .replay + .as_ref() + .unwrap() + .thought_signature + .as_deref(), + Some("signature") + ); +} + +#[tokio::test] +async fn truncated_stream_keeps_received_text_and_recovery_identity() { + let items=run_events(vec![created(), + json!({"type":"response.output_text.delta","sequence_number":1,"output_index":0,"content_index":0,"item_id":"msg_1","delta":"partial"}), + ],false).await; + let Some(ModelStreamItem::ProviderFailed(error)) = items.last() else { + panic!("truncation must fail"); + }; + assert_eq!(error.code.as_deref(), Some("truncated_stream")); + let partial = error.partial_response.as_ref().unwrap(); + assert_eq!(partial.text(), "partial"); + assert_eq!(partial.execution.as_ref().unwrap().id, "resp_1"); + assert_eq!(partial.execution.as_ref().unwrap().sequence_number, Some(1)); + assert_eq!( + items + .iter() + .filter(|i| matches!(i, ModelStreamItem::ProviderFailed(_))) + .count(), + 1 + ); +} + +#[tokio::test] +async fn malformed_events_and_changed_identities_fail_without_panicking() { + for bad in [ + json!(7), + json!({"type":"response.output_text.delta","output_index":0,"content_index":5000,"delta":"x"}), + json!({"type":"response.completed","response":{"id":"other","model":"openai/test","status":"completed","output":[]}}), + json!({"type":"response.output_item.done","output_index":0,"item":{"type":"message","content":42}}), + ] { + let items = run_events(vec![created(), bad], false).await; + assert!(matches!( + items.last(), + Some(ModelStreamItem::ProviderFailed(_)) + )); + } +} + +#[tokio::test] +async fn unknown_events_and_output_items_are_preserved() { + let unknown = json!({"type":"future_item","opaque":{"a":1}}); + let items=run_events(vec![created(),json!({"type":"future_event","sequence_number":1,"opaque":"value","response":"future representation"}), + json!({"type":"response.completed","sequence_number":2,"response":{"id":"resp_1","model":"openai/test","status":"completed","output":[unknown]}}) + ],false).await; + assert!(items.iter().any(|item|matches!(item,ModelStreamItem::OutputEvent(crate::ModelOutputEvent::Progress(crate::ModelProgress::Extension {event_type,..})) if event_type=="future_event"))); + let Some(ModelStreamItem::Completed(response)) = items.last() else { + panic!("missing completion"); + }; + assert!(matches!( + response.output[0].kind, + crate::ModelOutputKind::Extension { .. } + )); + assert!( + matches!(&response.execution.as_ref().unwrap().progress[0],crate::ModelProgress::Extension {data,..} if data["opaque"]=="value") + ); +} + +#[tokio::test] +async fn crlf_and_multiline_sse_data_are_decoded() { + let body = format!( + "event: response.created\r\ndata: {{\r\ndata: \"response\":{{\"id\":\"resp_1\",\"model\":\"openai/test\",\"status\":\"in_progress\",\"output\":[]}}\r\ndata: }}\r\n\r\ndata: {}\r\n\r\ndata: [DONE]\r\n\r\n", + json!({"type":"response.completed","response":complete()}) + ); + let incoming = http::Response::builder() + .header("content-type", "text/event-stream") + .body(body) + .unwrap() + .into(); + let (model, _) = fixture(vec![Ok(incoming)]); + let response = collect_model_stream( + model + .stream(&(), ModelRequest::new(vec![Message::user("q")])) + .await + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.text(), "answer"); +} + +#[tokio::test(start_paused = true)] +async fn stream_idle_timeout_and_event_limit_are_terminal_failures() { + let incoming = http::Response::builder() + .header("content-type", "text/event-stream") + .body(reqwest::Body::wrap_stream(futures::stream::pending::< + std::result::Result, + >())) + .unwrap() + .into(); + let (mut model, _) = fixture(vec![Ok(incoming)]); + Arc::get_mut(&mut model.inner).unwrap().config.idle_timeout = std::time::Duration::from_secs(1); + let items = model + .stream(&(), ModelRequest::new(vec![Message::user("q")])) + .await + .unwrap() + .collect::>() + .await; + assert!( + matches!(items.last(),Some(ModelStreamItem::ProviderFailed(error)) if error.code.as_deref()==Some("timeout")) + ); + let incoming = http::Response::builder() + .header("content-type", "text/event-stream") + .body(event(created())) + .unwrap() + .into(); + let (mut model, _) = fixture(vec![Ok(incoming)]); + Arc::get_mut(&mut model.inner) + .unwrap() + .config + .max_event_bytes = 16; + let items = model + .stream(&(), ModelRequest::new(vec![Message::user("q")])) + .await + .unwrap() + .collect::>() + .await; + assert!(matches!( + items.last(), + Some(ModelStreamItem::ProviderFailed(_)) + )); +} + +#[tokio::test] +async fn contradictory_final_text_keeps_the_text_already_delivered_in_partial_failure() { + for final_event in [ + json!({"type":"response.completed","sequence_number":2,"response":complete()}), + json!({"type":"response.output_text.done","sequence_number":2,"output_index":0,"content_index":0,"text":"different"}), + json!({"type":"response.content_part.done","sequence_number":2,"output_index":0,"content_index":0,"part":{"type":"output_text","text":"different"}}), + ] { + let items=run_events(vec![created(), + json!({"type":"response.output_text.delta","sequence_number":1,"output_index":0,"content_index":0,"item_id":"msg_1","delta":"delivered"}), + final_event + ],false).await; + let Some(ModelStreamItem::ProviderFailed(error)) = items.last() else { + panic!("contradiction must fail"); + }; + assert_eq!(error.partial_response.as_ref().unwrap().text(), "delivered"); + } +} + +#[tokio::test] +async fn dropping_a_stream_releases_its_body_without_remote_cancellation() { + let (sender, receiver) = futures::channel::mpsc::unbounded::(); + let incoming = http::Response::builder() + .header("content-type", "text/event-stream") + .body(reqwest::Body::wrap_stream( + receiver.map(Ok::<_, std::io::Error>), + )) + .unwrap() + .into(); + let (model, scripted) = fixture(vec![Ok(incoming)]); + let stream = model + .stream(&(), ModelRequest::new(vec![Message::user("q")])) + .await + .unwrap(); + assert!(!sender.is_closed()); + drop(stream); + assert!(sender.is_closed()); + assert_eq!(scripted.requests.lock().unwrap().len(), 1); +} + +#[tokio::test] +async fn terminal_failure_preserves_previously_streamed_work() { + let items=run_events(vec![created(), + json!({"type":"response.output_text.delta","sequence_number":1,"output_index":0,"content_index":0,"item_id":"msg_1","delta":"partial"}), + json!({"type":"response.failed","sequence_number":2,"response":{"id":"resp_1","model":"openai/test","status":"failed","output":[],"error":{"message":"failed"}}}) + ],false).await; + let Some(ModelStreamItem::ProviderFailed(error)) = items.last() else { + panic!("missing failure"); + }; + assert_eq!(error.partial_response.as_ref().unwrap().text(), "partial"); + assert_eq!( + error + .partial_response + .as_ref() + .unwrap() + .execution + .as_ref() + .unwrap() + .status, + crate::ExecutionStatus::Failed + ); +} + +#[tokio::test] +async fn reasoning_summary_fragments_stay_separate_from_visible_text() { + let reasoning = json!({"id":"reason_1","type":"reasoning","summary":[{"type":"summary_text","text":"considering"}]}); + let mut final_response = complete(); + final_response["output"] + .as_array_mut() + .unwrap() + .insert(0, reasoning.clone()); + let items=run_events(vec![created(), + json!({"type":"response.output_item.added","sequence_number":1,"output_index":0,"item":{"id":"reason_1","type":"reasoning","summary":[]}}), + json!({"type":"response.reasoning_summary_text.delta","sequence_number":2,"output_index":0,"summary_index":0,"delta":"considering"}), + json!({"type":"response.output_item.done","sequence_number":3,"output_index":0,"item":reasoning}), + json!({"type":"response.completed","sequence_number":4,"response":final_response}) + ],true).await; + let reasoning = items + .iter() + .filter_map(|i| { + if let ModelStreamItem::MessageDelta(d) = i { + Some(d.reasoning.as_str()) + } else { + None + } + }) + .collect::(); + assert_eq!(reasoning, "considering"); + let Some(ModelStreamItem::Completed(response)) = items.last() else { + panic!("missing completion: {items:?}"); + }; + assert_eq!(response.text(), "answer"); +} + +#[tokio::test(start_paused = true)] +async fn text_arrives_before_completion_and_collecting_preserves_the_full_response() { + let (sender, receiver) = futures::channel::mpsc::unbounded::(); + let incoming = http::Response::builder() + .header("content-type", "text/event-stream") + .body(reqwest::Body::wrap_stream( + receiver.map(Ok::<_, std::io::Error>), + )) + .unwrap() + .into(); + let (model, _) = fixture(vec![Ok(incoming)]); + sender + .unbounded_send(event(json!({"type":"response.created","sequence_number":0, + "response":{"id":"resp_1","model":"openai/test","status":"in_progress","output":[]}}))) + .unwrap(); + sender + .unbounded_send(event( + json!({"type":"response.output_item.added","sequence_number":1,"output_index":0, + "item":{"type":"message","id":"msg_1","role":"assistant","content":[]}}), + )) + .unwrap(); + sender + .unbounded_send(event( + json!({"type":"response.output_text.delta","sequence_number":2,"output_index":0, + "content_index":0,"item_id":"msg_1","delta":"answer"}), + )) + .unwrap(); + let mut stream = tokio::time::timeout( + std::time::Duration::from_secs(1), + model.stream(&(), ModelRequest::new(vec![Message::user("question")])), + ) + .await + .expect("opening the stream must not wait for its complete body") + .unwrap(); + assert!(matches!( + stream.next().await, + Some(ModelStreamItem::Started) + )); + loop { + match stream.next().await.unwrap() { + ModelStreamItem::MessageDelta(delta) => { + assert_eq!(delta.text, "answer"); + break; + } + ModelStreamItem::Completed(_) | ModelStreamItem::ProviderFailed(_) => { + panic!("stream ended before text was delivered") + } + _ => {} + } + } + while let Some(item) = stream.next().now_or_never() { + assert!(matches!(item, Some(ModelStreamItem::OutputEvent(_)))); + } + sender + .unbounded_send(event( + json!({"type":"response.completed","sequence_number":3,"response":complete()}), + )) + .unwrap(); + drop(sender); + let result = collect_model_stream(stream).await.unwrap(); + assert_eq!(result.text(), "answer"); + assert_eq!(result.execution.unwrap().sequence_number, Some(3)); +} diff --git a/crates/tinyinference-llm/src/providers/perplexity/transport.rs b/crates/tinyinference-llm/src/providers/perplexity/transport.rs new file mode 100644 index 00000000..2b87e1fe --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/transport.rs @@ -0,0 +1,393 @@ +use super::PerplexityConfig; +use crate::{Error, Result, model::ProviderError}; +use async_trait::async_trait; +use reqwest::header::{AUTHORIZATION, CONTENT_TYPE, HeaderValue}; +use serde_json::Value; +use std::{fmt, io::Write, sync::Arc, time::Duration}; +use tokio::time::Instant; + +#[derive(Clone, Copy)] +pub(super) enum Operation { + Create, + Read, + Cancel, +} + +#[async_trait] +pub(super) trait HttpTransport: Send + Sync { + async fn send(&self, request: reqwest::Request) -> Result; +} + +struct ReqwestTransport { + client: reqwest::Client, +} + +#[async_trait] +impl HttpTransport for ReqwestTransport { + async fn send(&self, request: reqwest::Request) -> Result { + self.client.execute(request).await.map_err(|_| { + failure( + "transport_interrupted", + "provider completion may be unknown", + None, + ) + }) + } +} + +pub(super) struct Transport { + pub config: PerplexityConfig, + pub sender: Arc, + client: reqwest::Client, + authorization: HeaderValue, + secrets: Vec, +} + +impl fmt::Debug for Transport { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("PerplexityTransport") + .field("config", &self.config) + .finish_non_exhaustive() + } +} + +impl Transport { + pub(super) fn new(key: String, mut config: PerplexityConfig) -> Result { + config.selection.validate()?; + if key.trim().is_empty() { + return Err(Error::Validation( + "Perplexity API key must not be blank".into(), + )); + } + if [config.timeout, config.idle_timeout, config.connect_timeout] + .iter() + .any(|duration| duration.is_zero() || Instant::now().checked_add(*duration).is_none()) + || config.max_body_bytes == 0 + || config.max_event_bytes == 0 + || config.max_file_bytes == 0 + || config.max_retries > tinyinference_core::MAX_RETRIES + || Instant::now().checked_add(config.timeout).is_none() + { + return Err(Error::Validation( + "invalid Perplexity resource limits".into(), + )); + } + let url = reqwest::Url::parse(config.base_url.trim()) + .map_err(|_| Error::Validation("invalid Perplexity endpoint".into()))?; + let loopback = url.host_str().is_some_and(|host| { + host == "localhost" + || host + .trim_matches(['[', ']']) + .parse::() + .is_ok_and(|ip| ip.is_loopback()) + }); + if url.host_str().is_none() + || !(url.scheme() == "https" || (url.scheme() == "http" && loopback)) + || !url.username().is_empty() + || url.password().is_some() + || url.query().is_some() + || url.fragment().is_some() + { + return Err(Error::Validation( + "Perplexity requires HTTPS or loopback HTTP without userinfo, query, or fragment" + .into(), + )); + } + config.base_url = url.as_str().trim_end_matches('/').into(); + let mut authorization = HeaderValue::from_str(&format!("Bearer {key}")) + .map_err(|_| Error::Validation("invalid Perplexity authorization header".into()))?; + authorization.set_sensitive(true); + let client = reqwest::Client::builder() + .connect_timeout(config.connect_timeout) + .redirect(reqwest::redirect::Policy::none()) + .build() + .map_err(|_| { + failure( + "client_setup", + "could not construct Perplexity HTTP client", + None, + ) + })?; + let mut secrets = vec![key]; + for credential in config.remote_credentials.values() { + if let Some(token) = &credential.authorization + && !token.is_empty() + { + secrets.push(token.clone()); + } + secrets.extend( + credential + .headers + .values() + .filter(|v| !v.is_empty()) + .cloned(), + ); + } + secrets.sort_by_key(|s| std::cmp::Reverse(s.len())); + secrets.dedup(); + Ok(Self { + config, + authorization, + sender: Arc::new(ReqwestTransport { + client: client.clone(), + }), + client, + secrets, + }) + } + + pub(super) fn deadline(&self, override_ms: Option) -> Result { + let duration = override_ms + .map(Duration::from_millis) + .unwrap_or(self.config.timeout); + if duration.is_zero() { + return Err(Error::Validation("timeout must be positive".into())); + } + Instant::now() + .checked_add(duration) + .ok_or_else(|| Error::Validation("timeout is too large".into())) + } + + pub(super) fn request( + &self, + method: reqwest::Method, + path: &str, + body: Option<&Value>, + ) -> Result { + let mut builder = self + .client + .request(method, format!("{}{}", self.config.base_url, path)) + .header(AUTHORIZATION, self.authorization.clone()); + if let Some(body) = body { + let mut writer = BoundedWriter { + bytes: Vec::new(), + limit: self.config.max_body_bytes, + }; + serde_json::to_writer(&mut writer, body).map_err(|_| { + Error::Validation("Perplexity request exceeds serialized size limit".into()) + })?; + builder = builder + .header(CONTENT_TYPE, "application/json") + .body(writer.bytes); + } + builder + .build() + .map_err(|_| Error::Validation("invalid Perplexity request".into())) + } + + pub(super) fn check_input_size(&self, request: &crate::model::ModelRequest) -> Result<()> { + let mut counter = ByteCounter { + count: 0, + limit: self.config.max_body_bytes, + }; + serde_json::to_writer(&mut counter, request) + .map_err(|_| Error::Validation("Perplexity input exceeds byte limit".into())) + } + + pub(super) async fn send( + &self, + request: reqwest::Request, + operation: Operation, + deadline: Instant, + ) -> Result { + crate::network_guard::ensure_network_models_allowed()?; + let work = async { + let mut retries = 0; + loop { + if Instant::now() >= deadline { + return Err(timeout()); + } + let outgoing = request + .try_clone() + .ok_or_else(|| Error::Validation("request cannot be replayed".into()))?; + let response = self.sender.send(outgoing).await?; + if response.status().is_success() { + return Ok(response); + } + let status = response.status().as_u16(); + let retry_after = response + .headers() + .get(reqwest::header::RETRY_AFTER) + .and_then(|v| v.to_str().ok()) + .map(str::to_owned); + let may_retry = match operation { + Operation::Create => status == 429, + Operation::Read => matches!(status, 429 | 500 | 502 | 503 | 504), + Operation::Cancel => false, + }; + if may_retry && retries < self.config.max_retries { + drop(response); + let delay = + tinyinference_core::backoff_ms_for_attempt(retries, retry_after.as_deref()); + retries += 1; + tokio::time::sleep(Duration::from_millis(delay)).await; + continue; + } + let body = read_limited(response, self.config.max_body_bytes.min(64 * 1024)) + .await + .unwrap_or_default(); + let raw = serde_json::from_slice::(&body) + .ok() + .map(|v| self.scrub_value(v)); + let detail = raw + .as_ref() + .and_then(|v| v.get("error")) + .unwrap_or(&Value::Null); + let message = detail + .get("message") + .and_then(Value::as_str) + .unwrap_or("Perplexity HTTP request failed"); + return Err(Error::Provider(Box::new(ProviderError { + provider: "perplexity".into(), + status: Some(status), + code: detail + .get("code") + .and_then(Value::as_str) + .map(str::to_owned), + message: self.scrub_text(message), + retryable: may_retry, + retry_after_ms: tinyinference_core::parse_retry_after_ms( + retry_after.as_deref(), + ), + raw, + ..Default::default() + }))); + } + }; + tokio::time::timeout_at(deadline, work) + .await + .map_err(|_| timeout())? + } + + pub(super) async fn json( + &self, + response: reqwest::Response, + deadline: Instant, + ) -> Result { + let bytes = + tokio::time::timeout_at(deadline, read_limited(response, self.config.max_body_bytes)) + .await + .map_err(|_| timeout())??; + let mut decoded = super::response::decode(&bytes)?; + decoded.value = self.scrub_value(decoded.value); + if let Some(observer) = &self.config.on_response { + observer(&decoded.value); + } + Ok(decoded) + } + + pub(super) fn scrub_text(&self, text: &str) -> String { + let mut text = text.to_owned(); + for secret in &self.secrets { + text = text.replace(secret, "[REDACTED]"); + } + tinyinference_core::sanitize::sanitize_api_error(&text) + } + + pub(super) fn scrub_value(&self, mut value: Value) -> Value { + fn walk(value: &mut Value, secrets: &[String]) { + match value { + Value::String(text) => { + for secret in secrets { + *text = text.replace(secret, "[REDACTED]"); + } + } + Value::Array(items) => { + for item in items { + walk(item, secrets); + } + } + Value::Object(object) => { + for value in object.values_mut() { + walk(value, secrets); + } + } + _ => {} + } + } + walk(&mut value, &self.secrets); + value + } +} + +struct BoundedWriter { + bytes: Vec, + limit: usize, +} + +struct ByteCounter { + count: usize, + limit: usize, +} +impl Write for ByteCounter { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + if bytes.len() > self.limit.saturating_sub(self.count) { + return Err(std::io::Error::other("input too large")); + } + self.count += bytes.len(); + Ok(bytes.len()) + } + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} +impl Write for BoundedWriter { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + if bytes.len() > self.limit.saturating_sub(self.bytes.len()) { + return Err(std::io::Error::other("request too large")); + } + self.bytes.extend_from_slice(bytes); + Ok(bytes.len()) + } + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +pub(super) async fn read_limited(mut response: reqwest::Response, limit: usize) -> Result> { + if response + .content_length() + .is_some_and(|length| length > limit as u64) + { + return Err(failure( + "response_too_large", + "response exceeds byte limit", + None, + )); + } + let mut bytes = Vec::new(); + while let Some(chunk) = response.chunk().await.map_err(|_| { + failure( + "transport_interrupted", + "response body was interrupted", + None, + ) + })? { + if chunk.len() > limit.saturating_sub(bytes.len()) { + return Err(failure( + "response_too_large", + "response exceeds byte limit", + None, + )); + } + bytes.extend_from_slice(&chunk); + } + Ok(bytes) +} + +pub(super) fn timeout() -> Error { + failure( + "timeout", + "Perplexity deadline expired; submitted work may still run", + None, + ) +} +pub(super) fn failure(code: &str, message: &str, status: Option) -> Error { + Error::Provider(Box::new(ProviderError { + provider: "perplexity".into(), + code: Some(code.into()), + message: message.into(), + status, + ..Default::default() + })) +} diff --git a/crates/tinyinference-llm/src/providers/perplexity/types.rs b/crates/tinyinference-llm/src/providers/perplexity/types.rs new file mode 100644 index 00000000..6883756b --- /dev/null +++ b/crates/tinyinference-llm/src/providers/perplexity/types.rs @@ -0,0 +1,16 @@ +//! Private wire records that are not part of the public provider contract. + +use serde::Deserialize; + +#[derive(Deserialize)] +pub(super) struct FileList { + pub data: Vec, +} + +#[derive(Deserialize)] +pub(super) struct FileRecord { + pub id: String, + pub filename: String, + pub bytes: u64, + pub created_at: u64, +} diff --git a/crates/tinyinference-llm/src/providers/types.rs b/crates/tinyinference-llm/src/providers/types.rs index cbb8d6a2..d8ac0b7d 100644 --- a/crates/tinyinference-llm/src/providers/types.rs +++ b/crates/tinyinference-llm/src/providers/types.rs @@ -123,6 +123,8 @@ pub trait BearerSource: Send + Sync { #[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] pub enum ProviderKind { + /// Native Perplexity Agent API, including model and preset selection. + Perplexity, /// Hosted OpenAI API. OpenAi, /// Anthropic via its OpenAI-compatible Chat Completions endpoint. @@ -159,6 +161,7 @@ impl ProviderKind { /// Stable provider identifier used in profiles, errors, and registry names. pub fn as_str(&self) -> &'static str { match self { + ProviderKind::Perplexity => "perplexity", ProviderKind::OpenAi => "openai", ProviderKind::Anthropic => "anthropic", ProviderKind::Ollama => "ollama", @@ -187,6 +190,7 @@ impl ProviderKind { if let Some((prefix, _)) = lower.split_once(':') { return match prefix { "openai" => Some(ProviderKind::OpenAi), + "perplexity" => Some(ProviderKind::Perplexity), "anthropic" => Some(ProviderKind::Anthropic), "ollama" => Some(ProviderKind::Ollama), "lmstudio" | "lm_studio" | "lm-studio" => Some(ProviderKind::LmStudio), @@ -232,7 +236,8 @@ pub struct ProviderSpec { pub kind: ProviderKind, /// Provider id written to profiles and normalized errors. pub provider: String, - /// Default provider model id. + /// Default provider model id. Empty for Perplexity, whose native adapter + /// requires an explicit model, fallback chain, or preset selection. pub model: String, /// API base URL without a trailing slash. pub base_url: String, @@ -248,6 +253,13 @@ impl ProviderSpec { /// Returns the default provider spec for a known provider. pub fn for_kind(kind: ProviderKind) -> Self { match kind { + ProviderKind::Perplexity => Self::new( + kind, + "", + "https://api.perplexity.ai/v1", + Some("PERPLEXITY_API_KEY"), + true, + ), ProviderKind::OpenAi => Self::new( kind, "gpt-4.1-mini", @@ -474,19 +486,17 @@ pub(crate) struct MockInner { /// # Placement of real providers /// /// Real network-backed providers live in sub-modules alongside this one. The -/// OpenAI (and OpenAI-compatible) adapter is always compiled; providers with a -/// different wire protocol would be gated behind their own Cargo feature: +/// OpenAI-compatible, native Anthropic, and native Perplexity adapters are +/// compiled using the workspace's existing dependencies: /// /// ```text /// pub mod openai; // always compiled -/// // #[cfg(feature = "anthropic")] pub mod anthropic; -/// // #[cfg(feature = "ollama")] pub mod ollama; +/// pub mod anthropic; +/// pub mod perplexity; /// ``` /// -/// Add the feature flag to `Cargo.toml` and implement -/// [`ChatModel`][crate::model::ChatModel] in the corresponding module. -/// No changes to `mod.rs` or `harness/mod.rs` are needed beyond enabling the -/// `pub mod` declaration. +/// Each adapter implements [`ChatModel`][crate::model::ChatModel] and owns its +/// protocol translation, while the caller owns provider-selection policy. pub struct MockModel { pub(crate) behavior: MockBehavior, pub(crate) inner: Mutex, diff --git a/crates/tinyinference-llm/src/tool.rs b/crates/tinyinference-llm/src/tool.rs index 6601d439..0b4c9c24 100644 --- a/crates/tinyinference-llm/src/tool.rs +++ b/crates/tinyinference-llm/src/tool.rs @@ -87,6 +87,9 @@ impl ToolSchema { /// A model request to invoke a tool. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] pub struct ToolCall { + /// Opaque signed-call replay context, scoped to its originating provider/model. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub replay: Option, /// Provider-assigned call identifier. pub id: String, /// Tool name. @@ -104,6 +107,7 @@ impl ToolCall { pub fn new(id: impl Into, name: impl Into, arguments: Value) -> Self { Self { id: id.into(), + replay: None, name: name.into(), arguments, invalid: None, @@ -119,6 +123,7 @@ impl ToolCall { ) -> Self { Self { id: id.into(), + replay: None, name: name.into(), arguments: Value::String(raw.into()), invalid: Some(reason.into()), @@ -131,6 +136,28 @@ impl ToolCall { } } +/// Context required to replay a signed function call without changing its bytes. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct ToolCallReplay { + /// Origin of the signature and argument representation. + pub origin: crate::message::MessageOrigin, + /// Provider output item ID, distinct from the call ID. + pub item_id: Option, + /// Original argument JSON text, retained even when parsing succeeds. + pub arguments: String, + /// Opaque provider signature; never reconstruct or modify it. + pub thought_signature: Option, +} + +/// Context for a native function result sent without replaying its preceding call. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct ToolResultContext { + /// Function name associated with the result's call ID. + pub name: String, + /// Original provider/model scope and signature when needed. + pub replay: Option, +} + /// An incremental tool-call fragment emitted by a model stream. #[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] pub struct ToolDelta { diff --git a/crates/tinyinference-llm/tests/perplexity_api.rs b/crates/tinyinference-llm/tests/perplexity_api.rs new file mode 100644 index 00000000..78ade3da --- /dev/null +++ b/crates/tinyinference-llm/tests/perplexity_api.rs @@ -0,0 +1,56 @@ +//! Public Perplexity operations must respect the host's network policy. + +use tinyinference_llm::{ + ChatModel, ExecutionStatus, Message, ModelExecution, ModelRequest, ModelResponse, + PerplexityModel, PerplexitySelection, deny_network_models, +}; + +#[tokio::test] +async fn every_public_network_operation_obeys_the_host_network_guard() { + let model = + PerplexityModel::new("test-key", PerplexitySelection::Model("openai/test".into())).unwrap(); + let mut snapshot = ModelResponse::assistant("answer"); + snapshot.message.origin = Some(tinyinference_llm::message::MessageOrigin { + provider: "perplexity".into(), + api: "agent".into(), + model: "openai/test".into(), + }); + snapshot.execution = Some(ModelExecution { + id: "resp_1".into(), + model: "openai/test".into(), + status: ExecutionStatus::Completed, + incomplete_reason: None, + sequence_number: None, + cost: None, + tool_usage: Default::default(), + progress: Vec::new(), + }); + let mut handle = model.response_handle(&snapshot).unwrap(); + handle.kind = Some("background".into()); + deny_network_models(); + let request = || ModelRequest::new(vec![Message::user("question")]); + let errors = [ + model.invoke(&(), request()).await.unwrap_err(), + model.stream(&(), request()).await.unwrap_err(), + model.submit_background(request()).await.unwrap_err(), + model.retrieve_response(&handle).await.unwrap_err(), + model.resume_background(&handle, Some(1)).await.unwrap_err(), + model.cancel_background(&handle).await.unwrap_err(), + model.list_response_files(&handle).await.unwrap_err(), + model + .download_response_file(&handle, "file_1") + .await + .unwrap_err(), + >::fetch_deferred(&model, &handle) + .await + .unwrap_err(), + ]; + for error in errors { + assert!( + error + .to_string() + .contains("network-backed model calls are denied"), + "{error}" + ); + } +} diff --git a/crates/tinyinference-local/src/service/model_rpc_tests.rs b/crates/tinyinference-local/src/service/model_rpc_tests.rs index 221347aa..ddfad3ef 100644 --- a/crates/tinyinference-local/src/service/model_rpc_tests.rs +++ b/crates/tinyinference-local/src/service/model_rpc_tests.rs @@ -36,6 +36,8 @@ fn local_model_selects_configured_provider() { #[test] fn model_outcome_enforces_empty_and_normalizes_usage() { let response = |text: &str, usage: Usage| ModelResponse { + output: Vec::new(), + execution: None, message: AssistantMessage { id: None, content: vec![ContentBlock::Text(text.to_string())], @@ -75,6 +77,8 @@ fn model_outcome_enforces_empty_and_normalizes_usage() { assert_eq!(populated.completion_tokens, Some(3)); let reasoning_only = ModelResponse { + output: Vec::new(), + execution: None, message: AssistantMessage { id: None, content: vec![ContentBlock::Thinking { diff --git a/docs/migrations/perplexity-agent.md b/docs/migrations/perplexity-agent.md new file mode 100644 index 00000000..5eeb9058 --- /dev/null +++ b/docs/migrations/perplexity-agent.md @@ -0,0 +1,187 @@ +# Perplexity Agent API contracts and migration + +The native provider lives in `tinyinference_llm::providers::perplexity` and is +also exported from the crate root. It uses `/v1/agent` by default; an explicit +`PerplexityConfig.responses_alias` enables `/v1/responses` for creation. +Lifecycle and file operations use the canonical Agent routes. + +## Choosing models and presets + +Construct `PerplexityModel::new(key, selection)` or use `with_config` for explicit +limits, tools and callbacks. Credentials are supplied by the application; the +adapter does not read environment variables. `build_perplexity_model` returns +an `Arc>` when the host needs the existing trait-object interface. + +Selections are `Model`, `Models` (one to five fallback IDs), and `Preset` with +optional model/fallback override. Preset names are not a frozen whitelist. +Current documented names are fast, low, medium, high and xhigh. Preset contents +can change server-side. Explicit Anthropic selections require a positive +`ModelRequest.max_tokens` or configured output cap. + +Use `PerplexityOptions::apply_to` for typed per-call options. Those override +configured options; `ModelRequest.model` overrides a single model while retaining +a preset. A model override alongside a fallback list is rejected as ambiguous. +Unset instructions/tools/steps remain absent so server defaults survive. +System messages replace preset instructions; hosted-tool overrides merge per tool. +Use an explicit tool-choice override when replacing a preset's auto/required policy. + +Hosted configuration includes web/image/finance/people search, URL fetching, +sandbox, remote MCP and existing connectors. MCP credentials are configured +separately by server label and exact server URL; they do not enter serialized +`ModelRequest.provider_options`. Remote calls execute on Perplexity's side. +The API does not enforce an MCP `require_approval` option; this adapter rejects it. +Do not use an empty remote allowlist to mean no tools: the provider treats it as all tools. + +Unsupported settings fail explicitly: profile/skill administration or request +configuration, seed/stop sequences, unsupported reasoning settings, local media +paths, audio/video input, and unresolved required-capability guarantees. Standard +text, image, and URL/base64 document inputs use existing message/media types. +The caller resolves local files. Provider-specific model/media limits still apply. + +Payload callbacks may adjust validated step/output budgets and sampling values. +Use typed options for other fields; callbacks cannot alter input, tools, credentials, +or execution mode behind the validated request. Response callbacks see sanitized data. + +## Rich results and streams + +`ModelResponse.output` is authoritative for providers with rich output. Each item +retains its index, optional item ID/status, and a `ModelOutputKind`. Known kinds +cover messages with annotations, reasoning, custom functions, search sources, +images, fetched pages, financial content, sandbox results, remote tool discovery +and execution, namespaces, and generated-file notices. + +`message` remains the compatibility projection: visible text, supplied reasoning, +and only functions for the host to execute. Hosted tools never become local +`ToolCall` values. Text markers and source IDs are preserved without rewriting. +Citation offsets carry `provider_characters`; they are not safe Rust byte indices. +Unknown items/content/annotations use the explicit provider-extension boundary. +Malformed known items return an error rather than an empty or opaque success. + +`ModelResponse.execution` records response ID, actual model, status, reconnect +cursor, reported costs/tool counts and stream-only `ModelProgress`. Stream items +include `OutputEvent` alongside existing text/tool/usage deltas. Collecting a stream +retains progress and unknown events; authoritative snapshots replace incremental +state without adding the same text, function arguments or usage twice. + +`Completed` marks stream termination, not necessarily a fully completed task: +inspect execution status and finish reason for incomplete output. Failed/cancelled +foreground runs produce `ProviderFailed`. That error retains `partial_response`, +including received output and recovery identifiers. EOF without a terminal response +is a failure. Refusals and unfamiliar content are preserved as extensions. + +Usage remains optional. Missing usage or charges are unknown, not zero. Provider +decimal costs retain their original number text inside the adapter; valid USD +totals are also projected to checked, nearest micro-dollars in +`Usage.charged_amount`. No local pricing calculation is performed. Tool counts and +cost breakdowns remain separate from token totals. + +## Functions and conversation state + +Declare application functions with the existing `ToolSchema`. A returned +`ToolCall` preserves `call_id` as `id` and keeps the distinct provider item ID, +original argument text and thought signature in optional `replay` metadata. +Malformed arguments remain explicitly invalid; the library never runs functions. + +Create results with `ToolMessage::for_call(&call, content)`. For manual history, +include the preceding assistant call and matching result; native function items +are sent instead of text imitations. Duplicate/unmatched/conflicting calls or +results fail validation. Signed replay requires its original provider/model. +Pin the response's actual model when continuing a dynamically selected preset. + +For custom function results, keep the original request's conversation and tools, +append `Message::Assistant(first.message)` followed by the result from `for_call`, +set `request.model` to the returned model, and leave `continuation_id` unset. +Retain the original call rather than reconstructing it: its argument bytes and +signature must survive. Change a forced tool choice back to `Auto` for another +turn, or `None` when only a final answer is wanted. Execute each function once; +resending its result does not require executing it again. + +The complete [function replay example](../../crates/tinyinference-llm/examples/perplexity_function_replay.rs) +uses these existing types. With `PERPLEXITY_API_KEY` set, run: + +```sh +cargo run -p tinyinference-llm --example perplexity_function_replay +``` + +For ordinary chat continuation, put the prior execution ID in +`ModelRequest.continuation_id` and send only the new user turn. The provider +requires a completed same-account response. Do not resend full assistant history +alongside the continuation ID. Unsupported signed reasoning/history blocks +are rejected rather than being silently stripped. + +Live checks on 2026-10-04 isolated a broader source-response problem: referencing +a response ending in a custom function call returned HTTP 400 for both +`google/gemini-3.1-flash-lite` and `openai/gpt-6-luna`, even for a plain user +follow-up. Stored assistant-text responses continued successfully. Direct HTTP +probes, including completed background calls, reproduced this independently of +the adapter. Full call/result replay works; the internal server cause remains +unconfirmed. +The adapter preserves the error and never silently substitutes another request. + +`store` is omitted by default. In Perplexity's documented behavior, false hides +retrieval but does **not** disable server persistence or continuation. Follow your +provider agreement for retention requirements. + +## Background recovery and generated files + +`submit_background` returns a serializable `DeferredHandle`. Persist it in the +calling application. Each retrieval is one explicit read; the library starts no +polling loop. `fetch_deferred` maps to pending, completed (including explicit +incomplete status), structured provider failure, or confirmed cancellation. + +`resume_background(handle, cursor)` reconnects after the last sequence number. +It emits only new events and returns the full authoritative final response. +If the reconnect window has expired, inspect `retrieve_response` instead of +creating a replacement job. Handles are validated against the configured endpoint; +they cannot redirect credentials to a URL embedded in persisted metadata. + +`cancel_background` returns `Cancelling` when acknowledged. Poll to observe +`Cancelled`; acknowledgement is not proof that execution has stopped. Dropping +a local stream stops local reads but does not silently cancel a durable remote job. +Cancellation is never automatically replayed. + +The live cancellation check received `cancelling`, followed by `incomplete`. +Inspect the returned status; cancellation acknowledgement is not proof of a +terminal `cancelled` response. + +Use `response_handle` for a foreground response's generated files. `list_response_files` +returns response-scoped file IDs, filenames, reported byte sizes and creation times. +`download_response_file` returns bounded bytes and media type. The library does not +write files, interpret filenames as paths, or fetch arbitrary output URLs. A failed +download does not regenerate the run. `share_file` notices preserve undocumented +metadata; the file-list endpoint provides authoritative descriptors. + +## Limits and retries + +Defaults are configurable: 600 seconds total, 30 seconds to connect, 60 seconds +of stream inactivity, 32 MiB for input/JSON/stream traffic, 4 MiB per SSE event, +and 64 MiB per downloaded file. `ModelRequest.timeout_ms` overrides the total +call budget. Limits are enforced during serialization and reads. Stream collection +is pull-driven; no detached producer continues after the caller drops the stream. + +Creation retries only explicit HTTP 429, up to three retries within the original +deadline. Transport and 5xx creation failures can have unknown completion/billing +and are not replayed. Read operations can retry 429/500/502/503/504. Use zero retries +when the caller needs to manage all repetition. Recover acknowledged jobs by ID. + +The adapter provides no automatic response-cache identity: presets evolve, +searches change, and remote tools may have side effects. The host owns caching +policy. Existing response serialization and cache storage preserve the rich fields. + +## Rust compatibility + +Existing serialized records remain readable through defaulted optional fields. +Rust struct literals and exhaustive matches require source updates: + +- `ModelResponse`: add `output: Vec::new()` and `execution: None` for legacy providers. +- `ProviderError`: add `partial_response: None`, or use its existing default. +- `ToolCall`: add `replay: None`, or use `ToolCall::new`/`invalid`. +- `ToolMessage`: add `call_context: None`, or use existing constructors/`for_call`. +- `ModelStreamItem`: handle `OutputEvent` while retaining the existing delta channels. +- `DeferredStatus`: handle structured `ProviderFailed` and `Cancelled` states. +- `ProviderKind`: handle `Perplexity`; the OpenAI-compatible constructor explicitly + rejects it so native requests cannot accidentally use Chat Completions. + +Other providers keep their wire behavior. No workspace release version is changed +by this feature. Library fixtures verify protocol behavior; live provider behavior, +model access and billing require a separately authorized Perplexity account test.