Skip to content

Contract drift audit 2026-09-21 - #35

Merged
anthturner merged 2 commits into
developfrom
contract-drift/2026-09-21
Sep 22, 2026
Merged

anthturner merged 2 commits into
developfrom
contract-drift/2026-09-21

Conversation

@claude

@claude claude Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Audited the 4 snapshots selected by the deterministic stage
(drift-report/contract-drift-report.json, budget 4, all never_live_verified: true):
openai-compat, openai, openrouter, azure-foundry. Per contracts/DRIFT-CHECK.md.

Summary table

Provider OK DRIFT DEPRECATION NEW-CAPABILITY UNVERIFIABLE
openai-compat 4 0 0 4 0
openai 5 1 0 6 2
openrouter 2 1 0 4 2
azure-foundry 6 1 0 3 0

Findings (most severe first)

openai: platform.openai.com/docs/changelog moved [DRIFT]

  • Snapshot says: cites https://platform.openai.com/docs/changelog as a source.
  • Live docs say: that URL now 301-redirects permanently to https://developers.openai.com/api/docs/changelog (fetched 2026-09-21, content unchanged in substance).
  • Impact: none on the wire contract itself; the citation was stale.
  • Proposed action: updated Upstream sources to record the redirect and the new URL. Applied.

azure-foundry: /reference upstream source no longer documents chat/embeddings [DRIFT]

  • Snapshot says: cites learn.microsoft.com/en-us/azure/ai-foundry/openai/reference as the source for the chat-completions wire contract.
  • Live docs say: that page (fetched 2026-09-21) now covers only image-generation and audio (transcription/translation) REST operations, and explicitly tells readers to go to https://learn.microsoft.com/en-us/rest/api/microsoft-foundry/azureopenai/chat for chat completions, embeddings, and "all other operations."
  • Impact: the snapshot's chat-completions assertions (max_completion_tokens, reasoning_effort, etc.) were not actually checkable against the cited page anymore — they still happen to be correct, reconfirmed against the real page.
  • Proposed action: added the actual chat reference URL as a source and re-verified against it directly. Applied.

openrouter: list-available-models source URL is dead [DRIFT]

  • Snapshot says: cites https://openrouter.ai/docs/api-reference/list-available-models.
  • Live docs say: 404 (confirmed both by the deterministic tripwire's source_checks and a live fetch 2026-09-21). Corrected URL: https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties — the response shape (context_length, pricing.{prompt,completion}, supported_parameters) is unchanged at the new location.
  • Impact: link rot only; no schema drift found.
  • Proposed action: corrected the URL. Applied.

openai: prompt-caching "VERIFY on next drift run" item — resolved

  • Snapshot said: cached tokens' inclusion in prompt_tokens/input_tokens and the caching threshold were both explicitly unverified.
  • Live docs say (developers.openai.com/api/docs/guides/prompt-caching, fetched 2026-09-21): cached tokens are confirmed still included in usage.input_tokens, broken out at usage.input_tokens_details.cached_tokens. Threshold is 1,024 visible input tokens for GPT-5.6+; earlier models vary by request shape (documented as such, not a gap in our check).
  • Impact: resolves a two-sprint-old open question; also surfaces a new cache_write_tokens field (Prompt Cache Diagnostics GA'd 2026-09-08) that isn't read by the adapter.
  • Proposed action: contract updated with citation. cache_write_tokens / diagnostics-endpoint support is a proposed adapter work item, not applied.

NEW-CAPABILITY findings (not urgent; feed the capability catalog / roadmap)

  • openai-compat / azure-foundry: stream_options.include_obfuscation, prompt_cache_key/prompt_cache_retention, and response usage.{prompt,completion}_tokens_details.* breakdowns exist on the wire and aren't read. Azure additionally exposes parallel_tool_calls, safety_identifier, store, modalities, prediction, and message-level annotations[].url_citation.
  • openai-compat / openai: error bodies now carry typed code values (slow_down for 429, server_is_overloaded for 503) — both already fall inside the existing retryable-status set, so no break, just finer-grained typing available if wanted.
  • openai: Responses API gained async tool calling, mid-turn steering, mid-conversation reasoning.effort changes, and expanded server-side tools (file_search, computer_use, image_generation, custom tools) beyond the web_search/code_interpreter pair already documented. GET /v1/models responses now include a shutdown_date field per model.
  • openrouter: /models gained filter/sort query params (min/max_intelligence_index, agentic_index, coding_index, tool_success_rate, output_price, age_days); attribution headers gained X-OpenRouter-Categories (and X-OpenRouter-Title is now the canonical name, with X-Title as an accepted alias); OpenRouter also GA'd a Responses-API-shaped endpoint (out of scope — this adapter only implements Chat Completions).
  • azure-foundry: confirmed the version-less /openai/v1 surface is now GA (resolves a standing Watchlist item), that both *.openai.azure.com and *.services.ai.azure.com base URL forms are valid, and that reasoning_effort now accepts xhigh.

Unverifiable

  • openai: the Responses API's full typed streaming-event catalog — the fetch this run confirmed response.output_text.delta/response.completed still exist but didn't enumerate the rest. Needs a more targeted fetch next run.
  • openai: platform.openai.com/docs/api-reference/{responses,models,chat} remain bot-walled (403) — verified instead via the developers.openai.com mirrors, the same substitution the embeddings section already used.
  • openrouter: the : OPENROUTER PROCESSING SSE keep-alive comment framing and the 402 insufficient-credits error shape weren't re-fetched this run; not contradicted, just not re-confirmed.

Left unapplied (adapter/code follow-up, out of scope for this workflow)

  • Reading usage.{prompt,completion}_tokens_details.* / cache_write_tokens for cache-aware pricing on the openai-compat, openai, and azure-foundry adapters.
  • Wiring prompt_cache_key for cache-affinity hints.
  • Declaring stream_options.include_obfuscation in ignored_parameters instead of silently dropping it.
  • Any of the newly-listed server-side Responses tools (file_search, computer_use, image_generation, custom tools).
  • GET /v1/models shutdown_date surfaced through model discovery.
  • OpenRouter /models filter params and the X-OpenRouter-Categories header.

Proposed contract edits

All applied directly in this PR, each with an inline citation and date. See the diff.

🤖 Generated with Claude Code

github-actions Bot and others added 2 commits September 21, 2026 13:50
Per-provider summary:
- openai-compat: added stream_options.include_obfuscation and prompt_cache_key
  request fields, usage.{prompt,completion}_tokens_details response fields, and
  the 429 slow_down / 503 server_is_overloaded error-code split (all
  NEW-CAPABILITY, none currently read by the adapter). Bot-walled
  platform.openai.com pages re-verified via the developers.openai.com mirrors.
- openai: resolved the standing "VERIFY on next drift run" prompt-caching item
  — cached tokens confirmed still counted in usage.input_tokens, threshold is
  1,024 tokens for GPT-5.6+. Added cache_write_tokens, expanded server-side
  tool list, async tool calling/mid-turn steering, /models shutdown_date, and
  the same error-code split as openai-compat (all NEW-CAPABILITY). Corrected
  the changelog source URL, which now 301s to developers.openai.com.
- openrouter: fixed a dead source URL (list-available-models -> the
  list-all-models-and-their-properties path) — DRIFT, link only, response
  shape unchanged. Added the new /models filter params, X-OpenRouter-Title/
  X-OpenRouter-Categories headers, and OpenRouter's new Responses-shaped
  endpoint as NEW-CAPABILITY.
- azure-foundry: the /reference upstream source has narrowed to image/audio
  only; added the actual current chat-completions reference URL and
  reconfirmed max_completion_tokens and reasoning_effort (now includes
  xhigh) directly against it. Confirmed the /openai/v1 surface is GA (not
  preview) and that both *.openai.azure.com and *.services.ai.azure.com
  base_url forms are valid. Noted Microsoft's ai-foundry -> foundry doc path
  rebrand (old links still resolve, not urgent) and the same
  prompt_cache_key/usage-details gaps as openai-compat.

All four snapshots were never_live_verified before this run; findings above
are the expected first-pass real drift, not noise. No adapter code changed —
every field addition here is a proposed follow-up work item.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…sing

The audit filed `usage.prompt_tokens_details.cached_tokens` and
`usage.completion_tokens_details.reasoning_tokens` as unread on both the
openai-compat and azure-foundry surfaces, and openai-compat.md went on to
say cache-aware pricing has no read path at all. Both are already wired:
`OpenAICompatAdapter._parse_usage` puts them in `Usage.cache_read_tokens`
and `Usage.reasoning_tokens`, `capabilities/pricing.py` reprices the
cached share against `cache_read_per_1m`, and `AzureFoundryAdapter`
inherits that parse rather than overriding it.

A snapshot is the record a later reader trusts about what the adapter
does today, so a gap recorded here that does not exist invites the work
to be done twice. The genuinely unread fields — the audio and prediction
breakdowns, and Azure's `annotations[].url_citation` — stay listed.
@anthturner
anthturner merged commit 6e9d022 into develop Sep 22, 2026
8 checks passed
@anthturner
anthturner deleted the contract-drift/2026-09-21 branch September 22, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant