Skip to content

Build ManagedCode.LlmTestServer with isolated deterministic API testing - #1

Merged
KSemenenko merged 12 commits into
mainfrom
codex/llm-test-server
Oct 20, 2026
Merged

KSemenenko merged 12 commits into
mainfrom
codex/llm-test-server

Conversation

@KSemenenko

@KSemenenko KSemenenko commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Build ManagedCode.LlmTestServer as a deterministic LLM API test backend with one identity across the runtime, 19 packages, provider adapters, control client, Aspire integration, samples and documentation.

The branch adds isolated sessions, conversation graphs and correlated tool results; bounded request query/export and workflow verification; seeded wire metadata and fixture clocks; and twelve typed function-tool presets. Cache keys preserve provider/session ownership, and capacity failures are rejected before cache admission.

OpenAI Responses supports bounded completed native web-search, file-search and code-interpreter fixtures, with ordered outputs, citations, SSE lifecycle, mixed function rounds and continuation. Anthropic Messages adds bounded native web search, citations, six documented in-band search errors and strict official SDK JSON/SSE parsing. Native tools return scripted data; they do not execute searches or code.

Official SDK integration now covers Anthropic, Google Gemini, AWS Bedrock and OpenAI-compatible chat routes, in addition to existing OpenAI/Azure/Foundry clients. Fixtures use current documented model IDs, including the original Anthropic SDK Sonnet 5.5 identifier. Gemini's SDK batch-embedding route validates requested dimensions atomically. Gemini function rounds preserve opaque thought signatures in JSON/SSE, history, cache identity and retention; first-call presence and last locally issued signature checks are explicit, with broader text/multimodal signature support outside this slice.

Tool-failure integrations distinguish literal null, empty and absent results; controlled invocation exceptions and cancellation; provider 500/delays; filtered tool results; and explicit recovery with cache/history assertions. Text SSE on OpenAI Chat Completions, OpenAI Responses, Anthropic Messages and Gemini supports bounded early clean EOF, a cancellable post-frame delay and actual connection abort after a selected nonempty text frame. Official SDK tests over real Kestrel/TCP distinguish committed inference/cache effects from delivery, verify missing terminals even after all text arrives, replay and explicit recovery, and reject unsupported request modes atomically. Gemini replay excludes transport capability metadata from inference identity while preserving credential/provider/session ownership and checking the current delivery mode before replay. Tests synchronize real server startup and cancellation, drain HTTP hosts before disposal, and retain strict cache/queue rollback checks. Other E6 delivery families and malformed streams remain unfinished.

Native text output-policy fixtures now distinguish successful output refusal/filtering from input rejection and broken transport. OpenAI Chat maps typed refusal and content_filter; Responses maps refusal parts/events or incomplete content_filter; strict Anthropic parses refusal stop details; Gemini emits a terminal SAFETY candidate without duplicated content; DeepSeek Chat returns native content_filter with its own mandatory final-choice usage layout. Microsoft.Extensions.AI preserves refusal as ErrorContent. Real SDK tests cover terminal evidence, explicit recovery, graph/cache/session ownership, canonical stored/manual history and bounded diagnostics; unsupported surfaces fail before consuming the fixture. Raw JSON configuration accepts the documented Filtered/Refused names, and invalid configuration tests prove their intended runtime rejection after a successful baseline. DeepSeek rejects non-streaming stream_options before consumption. Raw empty-header TCP tests synchronize actual endpoint startup and drain requests before disposal. This slice does not qualify policy behavior for the remaining provider roots.

Launch preparation includes GitHub Pages and .NET SDK-based container distribution with management configuration. Owner visual/analytics changes remain outside this code checkpoint.

Validation for code checkpoint 3ce0742:

  • Final code-only full server integrations: 1438/1438; production line coverage 92.58%, branch 79.62%.
  • Output-policy SDK/server cases: 79/79, including 19 DeepSeek cases and three successful documented raw JSON configuration cases; verification/lifecycle suite 54/54.
  • Canonical formatting and final Release build passed with zero warnings/errors, including the concurrent owner working tree.
  • All 19 packages packed; fresh installed-package Aspire/original SDK/Chromium consumer 16/16 with restored artifact SHA-512 verification. The .NET SDK-built linux/arm64 container passed standalone management, native tools and DeepSeek filtered JSON/SSE, final usage, invalid-options/no-consumption and recovery checks.

Exact-head GitHub CI and website checks for this new checkpoint are pending. Previous checkpoint 0aea3dc passed CI and website; those results do not qualify the new head.

This remains a draft for the complete v1.0.0 launch. Other E5 native contracts and E6–E13 remain unfinished; public NuGet/GHCR/Pages launch has not occurred. The owner merges when the full batch is ready. Release acceptance requires actual public package availability, anonymous GHCR pull and the public installed consumer. No intermediate release or version bump is included.

@KSemenenko KSemenenko changed the title Rename to ManagedCode.LlmTestServer and isolate test sessions Build ManagedCode.LlmTestServer with isolated deterministic API testing Oct 20, 2026
@KSemenenko
KSemenenko merged commit ef2fb46 into main Oct 20, 2026
2 of 3 checks passed

This branch had an error being deployed

1 failed deployment
github-pages — ef2fb465 Deployed Oct 20, 2026 by KSemenenko via deploy #12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant