[andrewklatzke]

Summary

• add launchdarkly-ai-litellm-messages using in-process LiteLLM completions
• add launchdarkly-ai-litellm-agents using the OpenAI Agents SDK LiteLLM model
• support tools, streaming, structured output, conversation history, native graphs, telemetry, and release configuration
• document installation, provider credentials, examples, and wildcard-handler behavior

Usage

from launchdarkly_ai_litellm_messages import litellm_messages

result = await litellm_messages(
    "launch-darkly-documentation-summarizer-messages",
    "How do AI Configs work?",
    {"kind": "user", "key": "user-123"},
)
print(result["response"])

Python runs LiteLLM in-process, so callers set the upstream provider credentials required by the evaluated model; no LiteLLM proxy is required.

Test plan

• uv run pytest
• uv run mypy across all package sources
• uv run ruff check .
• uv run ruff format --check .
• LiteLLM messages and agents integration scenarios

Made with Cursor

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[jeffdupont]

Reviewed with the 1.0 freeze in mind, alongside launchdarkly/js-ai-sdk#75 and the spec in launchdarkly/ai-sdks-monorepo#18. Tests pass at 3683c97 (uv run pytest: 1365 passed, 11 skipped, exit 0; make typecheck clean). The branch is 29 commits behind main (fee904a) and conflicts in .release-please-manifest.json and uv.lock. I resolved those by hand in a scratch tree and the merged result passes too (1445 passed, mypy clean). LiteLLM isn't in the GA plan, so the questions here are whether it's safe to ship and whether it ships as 1.0 or as experimental.

Four things I'd like settled before merge:

1. An AI Config can send the customer's provider key to any host. Both handlers pass model.parameters to LiteLLM, and LiteLLM takes connection settings as call arguments. I reproduced it at 3683c97: with ANTHROPIC_API_KEY set in the process and model.parameters = {api_base: "http://127.0.0.1:<port>"}, both create_litellm_messages_handler() and create_litellm_agents_handler() sent their request to my local listener with x-api-key: <the customer's key>. The agents model factory also reads base_url and api_key from the config on purpose (handler.py:117, native_graph.py:45). It's the same kind of problem as claude-agents on #107, and worse here: Python runs LiteLLM in-process, so no proxy sits in between. I'd use an allowlist per handler, as #107 does, and never forward connection or credential keys. Details inline.
2. The native graph emits the retired graph event. Since #114 (AIC-3211), every native adapter emits $ld:ai:graph:node once per node and no longer emits $ld:ai:graph:path. TESTING.md §2.2 on main says so. to_litellm_agents was copied before that change. After a merge it will be the only adapter still sending $ld:ai:graph:path, and the only one sending no node events. The merge doesn't conflict there and the tests still pass, so nothing will catch it.
3. Python and JS don't match (js #75). The transport difference is by design and the spec explains it (A.13). These other differences aren't explained:
   • Native graph telemetry: here you get a span and graph events. JS toLiteLLMAgents emits neither and takes no context.
   • Agent parameters: here, known snake_case keys go into ModelSettings and the rest into extra_args. JS passes the raw bag as modelSettings, where max_tokens and top_p are silently ignored. So one config gets a token limit in Python and none in JS.
   • Framework tracing: Python leaves the Agents SDK's own tracing on. JS turns it off for the whole process.
   • Factory name: create_litellm_agents_handler here, createLiteLLMAgentHandler in JS. The other agent packages use the same name in both languages.
   • Default gen_ai.provider.name when the config has no provider: litellm here, openai in JS. That's a span attribute Monitoring reads.
4. I think this should ship as experimental. Both packages start at 0.1.0 with bump-minor-pre-major, so they can stay 0.x when everything else goes to 1.0. The lifecycle spec (monorepo #33) only covers client features exported from launchdarkly_ai_server.experimental, not whole provider packages, so it needs a line for this case. My suggestion: keep both LiteLLM packages out of the 1.0 cut, say "experimental" in their READMEs, add them to the promotion register, and promote them once the two languages match.

Smaller notes:

• Runner.run is called without RunConfig(tracing_disabled=True). The Agents SDK's default exporter posts to https://api.openai.com/v1/traces/ingest (agents/tracing/processors.py:34). So if OPENAI_API_KEY is also set, a run that LiteLLM routes to Anthropic still sends its trace, prompts included, to OpenAI. I took that from the SDK source and haven't watched it happen at runtime. Spec §2.x.2 asks for suppression. Per-run tracing_disabled (agents/run_config.py:257) does that without touching the customer's own Agents usage.
• Messages streaming still records a cancelled stream as abandoned, with no launchdarkly.run.cancelled (Bugbot's last finding). The agents stream catches CancelledError; the messages stream doesn't.
• Once #121 lands, native graph spans get set_ld_span_attributes, and to_litellm_agents will need the same change.
• Content capture runs str(message["content"]) (litellm-messages/.../handler.py:362), so multimodal input shows up on the span as a Python list repr.

Bugbot's earlier findings (catalog entries, README install line, graph span leak, root content on stream) are fixed at this head. The history one in the convenience wrappers matches every other provider's wrapper, so it's not specific to this PR.

[andrewklatzke]

Addressed the review on 3cee111, then merged main in 642e4ca.

• Configs can only forward generation settings (temperature, top_p, max_tokens, max_completion_tokens, stop, seed, penalties, tool_choice, parallel_tool_calls, reasoning_effort). api_base, api_key, extra_headers, and the other connection keys never reach completion.
• Agents take base_url and api_key from the handler (or the environment). Dropped extra_args, extra_headers, extra_query, and extra_body.
• Native graphs emit $ld:ai:graph:node once per node. Runner.run / run_streamed pass RunConfig(tracing_disabled=True).
• A cancelled stream sets launchdarkly.run.cancelled and does not mark the run abandoned. Multimodal content uses content_to_text.
• Missing provider name defaults to litellm. Both READMEs say the packages stay on 0.x.

Left as-is: convenience wrappers still drop history, same as the other providers. set_ld_span_attributes is still only on agent spans on main, so native-graph spans were not changed. The promotion register lives in the monorepo, not this repo.

[cursor]

Stale Bugbot comment from a previous run.

[jeffdupont]

Re-checked at 642e4ca. Most of my 2 Oct points are fixed: the provider key can't be redirected, there's no $ld:ai:graph:path event, per-run tracing_disabled, cancelled-over-abandoned, and multimodal content text.

Still blocking:

1. Merge conflicts with main in .release-please-manifest.json, .sdk_metadata.json, AGENTS.md, release-please-config.json and uv.lock.
2. litellm_graph can't set connection settings. litellm-agents/.../graph.py:12-27 passes only model_factory and capture_content. base_url/api_key land in **options, and graph() rejects them with TypeError. Now that config can't supply URLs, graph users can't reach a proxy. JS litellmGraph forwards these.
3. Agents allowlist forwards metadata (handler.py:47), plus include_usage and top_logprobs, which messages doesn't allow. JS messages explicitly tests that metadata never reaches the request. Pick one list across both SDKs.

Smaller:

• reasoning goes into ModelSettings as a raw dict (handler.py:45), while the SDK types it as Reasoning, and messages reads reasoning_effort instead. I haven't checked whether LitellmModel accepts the dict.
• No test asserts RunConfig(tracing_disabled=True), so a regression would bring back trace export to OpenAI unnoticed.
• The spec in ai-sdks-monorepo#18 still describes the old denylist.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 94d481f. Configure here.}

[jeffdupont]

Looks good against the spec. One thing: _model_config only keeps snake_case keys, so topP/maxTurns get dropped, but spec §2.x.1 says either spelling is read (JS does). A quick normalize before the filter plus a test would cover it. Also, should messages take base_url/api_key like JS? A.20 is a bit ambiguous there.

[andrewklatzke]

Both fixed at 6aaeffb:

• `_model_config` normalizes `topP`/`maxTurns`/etc. to their snake_case name before filtering (snake_case wins when both are set), with a test. Covers `to_litellm_agents` too, since it reuses `_model_config`.
• `create_litellm_messages_handler` now takes `base_url`/`api_key`, forwarded to `acompletion` as `api_base`/`api_key`. `model.parameters` still can't set them. Clarified A.20 in ai-sdks-monorepo#18 (2a54015) to say both handler factories in both languages take these options, so it reads less ambiguously.