Skip to content

fix(litellm): preserve request context and streaming lifecycle - #246

Open
jjcav84 wants to merge 11 commits into
future-agi:devfrom
jjcav84:fix/litellm-input-value-context
Open

jjcav84 wants to merge 11 commits into
future-agi:devfrom
jjcav84:fix/litellm-input-value-context

Conversation

@jjcav84

@jjcav84 jjcav84 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Preserves the original contributor's multi-message input.value fix and completes the LiteLLM Python integration repair. Reuses PR #246 and its existing branch; dev was merged normally, without rebasing or force-pushing.

  • Preserve conversation text, tool-call IDs/arguments/results, and multimodal image references without changing provider requests or responses.
  • Apply input-image hiding and base64 length limits to every image representation. Tool-bearing conversations no longer put raw image bytes back into input.value; canonical image attributes end in image.url, and the aggregate uses the same configured mask.
  • Treat FI_HIDE_INPUT_TEXT and FI_HIDE_INPUT_MESSAGES as private embedding text even with published core 1.2.2; preserve audio and other non-text input blocks in structured input.value without exposing images.
  • Mask every indexed embedding input even with the published minimum core, fi-instrumentation-otel==1.2.2. Associate each batch input with its vector using the provider's embedding index; vectors have no unfiltered raw-output copies.
  • Preserve all completion choices and indices, including interleaved sync/async streaming choices, rather than collapsing n > 1 to the first alternative.
  • Bind positional arguments to the correct endpoint, retain completed response types, and support the retry helper after a runtime-only installation by declaring tenacity as a runtime dependency.
  • Finalize stream spans once on exhaustion, explicit close, construction/read errors, or cancellation. Close the real underlying provider stream on older LiteLLM versions, restore caller context, and preserve original exceptions.
  • Avoid duplicate spans during nested SDK/retry operations. Do not serialize provider credentials, headers, clients, or callbacks into invocation parameters, or let optional telemetry extraction break successful SDK calls.
  • Restore the owned Protect.protect descriptor on uninstrumentation.
  • Add floor/latest compatibility CI and loopback OTLP contract coverage. The README separates Future AGI telemetry credentials from the provider's BYOK inference credentials and warns against duplicate tracing routes.

Validation

All local inference tests run with outbound networking denied, an isolated home, and synthetic credentials. Provider contract tests use actual LiteLLM/OpenAI SDKs with synthetic HTTPX transports or LiteLLM's built-in mock responses. OTLP acceptance is loopback only, not live vendor or Future AGI Cloud acceptance.

The second independent review's five findings were reproduced before implementation changes: 7 failed, 4 passed on each SDK version. The expanded regression matrix covers explicit configuration and environment overrides, image hiding with/without tool history, default/custom image size limits and allowed-image controls, single/batched text and token embedding inputs, reordered embedding results, and sync/async multi-choice responses and interleaved streams.

The third review's two remaining defects were reproduced on the released core before changes (10 failures, 6 controls passing); all 16 targeted regression cases now pass on both SDK versions.

Current full package and framework suite, including prior lifecycle, privacy, suppression, README and loopback OTLP tests:

  • LiteLLM 1.43.0, Python 3.10, OpenAI 1.40.0, HTTPX 0.27.2, published core 1.2.2: 170 passed.
  • LiteLLM 1.106.6, Python 3.12, OpenAI 2.54.0, HTTPX 0.28.1, published core 1.2.2: 170 passed.
  • Previous Python 3.13 run (before third-round regressions), LiteLLM 1.106.6 and published core 1.2.2: 154 passed.
  • release_plan.py --check: 129 packages, versions and coverage consistent.
  • pnpm install --frozen-lockfile --ignore-scripts after the normal dev merge: pass.
  • Floor/latest uv pip check and repair patch git diff --check: pass.

These tests intentionally use the released core dependency, not editable repository core, so unpublished privacy fixes cannot conceal integration leaks. GitHub test workflows require fork-workflow approval; local results do not imply remote CI is green.

PyPI no longer serves LiteLLM 1.43.0. The floor matrix installs its immutable upstream release commit 4603eedcc13bfab2da5a82aed209e6ef496c384d. The LiteLLM floor is unchanged; the core floor is now 1.2.2. The deprecated acompletion_with_retries helper remains unpatched; supported async calls use acompletion.

Release / documentation

Updates the existing maintained package. No registry publication or PR merge is part of this repair. Update the customer guide's pinned version only after the repaired package is published; these changes are not represented as already available from PyPI.

Related: #151.

🤖 Generated with Claude Code

@jjcav84

jjcav84 commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Bump — fixes multi-message context truncation in input.value. Mergeable.

@nik13 nik13 changed the title fix(litellm): keep full multi-message context in input.value fix(litellm): preserve request context and streaming lifecycle Nov 2, 2026
@nik13
nik13 changed the base branch from main to dev November 2, 2026 08:03
jjcav84 added a commit to jjcav84/traceAI that referenced this pull request Nov 6, 2026
Third instance of the future-agi#151 defect family (openai future-agi#245, litellm future-agi#246):
each wrapped call built its attribute dict with _get_query last, where
query = eval_input[0] — the first message's text — overwriting the
joined eval_input that carried every message.

Remove the query write, its dead _get_query helper, and the "query" key
from _extract_image_data; move _get_eval_input to the end of the chain
so the joined multi-message text wins INPUT_VALUE, matching the openai,
litellm, dspy, and groq handlers.

Generated with Devin
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
jjcav84 and others added 11 commits November 5, 2026 18:38
Same defect as future-agi#151 (fixed for openai in future-agi#245): the completion
instrumentation wrote INPUT_VALUE three times, ending with
"query" = eval_input[0] — the first message's text only — after
already writing the joined text of every message. Multi-turn inputs
lost all but the first content piece.

Drop the query write and the now-unused "query" key from
_process_messages so the joined eval_input survives; image parts still
surface as {{input_images.N}} placeholders alongside INPUT_IMAGES.

Also repairs three stale attribute assertions in the litellm test file
that referenced pre-migration names (FI_SPAN_KIND -> GEN_AI_SPAN_KIND,
LLM_MODEL_NAME -> GEN_AI_REQUEST_MODEL) so the file actually runs green.

Generated with Devin
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Keep streaming context scoped to provider operations, finalize errors and cancellation, retain usage-only chunks, suppress nested calls, bind positional requests, and omit SDK credentials from telemetry. Restore owned Protect descriptors on uninstrument. Exercise minimum/latest SDKs and loopback OTLP export.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…context

Address recall review: keep embedding vectors exclusively in privacy-filtered attributes, preserve completed response types and tool conversations, bind async endpoint arguments correctly, and declare/test usable runtime dependencies and provider credentials.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Apply image privacy and size limits to every captured representation, retain tool linkage without raw image copies, and mask indexed embedding inputs against released core. Match batch vectors by provider index and preserve every completion choice for sync and async streams and responses.

Add real SDK transport regressions covering privacy configuration and environment overrides, supported input shapes, image limits, and multi-choice completions.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Trace optional Responses endpoints, preserve provider/model and detailed usage metadata, accumulate streaming tools, and finalize abandoned streams once. Cover real loopback SDK requests, privacy, suppression, errors and lifecycle behavior on current and floor dependencies.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Project connection-bearing tool settings before serialization, add the supported Responses runtime extra, trace sync and async text completions, and retain reported finish reasons by choice. Cover real HTTP streams, errors, privacy and SDK floor behavior in cloud-tested regressions.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Source core now fail-closes input.value to __REDACTED__ under
hide_input_images when the value is an opaque string, so the sanitized
conversation JSON was being dropped entirely. Emit the filtered messages
through the structured ChatInput API instead (same as fireworks), which
lets core mask per content item: images and tool-call payloads are
redacted while text, tool history, and non-text blocks survive.

Also teach core _content the input_audio part kind — speech blobs redact
their data under hide_input_text but are preserved under
hide_input_images, matching the test contract that non-image content
survives image hiding. Unknown part types still fail closed.

Generated with Devin

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@jjcav84
jjcav84 force-pushed the fix/litellm-input-value-context branch from 4ea170f to 662940f Compare November 6, 2026 00:55

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants