Skip to content

fix(traceai-openai): request model on stream/error, provider from base_url host, readable Responses + embedding spans (TH-8402) - #249

Merged
nik13 merged 5 commits into
devfrom
fix/th-8402-openai-compatible-provider-model
Oct 28, 2026
Merged

nik13 merged 5 commits into
devfrom
fix/th-8402-openai-compatible-provider-model

Conversation

@nik13

@nik13 nik13 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Current-head verification — 2026-10-28

Head: bde0419bf48c2c63aa95fb9bad43f5e0a6ee43a1 (dev merged normally).

  • Built the wheel and installed it into fresh Python 3.11 consumers. 51 tests pass on both openai==3.24.0 and the 1.69.0 SDK floor.
  • Nine real OpenAI API cases with gpt-5.4-nano / text-embedding-3-small: chat; synchronous and asynchronous context-managed chat streams; Responses plain and streamed; embeddings; invalid-model error; hidden chat; hidden embeddings. All nine exact trace IDs were read back in Future AGI Cloud, including model/provider, token usage, output and privacy assertions.
  • Fresh testing found and fixed two additional issues: context managers returned the unwrapped SDK stream and lost spans; embedding input text and vectors escaped the generic input/output privacy flags. Regression tests cover full, early, empty and exceptional context exits plus both embedding privacy flags.
  • Representative Cloud traces: chat 1eaae6aa-5561-4eaf-8efb-7fc934db49cf; async stream ccd73abf-48c6-4f95-8194-ea9595a191aa; Responses 5ce53526-4d4b-4d10-9491-e28e9e18970d; hidden embeddings 43071e0c-bf7a-4f56-88c3-4287b933fc5d.
  • Exact-head Python/package and integration CI is green. The earlier verification sections below describe their historical runs.

Summary

Fixes TH-8402 in traceai-openai, plus the Responses-API and embedding span problems that the same live E2E (TH-8103 provider recipes #234–#243) exposed in this package.

Problem (seen live in ClickHouse / Observe) Fix
Streamed and failed Chat Completions spans had no gen_ai.request.model, so Observe showed no Model. The requested model is recorded at span start. A successful non-streamed response still overwrites it with the served model, so existing behaviour there is unchanged.
gen_ai.provider.name was always openai. A request-level openai even overwrote the existing Azure/Google host detection. The provider comes from the client's base_url host via a small explicit table (_providers.py). Unknown hosts, proxies, loopback and look-alike domains stay openai; nothing is guessed.
Non-streamed Responses output.value was Python str(dict). A string input was JSON-quoted ("\"hi\""). The Observe Assistant card showed {}. output.value is valid JSON with output.mime_type=application/json, led by output_text. A string input is recorded as text. Text-only assistant messages are a flat message.content, the same shape as Chat Completions.
Embedding spans put the full vector (~69 KB) into output.value as str(dict) and had no span model. FI_HIDE_EMBEDDING_VECTORS left the vectors in the raw embedding.embeddings JSON. Items without index (Gemini's OpenAI-compatible API) were skipped. gen_ai.request.model = embedding model. output.value = compact JSON {model, embeddings, dimensions}. The hide flag also strips the raw JSON (each vector is replaced by its dimension count). A missing index falls back to the item's position. Token attributes stay omitted when the provider returns no usage.

The provider table has exact hosts plus suffixes only for per-tenant subdomains:

Host Provider
api.openai.com openai
*.openai.azure.com, *.services.ai.azure.com, *.cognitiveservices.azure.com azure
generativelanguage.googleapis.com google
*-aiplatform.googleapis.com vertexai
api.anthropic.com anthropic
api.mistral.ai mistralai
api.deepseek.com deepseek
api.x.ai xai
inference.baseten.co baseten
dashscope*.aliyuncs.com, *.maas.aliyuncs.com dashscope
*.cloud.databricks.com, *.gcp.databricks.com, *.azuredatabricks.net databricks
*.xenon.lepton.run lepton
api.perplexity.ai perplexity
api.orcarouter.ai orcarouter
api.anannas.ai, anannas.ai anannas
api.doubleword.ai doubleword

Cost lookup in the backend is keyed on the model, not the provider, so the new labels don't change pricing.

Tests (test-first)

New python/frameworks/openai/tests/test_openai_compatible_spans.py has 41 tests.

  • They drive the real openai SDK through httpx.MockTransport and export through a real OpenTelemetry SDK pipeline (InMemorySpanExporter).
  • They cover sync and async, chat, stream, error, the provider table (including look-alike and loopback hosts), Responses plain and streamed, hide_outputs, hide_llm_invocation_parameters, and embeddings with and without index, usage and hidden vectors.
  • 30 of them failed before the fix.
Python openai Result
3.12 3.24.0 41 passed
3.11 3.24.0 41 passed
3.11 1.69.0 (package floor) 41 passed

Command, from python/frameworks/openai:
uv run --no-project --python 3.11 --with 'openai==3.24.0' --with 'fi-instrumentation-otel==1.1.0' --with httpx --with 'wrapt<2' --with pytest --with opentelemetry-sdk pytest tests -q, with PYTHONPATH=..

Real E2E (a local Future AGI stack built from future-agi dev)

The recipe apps from #234 (Baseten), #239 (Perplexity, Responses API) and #237 (Databricks, embeddings) ran unchanged against this branch, installed editable.

  • Endpoints:
    • a local OpenAI-compatible endpoint on loopback, serving a real model; its host is unknown to the map, so provider is openai by design;
    • Gemini's OpenAI-compatible host generativelanguage.googleapis.com (gemini-3.1-flash-lite, gemini-embedding-001).
  • Spans were checked in ClickHouse (the store the Observe UI reads) and in Observe UI screenshots.
Run Span ClickHouse provider / model after Before this PR
Baseten recipe, loopback endpoint chat openai / requested model same
stream openai / requested model model empty
401 openai / requested model model empty
400 bad model openai / the bad model id model empty
Baseten recipe, Gemini host chat google / gemini-3.1-flash-lite provider openai
stream google / gemini-3.1-flash-lite openai / empty
404 bad model google / the bad model id openai / empty
Perplexity recipe (Responses API), loopback endpoint non-streamed output = JSON {"output_text":"Rome",...}, application/json; input unquoted; message.content="Rome" str(dict), "\"…\"", {} card
streamed message.content="1, 2, 3" {} card
Databricks recipe, embeddings on the Gemini host embedding google / gemini-embedding-001; output {"model": …, "embeddings": 1, "dimensions": 3072} (70 B) openai / empty; 69 KB float dump
embedding with FI_HIDE_EMBEDDING_VECTORS=true raw embedding.embeddings 106 B with no vectors; no indexed vector attribute vectors still exported

In the Observe UI:

  • the streamed chat and failed calls now show a Model chip;
  • the Gemini spans show Provider google with its logo;
  • the Responses Assistant card shows the answer ("Tokyo"), and the trace list Output renders {output_text: "Rome", …};
  • the embedding span shows Model gemini-embedding-001 and a three-line JSON summary as output.

A secret grep of the stored spans and run logs found 0 credential hits.

Follow-ups (not in this PR)

  • Recipe pins: order-independent. The provider recipes feat(examples): Baseten Model APIs recipe on traceai-openai (TH-8310) #234–feat(examples): PrismML Bonsai local server recipe on traceai-openai (TH-8308) #243 used to pin today's behaviour, to make this shared fix visible: gen_ai.provider.name == "openai", and no model on stream/error spans. They now accept both versions, and their READMEs say which version does what. Each recipe's suite passes against published traceAI-openai==0.1.10 and against this branch, so the PRs can merge in any order.
  • Frontend (future-agi):
    • the embedding token chips show 0 when no usage was sent, because ClickHouse defaults those columns; this package omits the token attributes;
    • error and token-less spans show a stray 0 next to the model badge;
    • structured message.contents.N.message_content.text is mis-parsed by the preview's .replace("content.", "") in getSpanData.js.

🤖 Generated with Claude Code

nik13 and others added 2 commits October 11, 2026 20:48
…ses and embedding spans (TH-8402)

Live E2E of the OpenAI-compatible provider recipes (#234-#243) showed:
- streamed and failed Chat Completions spans had no gen_ai.request.model;
- gen_ai.provider.name was always "openai" (the request-level label even
  overwrote the existing Azure/Google host detection);
- non-streamed Responses output.value was Python str(dict), a string input
  was JSON-quoted, and the Observe preview showed `{}` as the answer because
  assistant text was only emitted as message.contents.N.message_content.text;
- embedding spans put the full vector into output.value as str(dict), had no
  span model, ignored FI_HIDE_EMBEDDING_VECTORS for the raw
  embedding.embeddings JSON, and skipped items without `index` (Gemini).

Fixes, test-first (python/frameworks/openai/tests, real openai SDK through
httpx.MockTransport and a real OTel SDK exporter; 41 tests, openai 1.69.0 and
3.24.0):
- record the requested model at span start; served model still wins on a
  successful non-streamed response;
- map the client's base_url host to a provider via a small explicit table
  (_providers.py); unknown hosts, proxies and loopback stay "openai";
- Responses: JSON output.value + application/json mime, led by output_text;
  plain-string input recorded as text; text-only assistant messages emitted
  as a flat message.content;
- embeddings: gen_ai.request.model = embedding model, compact JSON summary
  in output.value, hide-vectors strips the raw JSON too, index falls back to
  position;
- mime types follow the recorded value; async request path also reads
  Responses `input`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 12, 2026
…ovider/model fix

The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is
always "openai" and streamed/failed spans carry no gen_ai.request.model.
#249 labels known vendor hosts and records the requested
model, so these pins would break whichever PR merged second. Accept both
behaviours (and the related Responses/embedding output changes where this
recipe checks them), and say in the README which version does what.

Passes against published traceAI-openai 0.1.10 and against #249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 12, 2026
…der/model fix

The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is
always "openai" and streamed/failed spans carry no gen_ai.request.model.
#249 labels known vendor hosts and records the requested
model, so these pins would break whichever PR merged second. Accept both
behaviours (and the related Responses/embedding output changes where this
recipe checks them), and say in the README which version does what.

Passes against published traceAI-openai 0.1.10 and against #249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 12, 2026
…er the provider/model fix

The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is
always "openai" and streamed/failed spans carry no gen_ai.request.model.
#249 labels known vendor hosts and records the requested
model, so these pins would break whichever PR merged second. Accept both
behaviours (and the related Responses/embedding output changes where this
recipe checks them), and say in the README which version does what.

Passes against published traceAI-openai 0.1.10 and against #249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 12, 2026
… provider/model fix

The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is
always "openai" and streamed/failed spans carry no gen_ai.request.model.
#249 labels known vendor hosts and records the requested
model, so these pins would break whichever PR merged second. Accept both
behaviours (and the related Responses/embedding output changes where this
recipe checks them), and say in the README which version does what.

Passes against published traceAI-openai 0.1.10 and against #249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 12, 2026
…vider/model fix

The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is
always "openai" and streamed/failed spans carry no gen_ai.request.model.
#249 labels known vendor hosts and records the requested
model, so these pins would break whichever PR merged second. Accept both
behaviours (and the related Responses/embedding output changes where this
recipe checks them), and say in the README which version does what.

Passes against published traceAI-openai 0.1.10 and against #249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 12, 2026
… provider/model fix

The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is
always "openai" and streamed/failed spans carry no gen_ai.request.model.
#249 labels known vendor hosts and records the requested
model, so these pins would break whichever PR merged second. Accept both
behaviours (and the related Responses/embedding output changes where this
recipe checks them), and say in the README which version does what.

Passes against published traceAI-openai 0.1.10 and against #249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 12, 2026
… provider/model fix

The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is
always "openai" and streamed/failed spans carry no gen_ai.request.model.
#249 labels known vendor hosts and records the requested
model, so these pins would break whichever PR merged second. Accept both
behaviours (and the related Responses/embedding output changes where this
recipe checks them), and say in the README which version does what.

Passes against published traceAI-openai 0.1.10 and against #249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 12, 2026
… provider/model fix

The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is
always "openai" and streamed/failed spans carry no gen_ai.request.model.
#249 labels known vendor hosts and records the requested
model, so these pins would break whichever PR merged second. Accept both
behaviours (and the related Responses/embedding output changes where this
recipe checks them), and say in the README which version does what.

Passes against published traceAI-openai 0.1.10 and against #249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 12, 2026
… provider/model fix

The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is
always "openai" and streamed/failed spans carry no gen_ai.request.model.
#249 labels known vendor hosts and records the requested
model, so these pins would break whichever PR merged second. Accept both
behaviours (and the related Responses/embedding output changes where this
recipe checks them), and say in the README which version does what.

Passes against published traceAI-openai 0.1.10 and against #249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 12, 2026
…ovider/model fix

The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is
always "openai" and streamed/failed spans carry no gen_ai.request.model.
#249 labels known vendor hosts and records the requested
model, so these pins would break whichever PR merged second. Accept both
behaviours (and the related Responses/embedding output changes where this
recipe checks them), and say in the README which version does what.

Passes against published traceAI-openai 0.1.10 and against #249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 19, 2026
setup_tracing() passed project_name or "orcarouter-example" to register(),
so the default was always truthy and FI_PROJECT_NAME was never read:
`python src/app.py` always exported to orcarouter-example. The Anannas
recipe reads FI_PROJECT_NAME with its name as the default; do the same
here. An explicit setup_tracing() argument still wins, and an unset or
empty FI_PROJECT_NAME keeps orcarouter-example (never register()'s
generic default).

- README: FI_PROJECT_NAME in the Configure table and the Run section;
  the Code snippet reads it the same way as src/app.py.
- Tests: setup_tracing() default / empty / env / explicit cases; the
  guarded app.py subprocess runs with FI_PROJECT_NAME unset and set; the
  README contract checks the new row and snippet.

Passes against repository source, published traceAI-openai 0.1.10 and
#249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 19, 2026
setup_tracing() passed project_name or "prismml-bonsai" to register(),
so the default was always truthy and FI_PROJECT_NAME was never read:
`python src/app.py` always exported to prismml-bonsai. The Anannas
recipe reads FI_PROJECT_NAME with its name as the default; do the same
here. An explicit setup_tracing() argument still wins, and an unset or
empty FI_PROJECT_NAME keeps prismml-bonsai (never register()'s generic
default).

- README: FI_PROJECT_NAME in the Configure table and the Run section;
  the Code snippet reads it the same way as src/app.py.
- Tests: setup_tracing() default / empty / env / explicit cases; the
  app.py subprocess runs with FI_PROJECT_NAME unset and set; the README
  contract checks the new row and snippet.

Passes against repository source, published traceAI-openai 0.1.10 and
#249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 19, 2026
setup_tracing() passed project_name or "baseten-openai" to register(),
so the default was always truthy and FI_PROJECT_NAME was never read:
`python src/app.py` always exported to baseten-openai. The Anannas
recipe reads FI_PROJECT_NAME with its name as the default; do the same
here. An explicit setup_tracing() argument still wins, and an unset or
empty FI_PROJECT_NAME keeps baseten-openai (never register()'s generic
default).

- README: FI_PROJECT_NAME in the Configure table and the Run section;
  the Code snippet reads it the same way as src/app.py.
- Tests: setup_tracing() default / empty / env / explicit cases; the
  guarded app.py subprocess runs with FI_PROJECT_NAME unset and set; the
  README contract checks the new row and snippet.

Passes against repository source, published traceAI-openai 0.1.10 and
#249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 19, 2026
setup_tracing() passed project_name or "doubleword-example" to
register(), so the default was always truthy and FI_PROJECT_NAME was
never read: `python src/app.py` always exported to doubleword-example.
The Anannas recipe reads FI_PROJECT_NAME with its name as the default;
do the same here. An explicit setup_tracing() argument still wins, and
an unset or empty FI_PROJECT_NAME keeps doubleword-example (never
register()'s generic default).

- README: FI_PROJECT_NAME in the Configure table and the Run section;
  the Code snippet reads it the same way as src/app.py.
- Tests: setup_tracing() default / empty / env / explicit cases; the
  guarded app.py subprocess runs with FI_PROJECT_NAME unset and set; the
  README contract checks the new row and snippet.

Passes against repository source, published traceAI-openai 0.1.10 and
#249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 19, 2026
setup_tracing() passed project_name or "qwen-openai-recipe" to
register(), so the default was always truthy and FI_PROJECT_NAME was
never read: `python src/app.py` always exported to qwen-openai-recipe.
The Anannas recipe reads FI_PROJECT_NAME with its name as the default;
do the same here. An explicit setup_tracing() argument still wins, and
an unset or empty FI_PROJECT_NAME keeps qwen-openai-recipe (never
register()'s generic default).

- README: FI_PROJECT_NAME in the Configure table and the Run section;
  the Code snippet reads it the same way as src/app.py.
- Tests: setup_tracing() default / empty / env / explicit cases; the
  guarded app.py subprocess runs with FI_PROJECT_NAME unset and set; the
  README contract checks the new row and snippet.

Passes against repository source, published traceAI-openai 0.1.10 and
#249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 19, 2026
setup_tracing() passed project_name or "azure-ai-foundry" to register(),
so the default was always truthy and FI_PROJECT_NAME was never read:
`python src/app.py` always exported to azure-ai-foundry. The Anannas
recipe reads FI_PROJECT_NAME with its name as the default; do the same
here. An explicit setup_tracing() argument still wins, and an unset or
empty FI_PROJECT_NAME keeps azure-ai-foundry (never register()'s generic
default).

- README: FI_PROJECT_NAME in the Configure table, the project-name note
  and the Run section; the Code snippet reads it the same way as
  src/app.py.
- Tests: setup_tracing() default / empty / env / explicit cases. The
  subprocess journeys already set FI_PROJECT_NAME but asserted that
  app.py exported to azure-ai-foundry anyway; they now expect the env
  project, and chat and stream also run with it unset. The README
  contract checks the new row and snippet.

Passes against repository source, published traceAI-openai 0.1.10 and
#249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Oct 19, 2026
setup_tracing() passed project_name or "databricks-openai-recipe" to
register(), so the default was always truthy and FI_PROJECT_NAME was
never read: `python src/app.py` always exported to
databricks-openai-recipe. The Anannas recipe reads FI_PROJECT_NAME with
its name as the default; do the same here. An explicit setup_tracing()
argument still wins, and an unset or empty FI_PROJECT_NAME keeps
databricks-openai-recipe (never register()'s generic default).

- README: FI_PROJECT_NAME in the Configure table and the Run section;
  the Code snippet reads it the same way as src/app.py.
- Tests: setup_tracing() default / empty / env / explicit cases; the
  guarded app.py subprocess runs with FI_PROJECT_NAME unset and set; the
  README contract checks the new row and snippet.

Passes against repository source, published traceAI-openai 0.1.10 and
#249.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 3 commits October 27, 2026 20:38
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
@nik13
nik13 merged commit b79e0db into dev Oct 28, 2026
18 checks passed
@nik13
nik13 deleted the fix/th-8402-openai-compatible-provider-model branch October 28, 2026 05:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant