Repository navigation
feat(examples): PrismML Bonsai local server recipe on traceai-openai (TH-8308) - #243
Merged
Merged
Conversation
Adds python/examples/prismml/: the official openai SDK pointed at the customer's own PrismML Bonsai OpenAI-compatible server (llama.cpp at http://localhost:8080/v1, MLX at http://localhost:8081/v1), traced by the existing traceai-openai instrumentor. No provider package. - The server does not check keys: PRISMML_API_KEY is optional and defaults to the placeholder not-needed; tests pin that it never reaches the Future AGI export and the Future AGI keys never reach the server. - PRISMML_BASE_URL is validated before tracing and never rewritten: exactly /v1 or /v1/ (the root is the built-in web chat), no query or fragment delimiter, no credentials, whitespace/control characters, non-ASCII or malformed hosts; exit 2. HTTP on loopback is expected; HTTP on another host is allowed with one stderr warning that the server is unauthenticated (quoting the vendor's warning). - Pins current traceai-openai behaviour: a successful non-streamed span records the model name the server reports, the requested name stays in gen_ai.request.parameters; streamed and failed spans omit the model; thinking_budget_tokens via extra_body reaches the server but is not in the parameters; tool-call ids, names and arguments are recorded. - Chat Completions output.value is the assistant text (empty for a textless stream); FI_HIDE_INPUTS/FI_HIDE_OUTPUTS with controls. - Loopback only: MockTransport at both documented URLs, a local OpenAI-shaped fake, the shared harness receiver and a socket guard with a loopback control. approved: Nikhil 2026-10-03 blanket Refs: TH-8308
The README contract test searched for internal project names, which put those names in the repository. The generic ticket-id and key checks on every recipe file stay. approved: Nikhil 2026-10-03 blanket Refs: TH-8308
…tion - check_base_url accepts underscores inside host labels, so container and Compose service names such as bonsai_server work (HTTP on such a host still prints the one-line unauthenticated-server warning); the refusal message names the allowed characters. - Only one trailing DNS dot is stripped (localhost.. is refused before tracing), and loopback is decided with ipaddress (127.0.0.0/8 and any ::1 spelling are not warned). - README: "Not these" Prism products surfaced after the summary; host rules and loopback wording updated; the server-docs citation is kept on vendor facts and the recorded-model claim is attributed to the tests. - Tests: new allowed and refused spellings; absence assertions that could not fail are removed or now point at reachable endpoints; the content scan reads only .py/.md/.txt outside __pycache__. approved: Nikhil 2026-10-03 blanket Refs: TH-8308
…; document --alias A live run against the real Bonsai server (PrismML llama.cpp fork, Bonsai-1.7B Q1_0, started with scripts/start_llama_server.sh) showed that llama-server returns the loaded GGUF file's absolute path as the response `model`. traceai-openai records that value in gen_ai.request.model, so a default setup exports a local path that can contain the user's name, and FI_HIDE_INPUTS/FI_HIDE_OUTPUTS do not mask it. Starting the server with `--alias bonsai` (passed through the demo script) makes the server, and therefore the span, report `bonsai`; verified live. Document both in "What you see", "Provider specifics" and "Privacy", and pin the advice in the README contract test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ovider/model fix The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is always "openai" and streamed/failed spans carry no gen_ai.request.model. #249 labels known vendor hosts and records the requested model, so these pins would break whichever PR merged second. Accept both behaviours (and the related Responses/embedding output changes where this recipe checks them), and say in the README which version does what. Passes against published traceAI-openai 0.1.10 and against #249. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setup_tracing() passed project_name or "prismml-bonsai" to register(), so the default was always truthy and FI_PROJECT_NAME was never read: `python src/app.py` always exported to prismml-bonsai. The Anannas recipe reads FI_PROJECT_NAME with its name as the default; do the same here. An explicit setup_tracing() argument still wins, and an unset or empty FI_PROJECT_NAME keeps prismml-bonsai (never register()'s generic default). - README: FI_PROJECT_NAME in the Configure table and the Run section; the Code snippet reads it the same way as src/app.py. - Tests: setup_tracing() default / empty / env / explicit cases; the app.py subprocess runs with FI_PROJECT_NAME unset and set; the README contract checks the new row and snippet. Passes against repository source, published traceAI-openai 0.1.10 and #249. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
nik13
added a commit
to jjcav84/traceAI
that referenced
this pull request
Nov 2, 2026
…ses and embedding spans (TH-8402) Live E2E of the OpenAI-compatible provider recipes (future-agi#234-future-agi#243) showed: - streamed and failed Chat Completions spans had no gen_ai.request.model; - gen_ai.provider.name was always "openai" (the request-level label even overwrote the existing Azure/Google host detection); - non-streamed Responses output.value was Python str(dict), a string input was JSON-quoted, and the Observe preview showed `{}` as the answer because assistant text was only emitted as message.contents.N.message_content.text; - embedding spans put the full vector into output.value as str(dict), had no span model, ignored FI_HIDE_EMBEDDING_VECTORS for the raw embedding.embeddings JSON, and skipped items without `index` (Gemini). Fixes, test-first (python/frameworks/openai/tests, real openai SDK through httpx.MockTransport and a real OTel SDK exporter; 41 tests, openai 1.69.0 and 3.24.0): - record the requested model at span start; served model still wins on a successful non-streamed response; - map the client's base_url host to a provider via a small explicit table (_providers.py); unknown hosts, proxies and loopback stay "openai"; - Responses: JSON output.value + application/json mime, led by output_text; plain-string input recorded as text; text-only assistant messages emitted as a flat message.content; - embeddings: gen_ai.request.model = embedding model, compact JSON summary in output.value, hide-vectors strips the raw JSON too, index falls back to position; - mime types follow the recorded value; async request path also reads Responses `input`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current-head verification — 2026-10-28
Head:
8bf24ab0(dev merged normally).prism-b10826-c9c34bc, with officialTernary-Bonsai-2-2B-PQ2_0.ggufweights checked against published SHA256SUMS, on Linux CPU. The server used its documented launcher with--alias bonsai.43181000-2c12-496b-a098-36090fdcaa5a; streamef49eb0d-1ba9-4712-8a6f-0687eadb0e3f; hidden content1f613d98-8953-4b60-8c91-6dc429424a95; error5b7afaeb-0538-43b8-9014-b86e6b9c4c4c. The published instrumentor's missing model/usage on default streams matches the documented limitation; no token values were invented.Summary
This PR adds a PrismML Bonsai local-server recipe at
python/examples/prismml/(Linear TH-8308, parent TH-8103). PrismML's demo repo ships an OpenAI-compatible server that the customer runs on their own machine: llama.cpp athttp://localhost:8080/v1, or MLX on Apple Silicon athttp://localhost:8081/v1(server docs). Customers who call it with the officialopenaiSDK get traces through the existingtraceai-openaiinstrumentor. The PR adds no package; it includes the recipe, shared test harness and focused recipe CI.src/app.py: callsregister()and instruments withOpenAIInstrumentorbefore creating the client.PRISMML_API_KEYis optional and defaults to the placeholdernot-needed.PRISMML_MODELdefaults tobonsai.PRISMML_BASE_URLdefaults tohttp://localhost:8080/v1.check_base_url()validates before tracing and never rewrites:/v1or/v1/on any host; the root is the built-in web chat, and/v1/chat/completionsand other paths are refused;bonsai_server); only one trailing dot is accepted;localhostand loopback IP addresses, decided withipaddress). HTTP on any other host is allowed with one stderr warning that the server is unauthenticated. The README quotes the vendor's own warning about exposing it.requirements.txt: the tested pinsopenai==3.24.0,traceAI-openai==0.1.10andfi-instrumentation-otel==1.1.0.README.md:tests/: a loopback contract on the shared harnessReceiver(test(harness): shared OTLP harness for TH-8103 contract tests #203), a socket guard with a loopback control, and a fake OpenAI server.What the tests pin
Every test runs offline. In-process tests use
httpx.MockTransportat both documented URLs, with exact request URLs. Subprocess tests use a fake on 127.0.0.1 under the guard. No Bonsai server is started and no weights are downloaded.ChatCompletionLLM span per call;gen_ai.provider.nameisopenai.bonsaiand returns the synthetic loaded-model nameTernary-Bonsai-2-27B-Q4_K_M.gguf. The non-streamed span records the returned name, andgen_ai.request.parameterskeepsbonsai. Streamed and failed spans have nogen_ai.request.model. The README carries the cost-lookup caveat.extra_body={"thinking_budget_tokens": …}reaches the server but is absent fromgen_ai.request.parameters, because the instrumentor reads the request beforeextra_bodyis merged.include_usageand a final usage chunk.output.valueis the assistant text; a stream with no text records''.FI_HIDE_INPUTS,FI_HIDE_OUTPUTSandFI_HIDE_LLM_INVOCATION_PARAMETERSare tested against visible controls, for chat and stream.bonsai_serverand other loopback forms, are returned unchanged. The non-loopback HTTP warning is printed exactly once, and instrumentation happens before the client.Tests (exact head
8dade7bfccfeeb1300c3ad419a3faa506fae373b, clean tree, actual exit codes + JUnit)openaitraceAI-openai==0.1.10+fi-instrumentation-otel==1.1.0(imported from site-packages)No files outside
python/examples/prismml/changed, and the head did not move during the run. The credential scan found 0 hits. Earlier heads74eaeafandf5d2754had the same 6/6 matrix with 86 tests.f5d2754removed a README test that searched for internal project names; the generic ticket-id and key checks remain.Decisions (approved: Nikhil 2026-10-03 blanket)
traceai-openaiwith no new package; the provider label staysopenai.not-neededplaceholder. A non-loopback HTTP host is allowed with a warning, not refused, so a trusted LAN setup still works.bonsai, and the tests pin which name lands where.Not covered
Review status
Review r1 (pr-reviewer
t_b258b908, claude-opus-5-5, atf5d2754): CHANGES_REQUESTED.bonsai_serverexited 2. That contradicts D-8308a, which allows non-loopback hosts with a warning.Rick fixed it test-first in
8dade7b(before: 4 failed; after: 91 passed), together with these P3s:ipaddress;__pycache__;R6 ("Full suite" wording) is declined: it is the same convention across all Wave F recipes, and this PR body gives exact counts. R8 (public commit trailers) follows the repository's ticket convention.
Verification r1 (pr-verifier
t_2047dda0, claude-opus-5-5, at8dade7b): VERIFIED, with no blocking findings. R1–R5 and R7 are fixed; the verifier agrees with declining R6 and R8. Retained P3s (not blocking; no third round):bonsaiboth with and without streaming.Video demo
A narrated CLI walkthrough, 6:08, recorded at the verified head
8dade7b: 1080p H.264/AAC, burned captions and 10 embedded chapters. The video, captions, transcript, chapters, preview and both media-verification notes are private attachments on the Linear issue TH-8308 (Future AGI workspace access required). Every command chapter runs./run_demo.sh <chapter>live in a real terminal and ends at exit 0. The documentedlocalhost:8080andlocalhost:8081are shown on purpose; ephemeral fake ports appear as<port>. Static waits in refusals and tests are shortened and disclosed in the captions; no command output was edited.traceai-openaitraces it unchanged; the catches are which model name is recorded, no server authentication, host rules for loopback, LAN and Compose, and other products named Prism (card)8dade7b, the recipe files, the env variable names and the optional placeholder keybonsaiexports the returned.ggufmodel name ingen_ai.request.modeland keepsbonsaiin parameters; usage; what Future AGI and the local server each received; the placeholder key is not exported and no Future AGI key reaches the serverlocalhost:8080and MLXlocalhost:8081, each with and without the trailing slash, give the exact SDK URL/v1/chat/completions; 0 external DNS/v1, endpoint paths, web chat, query or fragment, control characters, non-ASCII or malformed hosts, edge underscores, double dots, bad ports), each exit 2 with 0 requests and 0 spans; allowed URLs returned unchanged;http://bonsai_server:8080/v1is allowed with the exact warning printed once, and loopback spellings get no warninggen_ai.request.model,bonsaiin parameters and no keyFI_HIDE_INPUTS/FI_HIDE_OUTPUTSversus a control; the local server still receives the promptVerification sweep (2026-10-12)
dev, and the spans were checked in ClickHouse and in the Observe UI (hierarchy, span types, input/output, model and tokens, sessions, error status, and a secret scan of everything stored).Sweep fix (2026-10-18)
setup_tracing()passedproject_name or "prismml-bonsai"toregister(). That default is always truthy, soFI_PROJECT_NAMEwas never read andpython src/app.pyalways exported toprismml-bonsai. The Anannas recipe (feat(examples): Anannas AI recipe on traceai-openai (TH-8306) #241) readsFI_PROJECT_NAMEwith its own name as the default.project_name or os.environ.get("FI_PROJECT_NAME") or "prismml-bonsai", exactly as in feat(examples): Anannas AI recipe on traceai-openai (TH-8306) #241. An explicit argument still wins; an unset or empty variable keepsprismml-bonsai.FI_PROJECT_NAMEadded to the Configure table and the Run section; the Code snippet reads it the same way.setup_tracing()default / empty / env / explicit cases, theapp.pysubprocess (chat and stream) withFI_PROJECT_NAMEunset and set, and the README contract for the new row and snippet. Before the fix, the 3 env-project cases failed; after it, the suite is 97 passed on each of: repository source, publishedtraceAI-openai==0.1.10+fi-instrumentation-otel==1.1.0, and fix(traceai-openai): request model on stream/error, provider from base_url host, readable Responses + embedding spans (TH-8402) #249 (Python 3.11,openai==3.24.0).python src/app.pyandpython src/app.py --stream, plus one wrong-key run, with therequirements.txtpins, against a local Future AGI stack,FI_PROJECT_NAMEset to a dedicated project, andPRISMML_BASE_URLpointed at a local OpenAI-compatible stand-in (no Bonsai server). All 3 LLM spans landed in that project: chat with model and tokens (7f8acdc4-8e1f-42df-bceb-84c3cfb58d03), stream (cfa03804-148a-44bf-bca1-e1b145cd60ff), and the stand-in's 401 as ERROR (66dd78b6-ed8f-44bd-a718-06b5dfe5b278; a real Bonsai server does not check the key). The defaultprismml-bonsaiproject received no new spans. Secret scan of stored spans and output: 0 hits.🤖 Generated with Claude Code