From 74eaeaf42b9a3834ea61e981b07f8d1966654535 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Mon, 5 Oct 2026 23:38:19 -0700 Subject: [PATCH 1/8] feat(examples): PrismML Bonsai local server recipe on traceai-openai Adds python/examples/prismml/: the official openai SDK pointed at the customer's own PrismML Bonsai OpenAI-compatible server (llama.cpp at http://localhost:8080/v1, MLX at http://localhost:8081/v1), traced by the existing traceai-openai instrumentor. No provider package. - The server does not check keys: PRISMML_API_KEY is optional and defaults to the placeholder not-needed; tests pin that it never reaches the Future AGI export and the Future AGI keys never reach the server. - PRISMML_BASE_URL is validated before tracing and never rewritten: exactly /v1 or /v1/ (the root is the built-in web chat), no query or fragment delimiter, no credentials, whitespace/control characters, non-ASCII or malformed hosts; exit 2. HTTP on loopback is expected; HTTP on another host is allowed with one stderr warning that the server is unauthenticated (quoting the vendor's warning). - Pins current traceai-openai behaviour: a successful non-streamed span records the model name the server reports, the requested name stays in gen_ai.request.parameters; streamed and failed spans omit the model; thinking_budget_tokens via extra_body reaches the server but is not in the parameters; tool-call ids, names and arguments are recorded. - Chat Completions output.value is the assistant text (empty for a textless stream); FI_HIDE_INPUTS/FI_HIDE_OUTPUTS with controls. - Loopback only: MockTransport at both documented URLs, a local OpenAI-shaped fake, the shared harness receiver and a socket guard with a loopback control. approved: Nikhil 2026-10-03 blanket Refs: TH-8308 --- python/examples/prismml/README.md | 179 +++++ python/examples/prismml/requirements.txt | 4 + python/examples/prismml/src/app.py | 126 ++++ python/examples/prismml/tests/_fake_openai.py | 105 +++ .../tests/loopback_guard/sitecustomize.py | 42 ++ .../prismml/tests/test_prismml_recipe.py | 714 ++++++++++++++++++ 6 files changed, 1170 insertions(+) create mode 100644 python/examples/prismml/README.md create mode 100644 python/examples/prismml/requirements.txt create mode 100644 python/examples/prismml/src/app.py create mode 100644 python/examples/prismml/tests/_fake_openai.py create mode 100644 python/examples/prismml/tests/loopback_guard/sitecustomize.py create mode 100644 python/examples/prismml/tests/test_prismml_recipe.py diff --git a/python/examples/prismml/README.md b/python/examples/prismml/README.md new file mode 100644 index 000000000..03ad2d3cc --- /dev/null +++ b/python/examples/prismml/README.md @@ -0,0 +1,179 @@ +# PrismML (OpenAI-compatible) with traceAI + +PrismML here means the Bonsai local server from prismml.com; this recipe traces Chat Completions through the `openai` SDK at `http://localhost:8080/v1` ([server docs](https://docs.prismml.com/run/server)). It uses `traceai-openai`, with no PrismML instrumentation package. + +## Install + +From this recipe directory, install the pinned dependencies: + +```bash +pip install -r requirements.txt +``` + +The plain installation form is: + +```bash +pip install traceAI-openai fi-instrumentation-otel openai +``` + +The test versions and verification scope are listed below. + +## Configure + +Start your server first, following the [PrismML server docs](https://docs.prismml.com/run/server): `./scripts/start_llama_server.sh` serves llama.cpp on port 8080; `./scripts/start_mlx_server.sh` serves MLX on Apple Silicon on port 8081. Both expose the same OpenAI-compatible API. + +| Variable | Value or default | Destination | +|---|---|---| +| `FI_API_KEY` | Your Future AGI API key | Tracer only; never the local server | +| `FI_SECRET_KEY` | Your Future AGI secret key | Tracer only; never the local server | +| `PRISMML_API_KEY` | Optional; `not-needed` | OpenAI client only; never the Future AGI export | +| `PRISMML_MODEL` | Optional; `bonsai` | Model name sent by the OpenAI client | +| `PRISMML_BASE_URL` | Optional; `http://localhost:8080/v1` | OpenAI client; use `http://localhost:8081/v1` for MLX | +| `FI_BASE_URL` | Optional; the tracer's Future AGI collector default | Tracer export origin | +| `FI_HIDE_INPUTS` | Optional; `true` masks input attributes | Tracer only | +| `FI_HIDE_OUTPUTS` | Optional; `true` masks output attributes | Tracer only | +| `FI_HIDE_LLM_INVOCATION_PARAMETERS` | Optional; `true` omits invocation parameters | Tracer only | + +The local key is a placeholder, not a Future AGI secret. The OpenAI SDK requires a non-empty key, but the server does not check it ([server docs](https://docs.prismml.com/run/server)). An unset or empty `PRISMML_API_KEY` uses `not-needed`; an unset or empty `PRISMML_MODEL` uses `bonsai`. + +The recipe validates `PRISMML_BASE_URL` before tracing. Invalid URLs exit 2 and name that variable. An explicitly empty `--model` exits 2 and names `PRISMML_MODEL`. The recipe does not validate the Future AGI keys. + +## Run + +From this recipe directory, with your server already running: + +```bash +export FI_API_KEY="placeholder-fi-api-key" +export FI_SECRET_KEY="placeholder-fi-secret-key" +export PRISMML_API_KEY="not-needed" +export PRISMML_MODEL="bonsai" +export PRISMML_BASE_URL="http://localhost:8080/v1" +python src/app.py +python src/app.py --stream +``` + +Replace the Future AGI placeholders with your keys. `--prompt` changes the question and `--model` overrides `PRISMML_MODEL`. The app prints answer text and flushes the tracer. + +## Code + +Instrument before creating the client. Future AGI keys are read by the tracer from its environment. + +```python +from fi_instrumentation import register +from fi_instrumentation.fi_types import ProjectType +from openai import OpenAI +from traceai_openai import OpenAIInstrumentor +provider = register( + project_name="prismml-bonsai", project_type=ProjectType.OBSERVE, + set_global_tracer_provider=False, verbose=False, +) +OpenAIInstrumentor().instrument(tracer_provider=provider) +with OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed") as client: + response = client.chat.completions.create( + model="bonsai", messages=[{"role": "user", "content": "Say hello."}], stream=False, + ) + print(response.choices[0].message.content or "") +provider.force_flush() +``` + +## What you see in Future AGI + +The fixtures produce one `ChatCompletion` span per call with `gen_ai.span.kind=LLM`. The provider field says `openai`: `gen_ai.provider.name` is the shared instrumentor's label. + +Current `traceai-openai` behavior: + +- For a successful non-streamed call, `gen_ai.request.model` is the model name the server returns. The fixture requests `bonsai` and returns the synthetic loaded-model name `Ternary-Bonsai-2-27B-Q4_K_M.gguf`; the latter is recorded. The requested `bonsai` remains in the `gen_ai.request.parameters` JSON. +- Streamed and failed calls omit `gen_ai.request.model`. The requested model remains in `gen_ai.request.parameters`, unless invocation parameters are hidden. +- Non-streamed token usage is recorded as `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, and `gen_ai.usage.total_tokens` when the response supplies it. Missing usage is omitted, never replaced with zero. +- A stream accumulates answer text in `output.value`. Default fixture streams omit usage. With `stream_options={"include_usage": True}` and a final usage chunk, the same three token attributes are recorded. This verifies the instrumentor, not the real server's streaming usage support. +- Non-streamed `output.value` contains assistant content. Tool-call responses contain a readable function call instead. A stream with no text records `output.value=''`; Chat Completions have no raw-response fallback here. +- SDK errors set span status to ERROR and record an exception event. The 401 fixture is synthetic; it does not imply that this unauthenticated server checks a key. + +A cost lookup keyed on the requested name may miss the server's returned name. Cost may be unknown when usage or a matching price entry is unavailable. This recipe adds no model pricing. + +## Provider specifics + +According to the [PrismML server docs](https://docs.prismml.com/run/server): + +| Server | API base URL | +|---|---| +| llama.cpp | `http://localhost:8080/v1` | +| MLX on Apple Silicon | `http://localhost:8081/v1` | + +The SDK appends a trailing slash to its stored base URL and calls `/v1/chat/completions`. The validator returns allowed URLs unchanged. It permits exactly `/v1` or `/v1/` on any host and refuses the root web chat, a full `/v1/chat/completions` URL, other paths, credentials, query or fragment delimiters, whitespace, control characters, non-ASCII hosts and malformed IDNA. + +Loopback HTTP is allowed for `localhost`, `127.0.0.1` and `::1`. HTTP outside loopback is allowed with one stderr warning about the unauthenticated server. Use it only on a trusted LAN. HTTPS URLs still need the same API path. + +The [server docs](https://docs.prismml.com/run/server) warn: + +> The scripts bind to `127.0.0.1`, so the server is reachable only from the same machine and there is no authentication. Setting `BONSAI_HOST` to any non-loopback address (for example `0.0.0.0`) exposes an unauthenticated server to your network. + +For llama-server, any model string works because it serves the loaded model; `GET /v1/models` returns the real name. The documented Python example requests `bonsai` with streaming. The recorded non-streamed model name is what the server reports ([server docs](https://docs.prismml.com/run/server)). The loaded-model name used in tests is synthetic, not a promised filename. + +Thinking is controlled per request with `thinking_budget_tokens`; zero disables it ([server docs](https://docs.prismml.com/run/server)). Send it through the SDK's `extra_body`: + +```python +response = client.chat.completions.create( + model="bonsai", messages=[{"role": "user", "content": "Say hello."}], + extra_body={"thinking_budget_tokens": 0}, +) +``` + +The tests pin that this value reaches the HTTP request but is absent from `gen_ai.request.parameters`. The SDK merges `extra_body` after the instrumentor reads the request parameters. + +The 27B model supports native OpenAI-style `tool_calls` ([server docs](https://docs.prismml.com/run/server)). The fixture uses a standard `tools` array. It records definitions in `gen_ai.tool.definitions` and `gen_ai.tool.definitions.0.tool.json_schema`, and records the returned tool ID, function name and arguments under `gen_ai.output.messages.0.message.tool_calls.0.tool_call.*`. It does not execute a tool. + +## Privacy + +Prompt and answer text normally leave the machine in the Future AGI trace export. Set `FI_HIDE_INPUTS=true` to mask input attributes and `FI_HIDE_OUTPUTS=true` to mask output attributes. The tests verify each flag against visible controls for both chat and streaming. + +`FI_HIDE_INPUTS` masks only inputs. If the server echoes the prompt in its answer, that text remains in output attributes unless outputs are also hidden. Non-streamed output is assistant content, and streamed output is accumulated text, including an empty string when no text arrives. This recipe does not export a raw Chat Completions response in `output.value`. + +The local server still receives the prompt. Future AGI masking does not change the server's memory or its own logging. Key separation tests verify that `not-needed` and the custom local placeholder never reach the Future AGI export, while Future AGI keys never reach the local request. + +## Limits / not covered + +- The built-in web chat, Open WebUI, MCP, Responses API, embeddings and other API surfaces. +- Starting or stopping the server, distributing weights, live GPU jobs, MLX server differences, benchmarks or model quality. +- Tool execution and streaming tool calls. +- Other products called Prism: the [ssimplifi Prism proxy guide](https://ssimplifi.com/guides/openai-compatible-api), [prism-proxy](https://git.sovereign-society.org/prism/prism-proxy), and [Prism PHP](https://mintlify.wiki/prism-php/prism/providers/overview). +- This recipe is not tested against the live provider: the tests use a local fake of the OpenAI API. They do not start a Bonsai server or download weights. + +## Tests + +Run these commands from the repository root. All provider responses are synthetic: in-process MockTransport at both documented URLs, plus a server and OTLP receiver bound to `127.0.0.1`. A subprocess socket guard refuses non-loopback connections before DNS and has a loopback positive control. + +Repository source: + +```bash +env -u PYTHONPATH PYTHONDONTWRITEBYTECODE=1 \ + PYTHONPATH="python/examples/prismml/src:python:python/frameworks/openai:python/tests" \ + uv run --no-project --python 3.11 \ + --with 'openai==3.24.0' --with httpx --with 'wrapt<2' \ + --with opentelemetry-api --with opentelemetry-sdk --with opentelemetry-instrumentation \ + --with opentelemetry-exporter-otlp-proto-http --with requests --with protobuf \ + --with opentelemetry-proto --with pydantic --with jsonschema --with pytest \ + pytest python/examples/prismml/tests -q -p no:cacheprovider --noconftest -o addopts= -rfEs +``` + +Published packages: + +```bash +env -u PYTHONPATH PYTHONDONTWRITEBYTECODE=1 \ + PYTHONPATH="python/examples/prismml/src:python/tests" \ + uv run --no-project --python 3.11 \ + --with 'openai==3.24.0' --with httpx --with 'wrapt<2' \ + --with 'traceAI-openai==0.1.10' --with 'fi-instrumentation-otel==1.1.0' \ + --with opentelemetry-api --with opentelemetry-sdk --with opentelemetry-instrumentation \ + --with opentelemetry-exporter-otlp-proto-http --with requests --with protobuf \ + --with opentelemetry-proto --with pydantic --with jsonschema --with pytest \ + pytest python/examples/prismml/tests -q -p no:cacheprovider --noconftest -o addopts= -rfEs +``` + +To check the `traceai-openai` SDK floor, change the source command's OpenAI pin to `openai==1.69.0`. + +| Python | `openai` | traceAI packages | Result | +|---|---|---|---| +| 3.10, 3.11, 3.12, 3.13 | 3.24.0 | `traceai-openai` and `fi_instrumentation` from this repository | Full suite | +| 3.11 | 1.69.0 (the `traceai-openai` floor) | from this repository | Full suite | +| 3.11 | 3.24.0 | published `traceAI-openai==0.1.10` and `fi-instrumentation-otel==1.1.0` from PyPI | Full suite | diff --git a/python/examples/prismml/requirements.txt b/python/examples/prismml/requirements.txt new file mode 100644 index 000000000..2e789e40c --- /dev/null +++ b/python/examples/prismml/requirements.txt @@ -0,0 +1,4 @@ +# Installation pins; see README for source verification and the SDK floor. +openai==3.24.0 +traceAI-openai==0.1.10 +fi-instrumentation-otel==1.1.0 diff --git a/python/examples/prismml/src/app.py b/python/examples/prismml/src/app.py new file mode 100644 index 000000000..f9bf98754 --- /dev/null +++ b/python/examples/prismml/src/app.py @@ -0,0 +1,126 @@ +"""Trace the customer's local PrismML Bonsai Chat Completions server.""" + +import argparse +import ipaddress +import os +import re +import sys +from urllib.parse import urlsplit + +from fi_instrumentation import register +from fi_instrumentation.fi_types import ProjectType +from openai import OpenAI +from traceai_openai import OpenAIInstrumentor + +DEFAULT_BASE_URL = "http://localhost:8080/v1" +API_KEY_ENV = "PRISMML_API_KEY" +BASE_URL_ENV = "PRISMML_BASE_URL" +MODEL_ENV = "PRISMML_MODEL" +HTTP_WARNING = ( + "WARNING: PRISMML_BASE_URL uses HTTP on a non-loopback host; the Bonsai " + "server is unauthenticated. Keep it on loopback or a trusted LAN." +) + + +def check_base_url(url: str) -> str: + """Validate without rewriting; warn once for HTTP outside loopback.""" + def refuse(reason): + raise ValueError(f"{BASE_URL_ENV}: {reason}") + + if any(char.isspace() or not char.isprintable() for char in url): + refuse("remove whitespace and control characters from the URL") + try: + parsed = urlsplit(url) + host = parsed.hostname + port = parsed.port + except ValueError: + refuse("use a valid URL with a valid host and port") + if parsed.username is not None or parsed.password is not None: + refuse("remove embedded credentials; use PRISMML_API_KEY instead") + if parsed.scheme not in ("http", "https"): + refuse("use an http or https URL") + if not host or not host.isascii(): + refuse("use a plain ASCII host") + host = host.lower().rstrip(".") + try: + if ":" in host: + ipaddress.IPv6Address(host) + if "%" in host: + raise ValueError + else: + if len(host) > 253 or not all( + re.fullmatch(r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?", label) + for label in host.split(".") + ): + raise ValueError + host.encode("ascii").decode("idna") + except (ValueError, UnicodeError): + refuse("use a valid ASCII host without malformed IDNA") + if port == 0 or (parsed.netloc.endswith(":") and port is None): + refuse("use a valid non-zero port") + if "?" in url or "#" in url: + refuse("remove the query or fragment, including an empty delimiter") + if parsed.path not in ("/v1", "/v1/"): + refuse(f"use the API path /v1 or /v1/, for example {DEFAULT_BASE_URL}") + if parsed.scheme == "http" and host not in ("localhost", "127.0.0.1", "::1"): + print(HTTP_WARNING, file=sys.stderr) + return url + + +def make_client(base_url: str | None = None, api_key: str | None = None, + http_client=None) -> OpenAI: + url = check_base_url(base_url or os.environ.get(BASE_URL_ENV) or DEFAULT_BASE_URL) + key = api_key if api_key is not None else os.environ.get(API_KEY_ENV) or "not-needed" + if not key: + raise ValueError(f"{API_KEY_ENV}: use a non-empty placeholder such as not-needed") + return OpenAI(base_url=url, api_key=key, http_client=http_client) + + +def setup_tracing(project_name: str | None = None): + provider = register( + project_name=project_name or "prismml-bonsai", + project_type=ProjectType.OBSERVE, + set_global_tracer_provider=False, + verbose=False, + ) + OpenAIInstrumentor().instrument(tracer_provider=provider) + return provider + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--stream", action="store_true") + parser.add_argument("--prompt", default="Say hello.") + parser.add_argument("--model", default=os.environ.get(MODEL_ENV) or "bonsai") + args = parser.parse_args(argv) + try: + url = check_base_url(os.environ.get(BASE_URL_ENV) or DEFAULT_BASE_URL) + if not args.model.strip(): + raise ValueError(f"{MODEL_ENV}: use a non-empty model name or bonsai") + except ValueError as error: + print(error, file=sys.stderr) + return 2 + key = os.environ.get(API_KEY_ENV) or "not-needed" + provider = setup_tracing() + try: + # Validation above runs before tracing and emits the LAN warning only once. + with OpenAI(base_url=url, api_key=key) as client: + response = client.chat.completions.create( + model=args.model, + messages=[{"role": "user", "content": args.prompt}], + stream=args.stream, + ) + if args.stream: + for chunk in response: + if chunk.choices: + print(chunk.choices[0].delta.content or "", end="", flush=True) + print() + else: + print(response.choices[0].message.content or "") + finally: + provider.force_flush() + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/python/examples/prismml/tests/_fake_openai.py b/python/examples/prismml/tests/_fake_openai.py new file mode 100644 index 000000000..1a1f7f523 --- /dev/null +++ b/python/examples/prismml/tests/_fake_openai.py @@ -0,0 +1,105 @@ +"""Synthetic OpenAI fixtures; never start Bonsai or download model weights.""" + +import copy +import json +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +REQUEST_MODEL = "bonsai" +RESPONSE_MODEL = "Ternary-Bonsai-2-27B-Q4_K_M.gguf" +ANSWER = "Hello from the Bonsai fixture." +USAGE = {"prompt_tokens": 13, "completion_tokens": 7, "total_tokens": 20} +ERROR = {"error": {"message": "Synthetic fixture rejection", "type": "authentication_error", + "param": None, "code": "fixture_rejected"}} +TOOLS = [{"type": "function", "function": { + "name": "get_weather", "description": "Get the weather for a city.", + "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, + "required": ["city"]}, +}}] +TOOL_CALL = {"id": "call-fixture-weather", "type": "function", "function": { + "name": "get_weather", "arguments": '{"city":"Paris"}', +}} + + +def chat_response(include_usage=True, tool_call=False): + message = {"role": "assistant", "content": None if tool_call else ANSWER} + if tool_call: + message["tool_calls"] = [copy.deepcopy(TOOL_CALL)] + result = { + "id": "chatcmpl-fixture", "object": "chat.completion", "created": 1, + "model": RESPONSE_MODEL, + "choices": [{"index": 0, "message": message, + "finish_reason": "tool_calls" if tool_call else "stop"}], + } + if include_usage: + result["usage"] = USAGE.copy() + return result + + +def stream_body(include_usage=False, text=ANSWER): + common = {"id": "chatcmpl-stream-fixture", "object": "chat.completion.chunk", + "created": 1, "model": RESPONSE_MODEL} + chunks = [] + for part in (text[:len(text) // 2], text[len(text) // 2:]): + chunks.append({**common, "choices": [{"index": 0, + "delta": {"role": "assistant", "content": part}, + "finish_reason": None}]}) + chunks.append({**common, "choices": [{"index": 0, "delta": {}, + "finish_reason": "stop"}]}) + if include_usage: + chunks.append({**common, "choices": [], "usage": USAGE.copy()}) + return ("".join(f"data: {json.dumps(chunk)}\n\n" for chunk in chunks) + + "data: [DONE]\n\n").encode() + + +class FakeOpenAI: + """Threaded loopback server with chat, SSE and a synthetic error response.""" + + def __init__(self): + self._requests = [] + self._lock = threading.Lock() + owner = self + + class Handler(BaseHTTPRequestHandler): + def do_POST(self): + body = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) + with owner._lock: + owner._requests.append({"path": self.path, "body": body, + "headers": {key.lower(): value for key, value in self.headers.items()}}) + status, content_type = 200, "application/json" + if self.path != "/v1/chat/completions": + status, data = 404, json.dumps(ERROR).encode() + elif body["messages"][0]["content"] == "fixture-error": + status, data = 401, json.dumps(ERROR).encode() + elif body.get("stream"): + content_type = "text/event-stream" + data = stream_body(body.get("stream_options", {}).get("include_usage", False)) + else: + data = json.dumps(chat_response()).encode() + self.send_response(status) + self.send_header("Content-Type", content_type) + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + def log_message(self, _format, *args): + pass + + self._server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + self.origin = f"http://127.0.0.1:{self._server.server_port}" + self.base_url = self.origin + "/v1" + self._thread = threading.Thread( + target=lambda: self._server.serve_forever(poll_interval=0.01), daemon=True) + self._thread.start() + + def requests(self): + with self._lock: + return copy.deepcopy(self._requests) + + def __enter__(self): + return self + + def __exit__(self, *args): + self._server.shutdown() + self._server.server_close() + self._thread.join() diff --git a/python/examples/prismml/tests/loopback_guard/sitecustomize.py b/python/examples/prismml/tests/loopback_guard/sitecustomize.py new file mode 100644 index 000000000..a920eb502 --- /dev/null +++ b/python/examples/prismml/tests/loopback_guard/sitecustomize.py @@ -0,0 +1,42 @@ +"""Refuse non-loopback subprocess networking before DNS or socket connection.""" + +import os +from pathlib import Path +import socket + + +def _check(host): + if isinstance(host, bytes): + host = host.decode("ascii", errors="replace") + if host not in ("127.0.0.1", "localhost"): + with open(os.environ["LOOPBACK_GUARD_LOG"], "a", encoding="utf-8") as log: + log.write("BLOCKED non-loopback socket operation\n") + raise RuntimeError("loopback guard refused a non-loopback host before DNS/connect") + + +_getaddrinfo = socket.getaddrinfo +_connect = socket.socket.connect +_connect_ex = socket.socket.connect_ex + + +def _guarded_getaddrinfo(host, *args, **kwargs): + _check(host) + return _getaddrinfo(host, *args, **kwargs) + + +def _guarded_connect(sock, address): + if sock.family in (socket.AF_INET, socket.AF_INET6): + _check(address[0]) + return _connect(sock, address) + + +def _guarded_connect_ex(sock, address): + if sock.family in (socket.AF_INET, socket.AF_INET6): + _check(address[0]) + return _connect_ex(sock, address) + + +socket.getaddrinfo = _guarded_getaddrinfo +socket.socket.connect = _guarded_connect +socket.socket.connect_ex = _guarded_connect_ex +Path(os.environ["LOOPBACK_GUARD_READY"]).write_text("installed\n", encoding="utf-8") diff --git a/python/examples/prismml/tests/test_prismml_recipe.py b/python/examples/prismml/tests/test_prismml_recipe.py new file mode 100644 index 000000000..b8bf4a30b --- /dev/null +++ b/python/examples/prismml/tests/test_prismml_recipe.py @@ -0,0 +1,714 @@ +"""Contract tests against synthetic responses and loopback-only receivers.""" + +from contextlib import contextmanager +import ast +import json +import os +from pathlib import Path +import re +import socket +import subprocess +import sys +import uuid + +import httpx +import openai +import pytest +import fi_instrumentation +import traceai_openai +from harness import Receiver +from traceai_openai import OpenAIInstrumentor + +import app +from _fake_openai import ( + ANSWER, ERROR, REQUEST_MODEL, RESPONSE_MODEL, TOOL_CALL, TOOLS, USAGE, + FakeOpenAI, chat_response, stream_body, +) + +ROOT = Path(__file__).resolve().parents[1] +FI_API_KEY = "placeholder-fi-api-key" +FI_SECRET_KEY = "placeholder-fi-secret-key" +LOCAL_KEY = "not-needed" +EXPECTED_HTTP_WARNING = ( + "WARNING: PRISMML_BASE_URL uses HTTP on a non-loopback host; the Bonsai " + "server is unauthenticated. Keep it on loopback or a trusted LAN." +) +USAGE_ATTRIBUTES = { + "gen_ai.usage.input_tokens": 13, + "gen_ai.usage.output_tokens": 7, + "gen_ai.usage.total_tokens": 20, +} + + +@pytest.fixture(autouse=True) +def isolated_environment(monkeypatch): + # Do not inherit real credentials, tracing settings or an external proxy. + for name in list(os.environ): + if name.startswith(("FI_", "OTEL_", "OPENAI_", "PRISMML_")) or name.lower() in ( + "http_proxy", "https_proxy", "all_proxy", "no_proxy", + ): + monkeypatch.delenv(name, raising=False) + monkeypatch.setenv("FI_API_KEY", FI_API_KEY) + monkeypatch.setenv("FI_SECRET_KEY", FI_SECRET_KEY) + monkeypatch.setenv("FI_BASE_URL", "http://127.0.0.1:1") + monkeypatch.setenv("NO_PROXY", "*") + original_dns = socket.getaddrinfo + original_connect = socket.socket.connect + original_connect_ex = socket.socket.connect_ex + + def check(host): + assert host in ("127.0.0.1", "localhost"), "tests refuse non-loopback networking" + + def guarded_dns(host, *args, **kwargs): + check(host) + return original_dns(host, *args, **kwargs) + + def guarded_connect(sock, address): + if sock.family in (socket.AF_INET, socket.AF_INET6): + check(address[0]) + return original_connect(sock, address) + + def guarded_connect_ex(sock, address): + if sock.family in (socket.AF_INET, socket.AF_INET6): + check(address[0]) + return original_connect_ex(sock, address) + + monkeypatch.setattr(socket, "getaddrinfo", guarded_dns) + monkeypatch.setattr(socket.socket, "connect", guarded_connect) + monkeypatch.setattr(socket.socket, "connect_ex", guarded_connect_ex) + yield + instrumentor = OpenAIInstrumentor() + if instrumentor.is_instrumented_by_opentelemetry: + instrumentor.uninstrument() + + +@contextmanager +def tracing(monkeypatch): + with Receiver() as receiver: + monkeypatch.setenv("FI_BASE_URL", receiver.origin) + project = "prismml-fixture-" + uuid.uuid4().hex + provider = app.setup_tracing(project) + try: + yield receiver, provider, project + finally: + OpenAIInstrumentor().uninstrument() + provider.force_flush() + provider.shutdown() + + +@pytest.fixture +def traced(monkeypatch): + with tracing(monkeypatch) as context: + yield context + + +def attributes(span): + return {item["key"]: next(iter(item["value"].values())) + for item in span.get("attributes", [])} + + +def one_span(context): + receiver, provider, _ = context + assert provider.force_flush() + spans = receiver.spans() + assert len(spans) == 1 + span = spans[0] + assert span["name"] == "ChatCompletion" + attrs = attributes(span) + assert attrs["gen_ai.span.kind"] == "LLM" + assert attrs["gen_ai.provider.name"] == "openai" + return span, attrs + + +def assert_usage(attrs, expected): + actual = {key: int(value) for key, value in attrs.items() if key.startswith("gen_ai.usage.")} + assert actual == expected + + +def assert_export_separation(receiver, project, key=LOCAL_KEY): + requests = receiver.requests() + assert len(requests) == 1 + export = requests[0] + assert export["path"] == "/tracer/v1/traces" + assert export["headers"]["x-api-key"] == FI_API_KEY + assert export["headers"]["x-secret-key"] == FI_SECRET_KEY + assert export["resource_attributes"][0]["project_name"] == project + assert export["resource_attributes"][0]["project_type"] == "observe" + # The positive controls above and the bearer assertion at the server make + # these key-absence assertions meaningful. + assert key not in json.dumps(receiver.spans()) + assert key not in json.dumps(requests) + + +def mock_client(body=None, *, base_url=None, stream=False, status=200): + seen = [] + + def handler(request): + seen.append(request) + if stream: + return httpx.Response(status, headers={"Content-Type": "text/event-stream"}, content=body) + return httpx.Response(status, json=chat_response() if body is None else body) + + client = app.make_client(base_url=base_url, + http_client=httpx.Client(transport=httpx.MockTransport(handler))) + return client, seen + + +def assert_local_request(request, url, key=LOCAL_KEY): + assert str(request.url) == url + assert request.headers["authorization"] == f"Bearer {key}" + assert "x-api-key" not in request.headers + assert "x-secret-key" not in request.headers + assert FI_API_KEY not in json.dumps(dict(request.headers)) + assert FI_SECRET_KEY not in json.dumps(dict(request.headers)) + assert FI_API_KEY not in request.content.decode() + assert FI_SECRET_KEY not in request.content.decode() + assert json.loads(request.content)["model"] == REQUEST_MODEL + + +def test_documented_default_and_optional_key(monkeypatch): + assert app.DEFAULT_BASE_URL == "http://localhost:8080/v1" + assert app.API_KEY_ENV == "PRISMML_API_KEY" + assert app.BASE_URL_ENV == "PRISMML_BASE_URL" + assert app.MODEL_ENV == "PRISMML_MODEL" + assert app.DEFAULT_BASE_URL in (ROOT / "README.md").read_text() + with app.make_client() as client: + assert str(client.base_url) == "http://localhost:8080/v1/" + assert client.api_key == LOCAL_KEY + monkeypatch.setenv(app.BASE_URL_ENV, "http://localhost:8081/v1") + monkeypatch.setenv(app.API_KEY_ENV, "placeholder-prismml-key") + with app.make_client() as client: + assert str(client.base_url) == "http://localhost:8081/v1/" + assert client.api_key == "placeholder-prismml-key" + with app.make_client(base_url="http://127.0.0.1:8080/v1", api_key="explicit-placeholder") as client: + assert str(client.base_url) == "http://127.0.0.1:8080/v1/" + assert client.api_key == "explicit-placeholder" + + +@pytest.mark.parametrize("base_url", ["http://localhost:8080/v1", "http://localhost:8081/v1"]) +@pytest.mark.parametrize("key", [LOCAL_KEY, "placeholder-prismml-key"]) +def test_chat_model_usage_and_key_separation(traced, monkeypatch, base_url, key): + monkeypatch.setenv(app.API_KEY_ENV, key) + response = chat_response() + response["fixture_extra"] = "raw-response-marker" + client, seen = mock_client(response, base_url=base_url) + with client: + result = client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": "harmless fixture prompt"}]) + assert result.choices[0].message.content == ANSWER + assert len(seen) == 1 + assert_local_request(seen[0], base_url + "/chat/completions", key) + span, attrs = one_span(traced) + assert attrs["gen_ai.request.model"] == RESPONSE_MODEL + assert RESPONSE_MODEL != REQUEST_MODEL + assert json.loads(attrs["gen_ai.request.parameters"])["model"] == REQUEST_MODEL + assert attrs["output.value"] == ANSWER + assert "raw-response-marker" not in json.dumps(span) + assert_usage(attrs, USAGE_ATTRIBUTES) + assert_export_separation(traced[0], traced[2], key) + + +def test_usage_absent_is_omitted(traced): + client, seen = mock_client(chat_response(include_usage=False)) + with client: + client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": "usage fixture"}]) + assert len(seen) == 1 + _, attrs = one_span(traced) + assert attrs["gen_ai.request.model"] == RESPONSE_MODEL + assert_usage(attrs, {}) + + +@pytest.mark.parametrize("include_usage", [False, True]) +def test_stream_accumulation_and_usage(traced, include_usage): + client, seen = mock_client(stream_body(include_usage), stream=True) + options = {"stream_options": {"include_usage": True}} if include_usage else {} + with client: + response = client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": "stream fixture"}], stream=True, **options) + text = "".join(chunk.choices[0].delta.content or "" for chunk in response if chunk.choices) + assert text == ANSWER + assert len(seen) == 1 + assert_local_request(seen[0], "http://localhost:8080/v1/chat/completions") + body = json.loads(seen[0].content) + assert body["stream"] is True + assert body.get("stream_options") == ({"include_usage": True} if include_usage else None) + span, attrs = one_span(traced) + assert attrs["output.value"] == ANSWER + # traceai-openai records the model only from a non-streamed response; flip + # this when the instrumentor records the request model. + assert "gen_ai.request.model" not in attrs + assert json.loads(attrs["gen_ai.request.parameters"])["model"] == REQUEST_MODEL + assert_usage(attrs, USAGE_ATTRIBUTES if include_usage else {}) + assert any(event["name"] == "First Token Stream Event" for event in span["events"]) + assert_export_separation(traced[0], traced[2]) + + +def test_textless_stream_has_no_raw_response_fallback(traced): + client, seen = mock_client(stream_body(text=""), stream=True) + with client: + chunks = list(client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": "textless fixture"}], stream=True)) + assert chunks + assert len(seen) == 1 + _, attrs = one_span(traced) + assert attrs["output.value"] == "" + assert "gen_ai.request.model" not in attrs + assert json.loads(attrs["gen_ai.request.parameters"])["model"] == REQUEST_MODEL + assert_usage(attrs, {}) + + +def test_synthetic_401_records_error_without_key(traced): + # The real server is unauthenticated; this is an SDK error-shape fixture. + client, seen = mock_client(ERROR, status=401) + with client, pytest.raises(openai.AuthenticationError): + client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": "error fixture"}]) + assert len(seen) == 1 + assert_local_request(seen[0], "http://localhost:8080/v1/chat/completions") + span, attrs = one_span(traced) + assert span["status"]["code"] == "STATUS_CODE_ERROR" + exceptions = [event for event in span["events"] if event["name"] == "exception"] + assert len(exceptions) == 1 + assert "Synthetic fixture rejection" in json.dumps(exceptions) + assert LOCAL_KEY not in json.dumps(span) + # traceai-openai records the model only from a non-streamed response; flip + # this when the instrumentor records the request model. + assert "gen_ai.request.model" not in attrs + assert json.loads(attrs["gen_ai.request.parameters"])["model"] == REQUEST_MODEL + assert_usage(attrs, {}) + assert_export_separation(traced[0], traced[2]) + + +@pytest.mark.parametrize("stream", [False, True]) +def test_hide_inputs_with_control(monkeypatch, stream): + marker = "private-prompt-marker-" + uuid.uuid4().hex + for hidden in (False, True): + monkeypatch.setenv("FI_HIDE_INPUTS", str(hidden).lower()) + with tracing(monkeypatch) as context: + client, seen = mock_client(stream_body() if stream else None, stream=stream) + with client: + result = client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": marker}], stream=stream) + if stream: + list(result) + span, attrs = one_span(context) + assert json.loads(seen[0].content)["messages"][0]["content"] == marker + assert attrs["output.value"] == ANSWER + if hidden: + assert marker not in json.dumps(span) + assert attrs["input.value"] == "__REDACTED__" + else: + assert marker in json.dumps(span) + assert attrs["input.value"] == marker + + +@pytest.mark.parametrize("stream", [False, True]) +def test_hide_outputs_with_control(monkeypatch, stream): + marker = "private-output-marker-" + uuid.uuid4().hex + prompt = "visible-prompt-marker" + for hidden in (False, True): + monkeypatch.setenv("FI_HIDE_OUTPUTS", str(hidden).lower()) + response = chat_response() + response["choices"][0]["message"]["content"] = marker + with tracing(monkeypatch) as context: + client, seen = mock_client(stream_body(text=marker) if stream else response, stream=stream) + with client: + result = client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": prompt}], stream=stream) + received = ("".join(chunk.choices[0].delta.content or "" for chunk in result) + if stream else result.choices[0].message.content) + assert received == marker + assert len(seen) == 1 + span, attrs = one_span(context) + assert attrs["input.value"] == prompt + if hidden: + assert marker not in json.dumps(span) + assert attrs["output.value"] == "__REDACTED__" + else: + assert marker in json.dumps(span) + assert attrs["output.value"] == marker + + +def test_input_masking_does_not_mask_echoed_output(monkeypatch): + marker = "echoed-prompt-marker" + monkeypatch.setenv("FI_HIDE_INPUTS", "true") + response = chat_response() + response["choices"][0]["message"]["content"] = marker + with tracing(monkeypatch) as context: + client, seen = mock_client(response) + with client: + client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": marker}]) + _, attrs = one_span(context) + assert json.loads(seen[0].content)["messages"][0]["content"] == marker + assert attrs["input.value"] == "__REDACTED__" + assert attrs["output.value"] == marker + assert not any(key.startswith("gen_ai.input.messages") for key in attrs) + + +def test_invocation_parameters_can_be_hidden_with_control(monkeypatch): + for hidden in (False, True): + monkeypatch.setenv("FI_HIDE_LLM_INVOCATION_PARAMETERS", str(hidden).lower()) + with tracing(monkeypatch) as context: + client, seen = mock_client() + with client: + client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": "parameter fixture"}]) + assert json.loads(seen[0].content)["model"] == REQUEST_MODEL + _, attrs = one_span(context) + assert attrs["output.value"] == ANSWER + if hidden: + assert "gen_ai.request.parameters" not in attrs + else: + assert json.loads(attrs["gen_ai.request.parameters"])["model"] == REQUEST_MODEL + + +@pytest.mark.parametrize("budget", [0, 12]) +def test_thinking_extra_body_on_wire_but_absent_from_parameters(traced, budget): + client, seen = mock_client() + with client: + client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": "thinking fixture"}], + temperature=0.25, extra_body={"thinking_budget_tokens": budget}) + assert len(seen) == 1 + assert json.loads(seen[0].content)["thinking_budget_tokens"] == budget + _, attrs = one_span(traced) + params = json.loads(attrs["gen_ai.request.parameters"]) + assert params["model"] == REQUEST_MODEL + assert params["temperature"] == 0.25 + # The SDK merges extra_body after the instrumentor reads json_data. + assert "thinking_budget_tokens" not in params + + +def test_tool_call_records_definition_id_name_arguments_and_output(traced): + client, seen = mock_client(chat_response(tool_call=True)) + with client: + result = client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": "weather fixture"}], tools=TOOLS) + assert len(seen) == 1 + assert json.loads(seen[0].content)["tools"] == TOOLS + assert result.choices[0].message.tool_calls[0].id == TOOL_CALL["id"] + _, attrs = one_span(traced) + assert json.loads(attrs["gen_ai.tool.definitions"]) == TOOLS + assert json.loads(attrs["gen_ai.tool.definitions.0.tool.json_schema"]) == TOOLS[0] + prefix = "gen_ai.output.messages.0.message.tool_calls.0.tool_call." + assert attrs[prefix + "id"] == TOOL_CALL["id"] + assert attrs[prefix + "function.name"] == "get_weather" + assert attrs[prefix + "function.arguments"] == '{"city":"Paris"}' + assert attrs["output.value"] == 'Function: get_weather({"city":"Paris"})' + assert attrs["gen_ai.request.model"] == RESPONSE_MODEL + assert json.loads(attrs["gen_ai.request.parameters"])["model"] == REQUEST_MODEL + assert_usage(attrs, USAGE_ATTRIBUTES) + + +ALLOWED_URLS = [ + "http://localhost:8080/v1", "http://localhost:8081/v1/", + "http://LOCALHOST.:8080/v1", "HTTP://localhost:8080/v1/", + "http://127.0.0.1:8080/v1", "http://[::1]:8080/v1/", + "https://trusted-lan.example/v1", "http://trusted-lan.example:8080/v1", + "http://TRUSTED-LAN.EXAMPLE.:8080/v1/", +] + + +@pytest.mark.parametrize("url", ALLOWED_URLS) +def test_allowed_base_urls_are_unchanged(url, capsys): + result = app.check_base_url(url) + assert result is url + warning = "trusted-lan" in url.lower() and url.startswith("http:") + assert capsys.readouterr().err == (EXPECTED_HTTP_WARNING + "\n" if warning else "") + + +REFUSED_URLS = [ + ("http://localhost:8080", "API path"), + ("http://localhost:8080/", "API path"), + ("http://LOCALHOST.:8080/", "API path"), + ("http://localhost:8080/v1/chat/completions", "API path"), + ("http://trusted-lan.example/v1/chat/completions", "API path"), + ("https://trusted-lan.example/webchat", "API path"), + ("http://localhost:8080/V1", "API path"), + ("http://localhost:8080/v1//", "API path"), + ("http://localhost:8080/v1/../v1", "API path"), + ("http://localhost:8080/%76%31", "API path"), + ("http://localhost:8080/v1%2f", "API path"), + ("http://localhost:8080/v1?", "query or fragment"), + ("http://localhost:8080/v1#", "query or fragment"), + ("http://localhost:8080/v1?option=value", "query or fragment"), + ("http://localhost:8080/v1#chat", "query or fragment"), + ("https://trusted-lan.example/v1?", "query or fragment"), + ("http://trusted-lan.example/v1#", "query or fragment"), + ("http://credential-marker@localhost:8080/v1", "credentials"), + ("https://credential-marker:password-marker@trusted-lan.example/v1", "credentials"), + ("http://localhost:8080/\tv1", "whitespace"), + ("http://localhost:8080/v1\r", "whitespace"), + ("http://localhost:8080/v1\n", "whitespace"), + (" http://localhost:8080/v1", "whitespace"), + ("http://localhost:8080/v1 ", "whitespace"), + ("http://local\x00host:8080/v1", "control"), + ("http://localhost:8080/v1\x7f", "control"), + ("http://local\u00a0host:8080/v1", "whitespace"), + ("http://local\u200bhost:8080/v1", "control"), + ("http://local\u3002host:8080/v1", "ASCII host"), + ("http://local\uff0ehost:8080/v1", "ASCII host"), + ("http://local\uff61host:8080/v1", "ASCII host"), + ("http://caf\u00e9.example/v1", "ASCII host"), + ("http://xn--.example/v1", "IDNA"), + ("http://xn--a.example/v1", "IDNA"), + ("http://local%68ost:8080/v1", "ASCII host"), + ("http://local_host:8080/v1", "ASCII host"), + ("http://-localhost:8080/v1", "ASCII host"), + ("http://localhost..example/v1", "ASCII host"), + ("http://localhost:invalid/v1", "port"), + ("http://localhost:65536/v1", "port"), + ("http://localhost:0/v1", "port"), + ("http://localhost:/v1", "port"), + ("http://[::1/v1", "host"), + ("http://[::1%25zone]/v1", "ASCII host"), + ("ftp://localhost:8080/v1", "http or https"), + ("//localhost:8080/v1", "http or https"), + ("http:///v1", "ASCII host"), +] + + +@pytest.fixture(scope="module") +def refusal_endpoints(): + with Receiver() as receiver, FakeOpenAI() as fake: + yield receiver, fake + + +@pytest.mark.parametrize("url,reason", REFUSED_URLS) +def test_refused_base_urls_fail_before_tracing(url, reason, monkeypatch, capsys, refusal_endpoints): + receiver, fake = refusal_endpoints + monkeypatch.setenv("FI_BASE_URL", receiver.origin) + if "\x00" in url: + # OS environment variables cannot contain NUL; inject the mapping to + # exercise the same main() validation path without an OS-level failure. + monkeypatch.setattr(app.os, "environ", {app.BASE_URL_ENV: url}) + else: + monkeypatch.setenv(app.BASE_URL_ENV, url) + + def must_not_run(*args, **kwargs): + pytest.fail("invalid configuration reached tracing or client creation") + + monkeypatch.setattr(app, "setup_tracing", must_not_run) + monkeypatch.setattr(app, "OpenAI", must_not_run) + with pytest.raises(ValueError) as error: + app.check_base_url(url) + message = str(error.value) + assert app.BASE_URL_ENV in message + assert reason in message + assert "\n" not in message and "\r" not in message + assert "credential-marker" not in message and "password-marker" not in message + assert app.main([]) == 2 + output = capsys.readouterr() + assert output.out == "" + assert output.err == message + "\n" + assert fake.requests() == [] + assert receiver.spans() == [] + assert receiver.requests() == [] + + +def test_empty_explicit_key_is_actionable(): + with pytest.raises(ValueError, match="PRISMML_API_KEY"): + app.make_client(api_key="") + + +def test_empty_cli_model_exits_before_tracing(monkeypatch, capsys, refusal_endpoints): + receiver, fake = refusal_endpoints + + def must_not_run(*args, **kwargs): + pytest.fail("empty model reached tracing") + + monkeypatch.setattr(app, "setup_tracing", must_not_run) + assert app.main(["--model", ""]) == 2 + assert app.MODEL_ENV in capsys.readouterr().err + assert fake.requests() == [] + assert receiver.spans() == [] + + +def test_main_non_loopback_http_warns_once_and_instruments_before_client(traced, monkeypatch, capsys): + receiver, provider, project = traced + order, seen = [], [] + original_openai = app.OpenAI + original_flush = provider.force_flush + monkeypatch.setenv(app.BASE_URL_ENV, "http://trusted-lan.example:8080/v1") + + def setup(): + order.append("instrument") + return provider + + def flush(*args, **kwargs): + order.append("flush") + return original_flush(*args, **kwargs) + + def client(**kwargs): + order.append("client") + assert order == ["instrument", "client"] + + def handler(request): + seen.append(request) + return httpx.Response(200, json=chat_response()) + + return original_openai(**kwargs, http_client=httpx.Client(transport=httpx.MockTransport(handler))) + + monkeypatch.setattr(app, "setup_tracing", setup) + monkeypatch.setattr(app, "OpenAI", client) + monkeypatch.setattr(provider, "force_flush", flush) + assert app.main([]) == 0 + assert order == ["instrument", "client", "flush"] + output = capsys.readouterr() + assert output.err == EXPECTED_HTTP_WARNING + "\n" + assert output.out == ANSWER + "\n" + assert len(seen) == 1 + assert_local_request(seen[0], "http://trusted-lan.example:8080/v1/chat/completions") + one_span(traced) + assert_export_separation(receiver, project) + + +@pytest.mark.parametrize("cli_model", [None, "cli-model-fixture"]) +def test_main_model_environment_and_cli_override(traced, monkeypatch, cli_model): + expected = cli_model or "environment-model-fixture" + monkeypatch.setenv(app.MODEL_ENV, "environment-model-fixture") + monkeypatch.setattr(app, "setup_tracing", lambda: traced[1]) + original_openai = app.OpenAI + seen = [] + + def handler(request): + seen.append(request) + return httpx.Response(200, json=chat_response()) + + def client(**kwargs): + return original_openai(**kwargs, http_client=httpx.Client(transport=httpx.MockTransport(handler))) + + monkeypatch.setattr(app, "OpenAI", client) + assert app.main([] if cli_model is None else ["--model", cli_model]) == 0 + assert len(seen) == 1 + assert json.loads(seen[0].content)["model"] == expected + _, attrs = one_span(traced) + assert attrs["gen_ai.request.model"] == RESPONSE_MODEL + assert json.loads(attrs["gen_ai.request.parameters"])["model"] == expected + + +def subprocess_environment(tmp_path): + # Resolve imported package roots so the same test works with published wheels. + paths = [ROOT / "tests" / "loopback_guard", ROOT / "src", + Path(fi_instrumentation.__file__).resolve().parent.parent, + Path(traceai_openai.__file__).resolve().parent.parent, + Path(sys.modules[Receiver.__module__].__file__).resolve().parent.parent] + env = os.environ.copy() + env.update(PYTHONPATH=os.pathsep.join(dict.fromkeys(map(str, paths))), + PYTHONDONTWRITEBYTECODE="1", LOOPBACK_GUARD_LOG=str(tmp_path / "guard.log"), + LOOPBACK_GUARD_READY=str(tmp_path / "guard.ready")) + Path(env["LOOPBACK_GUARD_LOG"]).write_text("") + return env + + +@pytest.mark.parametrize("stream", [False, True]) +def test_app_subprocess_chat_and_stream(tmp_path, stream): + with Receiver() as receiver, FakeOpenAI() as fake: + env = subprocess_environment(tmp_path) + env.update(FI_BASE_URL=receiver.origin, PRISMML_BASE_URL=fake.base_url, + PRISMML_MODEL=REQUEST_MODEL) + if stream: + env["PRISMML_API_KEY"] = "placeholder-prismml-key" + # The non-streamed child exercises the default not-needed placeholder. + argv = [sys.executable, str(ROOT / "src" / "app.py"), "--prompt", "subprocess fixture"] + if stream: + argv.append("--stream") + result = subprocess.run(argv, env=env, capture_output=True, text=True, timeout=120) + assert result.returncode == 0, result.stderr + assert ANSWER in result.stdout + assert FI_API_KEY not in result.stdout + result.stderr + assert FI_SECRET_KEY not in result.stdout + result.stderr + key = env.get("PRISMML_API_KEY", LOCAL_KEY) + assert key not in result.stdout + result.stderr + assert Path(env["LOOPBACK_GUARD_READY"]).read_text() == "installed\n" + assert Path(env["LOOPBACK_GUARD_LOG"]).read_text() == "" + requests = fake.requests() + assert len(requests) == 1 + request = requests[0] + assert request["path"] == "/v1/chat/completions" + assert request["headers"]["authorization"] == f"Bearer {key}" + assert "x-api-key" not in request["headers"] + assert "x-secret-key" not in request["headers"] + assert FI_API_KEY not in json.dumps(request) + assert FI_SECRET_KEY not in json.dumps(request) + assert request["body"]["model"] == REQUEST_MODEL + assert request["body"]["stream"] is stream + assert request["body"]["messages"][0]["content"] == "subprocess fixture" + spans = receiver.spans() + assert len(spans) == 1 + assert spans[0]["name"] == "ChatCompletion" + attrs = attributes(spans[0]) + assert attrs["gen_ai.span.kind"] == "LLM" + assert attrs["gen_ai.provider.name"] == "openai" + assert attrs["output.value"] == ANSWER + assert json.loads(attrs["gen_ai.request.parameters"])["model"] == REQUEST_MODEL + if stream: + assert "gen_ai.request.model" not in attrs + else: + assert attrs["gen_ai.request.model"] == RESPONSE_MODEL + assert_usage(attrs, {} if stream else USAGE_ATTRIBUTES) + assert_export_separation(receiver, "prismml-bonsai", key) + + +@pytest.mark.parametrize("operation", ["getaddrinfo", "connect", "connect_ex"]) +def test_guard_refuses_before_dns_or_connect_with_loopback_control(tmp_path, operation): + env = subprocess_environment(tmp_path) + with FakeOpenAI() as fake: + port = int(fake.origin.rsplit(":", 1)[1]) + positive = subprocess.run([sys.executable, "-c", + f"import socket; s=socket.create_connection(('127.0.0.1', {port})); s.close()"], + env=env, capture_output=True, text=True, timeout=120) + assert positive.returncode == 0, positive.stderr + assert Path(env["LOOPBACK_GUARD_READY"]).read_text() == "installed\n" + assert Path(env["LOOPBACK_GUARD_LOG"]).read_text() == "" + # This local-server recipe has no remote provider API host. Use the public + # documentation host only as a guard target; it must never reach DNS. + expression = ("socket.getaddrinfo('docs.prismml.com', 443)" if operation == "getaddrinfo" + else f"socket.socket().{operation}(('docs.prismml.com', 443))") + negative = subprocess.run([sys.executable, "-c", "import socket; " + expression], + env=env, capture_output=True, text=True, timeout=120) + assert negative.returncode != 0 + assert "loopback guard refused" in negative.stderr + assert Path(env["LOOPBACK_GUARD_LOG"]).read_text() == "BLOCKED non-loopback socket operation\n" + + +def test_readme_and_requirements_contract(): + readme = (ROOT / "README.md").read_text() + assert readme.startswith("# PrismML (OpenAI-compatible) with traceAI\n") + for phrase in ("http://localhost:8080/v1", "http://localhost:8081/v1", + "provider field says `openai`", "gen_ai.request.model", "verbose=False", + "FI_API_KEY", "FI_SECRET_KEY", "FI_BASE_URL", "FI_HIDE_INPUTS", "FI_HIDE_OUTPUTS", + "thinking_budget_tokens", "tool_calls", "not-needed", "cost"): + assert phrase in readme + source = (ROOT / "src" / "app.py").read_text() + for name in (app.API_KEY_ENV, app.BASE_URL_ENV, app.MODEL_ENV): + assert name in readme + tree = ast.parse(source) + for node in ast.walk(tree): + if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute): + if ast.unparse(node.func).startswith("os.environ.") and node.args: + if isinstance(node.args[0], ast.Constant): + assert node.args[0].value in readme + headings = ["## Install", "## Configure", "## Run", "## Code", + "## What you see in Future AGI", "## Provider specifics", "## Privacy", + "## Limits / not covered", "## Tests"] + assert [readme.index(heading) for heading in headings] == sorted(readme.index(heading) for heading in headings) + for paths in ("python/examples/prismml/src:python:python/frameworks/openai:python/tests", + "python/examples/prismml/src:python/tests"): + assert f'PYTHONPATH="{paths}"' in readme + assert readme.count("pytest python/examples/prismml/tests -q -p no:cacheprovider --noconftest -o addopts= -rfEs") == 2 + assert "--with 'traceAI-openai==0.1.10'" in readme + assert "--with 'fi-instrumentation-otel==1.1.0'" in readme + assert "TBD" not in readme + assert "not tested against the live provider" in readme + requirements = (ROOT / "requirements.txt").read_text().splitlines() + assert [line for line in requirements if not line.startswith("#")] == [ + "openai==3.24.0", "traceAI-openai==0.1.10", "fi-instrumentation-otel==1.1.0"] + for path in ROOT.rglob("*"): + if path.is_file(): + text = path.read_text() + assert not re.search(r"\b[A-Z]{2,5}-\d{3,}\b", text), str(path) + assert not re.search(r"\bsk-[A-Za-z0-9]{16,}\b", text), str(path) + assert not re.search(r"\b(?:Rick|Linear|brief)\b|company-brain", readme) From f5d27542dab1dccbde74d4aecf7186bc46d0d76a Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Mon, 5 Oct 2026 23:45:15 -0700 Subject: [PATCH 2/8] test(examples/prismml): drop internal-name README check The README contract test searched for internal project names, which put those names in the repository. The generic ticket-id and key checks on every recipe file stay. approved: Nikhil 2026-10-03 blanket Refs: TH-8308 --- python/examples/prismml/tests/test_prismml_recipe.py | 1 - 1 file changed, 1 deletion(-) diff --git a/python/examples/prismml/tests/test_prismml_recipe.py b/python/examples/prismml/tests/test_prismml_recipe.py index b8bf4a30b..e7983528f 100644 --- a/python/examples/prismml/tests/test_prismml_recipe.py +++ b/python/examples/prismml/tests/test_prismml_recipe.py @@ -711,4 +711,3 @@ def test_readme_and_requirements_contract(): text = path.read_text() assert not re.search(r"\b[A-Z]{2,5}-\d{3,}\b", text), str(path) assert not re.search(r"\bsk-[A-Za-z0-9]{16,}\b", text), str(path) - assert not re.search(r"\b(?:Rick|Linear|brief)\b|company-brain", readme) From 8dade7bfccfeeb1300c3ad419a3faa506fae373b Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Tue, 6 Oct 2026 00:00:35 -0700 Subject: [PATCH 3/8] =?UTF-8?q?fix(examples/prismml):=20review=20r1=20?= =?UTF-8?q?=E2=80=94=20Compose=20host=20names,=20loopback=20detection?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - check_base_url accepts underscores inside host labels, so container and Compose service names such as bonsai_server work (HTTP on such a host still prints the one-line unauthenticated-server warning); the refusal message names the allowed characters. - Only one trailing DNS dot is stripped (localhost.. is refused before tracing), and loopback is decided with ipaddress (127.0.0.0/8 and any ::1 spelling are not warned). - README: "Not these" Prism products surfaced after the summary; host rules and loopback wording updated; the server-docs citation is kept on vendor facts and the recorded-model claim is attributed to the tests. - Tests: new allowed and refused spellings; absence assertions that could not fail are removed or now point at reachable endpoints; the content scan reads only .py/.md/.txt outside __pycache__. approved: Nikhil 2026-10-03 blanket Refs: TH-8308 --- python/examples/prismml/README.md | 8 ++++--- python/examples/prismml/src/app.py | 21 +++++++++++++++---- .../prismml/tests/test_prismml_recipe.py | 16 ++++++++++---- 3 files changed, 34 insertions(+), 11 deletions(-) diff --git a/python/examples/prismml/README.md b/python/examples/prismml/README.md index 03ad2d3cc..435c107ee 100644 --- a/python/examples/prismml/README.md +++ b/python/examples/prismml/README.md @@ -2,6 +2,8 @@ PrismML here means the Bonsai local server from prismml.com; this recipe traces Chat Completions through the `openai` SDK at `http://localhost:8080/v1` ([server docs](https://docs.prismml.com/run/server)). It uses `traceai-openai`, with no PrismML instrumentation package. +Not these: other products named Prism are not covered here, for example the [ssimplifi Prism proxy guide](https://ssimplifi.com/guides/openai-compatible-api), [prism-proxy](https://git.sovereign-society.org/prism/prism-proxy) and [Prism PHP](https://mintlify.wiki/prism-php/prism/providers/overview). + ## Install From this recipe directory, install the pinned dependencies: @@ -100,15 +102,15 @@ According to the [PrismML server docs](https://docs.prismml.com/run/server): | llama.cpp | `http://localhost:8080/v1` | | MLX on Apple Silicon | `http://localhost:8081/v1` | -The SDK appends a trailing slash to its stored base URL and calls `/v1/chat/completions`. The validator returns allowed URLs unchanged. It permits exactly `/v1` or `/v1/` on any host and refuses the root web chat, a full `/v1/chat/completions` URL, other paths, credentials, query or fragment delimiters, whitespace, control characters, non-ASCII hosts and malformed IDNA. +The SDK appends a trailing slash to its stored base URL and calls `/v1/chat/completions`. The validator returns allowed URLs unchanged. It permits exactly `/v1` or `/v1/` on any host and refuses the root web chat, a full `/v1/chat/completions` URL, other paths, credentials, query or fragment delimiters, whitespace, control characters, non-ASCII hosts and malformed IDNA. Host names may use letters, digits, hyphens and underscores inside a label, so container or Compose service names such as `bonsai_server` work. -Loopback HTTP is allowed for `localhost`, `127.0.0.1` and `::1`. HTTP outside loopback is allowed with one stderr warning about the unauthenticated server. Use it only on a trusted LAN. HTTPS URLs still need the same API path. +Loopback HTTP is allowed without a warning for `localhost` and loopback IP addresses (`127.0.0.0/8`, `::1`). HTTP outside loopback is allowed with one stderr warning about the unauthenticated server. Use it only on a trusted LAN. HTTPS URLs still need the same API path. The [server docs](https://docs.prismml.com/run/server) warn: > The scripts bind to `127.0.0.1`, so the server is reachable only from the same machine and there is no authentication. Setting `BONSAI_HOST` to any non-loopback address (for example `0.0.0.0`) exposes an unauthenticated server to your network. -For llama-server, any model string works because it serves the loaded model; `GET /v1/models` returns the real name. The documented Python example requests `bonsai` with streaming. The recorded non-streamed model name is what the server reports ([server docs](https://docs.prismml.com/run/server)). The loaded-model name used in tests is synthetic, not a promised filename. +For llama-server, any model string works because it serves the loaded model; `GET /v1/models` returns the real name. The documented Python example requests `bonsai` with streaming ([server docs](https://docs.prismml.com/run/server)). `traceai-openai` records the `model` field of the server's non-streamed response, pinned by the tests; the loaded-model name used in tests is synthetic, not a promised filename. Thinking is controlled per request with `thinking_budget_tokens`; zero disables it ([server docs](https://docs.prismml.com/run/server)). Send it through the SDK's `extra_body`: diff --git a/python/examples/prismml/src/app.py b/python/examples/prismml/src/app.py index f9bf98754..e36a695b8 100644 --- a/python/examples/prismml/src/app.py +++ b/python/examples/prismml/src/app.py @@ -22,6 +22,15 @@ ) +def _is_loopback(host: str) -> bool: + if host == "localhost": + return True + try: + return ipaddress.ip_address(host).is_loopback + except ValueError: + return False + + def check_base_url(url: str) -> str: """Validate without rewriting; warn once for HTTP outside loopback.""" def refuse(reason): @@ -41,28 +50,32 @@ def refuse(reason): refuse("use an http or https URL") if not host or not host.isascii(): refuse("use a plain ASCII host") - host = host.lower().rstrip(".") + host = host.lower() + host = host[:-1] if host.endswith(".") else host # One trailing DNS dot only. try: if ":" in host: ipaddress.IPv6Address(host) if "%" in host: raise ValueError else: + # Letters, digits and hyphens, plus underscores inside a label for + # container and Compose service names such as bonsai_server. if len(host) > 253 or not all( - re.fullmatch(r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?", label) + re.fullmatch(r"[a-z0-9](?:[a-z0-9_-]{0,61}[a-z0-9])?", label) for label in host.split(".") ): raise ValueError host.encode("ascii").decode("idna") except (ValueError, UnicodeError): - refuse("use a valid ASCII host without malformed IDNA") + refuse("use a valid ASCII host name (letters, digits, hyphens, inner underscores) " + "or IP address, without malformed IDNA") if port == 0 or (parsed.netloc.endswith(":") and port is None): refuse("use a valid non-zero port") if "?" in url or "#" in url: refuse("remove the query or fragment, including an empty delimiter") if parsed.path not in ("/v1", "/v1/"): refuse(f"use the API path /v1 or /v1/, for example {DEFAULT_BASE_URL}") - if parsed.scheme == "http" and host not in ("localhost", "127.0.0.1", "::1"): + if parsed.scheme == "http" and not _is_loopback(host): print(HTTP_WARNING, file=sys.stderr) return url diff --git a/python/examples/prismml/tests/test_prismml_recipe.py b/python/examples/prismml/tests/test_prismml_recipe.py index e7983528f..7904af663 100644 --- a/python/examples/prismml/tests/test_prismml_recipe.py +++ b/python/examples/prismml/tests/test_prismml_recipe.py @@ -408,14 +408,17 @@ def test_tool_call_records_definition_id_name_arguments_and_output(traced): "http://127.0.0.1:8080/v1", "http://[::1]:8080/v1/", "https://trusted-lan.example/v1", "http://trusted-lan.example:8080/v1", "http://TRUSTED-LAN.EXAMPLE.:8080/v1/", + "http://bonsai_server:8080/v1", # Compose-style service name: allowed, warned (not loopback). + "http://127.0.0.2:8080/v1", "http://[0:0:0:0:0:0:0:1]:8080/v1", # Other loopback spellings: no warning. ] +WARNED_HOSTS = ("trusted-lan", "bonsai_server") @pytest.mark.parametrize("url", ALLOWED_URLS) def test_allowed_base_urls_are_unchanged(url, capsys): result = app.check_base_url(url) assert result is url - warning = "trusted-lan" in url.lower() and url.startswith("http:") + warning = url.lower().startswith("http:") and any(host in url.lower() for host in WARNED_HOSTS) assert capsys.readouterr().err == (EXPECTED_HTTP_WARNING + "\n" if warning else "") @@ -455,7 +458,9 @@ def test_allowed_base_urls_are_unchanged(url, capsys): ("http://xn--.example/v1", "IDNA"), ("http://xn--a.example/v1", "IDNA"), ("http://local%68ost:8080/v1", "ASCII host"), - ("http://local_host:8080/v1", "ASCII host"), + ("http://_bonsai:8080/v1", "ASCII host"), + ("http://bonsai_:8080/v1", "ASCII host"), + ("http://localhost..:8080/v1", "ASCII host"), ("http://-localhost:8080/v1", "ASCII host"), ("http://localhost..example/v1", "ASCII host"), ("http://localhost:invalid/v1", "port"), @@ -503,7 +508,8 @@ def must_not_run(*args, **kwargs): output = capsys.readouterr() assert output.out == "" assert output.err == message + "\n" - assert fake.requests() == [] + # main() never received a reachable provider URL here; the must_not_run patches are + # the guards. The receiver is the configured Future AGI collector, so these can fail. assert receiver.spans() == [] assert receiver.requests() == [] @@ -520,6 +526,8 @@ def must_not_run(*args, **kwargs): pytest.fail("empty model reached tracing") monkeypatch.setattr(app, "setup_tracing", must_not_run) + monkeypatch.setenv("FI_BASE_URL", receiver.origin) + monkeypatch.setenv(app.BASE_URL_ENV, fake.base_url) # Reachable, so the absence checks below can fail. assert app.main(["--model", ""]) == 2 assert app.MODEL_ENV in capsys.readouterr().err assert fake.requests() == [] @@ -707,7 +715,7 @@ def test_readme_and_requirements_contract(): assert [line for line in requirements if not line.startswith("#")] == [ "openai==3.24.0", "traceAI-openai==0.1.10", "fi-instrumentation-otel==1.1.0"] for path in ROOT.rglob("*"): - if path.is_file(): + if path.is_file() and "__pycache__" not in path.parts and path.suffix in {".py", ".md", ".txt"}: text = path.read_text() assert not re.search(r"\b[A-Z]{2,5}-\d{3,}\b", text), str(path) assert not re.search(r"\bsk-[A-Za-z0-9]{16,}\b", text), str(path) From 7c803393d981c46efb55fca2900bcc1f63f3dd27 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Sun, 11 Oct 2026 20:01:22 -0700 Subject: [PATCH 4/8] docs(examples/prismml): real llama-server reports the model file path; document --alias A live run against the real Bonsai server (PrismML llama.cpp fork, Bonsai-1.7B Q1_0, started with scripts/start_llama_server.sh) showed that llama-server returns the loaded GGUF file's absolute path as the response `model`. traceai-openai records that value in gen_ai.request.model, so a default setup exports a local path that can contain the user's name, and FI_HIDE_INPUTS/FI_HIDE_OUTPUTS do not mask it. Starting the server with `--alias bonsai` (passed through the demo script) makes the server, and therefore the span, report `bonsai`; verified live. Document both in "What you see", "Provider specifics" and "Privacy", and pin the advice in the README contract test. Co-Authored-By: Claude Opus 5.5 --- python/examples/prismml/README.md | 6 ++++-- python/examples/prismml/tests/test_prismml_recipe.py | 5 ++++- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/python/examples/prismml/README.md b/python/examples/prismml/README.md index 435c107ee..e248be946 100644 --- a/python/examples/prismml/README.md +++ b/python/examples/prismml/README.md @@ -84,7 +84,7 @@ The fixtures produce one `ChatCompletion` span per call with `gen_ai.span.kind=L Current `traceai-openai` behavior: -- For a successful non-streamed call, `gen_ai.request.model` is the model name the server returns. The fixture requests `bonsai` and returns the synthetic loaded-model name `Ternary-Bonsai-2-27B-Q4_K_M.gguf`; the latter is recorded. The requested `bonsai` remains in the `gen_ai.request.parameters` JSON. +- For a successful non-streamed call, `gen_ai.request.model` is the model name the server returns. The fixture requests `bonsai` and returns the synthetic loaded-model name `Ternary-Bonsai-2-27B-Q4_K_M.gguf`; the latter is recorded. The requested `bonsai` remains in the `gen_ai.request.parameters` JSON. A real `llama-server` started by the demo scripts returns the loaded file's absolute path, such as `/Users/you/Bonsai-demo/models/gguf/1.7B/Bonsai-1.7B-Q1_0.gguf`, and that path is what gets recorded. Start the server with `./scripts/start_llama_server.sh --alias bonsai` to record `bonsai` instead; the script passes extra arguments to `llama-server`. - Streamed and failed calls omit `gen_ai.request.model`. The requested model remains in `gen_ai.request.parameters`, unless invocation parameters are hidden. - Non-streamed token usage is recorded as `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, and `gen_ai.usage.total_tokens` when the response supplies it. Missing usage is omitted, never replaced with zero. - A stream accumulates answer text in `output.value`. Default fixture streams omit usage. With `stream_options={"include_usage": True}` and a final usage chunk, the same three token attributes are recorded. This verifies the instrumentor, not the real server's streaming usage support. @@ -110,7 +110,7 @@ The [server docs](https://docs.prismml.com/run/server) warn: > The scripts bind to `127.0.0.1`, so the server is reachable only from the same machine and there is no authentication. Setting `BONSAI_HOST` to any non-loopback address (for example `0.0.0.0`) exposes an unauthenticated server to your network. -For llama-server, any model string works because it serves the loaded model; `GET /v1/models` returns the real name. The documented Python example requests `bonsai` with streaming ([server docs](https://docs.prismml.com/run/server)). `traceai-openai` records the `model` field of the server's non-streamed response, pinned by the tests; the loaded-model name used in tests is synthetic, not a promised filename. +For llama-server, any model string works because it serves the loaded model; `GET /v1/models` returns the real name: the loaded file's absolute path by default, or the name set with `--alias`. The documented Python example requests `bonsai` with streaming ([server docs](https://docs.prismml.com/run/server)). `traceai-openai` records the `model` field of the server's non-streamed response, pinned by the tests; the loaded-model name used in tests is synthetic, not a promised filename. Thinking is controlled per request with `thinking_budget_tokens`; zero disables it ([server docs](https://docs.prismml.com/run/server)). Send it through the SDK's `extra_body`: @@ -129,6 +129,8 @@ The 27B model supports native OpenAI-style `tool_calls` ([server docs](https://d Prompt and answer text normally leave the machine in the Future AGI trace export. Set `FI_HIDE_INPUTS=true` to mask input attributes and `FI_HIDE_OUTPUTS=true` to mask output attributes. The tests verify each flag against visible controls for both chat and streaming. +The server's returned model name is exported too, and these flags do not mask it. Without `--alias`, `llama-server` returns the model file's absolute path, which can include your user name or directory layout. Start the server with `--alias bonsai` to keep local paths out of the export. + `FI_HIDE_INPUTS` masks only inputs. If the server echoes the prompt in its answer, that text remains in output attributes unless outputs are also hidden. Non-streamed output is assistant content, and streamed output is accumulated text, including an empty string when no text arrives. This recipe does not export a raw Chat Completions response in `output.value`. The local server still receives the prompt. Future AGI masking does not change the server's memory or its own logging. Key separation tests verify that `not-needed` and the custom local placeholder never reach the Future AGI export, while Future AGI keys never reach the local request. diff --git a/python/examples/prismml/tests/test_prismml_recipe.py b/python/examples/prismml/tests/test_prismml_recipe.py index 7904af663..98554da74 100644 --- a/python/examples/prismml/tests/test_prismml_recipe.py +++ b/python/examples/prismml/tests/test_prismml_recipe.py @@ -688,7 +688,10 @@ def test_readme_and_requirements_contract(): for phrase in ("http://localhost:8080/v1", "http://localhost:8081/v1", "provider field says `openai`", "gen_ai.request.model", "verbose=False", "FI_API_KEY", "FI_SECRET_KEY", "FI_BASE_URL", "FI_HIDE_INPUTS", "FI_HIDE_OUTPUTS", - "thinking_budget_tokens", "tool_calls", "not-needed", "cost"): + "thinking_budget_tokens", "tool_calls", "not-needed", "cost", + # A real llama-server reports the loaded file's absolute path as `model` + # unless started with --alias; the README must say so. + "absolute path", "--alias bonsai"): assert phrase in readme source = (ROOT / "src" / "app.py").read_text() for name in (app.API_KEY_ENV, app.BASE_URL_ENV, app.MODEL_ENV): From 79b60364bd179ee7772508d854ddcb84332eb676 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Sun, 11 Oct 2026 21:32:44 -0700 Subject: [PATCH 5/8] test(examples/prismml): accept traceai-openai before and after the provider/model fix The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is always "openai" and streamed/failed spans carry no gen_ai.request.model. future-agi/traceAI#249 labels known vendor hosts and records the requested model, so these pins would break whichever PR merged second. Accept both behaviours (and the related Responses/embedding output changes where this recipe checks them), and say in the README which version does what. Passes against published traceAI-openai 0.1.10 and against #249. Co-Authored-By: Claude Opus 5.5 --- python/examples/prismml/README.md | 2 +- python/examples/prismml/tests/test_prismml_recipe.py | 12 ++++++++---- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/python/examples/prismml/README.md b/python/examples/prismml/README.md index e248be946..194470ed9 100644 --- a/python/examples/prismml/README.md +++ b/python/examples/prismml/README.md @@ -85,7 +85,7 @@ The fixtures produce one `ChatCompletion` span per call with `gen_ai.span.kind=L Current `traceai-openai` behavior: - For a successful non-streamed call, `gen_ai.request.model` is the model name the server returns. The fixture requests `bonsai` and returns the synthetic loaded-model name `Ternary-Bonsai-2-27B-Q4_K_M.gguf`; the latter is recorded. The requested `bonsai` remains in the `gen_ai.request.parameters` JSON. A real `llama-server` started by the demo scripts returns the loaded file's absolute path, such as `/Users/you/Bonsai-demo/models/gguf/1.7B/Bonsai-1.7B-Q1_0.gguf`, and that path is what gets recorded. Start the server with `./scripts/start_llama_server.sh --alias bonsai` to record `bonsai` instead; the script passes extra arguments to `llama-server`. -- Streamed and failed calls omit `gen_ai.request.model`. The requested model remains in `gen_ai.request.parameters`, unless invocation parameters are hidden. +- Up to `traceai-openai` 0.1.10, streamed and failed calls omit `gen_ai.request.model`; later versions (future-agi/traceAI#249) record the requested model there. The requested model remains in `gen_ai.request.parameters`, unless invocation parameters are hidden. - Non-streamed token usage is recorded as `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, and `gen_ai.usage.total_tokens` when the response supplies it. Missing usage is omitted, never replaced with zero. - A stream accumulates answer text in `output.value`. Default fixture streams omit usage. With `stream_options={"include_usage": True}` and a final usage chunk, the same three token attributes are recorded. This verifies the instrumentor, not the real server's streaming usage support. - Non-streamed `output.value` contains assistant content. Tool-call responses contain a readable function call instead. A stream with no text records `output.value=''`; Chat Completions have no raw-response fallback here. diff --git a/python/examples/prismml/tests/test_prismml_recipe.py b/python/examples/prismml/tests/test_prismml_recipe.py index 98554da74..e25adcb64 100644 --- a/python/examples/prismml/tests/test_prismml_recipe.py +++ b/python/examples/prismml/tests/test_prismml_recipe.py @@ -25,6 +25,10 @@ FakeOpenAI, chat_response, stream_body, ) +# traceai-openai 0.1.10 and earlier omit gen_ai.request.model on streamed and failed +# calls; later versions (future-agi/traceAI#249) record the requested model there. +# Accept both so this recipe passes on either version. + ROOT = Path(__file__).resolve().parents[1] FI_API_KEY = "placeholder-fi-api-key" FI_SECRET_KEY = "placeholder-fi-secret-key" @@ -237,7 +241,7 @@ def test_stream_accumulation_and_usage(traced, include_usage): assert attrs["output.value"] == ANSWER # traceai-openai records the model only from a non-streamed response; flip # this when the instrumentor records the request model. - assert "gen_ai.request.model" not in attrs + assert attrs.get("gen_ai.request.model") in (None, REQUEST_MODEL) assert json.loads(attrs["gen_ai.request.parameters"])["model"] == REQUEST_MODEL assert_usage(attrs, USAGE_ATTRIBUTES if include_usage else {}) assert any(event["name"] == "First Token Stream Event" for event in span["events"]) @@ -253,7 +257,7 @@ def test_textless_stream_has_no_raw_response_fallback(traced): assert len(seen) == 1 _, attrs = one_span(traced) assert attrs["output.value"] == "" - assert "gen_ai.request.model" not in attrs + assert attrs.get("gen_ai.request.model") in (None, REQUEST_MODEL) assert json.loads(attrs["gen_ai.request.parameters"])["model"] == REQUEST_MODEL assert_usage(attrs, {}) @@ -274,7 +278,7 @@ def test_synthetic_401_records_error_without_key(traced): assert LOCAL_KEY not in json.dumps(span) # traceai-openai records the model only from a non-streamed response; flip # this when the instrumentor records the request model. - assert "gen_ai.request.model" not in attrs + assert attrs.get("gen_ai.request.model") in (None, REQUEST_MODEL) assert json.loads(attrs["gen_ai.request.parameters"])["model"] == REQUEST_MODEL assert_usage(attrs, {}) assert_export_separation(traced[0], traced[2]) @@ -653,7 +657,7 @@ def test_app_subprocess_chat_and_stream(tmp_path, stream): assert attrs["output.value"] == ANSWER assert json.loads(attrs["gen_ai.request.parameters"])["model"] == REQUEST_MODEL if stream: - assert "gen_ai.request.model" not in attrs + assert attrs.get("gen_ai.request.model") in (None, REQUEST_MODEL) else: assert attrs["gen_ai.request.model"] == RESPONSE_MODEL assert_usage(attrs, {} if stream else USAGE_ATTRIBUTES) From 3ac395c4469c2e9062cf31813375adcc52d76dd5 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Sun, 18 Oct 2026 21:28:55 -0700 Subject: [PATCH 6/8] fix(examples/prismml): honour FI_PROJECT_NAME setup_tracing() passed project_name or "prismml-bonsai" to register(), so the default was always truthy and FI_PROJECT_NAME was never read: `python src/app.py` always exported to prismml-bonsai. The Anannas recipe reads FI_PROJECT_NAME with its name as the default; do the same here. An explicit setup_tracing() argument still wins, and an unset or empty FI_PROJECT_NAME keeps prismml-bonsai (never register()'s generic default). - README: FI_PROJECT_NAME in the Configure table and the Run section; the Code snippet reads it the same way as src/app.py. - Tests: setup_tracing() default / empty / env / explicit cases; the app.py subprocess runs with FI_PROJECT_NAME unset and set; the README contract checks the new row and snippet. Passes against repository source, published traceAI-openai 0.1.10 and future-agi/traceAI#249. Co-Authored-By: Claude Opus 5.5 --- python/examples/prismml/README.md | 8 ++-- python/examples/prismml/src/app.py | 2 +- .../prismml/tests/test_prismml_recipe.py | 39 +++++++++++++++++-- 3 files changed, 42 insertions(+), 7 deletions(-) diff --git a/python/examples/prismml/README.md b/python/examples/prismml/README.md index 194470ed9..56c17d707 100644 --- a/python/examples/prismml/README.md +++ b/python/examples/prismml/README.md @@ -32,6 +32,7 @@ Start your server first, following the [PrismML server docs](https://docs.prismm | `PRISMML_MODEL` | Optional; `bonsai` | Model name sent by the OpenAI client | | `PRISMML_BASE_URL` | Optional; `http://localhost:8080/v1` | OpenAI client; use `http://localhost:8081/v1` for MLX | | `FI_BASE_URL` | Optional; the tracer's Future AGI collector default | Tracer export origin | +| `FI_PROJECT_NAME` | Optional; `prismml-bonsai` | Tracer only; the Observe project name | | `FI_HIDE_INPUTS` | Optional; `true` masks input attributes | Tracer only | | `FI_HIDE_OUTPUTS` | Optional; `true` masks output attributes | Tracer only | | `FI_HIDE_LLM_INVOCATION_PARAMETERS` | Optional; `true` omits invocation parameters | Tracer only | @@ -54,19 +55,20 @@ python src/app.py python src/app.py --stream ``` -Replace the Future AGI placeholders with your keys. `--prompt` changes the question and `--model` overrides `PRISMML_MODEL`. The app prints answer text and flushes the tracer. +Replace the Future AGI placeholders with your keys. `--prompt` changes the question and `--model` overrides `PRISMML_MODEL`. Set `FI_PROJECT_NAME` to send the traces to another Observe project; unset or empty, they go to `prismml-bonsai`. The app prints answer text and flushes the tracer. ## Code Instrument before creating the client. Future AGI keys are read by the tracer from its environment. ```python +import os from fi_instrumentation import register from fi_instrumentation.fi_types import ProjectType from openai import OpenAI from traceai_openai import OpenAIInstrumentor -provider = register( - project_name="prismml-bonsai", project_type=ProjectType.OBSERVE, +provider = register( # FI_PROJECT_NAME overrides, as in src/app.py + project_name=os.environ.get("FI_PROJECT_NAME") or "prismml-bonsai", project_type=ProjectType.OBSERVE, set_global_tracer_provider=False, verbose=False, ) OpenAIInstrumentor().instrument(tracer_provider=provider) diff --git a/python/examples/prismml/src/app.py b/python/examples/prismml/src/app.py index e36a695b8..f7e98ef46 100644 --- a/python/examples/prismml/src/app.py +++ b/python/examples/prismml/src/app.py @@ -91,7 +91,7 @@ def make_client(base_url: str | None = None, api_key: str | None = None, def setup_tracing(project_name: str | None = None): provider = register( - project_name=project_name or "prismml-bonsai", + project_name=project_name or os.environ.get("FI_PROJECT_NAME") or "prismml-bonsai", project_type=ProjectType.OBSERVE, set_global_tracer_provider=False, verbose=False, diff --git a/python/examples/prismml/tests/test_prismml_recipe.py b/python/examples/prismml/tests/test_prismml_recipe.py index e25adcb64..afe9a54d2 100644 --- a/python/examples/prismml/tests/test_prismml_recipe.py +++ b/python/examples/prismml/tests/test_prismml_recipe.py @@ -212,6 +212,34 @@ def test_chat_model_usage_and_key_separation(traced, monkeypatch, base_url, key) assert_export_separation(traced[0], traced[2], key) +@pytest.mark.parametrize("env_project, explicit, expected", [ + (None, None, "prismml-bonsai"), + ("", None, "prismml-bonsai"), + ("prismml-from-env", None, "prismml-from-env"), + ("prismml-from-env", "prismml-explicit", "prismml-explicit"), +], ids=["default", "empty", "env", "explicit"]) +def test_setup_tracing_project_name(monkeypatch, env_project, explicit, expected): + # FI_PROJECT_NAME overrides the recipe default; an explicit argument wins over both. + # Never fall through to register()'s generic DEFAULT_PROJECT_NAME. + if env_project is not None: + monkeypatch.setenv("FI_PROJECT_NAME", env_project) + with Receiver() as receiver: + monkeypatch.setenv("FI_BASE_URL", receiver.origin) + provider = app.setup_tracing(explicit) + try: + client, seen = mock_client() + with client: + client.chat.completions.create(model=REQUEST_MODEL, + messages=[{"role": "user", "content": "project fixture"}]) + assert len(seen) == 1 + one_span((receiver, provider, expected)) + assert_export_separation(receiver, expected) + finally: + OpenAIInstrumentor().uninstrument() + provider.force_flush() + provider.shutdown() + + def test_usage_absent_is_omitted(traced): client, seen = mock_client(chat_response(include_usage=False)) with client: @@ -615,12 +643,15 @@ def subprocess_environment(tmp_path): return env +@pytest.mark.parametrize("env_project", [None, "prismml-subprocess-env"], ids=["default-project", "env-project"]) @pytest.mark.parametrize("stream", [False, True]) -def test_app_subprocess_chat_and_stream(tmp_path, stream): +def test_app_subprocess_chat_and_stream(tmp_path, stream, env_project): with Receiver() as receiver, FakeOpenAI() as fake: env = subprocess_environment(tmp_path) env.update(FI_BASE_URL=receiver.origin, PRISMML_BASE_URL=fake.base_url, PRISMML_MODEL=REQUEST_MODEL) + if env_project: + env["FI_PROJECT_NAME"] = env_project if stream: env["PRISMML_API_KEY"] = "placeholder-prismml-key" # The non-streamed child exercises the default not-needed placeholder. @@ -661,7 +692,7 @@ def test_app_subprocess_chat_and_stream(tmp_path, stream): else: assert attrs["gen_ai.request.model"] == RESPONSE_MODEL assert_usage(attrs, {} if stream else USAGE_ATTRIBUTES) - assert_export_separation(receiver, "prismml-bonsai", key) + assert_export_separation(receiver, env_project or "prismml-bonsai", key) @pytest.mark.parametrize("operation", ["getaddrinfo", "connect", "connect_ex"]) @@ -691,7 +722,9 @@ def test_readme_and_requirements_contract(): assert readme.startswith("# PrismML (OpenAI-compatible) with traceAI\n") for phrase in ("http://localhost:8080/v1", "http://localhost:8081/v1", "provider field says `openai`", "gen_ai.request.model", "verbose=False", - "FI_API_KEY", "FI_SECRET_KEY", "FI_BASE_URL", "FI_HIDE_INPUTS", "FI_HIDE_OUTPUTS", + "FI_API_KEY", "FI_SECRET_KEY", "FI_BASE_URL", "FI_PROJECT_NAME", "FI_HIDE_INPUTS", "FI_HIDE_OUTPUTS", + "| `FI_PROJECT_NAME` | Optional; `prismml-bonsai` |", + 'project_name=os.environ.get("FI_PROJECT_NAME") or "prismml-bonsai"', "thinking_budget_tokens", "tool_calls", "not-needed", "cost", # A real llama-server reports the loaded file's absolute path as `model` # unless started with --alias; the README must say so. From d45b642a946d824c0166424bb4a747bf482b5c0e Mon Sep 17 00:00:00 2001 From: Nikhil Pareek <8347993+nik13@users.noreply.github.com> Date: Tue, 27 Oct 2026 21:01:56 -0700 Subject: [PATCH 7/8] test(prismml): add recipe CI and record real Bonsai server verification Co-Authored-By: Claude Code --- .github/workflows/prismml-recipe-tests.yml | 45 ++++++++++++++++++++++ python/examples/prismml/README.md | 20 +++++++++- 2 files changed, 64 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/prismml-recipe-tests.yml diff --git a/.github/workflows/prismml-recipe-tests.yml b/.github/workflows/prismml-recipe-tests.yml new file mode 100644 index 000000000..a7ca8deaf --- /dev/null +++ b/.github/workflows/prismml-recipe-tests.yml @@ -0,0 +1,45 @@ +name: PrismML recipe tests + +on: + pull_request: + branches: [dev, main] + paths: + - 'python/examples/prismml/**' + - 'python/frameworks/openai/**' + - 'python/fi_instrumentation/**' + - 'python/tests/harness/**' + - '.github/workflows/prismml-recipe-tests.yml' + workflow_dispatch: + +permissions: + contents: read + +jobs: + recipe: + name: PrismML recipe (${{ matrix.packages }}) + runs-on: ubuntu-latest + timeout-minutes: 10 + strategy: + fail-fast: false + matrix: + packages: [published, source] + env: + FI_BASE_URL: http://127.0.0.1:9 + FI_GRPC_URL: http://127.0.0.1:9 + PYTHONDONTWRITEBYTECODE: '1' + PYTHONPATH: python/examples/prismml/src:python/tests + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ github.event.pull_request.head.sha || github.sha }} + persist-credentials: false + - uses: actions/setup-python@v5 + with: + python-version: '3.11' + - name: Install published recipe pins and test tools + run: pip install -r python/examples/prismml/requirements.txt pytest httpx 'wrapt<2' + - name: Install source distributions + if: matrix.packages == 'source' + run: pip install ./python ./python/frameworks/openai + - name: Loopback-only recipe contracts + run: pytest python/examples/prismml/tests -q -p no:cacheprovider --noconftest -o addopts= -rfEs diff --git a/python/examples/prismml/README.md b/python/examples/prismml/README.md index 56c17d707..6a9470ada 100644 --- a/python/examples/prismml/README.md +++ b/python/examples/prismml/README.md @@ -143,7 +143,25 @@ The local server still receives the prompt. Future AGI masking does not change t - Starting or stopping the server, distributing weights, live GPU jobs, MLX server differences, benchmarks or model quality. - Tool execution and streaming tool calls. - Other products called Prism: the [ssimplifi Prism proxy guide](https://ssimplifi.com/guides/openai-compatible-api), [prism-proxy](https://git.sovereign-society.org/prism/prism-proxy), and [Prism PHP](https://mintlify.wiki/prism-php/prism/providers/overview). -- This recipe is not tested against the live provider: the tests use a local fake of the OpenAI API. They do not start a Bonsai server or download weights. +- The automated contract suite uses a local fake of the OpenAI API and does not download weights. The real-server check below is separate. + +## Real-server check + +On 2026-10-28 this recipe ran against the official PrismML `llama-server` +release `prism-b10826-c9c34bc`, with the official +`Ternary-Bonsai-2-2B-PQ2_0.gguf` weights (verified against the published SHA256SUMS). +The Linux CPU server used `BONSAI_MODEL=2B`, `BONSAI_NGL=0`, `BONSAI_CTX=4096`, +`BONSAI_THINKING=0`, and `--alias bonsai`. The model and binary came from the +[official demo](https://github.com/PrismML-Eng/Bonsai-demo) and its linked releases. + +With the pinned `openai==3.24.0`, `traceAI-openai==0.1.10`, and +`fi-instrumentation-otel==1.1.0`, both the normal and `--stream` recipe commands +returned answers. Their exported spans were read back in Future AGI Cloud: +non-streamed calls had model `bonsai` and token usage; the default stream had +answer text but no model or usage attributes, matching the published-version +behavior described above. A separate call with input/output/invocation-parameter +hiding enabled exported no prompt or answer text. This check covers the 2B text +model on the llama.cpp server, not 27B tool calling, vision, or MLX. ## Tests From 8bf24ab0abae138af9e0749cfad47f2d5ad70fb6 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek <8347993+nik13@users.noreply.github.com> Date: Tue, 27 Oct 2026 21:12:03 -0700 Subject: [PATCH 8/8] test(prismml): pin real-server verification scope in README contract Co-Authored-By: Claude Code --- python/examples/prismml/tests/test_prismml_recipe.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/python/examples/prismml/tests/test_prismml_recipe.py b/python/examples/prismml/tests/test_prismml_recipe.py index afe9a54d2..29be3876a 100644 --- a/python/examples/prismml/tests/test_prismml_recipe.py +++ b/python/examples/prismml/tests/test_prismml_recipe.py @@ -750,7 +750,10 @@ def test_readme_and_requirements_contract(): assert "--with 'traceAI-openai==0.1.10'" in readme assert "--with 'fi-instrumentation-otel==1.1.0'" in readme assert "TBD" not in readme - assert "not tested against the live provider" in readme + assert "## Real-server check" in readme + assert "Ternary-Bonsai-2-2B-PQ2_0.gguf" in readme + assert "The automated contract suite uses a local fake" in readme + assert "not 27B tool calling, vision, or MLX" in readme requirements = (ROOT / "requirements.txt").read_text().splitlines() assert [line for line in requirements if not line.startswith("#")] == [ "openai==3.24.0", "traceAI-openai==0.1.10", "fi-instrumentation-otel==1.1.0"]