From 014157414cc1ab1bf5d8aa0c285548020c4dcf7c Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Mon, 5 Oct 2026 18:05:00 -0700 Subject: [PATCH 1/7] feat(examples): Baseten Model APIs recipe on traceai-openai Add python/examples/baseten: trace Baseten Model APIs Chat Completions made with the official openai SDK (base_url https://inference.baseten.co/v1) by using the existing traceai-openai instrumentor. No Baseten package. - src/app.py: register() + OpenAIInstrumentor before the client; the Baseten key goes only to the OpenAI client, Future AGI keys only to the tracer. check_base_url() refuses the Anthropic-beta root (no /v1) and dedicated-deployment hosts (model-*.api.baseten.co); it never rewrites. - tests: loopback contract on the shared harness Receiver: request URL and bearer key at the documented host (MockTransport), one LLM span, model, usage (omitted when absent), provider label openai (D2), project resource and FI headers, no vendor key in any exported field, streaming, 401 error span, FI_HIDE_INPUTS/OUTPUTS with control, using_session, app subprocess under a socket guard, refused URLs exit 2 before tracing. - Pins current traceai-openai behaviour: streamed and failed calls have no model attribute; the default stream has no usage attributes. - README: install, configure, run, code, what you see, Baseten specifics, privacy, limits, tests and tested versions. approved: Nikhil 2026-10-03 blanket Refs: TH-8310 --- python/examples/baseten/README.md | 163 ++++++ python/examples/baseten/requirements.txt | 4 + python/examples/baseten/src/app.py | 95 ++++ python/examples/baseten/tests/_fake_openai.py | 99 ++++ .../tests/loopback_guard/sitecustomize.py | 42 ++ .../baseten/tests/test_baseten_recipe.py | 465 ++++++++++++++++++ 6 files changed, 868 insertions(+) create mode 100644 python/examples/baseten/README.md create mode 100644 python/examples/baseten/requirements.txt create mode 100644 python/examples/baseten/src/app.py create mode 100644 python/examples/baseten/tests/_fake_openai.py create mode 100644 python/examples/baseten/tests/loopback_guard/sitecustomize.py create mode 100644 python/examples/baseten/tests/test_baseten_recipe.py diff --git a/python/examples/baseten/README.md b/python/examples/baseten/README.md new file mode 100644 index 000000000..ef2273bc6 --- /dev/null +++ b/python/examples/baseten/README.md @@ -0,0 +1,163 @@ +# Baseten (OpenAI-compatible) with traceAI + +Trace Chat Completions made through the official `openai` Python SDK at +`https://inference.baseten.co/v1`. This recipe uses `traceai-openai`; it does +not need a Baseten tracing package. + +## Install + +From this directory, install the tested pins: + +```bash +pip install -r requirements.txt +``` + +The plain install form is: + +```bash +pip install traceAI-openai fi-instrumentation-otel openai +``` + +## Configure + +| Environment variable | Purpose | Destination | +| --- | --- | --- | +| `FI_API_KEY` | Future AGI API key | Tracer only | +| `FI_SECRET_KEY` | Future AGI secret key | Tracer only | +| `BASETEN_API_KEY` | Baseten API key | OpenAI client only | +| `BASETEN_MODEL` | Required Model APIs slug; `--model` overrides it | Baseten request | +| `BASETEN_BASE_URL` | Optional override; defaults to `https://inference.baseten.co/v1` | OpenAI client | +| `FI_BASE_URL` | Optional Future AGI collector origin | Tracer only | + +Choose a slug from Baseten's current [Model APIs table](https://docs.baseten.co/inference/model-apis/overview). +`zai-org/GLM-5.2` is a documentation example, not a model availability guarantee. + +## Run + +Run from this directory. Replace the placeholders with your own keys: + +```bash +export FI_API_KEY="placeholder-futureagi-api-key" +export FI_SECRET_KEY="placeholder-futureagi-secret-key" +export BASETEN_API_KEY="placeholder-baseten-key" +export BASETEN_MODEL="zai-org/GLM-5.2" +python src/app.py +python src/app.py --stream +``` + +You can also pass `--model ` and `--prompt "What is the capital of France?"`. +The app flushes traces before returning. The SDK appends a trailing slash to +`client.base_url`: `https://inference.baseten.co/v1/`. Its request goes to +`https://inference.baseten.co/v1/chat/completions`. + +## Code + +Instrument before creating the client. Future AGI keys are read by the tracer; +only the Baseten key is passed to `OpenAI`: + +```python +import os +from fi_instrumentation import register +from fi_instrumentation.fi_types import ProjectType +from openai import OpenAI +from traceai_openai import OpenAIInstrumentor +provider = register(project_name="baseten-openai", project_type=ProjectType.OBSERVE, + set_global_tracer_provider=False, verbose=False) +OpenAIInstrumentor().instrument(tracer_provider=provider) +with OpenAI(base_url="https://inference.baseten.co/v1", + api_key=os.environ["BASETEN_API_KEY"]) as client: + response = client.chat.completions.create(model=os.environ["BASETEN_MODEL"], + messages=[{"role": "user", "content": "What is the capital of France?"}]) + print(response.choices[0].message.content) +provider.force_flush() +``` + +See [src/app.py](src/app.py) for the runnable version and URL checks. + +## What you see in Future AGI + +For this Chat Completions flow, expect one `ChatCompletion` LLM span per call. +On normal calls, `gen_ai.request.model` is the model id the provider returns +in its response. Token usage appears only when the response has `usage`; +missing usage is omitted, never reported as zero. + +Streamed calls record accumulated output text but have no model attribute. +The default stream tested here returns no usage, so it exports +no `gen_ai.usage.*` attributes. Failed calls produce an error span with an +exception event and no model attribute. These are current `traceai-openai` behaviour. + +The provider field says `openai`: `gen_ai.provider.name` is `openai`. This is +the shared instrumentor's label, including when the client calls Baseten. + +## Provider specifics + +This recipe covers Baseten Model APIs only. Dedicated deployments use their +own per-deployment URL and are not covered. The app rejects hosts matching +`model-*.api.baseten.co`, including +`https://model-abc123.api.baseten.co/environments/production/sync/v1`. + +The Anthropic Messages beta at `https://inference.baseten.co` uses a different +client and is not covered. The app rejects that root and tells you to use +`/v1` for the OpenAI SDK. It returns allowed URLs unchanged, including custom +proxy URLs; it does not rewrite rejected URLs. Baseten CLI is out of scope. + +`x-session-affinity` is a Baseten routing header. It is NOT a Future AGI +session id; there is no bridge between them. For a Future AGI session, use +the context helper exported by `fi_instrumentation`: + +```python +from fi_instrumentation import using_session + +with using_session("s-1"): + response = client.chat.completions.create( + model=os.environ["BASETEN_MODEL"], + messages=[{"role": "user", "content": "Hello"}], + ) +``` + +After tracing setup, this context adds `session.id="s-1"` to the span. + +## Privacy + +Set `FI_HIDE_INPUTS=true` to keep prompt text out of exported spans. Set +`FI_HIDE_OUTPUTS=true` to keep response text out. Set these before tracing +setup. The provider still receives the prompt. Future AGI masking does not +change Baseten's own logging. + +## Limits / not covered + +- Anthropic SDK calls to Baseten and its Anthropic Messages beta. +- Dedicated deployments and the Baseten CLI. +- Historical import of past requests. +- Not tested against the live provider (no paid call); the tests use a local fake of the OpenAI API. + +## Tests + +The tests pin the current absence of model attributes on streaming and error +spans. They also pin the absence of usage attributes on the default stream. + +From the repository root: + +```bash +env -u PYTHONPATH PYTHONDONTWRITEBYTECODE=1 \ + PYTHONPATH="python/examples/baseten/src:python:python/frameworks/openai:python/tests" \ + uv run --no-project --python 3.11 \ + --with 'openai==3.24.0' --with httpx --with 'wrapt<2' \ + --with opentelemetry-api --with opentelemetry-sdk --with opentelemetry-instrumentation \ + --with opentelemetry-exporter-otlp-proto-http --with requests --with protobuf \ + --with opentelemetry-proto --with pydantic --with jsonschema --with pytest \ + pytest python/examples/baseten/tests -q -p no:cacheprovider --noconftest -o addopts= -rfEs +``` + +The tests use MockTransport at the documented host and local HTTP servers. +Subprocesses install a socket guard that blocks external hosts before DNS. + +| Python | `openai` | traceAI packages | Result | +| --- | --- | --- | --- | +| 3.10, 3.11, 3.12, 3.13 | 3.24.0 | `traceai-openai` and `fi_instrumentation` from this repository | Full suite | +| 3.11 | 1.69.0 (the `traceai-openai` floor) | from this repository | Full suite | +| 3.11 | 3.24.0 | published `traceAI-openai==0.1.10` and `fi-instrumentation-otel==1.1.0` from PyPI | Full suite | + +To run another row, change the `openai` pin in the command above, or replace the +repository paths on `PYTHONPATH` with `--with 'traceAI-openai==0.1.10' --with +'fi-instrumentation-otel==1.1.0'`. diff --git a/python/examples/baseten/requirements.txt b/python/examples/baseten/requirements.txt new file mode 100644 index 000000000..3a32ac062 --- /dev/null +++ b/python/examples/baseten/requirements.txt @@ -0,0 +1,4 @@ +# Recipe pins; tested against the matching traceAI sources in this worktree. +openai==3.24.0 +traceAI-openai==0.1.10 +fi-instrumentation-otel==1.1.0 diff --git a/python/examples/baseten/src/app.py b/python/examples/baseten/src/app.py new file mode 100644 index 000000000..9b2c0b0f6 --- /dev/null +++ b/python/examples/baseten/src/app.py @@ -0,0 +1,95 @@ +"""Trace Baseten Model APIs Chat Completions with the OpenAI SDK.""" + +import argparse +import os +import re +import sys +from urllib.parse import urlsplit + +from fi_instrumentation import register +from fi_instrumentation.fi_types import ProjectType +from openai import OpenAI +from traceai_openai import OpenAIInstrumentor + +DEFAULT_BASE_URL = "https://inference.baseten.co/v1" +API_KEY_ENV = "BASETEN_API_KEY" +BASE_URL_ENV = "BASETEN_BASE_URL" +MODEL_ENV = "BASETEN_MODEL" + + +def check_base_url(url: str) -> str: + """Reject the two Baseten surfaces that need a different recipe.""" + parsed = urlsplit(url) + host = parsed.hostname or "" + if host == "inference.baseten.co" and parsed.path in ("", "/"): + raise ValueError( + "Baseten's root is the Anthropic Messages beta endpoint; " + "use https://inference.baseten.co/v1 for the OpenAI SDK." + ) + if re.fullmatch(r"model-[^.]+\.api\.baseten\.co", host): + raise ValueError( + "Baseten dedicated deployments use a per-deployment URL and are " + "out of scope for this Model APIs recipe." + ) + return url + + +def make_client(base_url: str | None = None, api_key: str | None = None, + http_client=None) -> OpenAI: + base_url = check_base_url( + base_url if base_url is not None + else os.environ.get(BASE_URL_ENV) or DEFAULT_BASE_URL + ) + api_key = api_key if api_key is not None else os.environ[API_KEY_ENV] + return OpenAI(base_url=base_url, api_key=api_key, http_client=http_client) + + +def setup_tracing(project_name: str | None = None): + provider = register( + project_name=project_name or "baseten-openai", + project_type=ProjectType.OBSERVE, + set_global_tracer_provider=False, + verbose=False, + ) + OpenAIInstrumentor().instrument(tracer_provider=provider) + return provider + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--stream", action="store_true") + parser.add_argument("--prompt", default="What is the capital of France?") + parser.add_argument("--model", help=f"Model slug; otherwise set {MODEL_ENV}") + args = parser.parse_args(argv) + try: + base_url = check_base_url(os.environ.get(BASE_URL_ENV) or DEFAULT_BASE_URL) + model = args.model or os.environ[MODEL_ENV] + api_key = os.environ[API_KEY_ENV] + except ValueError as error: + print(str(error), file=sys.stderr) + return 2 + except KeyError as error: + print(f"Set {error.args[0]} before running this example.", file=sys.stderr) + return 2 + + provider = setup_tracing() + try: + with make_client(base_url=base_url, api_key=api_key) as client: + response = client.chat.completions.create( + model=model, messages=[{"role": "user", "content": args.prompt}], + stream=args.stream, + ) + if args.stream: + for chunk in response: + if chunk.choices: + print(chunk.choices[0].delta.content or "", end="", flush=True) + print() + else: + print(response.choices[0].message.content or "") + finally: + provider.force_flush() + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/python/examples/baseten/tests/_fake_openai.py b/python/examples/baseten/tests/_fake_openai.py new file mode 100644 index 000000000..de2c4b9fa --- /dev/null +++ b/python/examples/baseten/tests/_fake_openai.py @@ -0,0 +1,99 @@ +"""OpenAI-shaped fixtures and a loopback server; never forwards requests.""" + +import copy +import json +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +ANSWER = "Paris is the capital of France." +STREAM_PARTS = ("Paris is ", "the capital ", "of France.") +USAGE = {"prompt_tokens": 9, "completion_tokens": 7, "total_tokens": 16} +ERROR = {"error": {"message": "Invalid API key", "type": "authentication_error"}} + + +def completion(model, *, usage=True): + body = { + "id": "chatcmpl-baseten-fixture", "object": "chat.completion", + "created": 1, "model": model, + "choices": [{"index": 0, "message": {"role": "assistant", "content": ANSWER}, + "finish_reason": "stop"}], + } + if usage: + body["usage"] = dict(USAGE) + return body + + +def stream_body(model): + chunks = [] + for index, text in enumerate(STREAM_PARTS): + delta = {"content": text} + if index == 0: + delta["role"] = "assistant" + chunks.append({ + "id": "chatcmpl-baseten-fixture", "object": "chat.completion.chunk", + "created": 1, "model": model, + "choices": [{"index": 0, "delta": delta, "finish_reason": None}], + }) + chunks.append({ + "id": "chatcmpl-baseten-fixture", "object": "chat.completion.chunk", + "created": 1, "model": model, + "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}], + }) + return ("".join(f"data: {json.dumps(chunk)}\n\n" for chunk in chunks) + + "data: [DONE]\n\n").encode() + + +class FakeOpenAI: + def __init__(self): + self._requests = [] + self._lock = threading.Lock() + owner = self + + class Handler(BaseHTTPRequestHandler): + def do_POST(self): + body = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) + with owner._lock: + owner._requests.append({ + "path": self.path, + "headers": {key.lower(): value for key, value in self.headers.items()}, + "body": body, + }) + if self.path != "/v1/chat/completions": + self.send_error(404) + return + unauthorized = self.headers.get("Authorization") != ( + "Bearer placeholder-baseten-key" + ) + is_stream = body.get("stream", False) and not unauthorized + if unauthorized: + payload = json.dumps(ERROR).encode() + elif is_stream: + payload = stream_body(body["model"]) + else: + payload = json.dumps(completion(body["model"])).encode() + self.send_response(401 if unauthorized else 200) + self.send_header("Content-Type", "text/event-stream" if is_stream + else "application/json") + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + + def log_message(self, _format, *args): + pass + + self._server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + self.base_url = f"http://127.0.0.1:{self._server.server_port}/v1" + self._thread = threading.Thread(target=self._server.serve_forever, daemon=True) + self._thread.start() + + def __enter__(self): + return self + + def __exit__(self, *_args): + self._server.shutdown() + self._server.server_close() + self._thread.join() + + def requests(self): + with self._lock: + return copy.deepcopy(self._requests) diff --git a/python/examples/baseten/tests/loopback_guard/sitecustomize.py b/python/examples/baseten/tests/loopback_guard/sitecustomize.py new file mode 100644 index 000000000..ebda0202e --- /dev/null +++ b/python/examples/baseten/tests/loopback_guard/sitecustomize.py @@ -0,0 +1,42 @@ +"""Installed at Python startup in tests to reject external connections before DNS.""" + +import os +import socket +from pathlib import Path + +_connect = socket.socket.connect +_connect_ex = socket.socket.connect_ex +_getaddrinfo = socket.getaddrinfo + + +def _check(host): + if isinstance(host, bytes): + host = host.decode("ascii", errors="replace") + if host not in ("127.0.0.1", "localhost"): + log = os.environ.get("LOOPBACK_GUARD_LOG") + if log: + with open(log, "a", encoding="utf-8") as output: + output.write("blocked non-loopback connection\n") + raise OSError("loopback guard refused a non-loopback host before DNS") + + +def _guard_connect(self, address): + _check(address[0] if isinstance(address, tuple) else address) + return _connect(self, address) + + +def _guard_connect_ex(self, address): + _check(address[0] if isinstance(address, tuple) else address) + return _connect_ex(self, address) + + +def _guard_getaddrinfo(host, *args, **kwargs): + _check(host) + return _getaddrinfo(host, *args, **kwargs) + + +socket.socket.connect = _guard_connect +socket.socket.connect_ex = _guard_connect_ex +socket.getaddrinfo = _guard_getaddrinfo +if ready := os.environ.get("LOOPBACK_GUARD_READY"): + Path(ready).write_text("loopback guard installed\n", encoding="utf-8") diff --git a/python/examples/baseten/tests/test_baseten_recipe.py b/python/examples/baseten/tests/test_baseten_recipe.py new file mode 100644 index 000000000..ed3215b9f --- /dev/null +++ b/python/examples/baseten/tests/test_baseten_recipe.py @@ -0,0 +1,465 @@ +"""Offline contract tests for the Baseten recipe and its exported OTLP spans.""" + +import inspect +import json +import os +import re +import socket +import subprocess +import sys +import uuid +from pathlib import Path + +import fi_instrumentation +import httpx +import openai +import pytest +import traceai_openai +from fi_instrumentation import using_session +from harness import Receiver +from traceai_openai import OpenAIInstrumentor + +import app +from _fake_openai import ANSWER, ERROR, STREAM_PARTS, USAGE, FakeOpenAI, completion, stream_body + +RECIPE = Path(__file__).resolve().parents[1] +ROOT = RECIPE.parents[2] +MODEL = "zai-org/GLM-5.2" +VENDOR_KEY = "placeholder-baseten-key" +FI_KEY = "placeholder-futureagi-api-key" +FI_SECRET = "placeholder-futureagi-secret-key" +DOCUMENTED_URL = "https://inference.baseten.co/v1" +REFUSED = ( + ("https://inference.baseten.co", "Anthropic Messages beta"), + ("https://inference.baseten.co/", "use https://inference.baseten.co/v1"), + ("https://model-abc123.api.baseten.co/environments/production/sync/v1", + "dedicated deployments"), +) +TEST_COMMAND = """env -u PYTHONPATH PYTHONDONTWRITEBYTECODE=1 \\ + PYTHONPATH="python/examples/baseten/src:python:python/frameworks/openai:python/tests" \\ + uv run --no-project --python 3.11 \\ + --with 'openai==3.24.0' --with httpx --with 'wrapt<2' \\ + --with opentelemetry-api --with opentelemetry-sdk --with opentelemetry-instrumentation \\ + --with opentelemetry-exporter-otlp-proto-http --with requests --with protobuf \\ + --with opentelemetry-proto --with pydantic --with jsonschema --with pytest \\ + pytest python/examples/baseten/tests -q -p no:cacheprovider --noconftest -o addopts= -rfEs""" + + +@pytest.fixture(autouse=True) +def offline_environment(monkeypatch): + """Protect the pytest process too; MockTransport never needs DNS.""" + original_connect = socket.socket.connect + original_connect_ex = socket.socket.connect_ex + original_getaddrinfo = socket.getaddrinfo + + def check(host): + assert host in ("127.0.0.1", "localhost", b"127.0.0.1", b"localhost"), ( + "tests must not resolve or connect to an external host" + ) + + def connect(sock, address): + check(address[0]) + return original_connect(sock, address) + + def connect_ex(sock, address): + check(address[0]) + return original_connect_ex(sock, address) + + def getaddrinfo(host, *args, **kwargs): + check(host) + return original_getaddrinfo(host, *args, **kwargs) + + monkeypatch.setattr(socket.socket, "connect", connect) + monkeypatch.setattr(socket.socket, "connect_ex", connect_ex) + monkeypatch.setattr(socket, "getaddrinfo", getaddrinfo) + # Remove inherited masking, SDK metadata, collector tuning, and proxies. + for name in tuple(os.environ): + if (name.startswith(("FI_", "OPENAI_", "OTEL_", "BASETEN_")) + or name.lower() in ("http_proxy", "https_proxy", "all_proxy", "no_proxy")): + monkeypatch.delenv(name) + monkeypatch.setenv("NO_PROXY", "*") + monkeypatch.setenv("FI_API_KEY", FI_KEY) + monkeypatch.setenv("FI_SECRET_KEY", FI_SECRET) + monkeypatch.setenv(app.API_KEY_ENV, VENDOR_KEY) + monkeypatch.setenv(app.MODEL_ENV, MODEL) + monkeypatch.setenv("PYTHONDONTWRITEBYTECODE", "1") + + +@pytest.fixture +def receiver(monkeypatch): + with Receiver() as sink: + monkeypatch.setenv("FI_BASE_URL", sink.origin) + yield sink + + +@pytest.fixture +def start_tracing(receiver, request): + providers = [] + + def start(): + project = f"baseten-{request.node.name}-{uuid.uuid4().hex}" + provider = app.setup_tracing(project) + providers.append(provider) + return provider, project + + yield start + instrumentor = OpenAIInstrumentor() + if instrumentor.is_instrumented_by_opentelemetry: + instrumentor.uninstrument() + for provider in providers: + provider.force_flush() + provider.shutdown() + + +def attributes(span): + return {item["key"]: next(iter(item["value"].values())) + for item in span.get("attributes", [])} + + +def assert_no_vendor_key(receiver): + assert VENDOR_KEY not in json.dumps(receiver.spans()) + assert VENDOR_KEY not in json.dumps(receiver.requests()) + + +def one_span(receiver, project=None, model=MODEL): + spans = receiver.spans() + assert len(spans) == 1 + span = spans[0] + attrs = attributes(span) + assert span["name"] == "ChatCompletion" + assert attrs["gen_ai.span.kind"] == "LLM" + assert attrs["gen_ai.provider.name"] == "openai" + requests = receiver.requests() + assert len(requests) == 1 + export = requests[0] + assert export["path"] == "/tracer/v1/traces" + assert export["headers"]["x-api-key"] == FI_KEY + assert export["headers"]["x-secret-key"] == FI_SECRET + if project is not None: + assert export["resource_attributes"][0]["project_name"] == project + assert export["resource_attributes"][0]["project_type"] == "observe" + assert_no_vendor_key(receiver) + if model is None: + # traceai-openai records the model only from a non-streamed response; + # flip this when the instrumentor records the request model + assert "gen_ai.request.model" not in attrs + else: + assert attrs["gen_ai.request.model"] == model + return span, attrs + + +def chat(client, **kwargs): + return client.chat.completions.create( + model=MODEL, messages=[{"role": "user", "content": "Hello fixture"}], **kwargs + ) + + +def mock_client(handler): + return app.make_client(http_client=httpx.Client( + transport=httpx.MockTransport(handler), trust_env=False + )) + + +def subprocess_env(tmp_path): + # Follow the imported packages so wheel tests cannot fall back to repo sources. + paths = [ + str(RECIPE / "tests" / "loopback_guard"), + str(RECIPE / "src"), + str(ROOT / "python" / "tests"), + str(Path(fi_instrumentation.__file__).resolve().parent.parent), + str(Path(traceai_openai.__file__).resolve().parent.parent), + ] + child_env = os.environ.copy() + child_env.update({ + "PYTHONPATH": os.pathsep.join(dict.fromkeys(paths)), + "PYTHONDONTWRITEBYTECODE": "1", + "LOOPBACK_GUARD_LOG": str(tmp_path / "guard.log"), + "LOOPBACK_GUARD_READY": str(tmp_path / "guard.ready"), + }) + (tmp_path / "guard.log").write_text("", encoding="utf-8") + return child_env + + +def test_documented_base_url(): + assert app.DEFAULT_BASE_URL == DOCUMENTED_URL + assert DOCUMENTED_URL in (RECIPE / "README.md").read_text() + with app.make_client() as client: + assert str(client.base_url) == DOCUMENTED_URL + "/" + + +def test_client_overrides(monkeypatch): + monkeypatch.setenv(app.BASE_URL_ENV, "http://localhost:1234/proxy") + with app.make_client() as client: + assert str(client.base_url) == "http://localhost:1234/proxy/" + with app.make_client(base_url="http://127.0.0.1:1235/v1", + api_key="placeholder-explicit-key") as client: + assert str(client.base_url) == "http://127.0.0.1:1235/v1/" + assert client.api_key == "placeholder-explicit-key" + monkeypatch.setenv(app.BASE_URL_ENV, "") + with app.make_client() as client: + assert str(client.base_url) == DOCUMENTED_URL + "/" + + +def test_missing_vendor_key_names_variable(monkeypatch): + monkeypatch.delenv(app.API_KEY_ENV) + with pytest.raises(KeyError, match=app.API_KEY_ENV): + app.make_client() + + +def test_chat_export_and_key_separation(receiver, start_tracing): + provider, project = start_tracing() + seen = [] + + def handler(request): + seen.append(request) + return httpx.Response(200, json=completion(json.loads(request.content)["model"])) + + with mock_client(handler) as client: + assert chat(client).choices[0].message.content == ANSWER + assert provider.force_flush() + assert len(seen) == 1 + request = seen[0] + assert str(request.url) == DOCUMENTED_URL + "/chat/completions" + assert request.headers["authorization"] == "Bearer " + VENDOR_KEY + assert "x-api-key" not in request.headers + assert "x-secret-key" not in request.headers + assert FI_KEY not in str(request.headers) + assert FI_SECRET not in str(request.headers) + assert json.loads(request.content)["model"] == MODEL + _, attrs = one_span(receiver, project) + assert attrs["gen_ai.usage.input_tokens"] == str(USAGE["prompt_tokens"]) + assert attrs["gen_ai.usage.output_tokens"] == str(USAGE["completion_tokens"]) + assert attrs["gen_ai.usage.total_tokens"] == str(USAGE["total_tokens"]) + + +def test_usage_absent_is_omitted(receiver, start_tracing): + provider, project = start_tracing() + with mock_client(lambda _: httpx.Response(200, json=completion(MODEL, usage=False))) as client: + chat(client) + assert provider.force_flush() + _, attrs = one_span(receiver, project) + assert not any(key.startswith("gen_ai.usage.") for key in attrs) + + +def test_stream_accumulates_output(receiver, start_tracing): + provider, project = start_tracing() + with mock_client(lambda _: httpx.Response( + 200, headers={"content-type": "text/event-stream"}, content=stream_body(MODEL) + )) as client: + text = "".join(chunk.choices[0].delta.content or "" for chunk in chat(client, stream=True)) + assert text == "".join(STREAM_PARTS) == ANSWER + assert provider.force_flush() + _, attrs = one_span(receiver, project, model=None) + assert text in attrs["output.value"] + assert not any(key.startswith("gen_ai.usage.") for key in attrs) + + +def test_authentication_error_is_exported(receiver, start_tracing): + provider, project = start_tracing() + with mock_client(lambda _: httpx.Response(401, json=ERROR)) as client: + with pytest.raises(openai.AuthenticationError, match="Invalid API key"): + chat(client) + assert provider.force_flush() + span, _ = one_span(receiver, project, model=None) + assert span["status"]["code"] == "STATUS_CODE_ERROR" + exceptions = [event for event in span["events"] if event["name"] == "exception"] + assert exceptions + assert "Invalid API key" in json.dumps(exceptions) + assert VENDOR_KEY not in json.dumps(span["status"]) + assert VENDOR_KEY not in json.dumps(exceptions) + + +@pytest.mark.parametrize("hidden", [False, True], ids=["control", "hidden"]) +def test_hide_inputs_has_control(hidden, monkeypatch, receiver, start_tracing): + marker = "unique-baseten-private-prompt-marker" + monkeypatch.setenv("FI_HIDE_INPUTS", str(hidden).lower()) + provider, project = start_tracing() + seen = [] + + def handler(request): + seen.append(json.loads(request.content)) + return httpx.Response(200, json=completion(MODEL)) + + with mock_client(handler) as client: + client.chat.completions.create( + model=MODEL, messages=[{"role": "user", "content": marker}] + ) + assert provider.force_flush() + span, _ = one_span(receiver, project) + assert (marker in json.dumps(span)) is (not hidden) + assert seen[0]["messages"][0]["content"] == marker + + +def test_hide_outputs(monkeypatch, receiver, start_tracing): + monkeypatch.setenv("FI_HIDE_OUTPUTS", "true") + provider, project = start_tracing() + with mock_client(lambda _: httpx.Response(200, json=completion(MODEL))) as client: + assert chat(client).choices[0].message.content == ANSWER + assert provider.force_flush() + span, _ = one_span(receiver, project) + assert ANSWER not in json.dumps(span) + + +def test_session_context_is_independent_of_routing_header(receiver, start_tracing): + provider, project = start_tracing() + seen = [] + + def handler(request): + seen.append(request) + return httpx.Response(200, json=completion(MODEL)) + + with mock_client(handler) as client: + chat(client, extra_headers={"x-session-affinity": "routing-1"}) + assert provider.force_flush() + _, attrs = one_span(receiver, project) + assert "session.id" not in attrs + receiver.clear() + with using_session("s-1"): + chat(client, extra_headers={"x-session-affinity": "routing-1"}) + assert provider.force_flush() + _, attrs = one_span(receiver, project) + assert attrs["session.id"] == "s-1" + assert len(seen) == 2 + assert all(request.headers["x-session-affinity"] == "routing-1" for request in seen) + assert all("s-1" not in str(request.headers) for request in seen) + + +@pytest.mark.parametrize("stream", [False, True], ids=["chat", "stream"]) +def test_app_subprocess_loopback(stream, receiver, tmp_path): + with FakeOpenAI() as fake: + child_env = subprocess_env(tmp_path) + child_env[app.BASE_URL_ENV] = fake.base_url + command = [sys.executable, str(RECIPE / "src" / "app.py")] + if stream: + command.append("--stream") + result = subprocess.run(command, env=child_env, cwd=ROOT, capture_output=True, + text=True, timeout=30) + assert result.returncode == 0, result.stderr + assert result.stdout.strip() == ANSWER + assert VENDOR_KEY not in result.stdout + result.stderr + _, attrs = one_span(receiver, "baseten-openai", model=None if stream else MODEL) + if stream: + assert ANSWER in attrs["output.value"] + assert not any(key.startswith("gen_ai.usage.") for key in attrs) + requests = fake.requests() + assert len(requests) == 1 + assert requests[0]["path"] == "/v1/chat/completions" + assert requests[0]["headers"]["authorization"] == "Bearer " + VENDOR_KEY + assert "x-api-key" not in requests[0]["headers"] + assert "x-secret-key" not in requests[0]["headers"] + assert requests[0]["body"]["model"] == MODEL + assert requests[0]["body"]["stream"] is stream + assert (tmp_path / "guard.log").read_text() == "" + assert (tmp_path / "guard.ready").read_text() == "loopback guard installed\n" + + +@pytest.mark.parametrize("operation", ["connect", "connect_ex", "getaddrinfo"]) +def test_guard_refuses_provider_before_dns(operation, tmp_path): + code = { + "connect": "socket.socket().connect(('inference.baseten.co', 443))", + "connect_ex": "socket.socket().connect_ex(('inference.baseten.co', 443))", + "getaddrinfo": "socket.getaddrinfo('inference.baseten.co', 443)", + }[operation] + result = subprocess.run([sys.executable, "-c", "import socket; " + code], + env=subprocess_env(tmp_path), cwd=ROOT, capture_output=True, + text=True, timeout=10) + assert result.returncode != 0 + assert "loopback guard refused a non-loopback host before DNS" in result.stderr + assert (tmp_path / "guard.log").read_text() == "blocked non-loopback connection\n" + assert (tmp_path / "guard.ready").is_file() + + +@pytest.mark.parametrize("url", [ + DOCUMENTED_URL, DOCUMENTED_URL + "/", "http://127.0.0.1:1234/v1", + "http://localhost:1234/custom/path/", "https://customer.example/proxy/v1?route=one", + "https://model-abc123.api.baseten.co.customer.example/v1", +]) +def test_allowed_base_urls_are_unchanged(url): + assert app.check_base_url(url) == url + + +@pytest.mark.parametrize("url,reason", REFUSED) +def test_out_of_scope_urls_are_refused(url, reason): + with pytest.raises(ValueError, match=re.escape(reason)) as raised: + app.check_base_url(url) + assert "\n" not in str(raised.value) + with pytest.raises(ValueError, match=re.escape(reason)): + app.make_client(base_url=url) + + +@pytest.mark.parametrize("url,reason", REFUSED) +def test_main_rejects_url_before_tracing(url, reason, monkeypatch, receiver, capsys): + def unexpected_tracing(*_args, **_kwargs): + pytest.fail("a refused URL must fail before tracing setup") + + monkeypatch.setattr(app, "setup_tracing", unexpected_tracing) + monkeypatch.setenv(app.BASE_URL_ENV, url) + with FakeOpenAI() as fake: + assert app.main([]) == 2 + assert fake.requests() == [] + output = capsys.readouterr() + assert reason in output.err + assert output.out == "" + assert VENDOR_KEY not in output.err + assert receiver.spans() == [] + assert receiver.requests() == [] + + +@pytest.mark.parametrize("variable", [app.MODEL_ENV, app.API_KEY_ENV]) +def test_main_missing_configuration_names_variable(variable, monkeypatch, capsys): + monkeypatch.delenv(variable) + monkeypatch.setattr(app, "setup_tracing", lambda: pytest.fail("missing configuration")) + assert app.main([]) == 2 + output = capsys.readouterr() + assert variable in output.err + assert VENDOR_KEY not in output.err + + +def test_model_and_prompt_cli_override(monkeypatch, receiver, tmp_path): + with FakeOpenAI() as fake: + child_env = subprocess_env(tmp_path) + child_env[app.BASE_URL_ENV] = fake.base_url + child_env.pop(app.MODEL_ENV) + result = subprocess.run( + [sys.executable, str(RECIPE / "src" / "app.py"), "--model", MODEL, + "--prompt", "A harmless custom prompt"], + env=child_env, cwd=ROOT, capture_output=True, text=True, timeout=30, + ) + assert result.returncode == 0, result.stderr + assert result.stdout.strip() == ANSWER + assert fake.requests()[0]["body"]["messages"][0]["content"] == "A harmless custom prompt" + one_span(receiver, "baseten-openai") + assert (tmp_path / "guard.log").read_text() == "" + assert (tmp_path / "guard.ready").is_file() + + +def test_readme_contract_and_pins(): + readme = (RECIPE / "README.md").read_text() + assert "# Baseten (OpenAI-compatible) with traceAI" in readme + headings = ["Install", "Configure", "Run", "Code", "What you see in Future AGI", + "Provider specifics", "Privacy", "Limits / not covered", "Tests"] + positions = [readme.index("## " + heading + "\n") for heading in headings] + assert positions == sorted(positions) + assert DOCUMENTED_URL in readme + assert "provider field says `openai`" in readme + assert "model id the provider returns" in readme + assert "no model attribute" in readme + assert "no `gen_ai.usage.*` attributes" in readme + assert "current `traceai-openai` behaviour" in readme + for variable in (app.API_KEY_ENV, app.BASE_URL_ENV, app.MODEL_ENV, "FI_API_KEY", + "FI_SECRET_KEY", "FI_BASE_URL", "FI_HIDE_INPUTS", "FI_HIDE_OUTPUTS"): + assert variable in readme + # Literal environment lookups in the app must also be documented. + for variable in re.findall(r'os\.environ(?:\.get\(|\[)["\']([^"\']+)', inspect.getsource(app)): + assert variable in readme + assert TEST_COMMAND in readme + assert "x-session-affinity" in readme + assert "NOT a Future AGI" in readme + assert "using_session" in readme + assert "1.69.0" in readme + assert "not tested against the live provider" in readme.lower() + assert not re.search(r"\b(?:sk-(?:proj-)?|fi-)[A-Za-z0-9_]{20,}\b", readme) + pins = (RECIPE / "requirements.txt").read_text().splitlines() + assert pins[0].startswith("#") + assert pins[1:] == ["openai==3.24.0", "traceAI-openai==0.1.10", + "fi-instrumentation-otel==1.1.0"] From 34ed9de8daab7fde731450ac106859e7a5316d34 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Mon, 5 Oct 2026 18:27:08 -0700 Subject: [PATCH 2/7] fix(examples/baseten): review-r1 P3s (trailing-dot hosts, controls, README) From pr-reviewer t_6453b6cd (APPROVED, P3 follow-ups): - R1: check_base_url compares the host without an absolute-FQDN trailing dot, so https://inference.baseten.co. and model-*.api.baseten.co. are refused too (the URL is still returned/refused unchanged, never rewritten). Tests pin trailing-dot, upper-case, userinfo and port variants; the two trailing-dot cases failed before this change. - R2: test_hide_outputs gains a non-hidden control. - R3: README names gen_ai.request.model and notes the requested model stays in gen_ai.request.parameters on streamed and failed spans; tests pin it. - R4: requirements.txt comment no longer mentions a worktree. - R6: test_main_rejects_url_before_tracing replaces a vacuous fake-server check with a client-constructor spy. approved: Nikhil 2026-10-03 blanket Refs: TH-8310 --- python/examples/baseten/README.md | 10 ++++--- python/examples/baseten/requirements.txt | 2 +- python/examples/baseten/src/app.py | 4 ++- .../baseten/tests/test_baseten_recipe.py | 30 ++++++++++++++----- 4 files changed, 32 insertions(+), 14 deletions(-) diff --git a/python/examples/baseten/README.md b/python/examples/baseten/README.md index ef2273bc6..edeed7dd9 100644 --- a/python/examples/baseten/README.md +++ b/python/examples/baseten/README.md @@ -81,10 +81,12 @@ On normal calls, `gen_ai.request.model` is the model id the provider returns in its response. Token usage appears only when the response has `usage`; missing usage is omitted, never reported as zero. -Streamed calls record accumulated output text but have no model attribute. +Streamed calls record accumulated output text but have no `gen_ai.request.model` attribute. The default stream tested here returns no usage, so it exports no `gen_ai.usage.*` attributes. Failed calls produce an error span with an -exception event and no model attribute. These are current `traceai-openai` behaviour. +exception event and no `gen_ai.request.model` attribute. On both, the model you +requested is still visible inside the `gen_ai.request.parameters` JSON. These are +current `traceai-openai` behaviour. The provider field says `openai`: `gen_ai.provider.name` is `openai`. This is the shared instrumentor's label, including when the client calls Baseten. @@ -133,8 +135,8 @@ change Baseten's own logging. ## Tests -The tests pin the current absence of model attributes on streaming and error -spans. They also pin the absence of usage attributes on the default stream. +The tests pin the current absence of `gen_ai.request.model` on streaming and +error spans. They also pin the absence of usage attributes on the default stream. From the repository root: diff --git a/python/examples/baseten/requirements.txt b/python/examples/baseten/requirements.txt index 3a32ac062..0b7c2c8f4 100644 --- a/python/examples/baseten/requirements.txt +++ b/python/examples/baseten/requirements.txt @@ -1,4 +1,4 @@ -# Recipe pins; tested against the matching traceAI sources in this worktree. +# Tested pins (see the README for the full version matrix). openai==3.24.0 traceAI-openai==0.1.10 fi-instrumentation-otel==1.1.0 diff --git a/python/examples/baseten/src/app.py b/python/examples/baseten/src/app.py index 9b2c0b0f6..bfe728333 100644 --- a/python/examples/baseten/src/app.py +++ b/python/examples/baseten/src/app.py @@ -20,7 +20,9 @@ def check_base_url(url: str) -> str: """Reject the two Baseten surfaces that need a different recipe.""" parsed = urlsplit(url) - host = parsed.hostname or "" + # Compare the host without an absolute-FQDN trailing dot; the URL itself is + # returned unchanged. + host = (parsed.hostname or "").rstrip(".") if host == "inference.baseten.co" and parsed.path in ("", "/"): raise ValueError( "Baseten's root is the Anthropic Messages beta endpoint; " diff --git a/python/examples/baseten/tests/test_baseten_recipe.py b/python/examples/baseten/tests/test_baseten_recipe.py index ed3215b9f..c25a0f499 100644 --- a/python/examples/baseten/tests/test_baseten_recipe.py +++ b/python/examples/baseten/tests/test_baseten_recipe.py @@ -34,6 +34,12 @@ ("https://inference.baseten.co/", "use https://inference.baseten.co/v1"), ("https://model-abc123.api.baseten.co/environments/production/sync/v1", "dedicated deployments"), + # Host spelling variants must not bypass the refusals. + ("https://inference.baseten.co.", "Anthropic Messages beta"), + ("https://INFERENCE.Baseten.CO/", "Anthropic Messages beta"), + ("https://user@inference.baseten.co:443", "Anthropic Messages beta"), + ("https://model-abc123.api.baseten.co./v1", "dedicated deployments"), + ("https://MODEL-abc123.API.baseten.co:8443/v1", "dedicated deployments"), ) TEST_COMMAND = """env -u PYTHONPATH PYTHONDONTWRITEBYTECODE=1 \\ PYTHONPATH="python/examples/baseten/src:python:python/frameworks/openai:python/tests" \\ @@ -252,6 +258,8 @@ def test_stream_accumulates_output(receiver, start_tracing): _, attrs = one_span(receiver, project, model=None) assert text in attrs["output.value"] assert not any(key.startswith("gen_ai.usage.") for key in attrs) + # The requested model is still exported inside the request parameters. + assert json.loads(attrs["gen_ai.request.parameters"])["model"] == MODEL def test_authentication_error_is_exported(receiver, start_tracing): @@ -260,7 +268,8 @@ def test_authentication_error_is_exported(receiver, start_tracing): with pytest.raises(openai.AuthenticationError, match="Invalid API key"): chat(client) assert provider.force_flush() - span, _ = one_span(receiver, project, model=None) + span, attrs = one_span(receiver, project, model=None) + assert json.loads(attrs["gen_ai.request.parameters"])["model"] == MODEL assert span["status"]["code"] == "STATUS_CODE_ERROR" exceptions = [event for event in span["events"] if event["name"] == "exception"] assert exceptions @@ -290,14 +299,15 @@ def handler(request): assert seen[0]["messages"][0]["content"] == marker -def test_hide_outputs(monkeypatch, receiver, start_tracing): - monkeypatch.setenv("FI_HIDE_OUTPUTS", "true") +@pytest.mark.parametrize("hidden", [False, True], ids=["control", "hidden"]) +def test_hide_outputs(hidden, monkeypatch, receiver, start_tracing): + monkeypatch.setenv("FI_HIDE_OUTPUTS", str(hidden).lower()) provider, project = start_tracing() with mock_client(lambda _: httpx.Response(200, json=completion(MODEL))) as client: assert chat(client).choices[0].message.content == ANSWER assert provider.force_flush() span, _ = one_span(receiver, project) - assert ANSWER not in json.dumps(span) + assert (ANSWER in json.dumps(span)) is (not hidden) def test_session_context_is_independent_of_routing_header(receiver, start_tracing): @@ -392,11 +402,14 @@ def test_main_rejects_url_before_tracing(url, reason, monkeypatch, receiver, cap def unexpected_tracing(*_args, **_kwargs): pytest.fail("a refused URL must fail before tracing setup") + def unexpected_client(*_args, **_kwargs): + pytest.fail("a refused URL must fail before a client is created") + monkeypatch.setattr(app, "setup_tracing", unexpected_tracing) + monkeypatch.setattr(app, "make_client", unexpected_client) + monkeypatch.setattr(app, "OpenAI", unexpected_client) monkeypatch.setenv(app.BASE_URL_ENV, url) - with FakeOpenAI() as fake: - assert app.main([]) == 2 - assert fake.requests() == [] + assert app.main([]) == 2 output = capsys.readouterr() assert reason in output.err assert output.out == "" @@ -443,7 +456,8 @@ def test_readme_contract_and_pins(): assert DOCUMENTED_URL in readme assert "provider field says `openai`" in readme assert "model id the provider returns" in readme - assert "no model attribute" in readme + assert "no `gen_ai.request.model` attribute" in readme + assert "gen_ai.request.parameters" in readme assert "no `gen_ai.usage.*` attributes" in readme assert "current `traceai-openai` behaviour" in readme for variable in (app.API_KEY_ENV, app.BASE_URL_ENV, app.MODEL_ENV, "FI_API_KEY", From 0cd35c74f1858754e9ccecc74ae5a08002b6a216 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Sat, 10 Oct 2026 01:41:53 -0700 Subject: [PATCH 3/7] test(examples): pin a response model that differs from the request The chat fixture echoed the request model, so a green span could not show that traceai-openai stores completion.model. One installed-wheel case now returns a synthetic response id, asserts the OpenAI chat.completion shape, and checks the OTLP span stores that id while the request body keeps the request id. No live vendor call. Host rules are unchanged. RED: response.model equalled the request id. GREEN: the new case passes on the installed traceAI-openai 0.1.10 wheel. approved: Nikhil 2026-10-03 blanket Refs: TH-8310 --- .../baseten/tests/test_baseten_recipe.py | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/python/examples/baseten/tests/test_baseten_recipe.py b/python/examples/baseten/tests/test_baseten_recipe.py index c25a0f499..c0049be8c 100644 --- a/python/examples/baseten/tests/test_baseten_recipe.py +++ b/python/examples/baseten/tests/test_baseten_recipe.py @@ -25,6 +25,9 @@ RECIPE = Path(__file__).resolve().parents[1] ROOT = RECIPE.parents[2] MODEL = "zai-org/GLM-5.2" +# Synthetic. Not a documented Baseten model id. It exists so the span can be +# told apart from the request slug. No live Baseten call is made. +RESPONSE_MODEL = "fixture-served/not-a-live-baseten-model" VENDOR_KEY = "placeholder-baseten-key" FI_KEY = "placeholder-futureagi-api-key" FI_SECRET = "placeholder-futureagi-secret-key" @@ -238,6 +241,30 @@ def handler(request): assert attrs["gen_ai.usage.total_tokens"] == str(USAGE["total_tokens"]) +def test_response_model_differs_from_request(receiver, start_tracing): + """Span model is the response field, not the request slug.""" + assert RESPONSE_MODEL != MODEL + provider, project = start_tracing() + seen = [] + + def handler(request): + seen.append(request) + return httpx.Response(200, json=completion(RESPONSE_MODEL)) + + with mock_client(handler) as client: + response = chat(client) + assert response.model == RESPONSE_MODEL + assert provider.force_flush() + assert json.loads(seen[0].content)["model"] == MODEL + _, attrs = one_span(receiver, project, model=RESPONSE_MODEL) + assert attrs["gen_ai.request.model"] != MODEL + body = completion(RESPONSE_MODEL) + assert body["object"] == "chat.completion" + assert body["model"] == RESPONSE_MODEL + assert set(body["usage"]) == {"prompt_tokens", "completion_tokens", "total_tokens"} + assert body["choices"][0]["message"]["content"] == ANSWER + + def test_usage_absent_is_omitted(receiver, start_tracing): provider, project = start_tracing() with mock_client(lambda _: httpx.Response(200, json=completion(MODEL, usage=False))) as client: From 68fe7bc81446bdd6864f166a7d2db34cb38d160b Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Sun, 11 Oct 2026 21:32:19 -0700 Subject: [PATCH 4/7] test(examples/baseten): accept traceai-openai before and after the provider/model fix The recipe pinned traceai-openai 0.1.10 behaviour: gen_ai.provider.name is always "openai" and streamed/failed spans carry no gen_ai.request.model. future-agi/traceAI#249 labels known vendor hosts and records the requested model, so these pins would break whichever PR merged second. Accept both behaviours (and the related Responses/embedding output changes where this recipe checks them), and say in the README which version does what. Passes against published traceAI-openai 0.1.10 and against #249. Co-Authored-By: Claude Opus 5.5 --- python/examples/baseten/README.md | 14 ++++++++------ .../examples/baseten/tests/test_baseten_recipe.py | 12 +++++++++--- 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/python/examples/baseten/README.md b/python/examples/baseten/README.md index edeed7dd9..0cdb43f0a 100644 --- a/python/examples/baseten/README.md +++ b/python/examples/baseten/README.md @@ -81,15 +81,17 @@ On normal calls, `gen_ai.request.model` is the model id the provider returns in its response. Token usage appears only when the response has `usage`; missing usage is omitted, never reported as zero. -Streamed calls record accumulated output text but have no `gen_ai.request.model` attribute. +With `traceai-openai` 0.1.10 and earlier, streamed calls record accumulated output text but have no `gen_ai.request.model` attribute. The default stream tested here returns no usage, so it exports no `gen_ai.usage.*` attributes. Failed calls produce an error span with an exception event and no `gen_ai.request.model` attribute. On both, the model you requested is still visible inside the `gen_ai.request.parameters` JSON. These are -current `traceai-openai` behaviour. +current `traceai-openai` behaviour up to 0.1.10; later versions (future-agi/traceAI#249) +record the requested model on streamed and failed calls. -The provider field says `openai`: `gen_ai.provider.name` is `openai`. This is -the shared instrumentor's label, including when the client calls Baseten. +Up to `traceai-openai` 0.1.10, the provider field says `openai`: `gen_ai.provider.name` is `openai`. This is +the shared instrumentor's label, including when the client calls Baseten. Later versions +(future-agi/traceAI#249) say `baseten` for `inference.baseten.co`; other base URLs keep `openai`. ## Provider specifics @@ -135,8 +137,8 @@ change Baseten's own logging. ## Tests -The tests pin the current absence of `gen_ai.request.model` on streaming and -error spans. They also pin the absence of usage attributes on the default stream. +The tests accept both `traceai-openai` behaviours for the provider label and for +`gen_ai.request.model` on streaming and error spans (absent up to 0.1.10, the requested model later). They also pin the absence of usage attributes on the default stream. From the repository root: diff --git a/python/examples/baseten/tests/test_baseten_recipe.py b/python/examples/baseten/tests/test_baseten_recipe.py index c0049be8c..09c135e74 100644 --- a/python/examples/baseten/tests/test_baseten_recipe.py +++ b/python/examples/baseten/tests/test_baseten_recipe.py @@ -22,6 +22,12 @@ import app from _fake_openai import ANSWER, ERROR, STREAM_PARTS, USAGE, FakeOpenAI, completion, stream_body +# traceai-openai 0.1.10 and earlier label every span "openai"; later versions +# (future-agi/traceAI#249) use the vendor's name for its documented API host. +# They also omit gen_ai.request.model on streamed and failed calls, where later +# versions record the requested model. Accept both so this recipe passes on either version. +PROVIDER_NAMES = ("openai", "baseten") + RECIPE = Path(__file__).resolve().parents[1] ROOT = RECIPE.parents[2] MODEL = "zai-org/GLM-5.2" @@ -137,7 +143,7 @@ def one_span(receiver, project=None, model=MODEL): attrs = attributes(span) assert span["name"] == "ChatCompletion" assert attrs["gen_ai.span.kind"] == "LLM" - assert attrs["gen_ai.provider.name"] == "openai" + assert attrs["gen_ai.provider.name"] in PROVIDER_NAMES requests = receiver.requests() assert len(requests) == 1 export = requests[0] @@ -150,8 +156,8 @@ def one_span(receiver, project=None, model=MODEL): assert_no_vendor_key(receiver) if model is None: # traceai-openai records the model only from a non-streamed response; - # flip this when the instrumentor records the request model - assert "gen_ai.request.model" not in attrs + # 0.1.10 records the model only from non-streamed responses; later versions record the requested one. + assert attrs.get("gen_ai.request.model") in (None, MODEL) else: assert attrs["gen_ai.request.model"] == model return span, attrs From 7aac5c89e3928e010be1484a92d673ba1bd0a785 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Sun, 18 Oct 2026 21:38:16 -0700 Subject: [PATCH 5/7] fix(examples/baseten): honour FI_PROJECT_NAME setup_tracing() passed project_name or "baseten-openai" to register(), so the default was always truthy and FI_PROJECT_NAME was never read: `python src/app.py` always exported to baseten-openai. The Anannas recipe reads FI_PROJECT_NAME with its name as the default; do the same here. An explicit setup_tracing() argument still wins, and an unset or empty FI_PROJECT_NAME keeps baseten-openai (never register()'s generic default). - README: FI_PROJECT_NAME in the Configure table and the Run section; the Code snippet reads it the same way as src/app.py. - Tests: setup_tracing() default / empty / env / explicit cases; the guarded app.py subprocess runs with FI_PROJECT_NAME unset and set; the README contract checks the new row and snippet. Passes against repository source, published traceAI-openai 0.1.10 and future-agi/traceAI#249. Co-Authored-By: Claude Opus 5.5 --- python/examples/baseten/README.md | 7 ++-- python/examples/baseten/src/app.py | 2 +- .../baseten/tests/test_baseten_recipe.py | 34 +++++++++++++++++-- 3 files changed, 37 insertions(+), 6 deletions(-) diff --git a/python/examples/baseten/README.md b/python/examples/baseten/README.md index 0cdb43f0a..e48e75aba 100644 --- a/python/examples/baseten/README.md +++ b/python/examples/baseten/README.md @@ -28,6 +28,7 @@ pip install traceAI-openai fi-instrumentation-otel openai | `BASETEN_MODEL` | Required Model APIs slug; `--model` overrides it | Baseten request | | `BASETEN_BASE_URL` | Optional override; defaults to `https://inference.baseten.co/v1` | OpenAI client | | `FI_BASE_URL` | Optional Future AGI collector origin | Tracer only | +| `FI_PROJECT_NAME` | Optional Observe project name; defaults to `baseten-openai` | Tracer only | Choose a slug from Baseten's current [Model APIs table](https://docs.baseten.co/inference/model-apis/overview). `zai-org/GLM-5.2` is a documentation example, not a model availability guarantee. @@ -46,6 +47,8 @@ python src/app.py --stream ``` You can also pass `--model ` and `--prompt "What is the capital of France?"`. +Set `FI_PROJECT_NAME` to send the traces to another Observe project; unset or +empty, they go to `baseten-openai`. The app flushes traces before returning. The SDK appends a trailing slash to `client.base_url`: `https://inference.baseten.co/v1/`. Its request goes to `https://inference.baseten.co/v1/chat/completions`. @@ -61,8 +64,8 @@ from fi_instrumentation import register from fi_instrumentation.fi_types import ProjectType from openai import OpenAI from traceai_openai import OpenAIInstrumentor -provider = register(project_name="baseten-openai", project_type=ProjectType.OBSERVE, - set_global_tracer_provider=False, verbose=False) +provider = register(project_name=os.environ.get("FI_PROJECT_NAME") or "baseten-openai", # as in src/app.py + project_type=ProjectType.OBSERVE, set_global_tracer_provider=False, verbose=False) OpenAIInstrumentor().instrument(tracer_provider=provider) with OpenAI(base_url="https://inference.baseten.co/v1", api_key=os.environ["BASETEN_API_KEY"]) as client: diff --git a/python/examples/baseten/src/app.py b/python/examples/baseten/src/app.py index bfe728333..6584c5ac6 100644 --- a/python/examples/baseten/src/app.py +++ b/python/examples/baseten/src/app.py @@ -48,7 +48,7 @@ def make_client(base_url: str | None = None, api_key: str | None = None, def setup_tracing(project_name: str | None = None): provider = register( - project_name=project_name or "baseten-openai", + project_name=project_name or os.environ.get("FI_PROJECT_NAME") or "baseten-openai", project_type=ProjectType.OBSERVE, set_global_tracer_provider=False, verbose=False, diff --git a/python/examples/baseten/tests/test_baseten_recipe.py b/python/examples/baseten/tests/test_baseten_recipe.py index 09c135e74..52b906c24 100644 --- a/python/examples/baseten/tests/test_baseten_recipe.py +++ b/python/examples/baseten/tests/test_baseten_recipe.py @@ -271,6 +271,29 @@ def handler(request): assert body["choices"][0]["message"]["content"] == ANSWER +@pytest.mark.parametrize("env_project, explicit, expected", [ + (None, None, "baseten-openai"), + ("", None, "baseten-openai"), + ("baseten-from-env", None, "baseten-from-env"), + ("baseten-from-env", "baseten-explicit", "baseten-explicit"), +], ids=["default", "empty", "env", "explicit"]) +def test_setup_tracing_project_name(env_project, explicit, expected, monkeypatch, receiver): + # FI_PROJECT_NAME overrides the recipe default; an explicit argument wins over both. + # Never fall through to register()'s generic DEFAULT_PROJECT_NAME. + if env_project is not None: + monkeypatch.setenv("FI_PROJECT_NAME", env_project) + provider = app.setup_tracing(explicit) + try: + with mock_client(lambda request: httpx.Response(200, json=completion(MODEL))) as client: + chat(client) + assert provider.force_flush() + one_span(receiver, expected) + finally: + OpenAIInstrumentor().uninstrument() + provider.force_flush() + provider.shutdown() + + def test_usage_absent_is_omitted(receiver, start_tracing): provider, project = start_tracing() with mock_client(lambda _: httpx.Response(200, json=completion(MODEL, usage=False))) as client: @@ -367,11 +390,14 @@ def handler(request): assert all("s-1" not in str(request.headers) for request in seen) +@pytest.mark.parametrize("env_project", [None, "baseten-subprocess-env"], ids=["default-project", "env-project"]) @pytest.mark.parametrize("stream", [False, True], ids=["chat", "stream"]) -def test_app_subprocess_loopback(stream, receiver, tmp_path): +def test_app_subprocess_loopback(stream, env_project, receiver, tmp_path): with FakeOpenAI() as fake: child_env = subprocess_env(tmp_path) child_env[app.BASE_URL_ENV] = fake.base_url + if env_project: + child_env["FI_PROJECT_NAME"] = env_project command = [sys.executable, str(RECIPE / "src" / "app.py")] if stream: command.append("--stream") @@ -380,7 +406,7 @@ def test_app_subprocess_loopback(stream, receiver, tmp_path): assert result.returncode == 0, result.stderr assert result.stdout.strip() == ANSWER assert VENDOR_KEY not in result.stdout + result.stderr - _, attrs = one_span(receiver, "baseten-openai", model=None if stream else MODEL) + _, attrs = one_span(receiver, env_project or "baseten-openai", model=None if stream else MODEL) if stream: assert ANSWER in attrs["output.value"] assert not any(key.startswith("gen_ai.usage.") for key in attrs) @@ -494,8 +520,10 @@ def test_readme_contract_and_pins(): assert "no `gen_ai.usage.*` attributes" in readme assert "current `traceai-openai` behaviour" in readme for variable in (app.API_KEY_ENV, app.BASE_URL_ENV, app.MODEL_ENV, "FI_API_KEY", - "FI_SECRET_KEY", "FI_BASE_URL", "FI_HIDE_INPUTS", "FI_HIDE_OUTPUTS"): + "FI_SECRET_KEY", "FI_BASE_URL", "FI_PROJECT_NAME", "FI_HIDE_INPUTS", "FI_HIDE_OUTPUTS"): assert variable in readme + assert "| `FI_PROJECT_NAME` | Optional Observe project name; defaults to `baseten-openai` |" in readme + assert 'project_name=os.environ.get("FI_PROJECT_NAME") or "baseten-openai"' in readme # Literal environment lookups in the app must also be documented. for variable in re.findall(r'os\.environ(?:\.get\(|\[)["\']([^"\']+)', inspect.getsource(app)): assert variable in readme From c44b95f9735e0509c01158bec63f55cd00018861 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek <8347993+nik13@users.noreply.github.com> Date: Wed, 28 Oct 2026 01:03:12 -0700 Subject: [PATCH 6/7] fix(examples): align Baseten validation and CI with sibling recipes Reject malformed and non-base Baseten URLs before tracing, reject empty configuration, close streams, document token pricing and System One scope, and add Baseten to the three-cell recipe CI matrix. Co-Authored-By: Claude Code --- .../openai-compatible-recipe-tests.yml | 2 +- python/examples/baseten/README.md | 33 ++++++-- python/examples/baseten/src/app.py | 79 +++++++++++++------ .../baseten/tests/test_baseten_recipe.py | 79 ++++++++++++++++++- 4 files changed, 158 insertions(+), 35 deletions(-) diff --git a/.github/workflows/openai-compatible-recipe-tests.yml b/.github/workflows/openai-compatible-recipe-tests.yml index 0e2587e85..b82851e82 100644 --- a/.github/workflows/openai-compatible-recipe-tests.yml +++ b/.github/workflows/openai-compatible-recipe-tests.yml @@ -39,7 +39,7 @@ jobs: import json, os from pathlib import Path matrix = [] - for recipe in ('databricks', 'lepton', 'orcarouter', 'anannas-ai', 'doubleword'): + for recipe in ('databricks', 'lepton', 'orcarouter', 'anannas-ai', 'doubleword', 'baseten'): root = Path('python/examples') / recipe if not root.exists(): continue # Recipes may land independently. diff --git a/python/examples/baseten/README.md b/python/examples/baseten/README.md index e48e75aba..b1da0701b 100644 --- a/python/examples/baseten/README.md +++ b/python/examples/baseten/README.md @@ -6,7 +6,7 @@ not need a Baseten tracing package. ## Install -From this directory, install the tested pins: +From this directory, install the dependency pins: ```bash pip install -r requirements.txt @@ -30,8 +30,17 @@ pip install traceAI-openai fi-instrumentation-otel openai | `FI_BASE_URL` | Optional Future AGI collector origin | Tracer only | | `FI_PROJECT_NAME` | Optional Observe project name; defaults to `baseten-openai` | Tracer only | -Choose a slug from Baseten's current [Model APIs table](https://docs.baseten.co/inference/model-apis/overview). -`zai-org/GLM-5.2` is a documentation example, not a model availability guarantee. +Choose a language-model slug from Baseten's current [Model APIs catalog](https://docs.baseten.co/inference/model-apis/overview#list-available-models) +or `client.models.list()`. `zai-org/GLM-5.2` is listed on Baseten's +[pricing page](https://www.baseten.co/pricing/); availability depends on your account. +Model APIs use [per-token pricing](https://docs.baseten.co/inference/model-apis/pricing-and-limits#pricing). +A $0/month Basic plan is pay-as-you-go, not a promise of free inference. +The decision models `elephant-1` and `mammoth-1` use the separate +[System One API](https://docs.baseten.co/inference/model-apis/decision-models), +not this Chat Completions recipe. + +The app exits 2 for a missing or empty model or Baseten key, or an invalid base +URL, before tracing or network calls. It does not check the Future AGI keys itself. ## Run @@ -104,9 +113,15 @@ own per-deployment URL and are not covered. The app rejects hosts matching `https://model-abc123.api.baseten.co/environments/production/sync/v1`. The Anthropic Messages beta at `https://inference.baseten.co` uses a different -client and is not covered. The app rejects that root and tells you to use -`/v1` for the OpenAI SDK. It returns allowed URLs unchanged, including custom -proxy URLs; it does not rewrite rejected URLs. Baseten CLI is out of scope. +client and is not covered ([client configuration](https://docs.baseten.co/inference/model-apis/overview#migrate)). +The app rejects that root and tells you to use `/v1` for the OpenAI SDK. +On `inference.baseten.co`, the URL must use HTTPS and the path must be exactly +`/v1` or `/v1/`. Endpoint paths, repeated trailing slashes, dot segments, queries, +fragments, URL credentials, whitespace, control characters, and invalid host +spellings fail early. Uppercase hosts and a trailing DNS dot do not bypass the +checks. Customer proxies and loopback servers may use their own HTTP(S) paths; +proxy labels may contain internal underscores. Allowed URLs are returned +unchanged. Baseten CLI and the System One API are out of scope. `x-session-affinity` is a Baseten routing header. It is NOT a Future AGI session id; there is no bridge between them. For a Future AGI session, use @@ -136,7 +151,7 @@ change Baseten's own logging. - Anthropic SDK calls to Baseten and its Anthropic Messages beta. - Dedicated deployments and the Baseten CLI. - Historical import of past requests. -- Not tested against the live provider (no paid call); the tests use a local fake of the OpenAI API. +- System One decision-model requests, Responses API calls, and pricing-table changes. ## Tests @@ -158,10 +173,12 @@ env -u PYTHONPATH PYTHONDONTWRITEBYTECODE=1 \ The tests use MockTransport at the documented host and local HTTP servers. Subprocesses install a socket guard that blocks external hosts before DNS. +CI runs the three cells below via +[OpenAI-compatible recipe tests](../../../.github/workflows/openai-compatible-recipe-tests.yml). | Python | `openai` | traceAI packages | Result | | --- | --- | --- | --- | -| 3.10, 3.11, 3.12, 3.13 | 3.24.0 | `traceai-openai` and `fi_instrumentation` from this repository | Full suite | +| 3.11 | 3.24.0 | `traceai-openai` and `fi_instrumentation` from this repository | Full suite | | 3.11 | 1.69.0 (the `traceai-openai` floor) | from this repository | Full suite | | 3.11 | 3.24.0 | published `traceAI-openai==0.1.10` and `fi-instrumentation-otel==1.1.0` from PyPI | Full suite | diff --git a/python/examples/baseten/src/app.py b/python/examples/baseten/src/app.py index 6584c5ac6..8f1609c05 100644 --- a/python/examples/baseten/src/app.py +++ b/python/examples/baseten/src/app.py @@ -1,6 +1,7 @@ """Trace Baseten Model APIs Chat Completions with the OpenAI SDK.""" import argparse +import ipaddress import os import re import sys @@ -18,21 +19,51 @@ def check_base_url(url: str) -> str: - """Reject the two Baseten surfaces that need a different recipe.""" - parsed = urlsplit(url) - # Compare the host without an absolute-FQDN trailing dot; the URL itself is - # returned unchanged. - host = (parsed.hostname or "").rstrip(".") - if host == "inference.baseten.co" and parsed.path in ("", "/"): - raise ValueError( - "Baseten's root is the Anthropic Messages beta endpoint; " - "use https://inference.baseten.co/v1 for the OpenAI SDK." - ) - if re.fullmatch(r"model-[^.]+\.api\.baseten\.co", host): - raise ValueError( - "Baseten dedicated deployments use a per-deployment URL and are " - "out of scope for this Model APIs recipe." - ) + """Validate without rewriting; permit customer proxies and loopback servers.""" + def refuse(reason): + raise ValueError(f"{BASE_URL_ENV}: {reason}") + + if not url or any(c.isspace() or not c.isprintable() for c in url): + refuse("use an absolute URL without whitespace or control characters") + try: + parts = urlsplit(url) + host = parts.hostname + parts.port # Reject malformed ports before tracing. + except ValueError: + refuse("use a valid absolute HTTP(S) URL") + if parts.username is not None or parts.password is not None: + refuse("remove URL credentials; supply the key through BASETEN_API_KEY") + if parts.scheme not in ("http", "https") or not host: + refuse("use an absolute HTTP(S) URL with a host") + if not host.isascii(): + refuse("use a plain ASCII host with valid IDNA spelling") + comparable_host = host.lower().rstrip(".") + structural_host = host.lower().removesuffix(".") + try: + ipaddress.ip_address(structural_host) + except ValueError: + try: + structural_host.encode("ascii").decode("idna") + except UnicodeError: + refuse("use a plain ASCII host with valid IDNA spelling") + if len(structural_host) > 253 or not all( + re.fullmatch(r"[a-zA-Z0-9](?:[a-zA-Z0-9_-]{0,61}[a-zA-Z0-9])?", label) + for label in structural_host.split(".") + ): + refuse("use a valid ASCII host") + if re.fullmatch(r"model-[^.]+\.api\.baseten\.co", comparable_host): + refuse("Baseten dedicated deployments use a per-deployment URL and are " + "out of scope for this Model APIs recipe") + if comparable_host == "inference.baseten.co": + if parts.scheme != "https": + refuse(f"use HTTPS: {DEFAULT_BASE_URL}") + if "?" in url or "#" in url: + refuse(f"remove the query or fragment; use {DEFAULT_BASE_URL}") + if parts.path in ("", "/"): + refuse("Baseten's root is the Anthropic Messages beta endpoint; " + "use https://inference.baseten.co/v1 for the OpenAI SDK") + if parts.path not in ("/v1", "/v1/"): + refuse(f"use the base URL {DEFAULT_BASE_URL}, without an endpoint path") return url @@ -65,14 +96,15 @@ def main(argv=None) -> int: args = parser.parse_args(argv) try: base_url = check_base_url(os.environ.get(BASE_URL_ENV) or DEFAULT_BASE_URL) - model = args.model or os.environ[MODEL_ENV] - api_key = os.environ[API_KEY_ENV] + model = args.model or os.environ.get(MODEL_ENV) + if not model: + raise ValueError(f"{MODEL_ENV}: set a catalog model id or pass --model") + api_key = os.environ.get(API_KEY_ENV) + if not api_key: + raise ValueError(f"{API_KEY_ENV}: set your Baseten API key") except ValueError as error: print(str(error), file=sys.stderr) return 2 - except KeyError as error: - print(f"Set {error.args[0]} before running this example.", file=sys.stderr) - return 2 provider = setup_tracing() try: @@ -82,9 +114,10 @@ def main(argv=None) -> int: stream=args.stream, ) if args.stream: - for chunk in response: - if chunk.choices: - print(chunk.choices[0].delta.content or "", end="", flush=True) + with response: + for chunk in response: + if chunk.choices: + print(chunk.choices[0].delta.content or "", end="", flush=True) print() else: print(response.choices[0].message.content or "") diff --git a/python/examples/baseten/tests/test_baseten_recipe.py b/python/examples/baseten/tests/test_baseten_recipe.py index 52b906c24..6b0846921 100644 --- a/python/examples/baseten/tests/test_baseten_recipe.py +++ b/python/examples/baseten/tests/test_baseten_recipe.py @@ -46,7 +46,7 @@ # Host spelling variants must not bypass the refusals. ("https://inference.baseten.co.", "Anthropic Messages beta"), ("https://INFERENCE.Baseten.CO/", "Anthropic Messages beta"), - ("https://user@inference.baseten.co:443", "Anthropic Messages beta"), + ("https://user@inference.baseten.co:443", "URL credentials"), ("https://model-abc123.api.baseten.co./v1", "dedicated deployments"), ("https://MODEL-abc123.API.baseten.co:8443/v1", "dedicated deployments"), ) @@ -439,7 +439,9 @@ def test_guard_refuses_provider_before_dns(operation, tmp_path): @pytest.mark.parametrize("url", [ - DOCUMENTED_URL, DOCUMENTED_URL + "/", "http://127.0.0.1:1234/v1", + DOCUMENTED_URL, DOCUMENTED_URL + "/", "https://INFERENCE.BASETEN.CO/v1/", + "https://inference.baseten.co./v1", "https://inference.baseten.co:443/v1", + "http://127.0.0.1:1234/v1", "http://[::1]:1234/v1", "http://myproj_proxy_1:1234/v1", "http://localhost:1234/custom/path/", "https://customer.example/proxy/v1?route=one", "https://model-abc123.api.baseten.co.customer.example/v1", ]) @@ -505,6 +507,74 @@ def test_model_and_prompt_cli_override(monkeypatch, receiver, tmp_path): assert (tmp_path / "guard.ready").is_file() +def strict_refused_urls(): + cases = [] + for host in ("inference.baseten.co", "INFERENCE.BASETEN.CO", "inference.baseten.co."): + for path in ("/v1/chat/completions", "/chat/completions", "/v1/embeddings", + "/v1/responses", "/v1/systemone", "/V1", "/v1//", "/%76%31", + "/v1/../v1", "/v1/..", "/."): + cases.append((f"https://{host}{path}", "base URL")) + cases.append((f"http://{host}/v1", "HTTPS")) + for suffix in ("?", "#", "?route=x", "#anchor"): + cases.append((f"https://{host}/v1{suffix}", "query or fragment")) + for char in (" ", "\t", "\r", "\n", "\x00", "\x7f", " ", "​"): + cases.append((f"https://inference.baseten.co/v1{char}", "whitespace or control")) + for dot in ("。", ".", "。"): + cases.append((f"https://inference{dot}baseten.co/v1", "ASCII host")) + for host in ("xn--.example", "inference..baseten.co", "-bad.example", "bad_.example", + "inference.baseten.co%2e", "inference.baseten.co.."): + cases.append((f"https://{host}/v1", "host")) + for host in ("inference.baseten.co", "proxy.example.test", "127.0.0.1:12345"): + cases.append((f"https://private-credential-marker:private-password-marker@{host}/v1", "credentials")) + cases.append((f"https://@{host}/v1", "credentials")) + cases.extend(( + ("", "absolute URL"), ("inference.baseten.co/v1", "absolute"), + ("ftp://inference.baseten.co/v1", "HTTP(S)"), + ("https:///v1", "host"), ("https://[invalid/v1", "valid absolute"), + ("https://inference.baseten.co:invalid/v1", "valid absolute"), + )) + return cases + + +def test_strict_urls_exit_before_tracing_or_client(monkeypatch, receiver, capsys): + def unexpected(*args, **kwargs): + pytest.fail("invalid URL must exit before tracing or client creation") + + monkeypatch.setattr(app, "setup_tracing", unexpected) + monkeypatch.setattr(app, "make_client", unexpected) + for url, reason in strict_refused_urls(): + with pytest.raises(ValueError) as caught: + app.check_base_url(url) + message = str(caught.value) + assert app.BASE_URL_ENV in message + assert reason in message + assert "\n" not in message + assert "private-credential-marker" not in message + assert "private-password-marker" not in message + if not url: + continue # An empty environment override selects the default. + environment = dict(os.environ) + environment[app.BASE_URL_ENV] = url + with monkeypatch.context() as context: + # Exercise NUL without trying to put it in the real OS environment. + context.setattr(app.os, "environ", environment) + assert app.main([]) == 2 + output = capsys.readouterr() + assert output.out == "" + assert message in output.err + assert "private-credential-marker" not in output.err + assert receiver.spans() == [] + assert receiver.requests() == [] + + +@pytest.mark.parametrize("variable", [app.MODEL_ENV, app.API_KEY_ENV]) +def test_empty_configuration_exits_before_tracing(variable, monkeypatch, capsys): + monkeypatch.setenv(variable, "") + monkeypatch.setattr(app, "setup_tracing", lambda: pytest.fail("empty configuration")) + assert app.main([]) == 2 + assert variable in capsys.readouterr().err + + def test_readme_contract_and_pins(): readme = (RECIPE / "README.md").read_text() assert "# Baseten (OpenAI-compatible) with traceAI" in readme @@ -532,7 +602,10 @@ def test_readme_contract_and_pins(): assert "NOT a Future AGI" in readme assert "using_session" in readme assert "1.69.0" in readme - assert "not tested against the live provider" in readme.lower() + assert "per-token pricing" in readme + assert "System One API" in readme + assert "path must be exactly" in readme + assert "`/v1` or `/v1/`" in readme assert not re.search(r"\b(?:sk-(?:proj-)?|fi-)[A-Za-z0-9_]{20,}\b", readme) pins = (RECIPE / "requirements.txt").read_text().splitlines() assert pins[0].startswith("#") From 05c51dbfee45c70ab6c1718d5ffdc4d5bb20f936 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek <8347993+nik13@users.noreply.github.com> Date: Wed, 28 Oct 2026 03:17:32 -0700 Subject: [PATCH 7/7] docs(baseten): identify default-stream usage as an offline fixture Co-Authored-By: Claude Code --- python/examples/baseten/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/examples/baseten/README.md b/python/examples/baseten/README.md index b1da0701b..46efcf6a6 100644 --- a/python/examples/baseten/README.md +++ b/python/examples/baseten/README.md @@ -94,7 +94,7 @@ in its response. Token usage appears only when the response has `usage`; missing usage is omitted, never reported as zero. With `traceai-openai` 0.1.10 and earlier, streamed calls record accumulated output text but have no `gen_ai.request.model` attribute. -The default stream tested here returns no usage, so it exports +The offline default-stream fixture returns no usage, so that fixture exports no `gen_ai.usage.*` attributes. Failed calls produce an error span with an exception event and no `gen_ai.request.model` attribute. On both, the model you requested is still visible inside the `gen_ai.request.parameters` JSON. These are