diff --git a/examples/traceai/typesafe/.gitignore b/examples/traceai/typesafe/.gitignore new file mode 100644 index 000000000..77ac75498 --- /dev/null +++ b/examples/traceai/typesafe/.gitignore @@ -0,0 +1,3 @@ +.venv/ +__pycache__/ +*.pyc diff --git a/examples/traceai/typesafe/README.md b/examples/traceai/typesafe/README.md new file mode 100644 index 000000000..f263d4eed --- /dev/null +++ b/examples/traceai/typesafe/README.md @@ -0,0 +1,54 @@ +# TypeSafe (Jev) tracing example + +Runnable companion to the docs page `src/pages/docs/integrations/traceai/typesafe.mdx` +(`/docs/integrations/traceai/typesafe`). It traces TypeSafe System One decision calls made with the +official `typesafe-sdk` Python client using the +[`traceAI-typesafe`](https://pypi.org/project/traceAI-typesafe/) instrumentor. + +Status: verified against the live TypeSafe API on 2026-10-17 (typesafe-sdk 0.7.5, model `jev-1.13.0`): two real +decisions (sync and async), a real 422 and a real 401, traced into a local Future AGI. The default `./run_e2e.sh` +(no `TYPESAFE_API_KEY`) still uses the local STANDIN server below, so it needs no account. Not yet seen live: 429 +(`retry_after_ms`), `box` answers, image assets. + +## Contents + +| Path | What it is | +| --- | --- | +| `app.py` | The recipe the page shows (between the `--- ... start/end ---` markers) inside a small support-ticket triage app, with scenarios `ok`, `fail-422`, `fail-auth`, `fail-asset` | +| `run_e2e.sh` | Creates a venv, runs the tests, runs `app.py` against a Future AGI collector, optionally the privacy comparison and the real-API 401 probe | +| `standin/systemone_stub.py` | Local **STAND-IN** shaped like TypeSafe's published OpenAPI document (`POST /v1/systemone`, `GET /v1/models`, the real error bodies). Answers are hash-derived numbers, **not Jev** | +| `tests/` | pytest: the stand-in against the real SDK, `app.py` against a recording OTLP collector (spans, headers, no gzip, privacy markers), and page/example consistency | +| `requirements.txt` | Pinned versions the example was tested with | + +## Run it + +Python 3.10 to 3.13 (traceAI-typesafe 0.1.0 requires Python below 3.14). + +```bash +export FI_API_KEY=... FI_SECRET_KEY=... +export FI_BASE_URL=http://localhost:4318 # your Future AGI collector (run_e2e.sh defaults to this) +./run_e2e.sh # tests, then app.py against the stand-in. No TypeSafe needed +PRIVACY=1 ./run_e2e.sh # also: hide flags on vs default, with a unique marker +REAL_AUTH_PROBE=1 ./run_e2e.sh # also: one invalid-key call to the real api.typesafe.ai (HTTP 401) +``` + +Against the real TypeSafe API (needs your own account and key; a run is billed by TypeSafe, two decisions of a few hundred tokens each): + +```bash +export TYPESAFE_API_KEY=... # goes to the SDK only +./run_e2e.sh +``` + +`run_e2e.sh` installs `traceAI-typesafe==0.1.0` from PyPI. `TRACEAI_TYPESAFE` overrides +that pip requirement, for example with a local checkout: +`TRACEAI_TYPESAFE=/path/to/traceAI/python/frameworks/typesafe`. + +## What is real and what is a stand-in + +| Piece | Real or stand-in | +| --- | --- | +| `typesafe-sdk` client (0.7.5), `traceAI-typesafe`, `fi-instrumentation-otel` (`register()`), OTLP export | Real | +| Future AGI collector, storage and Observe UI | Real (a local Future AGI) | +| System One answers (`noul`, `choice`, `score`) | **Real** `api.typesafe.ai` when `TYPESAFE_API_KEY` is set (verified 2026-10-17); **STANDIN** (`standin/systemone_stub.py`, hash-derived numbers, not Jev) otherwise, and in `tests/` | +| HTTP 401 for an invalid key | Real `api.typesafe.ai` (`REAL_AUTH_PROBE=1`, or any run with a key set); the stand-in otherwise | +| HTTP 422 validation error | Real `api.typesafe.ai` when a key is set; STANDIN otherwise (FastAPI error shape from TypeSafe's OpenAPI document) | diff --git a/examples/traceai/typesafe/app.py b/examples/traceai/typesafe/app.py new file mode 100644 index 000000000..e27731189 --- /dev/null +++ b/examples/traceai/typesafe/app.py @@ -0,0 +1,171 @@ +"""Trace TypeSafe (Jev) System One decisions in Future AGI with traceAI-typesafe. + +The block between "recipe start" and "recipe end" is the code the docs page +shows. The rest is a small support-ticket triage app around it, with failure +scenarios, used by run_e2e.sh. + +Environment: FI_API_KEY, FI_SECRET_KEY (Future AGI), TYPESAFE_API_KEY +(TypeSafe), optional TYPESAFE_BASE_URL (another System One host) and +FI_BASE_URL (a self-hosted Future AGI collector). + + python app.py --scenario ok # triage one ticket: sync + async decisions, a session + python app.py --scenario fail-422 # the API rejects a malformed question (HTTP 422) + python app.py --scenario fail-auth # a wrong API key (HTTP 401) + python app.py --scenario fail-asset # an image file that does not exist (client-side error) + +RUN_MARKER (optional) is added to the ticket text (also as an email address) and the +question wording, so a privacy check can search the stored traces for it. +""" + +from __future__ import annotations + +import argparse +import asyncio +import tempfile +import uuid +from pathlib import Path + +from fi_instrumentation import FITracer, using_session +from fi_instrumentation.fi_types import FiSpanKindValues +from opentelemetry.trace import Status, StatusCode + +# --- recipe start --- +import os + +from fi_instrumentation import register +from fi_instrumentation.fi_types import ProjectType +from traceai_typesafe import TypeSafeInstrumentor +from typesafe_sdk import AsyncTypeSafeClient, Choice, Noul, Score, TypeSafeClient + +trace_provider = register( + project_type=ProjectType.OBSERVE, + project_name=os.environ.get("FI_PROJECT_NAME", "support-triage"), +) +TypeSafeInstrumentor().instrument(tracer_provider=trace_provider) +# --- recipe end --- + +# --- questions start --- +TICKET = ( + "Order 1042: I was charged twice for the same order and nobody answers my emails. " + "Please refund the duplicate today." +) + +QUESTIONS = { + "billing": Noul(instructions="Is this message about billing?"), + "tone": Choice( + instructions="What is the tone of this message?", + criteria={"calm": None, "angry": "Upset or hostile", "excited": None}, + ), + "urgency": Score( + instructions="How urgent is this message?", + criteria=["Can wait", "Needs attention this week", "Needs attention today"], + ), +} +# --- questions end --- + +# Privacy check only (not part of the page): put a unique marker in the ticket text, +# an email address, a question instruction and a criteria description. +MARKER = os.environ.get("RUN_MARKER", "") +if MARKER: + TICKET += " {0} Reply to {0}@example.com.".format(MARKER) + QUESTIONS["billing"] = Noul(instructions="Is this message about billing? {0}".format(MARKER)) + QUESTIONS["tone"] = Choice( + instructions="What is the tone of this message?", + criteria={"calm": None, "angry": "Upset or hostile {0}".format(MARKER), "excited": None}, + ) + +tracer = FITracer(trace_provider.get_tracer("support-triage")) + + +def triage_sync() -> None: + # --- sync call start --- + with TypeSafeClient() as client: + result = client.system_one(state=TICKET, questions=QUESTIONS) + print("billing", result.nouls["billing"].noul) + print("tone", result.choices["tone"].choice, result.choices["tone"].confidence) + print("urgency", result.scores["urgency"].score, result.scores["urgency"].confidence) + # --- sync call end --- + print("RESULT model", result.model, "tokens", result.usage.input_tokens, result.usage.output_tokens) + with TypeSafeClient() as client: + models = client.models.list() # GET /v1/models: not a decision, no span + print("RESULT models", [model.name for model in models.models]) + + +async def triage_async() -> None: + # --- async call start --- + async with AsyncTypeSafeClient() as client: + result = await client.system_one( + state={"ticket": TICKET, "customer_tier": "gold"}, + questions={"escalate": Noul(instructions="Should a human agent take this ticket now?")}, + ) + print("escalate", result.nouls["escalate"].noul) + # --- async call end --- + + +def fail_422() -> None: + with TypeSafeClient() as client: + # A choice question needs a mapping of criteria; a list passes the SDK's + # own check and is rejected by the API. + client.system_one( + state=TICKET, + questions={"tone": {"type": "choice", "instructions": "Tone?", "criteria": ["calm", "angry"]}}, + ) + + +def fail_auth() -> None: + with TypeSafeClient(api_key="ts-invalid-e2e-key") as client: + client.system_one(state=TICKET, questions={"billing": QUESTIONS["billing"]}) + + +def fail_asset() -> None: + from typesafe_sdk import Image + + missing = Path(tempfile.gettempdir()) / "typesafe-e2e-{0}".format(MARKER or "x") / "photo.png" + photo = Image(missing) + with TypeSafeClient() as client: + client.system_one( + state="Product photo: {0}".format(photo), + questions={"damaged": Noul(instructions="Is the product damaged?")}, + assets={"photo": photo}, + ) + + +SCENARIOS = { + "ok": lambda: (triage_sync(), asyncio.run(triage_async())), + "fail-422": fail_422, + "fail-auth": fail_auth, + "fail-asset": fail_asset, +} + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--scenario", choices=sorted(SCENARIOS), default="ok") + args = parser.parse_args() + session_id = "triage-" + uuid.uuid4().hex[:12] + outcome = "ok" + with using_session(session_id): + # The app's own parent span. It records no exception text of its own. + with tracer.start_as_current_span( + "triage_ticket", + fi_span_kind=FiSpanKindValues.CHAIN, + record_exception=False, + set_status_on_exception=False, + ) as span: + span.set_input(TICKET) + trace_id = "{0:032x}".format(span.get_span_context().trace_id) + try: + SCENARIOS[args.scenario]() + span.set_output("triaged") + span.set_status(Status(StatusCode.OK)) + except Exception as error: # the failure scenarios end here + outcome = type(error).__name__ + span.set_status(Status(StatusCode.ERROR, outcome)) + trace_provider.force_flush() + print("RESULT scenario={0} outcome={1} trace_id={2} session_id={3}".format(args.scenario, outcome, trace_id, session_id)) + trace_provider.shutdown() + return 0 if (outcome == "ok") == (args.scenario == "ok") else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/traceai/typesafe/pytest.ini b/examples/traceai/typesafe/pytest.ini new file mode 100644 index 000000000..0e02926da --- /dev/null +++ b/examples/traceai/typesafe/pytest.ini @@ -0,0 +1,3 @@ +[pytest] +testpaths = tests +addopts = -p no:cacheprovider diff --git a/examples/traceai/typesafe/requirements.txt b/examples/traceai/typesafe/requirements.txt new file mode 100644 index 000000000..882b0264a --- /dev/null +++ b/examples/traceai/typesafe/requirements.txt @@ -0,0 +1,7 @@ +# Python 3.10 to 3.13 (traceAI-typesafe 0.1.0 requires Python < 3.14). +# traceAI-typesafe itself is installed by run_e2e.sh (TRACEAI_TYPESAFE, default traceAI-typesafe==0.1.0). +typesafe-sdk==0.7.5 +fi-instrumentation-otel==1.1.0 +# tests +pytest>=8 +opentelemetry-proto>=1.27 diff --git a/examples/traceai/typesafe/run_e2e.sh b/examples/traceai/typesafe/run_e2e.sh new file mode 100755 index 000000000..d04a579a6 --- /dev/null +++ b/examples/traceai/typesafe/run_e2e.sh @@ -0,0 +1,94 @@ +#!/usr/bin/env bash +# End-to-end run of the TypeSafe (Jev) tracing example against a Future AGI collector. +# +# FI_API_KEY=... FI_SECRET_KEY=... FI_BASE_URL=http://localhost:4318 ./run_e2e.sh +# +# What it does +# 1. creates ./.venv (Python 3.10 to 3.13) and installs requirements.txt plus traceAI-typesafe +# (TRACEAI_TYPESAFE: a pip requirement; set VENV_DIR to reuse an existing environment) +# 2. runs the tests (local stand-in and a recording collector; no Future AGI, no TypeSafe) +# 3. runs app.py: scenario ok (sync + async decisions, models.list), then the failure +# scenarios fail-422, fail-auth and fail-asset, tracing to FI_BASE_URL +# 4. PRIVACY=1: runs ok, fail-422 and fail-asset again with a unique RUN_MARKER, once with the +# hide flags on (FI_HIDE_INPUTS/FI_HIDE_OUTPUTS=true) into -hidden and once with the +# default into -default, and prints the markers so you can search your store +# 5. REAL_AUTH_PROBE=1: one fail-auth call to the real https://api.typesafe.ai with a +# deliberately invalid key (no account needed, nothing is billed) +# +# Which API it calls +# default the local stand-in in standin/ (NOT TypeSafe) +# TYPESAFE_API_KEY (TYPESAFE_BASE_URL unset) the real TypeSafe API, https://api.typesafe.ai. +# Needs your own account and key; a real run is billed. +# TYPESAFE_API_KEY + TYPESAFE_BASE_URL any System One host you choose +# +# Other knobs: FI_PROJECT_NAME (default typesafe-e2e), SKIP_TESTS=1, SCENARIOS="ok fail-422". +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +cd "$HERE" +: "${FI_API_KEY:?set FI_API_KEY}" "${FI_SECRET_KEY:?set FI_SECRET_KEY}" +export FI_BASE_URL="${FI_BASE_URL:-http://localhost:4318}" +export FI_PROJECT_NAME="${FI_PROJECT_NAME:-typesafe-e2e}" +# The released package from PyPI; set TRACEAI_TYPESAFE to another pip requirement (for example a +# local traceAI checkout) to test an unreleased build. +TRACEAI_TYPESAFE="${TRACEAI_TYPESAFE:-traceAI-typesafe==0.1.0}" + +VENV_DIR="${VENV_DIR:-$HERE/.venv}" +if [ ! -x "$VENV_DIR/bin/python" ]; then + PY="${PYTHON:-$(command -v python3.12 || command -v python3.13 || command -v python3.11 || command -v python3.10 || true)}" + [ -n "$PY" ] || { echo "need python3.10 to python3.13 on PATH (or set PYTHON)"; exit 2; } + "$PY" -m venv "$VENV_DIR" + "$VENV_DIR/bin/python" -m pip install --quiet --upgrade pip + "$VENV_DIR/bin/python" -m pip install --quiet -r requirements.txt "$TRACEAI_TYPESAFE" +fi +PYBIN="$VENV_DIR/bin/python" +"$PYBIN" - <<'PY' +import importlib.metadata as m, sys +names = ["traceai-typesafe", "typesafe-sdk", "fi-instrumentation-otel", "opentelemetry-sdk"] +print("versions: python", sys.version.split()[0], *(f"{n}=={m.version(n)}" for n in names)) +PY + +if [ "${SKIP_TESTS:-0}" != "1" ]; then + "$PYBIN" -m pytest -q +fi + +STUB_PID="" +cleanup() { [ -n "$STUB_PID" ] && kill "$STUB_PID" 2>/dev/null || true; } +trap cleanup EXIT + +if [ -n "${TYPESAFE_API_KEY:-}" ]; then + echo "target: ${TYPESAFE_BASE_URL:-REAL TypeSafe, https://api.typesafe.ai}" +else + PORT="$("$PYBIN" -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1])')" + "$PYBIN" standin/systemone_stub.py --port "$PORT" >/dev/null 2>&1 & + STUB_PID=$! + for _ in $(seq 1 50); do curl -fs "http://127.0.0.1:$PORT/healthz" >/dev/null 2>&1 && break; sleep 0.2; done + export TYPESAFE_BASE_URL="http://127.0.0.1:$PORT" TYPESAFE_API_KEY="standin-key" + echo "target: STAND-IN (not TypeSafe) at $TYPESAFE_BASE_URL" +fi + +echo "== app.py -> System One, traces -> $FI_BASE_URL project=$FI_PROJECT_NAME" +for scenario in ${SCENARIOS:-ok fail-422 fail-auth fail-asset}; do + "$PYBIN" app.py --scenario "$scenario" | grep -E '^(RESULT|billing|tone|urgency|escalate)' +done + +if [ "${PRIVACY:-0}" = "1" ]; then + for mode in hidden default; do + MARK="TSPRIV-$(date +%s)-$RANDOM$RANDOM" + echo "== privacy run: mode=$mode marker=$MARK project=$FI_PROJECT_NAME-$mode" + for scenario in ok fail-422 fail-asset; do + if [ "$mode" = "hidden" ]; then + FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true RUN_MARKER="$MARK" FI_PROJECT_NAME="$FI_PROJECT_NAME-$mode" \ + "$PYBIN" app.py --scenario "$scenario" | grep -E '^RESULT scenario' + else + RUN_MARKER="$MARK" FI_PROJECT_NAME="$FI_PROJECT_NAME-$mode" "$PYBIN" app.py --scenario "$scenario" | grep -E '^RESULT scenario' + fi + done + done + echo "Search your Future AGI project -hidden for the hidden marker: it must not appear anywhere." +fi + +if [ "${REAL_AUTH_PROBE:-0}" = "1" ]; then + echo "== real api.typesafe.ai, invalid key (expects HTTP 401)" + env -u TYPESAFE_BASE_URL "$PYBIN" app.py --scenario fail-auth | grep -E '^RESULT' +fi diff --git a/examples/traceai/typesafe/standin/systemone_stub.py b/examples/traceai/typesafe/standin/systemone_stub.py new file mode 100644 index 000000000..fbbbdd2bc --- /dev/null +++ b/examples/traceai/typesafe/standin/systemone_stub.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python3 +"""STAND-IN for the TypeSafe System One API. This is NOT TypeSafe and NOT Jev. + +A small local server shaped like TypeSafe's published OpenAPI document +(https://api.typesafe.ai/openapi.json, version 0.2.0, read 2026-10-16) so the +real ``typesafe-sdk`` client can run end to end without a TypeSafe key: + +* ``POST /v1/systemone``: ``{model, state, questions[, assets]}`` in, + ``{model, answers, usage}`` out. ``noul`` answers carry only ``noul``; + ``choice`` answers ``choice``, ``confidence`` and ``probabilities``; + ``score`` answers ``score``, ``confidence``, ``legend`` and + ``probabilities``; ``box`` answers ``box`` and ``confidence: null``. +* ``GET /v1/models``: ``{"models": [{name, description, release_date}]}``. +* Errors use the bodies api.typesafe.ai returned on 2026-10-16: 403 with no + key, 401 for a key starting ``ts-invalid``, and FastAPI-style 422 + ``{"detail": [{loc, msg, type}]}`` for a body that breaks the schema. +* Every response has an ``x-typesafe-request-id`` header. + +Answers are deterministic numbers derived from a hash of the request. They +are not model output and say nothing about how Jev decides. ``jev-latest`` +resolves to ``jev-1.13.0`` in the response, as the API documents for aliases. + +Usage: ``python systemone_stub.py --port 8765``. Binds 127.0.0.1 only. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import uuid +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from typing import Any, Dict, List, Optional, Tuple + +MODELS = [ + {"name": "jev-latest", "description": "General-purpose system one model.", "release_date": "2026-09-15"}, + {"name": "jev-1.13.0", "description": "General-purpose system one model.", "release_date": "2026-09-15"}, +] +ALIASES = {"jev-latest": "jev-1.13.0"} +QUESTION_TYPES = ("noul", "choice", "score", "box") +NO_KEY = {"detail": {"error_type": "authentication_error", "message": "Must supply an API key! Check your request and try again."}} +BAD_KEY = {"detail": {"error_type": "authentication_error", "message": "Cannot authenticate with the server. Please check your API key and try again."}} + + +def _unit(*parts: str) -> float: + """A deterministic number in (0, 1) from the request.""" + digest = hashlib.sha256("\x1f".join(parts).encode()).digest() + return (int.from_bytes(digest[:4], "big") % 9973 + 1) / 9975 + + +def _distribution(names: List[str], seed: str) -> Dict[str, float]: + weights = [_unit(seed, name) ** 2 for name in names] + total = sum(weights) + return {name: round(weight / total, 4) for name, weight in zip(names, weights)} + + +def _answer(name: str, question: Dict[str, Any], state_key: str) -> Dict[str, Any]: + kind = question["type"] + seed = "{0}|{1}|{2}".format(state_key, name, json.dumps(question, sort_keys=True)) + if kind == "noul": + return {"type": "noul", "noul": round(_unit(seed), 4)} + if kind == "choice": + probabilities = _distribution(list(question["criteria"]), seed) + choice = max(probabilities, key=probabilities.get) + return {"type": "choice", "choice": choice, "confidence": probabilities[choice], "probabilities": probabilities} + if kind == "score": + levels = [str(index) for index in range(len(question["criteria"]))] + probabilities = _distribution(levels, seed) + score = round(sum(int(level) * p for level, p in probabilities.items()), 4) + return { + "type": "score", + "score": score, + "confidence": max(probabilities.values()), + "legend": dict(zip(levels, question["criteria"])), + "probabilities": probabilities, + } + top, left = round(_unit(seed, "t") / 2, 4), round(_unit(seed, "l") / 2, 4) + return {"type": "box", "box": {"top": top, "bottom": round(top + 0.25, 4), "left": left, "right": round(left + 0.25, 4)}, "confidence": None} + + +def validate(body: Any) -> List[Dict[str, Any]]: + """FastAPI-style validation errors for the SystemOneRequest schema (empty when valid).""" + if not isinstance(body, dict): + return [{"loc": ["body"], "msg": "Input should be a valid dictionary", "type": "dict_type"}] + errors: List[Dict[str, Any]] = [] + for field in ("model", "state", "questions"): + if field not in body: + errors.append({"loc": ["body", field], "msg": "Field required", "type": "missing"}) + if "model" in body and not isinstance(body["model"], str): + errors.append({"loc": ["body", "model"], "msg": "Input should be a valid string", "type": "string_type"}) + if "state" in body and not isinstance(body["state"], (str, dict, list)): + errors.append({"loc": ["body", "state"], "msg": "Input should be a valid string", "type": "string_type"}) + questions = body.get("questions") + if "questions" in body: + if not isinstance(questions, dict): + errors.append({"loc": ["body", "questions"], "msg": "Input should be a valid dictionary", "type": "dict_type"}) + elif not questions: + errors.append({"loc": ["body", "questions"], "msg": "Dictionary should have at least 1 item after validation, not 0", "type": "too_short"}) + else: + for name, question in questions.items(): + kind = question.get("type") if isinstance(question, dict) else None + if kind not in QUESTION_TYPES: + errors.append({"loc": ["body", "questions", name], "msg": "Input tag does not match any of the expected tags", "type": "union_tag_invalid"}) + elif kind == "choice" and "criteria" not in question: + errors.append({"loc": ["body", "questions", name, "choice", "criteria"], "msg": "Field required", "type": "missing"}) + elif kind == "choice" and not isinstance(question["criteria"], dict): + errors.append({"loc": ["body", "questions", name, "choice", "criteria"], "msg": "Input should be a valid dictionary", "type": "dict_type"}) + elif kind == "score" and not (isinstance(question.get("criteria"), list) and question["criteria"]): + errors.append({"loc": ["body", "questions", name, "score", "criteria"], "msg": "List should have at least 1 item after validation, not 0", "type": "too_short"}) + return errors + + +def _decode_body(content: bytes, encoding: str, content_type: str) -> Any: + if encoding == "zstd": + try: + from compression.zstd import decompress # Python 3.14 + except ImportError: + from backports.zstd import decompress # typesafe-sdk[zstd] on Python < 3.14 + content = decompress(content) + if content_type.startswith("multipart/form-data"): + boundary = content_type.split("boundary=", 1)[1].strip('"').encode() + for part in content.split(b"--" + boundary): + head, _, data = part.partition(b"\r\n\r\n") + if b'name="request"' in head: + return json.loads(data.rstrip(b"\r\n")) + raise ValueError("no request part") + return json.loads(content) + + +class Handler(BaseHTTPRequestHandler): + server_version = "systemone-standin/1" + + def log_message(self, fmt: str, *args: Any) -> None: # quiet: never log bodies or keys + return + + def _send(self, status: int, body: Any) -> None: + payload = json.dumps(body).encode() + self.send_response(status) + self.send_header("content-type", "application/json") + self.send_header("content-length", str(len(payload))) + self.send_header("x-typesafe-request-id", "req_standin_" + uuid.uuid4().hex) + self.end_headers() + self.wfile.write(payload) + + def _auth(self) -> Optional[Tuple[int, Any]]: + header = self.headers.get("authorization", "") + if not header.lower().startswith("bearer ") or not header[7:].strip(): + return 403, NO_KEY + if header[7:].strip().startswith("ts-invalid"): + return 401, BAD_KEY + return None + + def do_GET(self) -> None: # noqa: N802 + if self.path == "/healthz": + return self._send(200, {"ok": True, "standin": True}) + if self.path.split("?")[0] != "/v1/models": + return self._send(404, {"detail": "Not Found"}) + failure = self._auth() + if failure: + return self._send(*failure) + return self._send(200, {"models": MODELS}) + + def do_POST(self) -> None: # noqa: N802 + if self.path.split("?")[0] != "/v1/systemone": + return self._send(404, {"detail": "Not Found"}) + failure = self._auth() + if failure: + return self._send(*failure) + length = int(self.headers.get("content-length") or 0) + raw = self.rfile.read(length) + try: + body = _decode_body(raw, self.headers.get("content-encoding", ""), self.headers.get("content-type", "")) + except Exception: + return self._send(422, {"detail": [{"loc": ["body"], "msg": "JSON decode error", "type": "json_invalid"}]}) + errors = validate(body) + if errors: + return self._send(422, {"detail": errors}) + state_key = json.dumps(body["state"], sort_keys=True) + answers = {name: _answer(name, question, state_key) for name, question in body["questions"].items()} + self._send( + 200, + { + "model": ALIASES.get(body["model"], body["model"]), + "answers": answers, + "usage": {"input_tokens": max(1, len(raw) // 4), "output_tokens": 2 * len(answers)}, + }, + ) + + +def serve(port: int) -> ThreadingHTTPServer: + return ThreadingHTTPServer(("127.0.0.1", port), Handler) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--port", type=int, default=8765) + args = parser.parse_args() + server = serve(args.port) + print("STANDIN System One (not TypeSafe) on http://127.0.0.1:{0}".format(server.server_address[1]), flush=True) + server.serve_forever() + + +if __name__ == "__main__": + main() diff --git a/examples/traceai/typesafe/tests/__init__.py b/examples/traceai/typesafe/tests/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/examples/traceai/typesafe/tests/conftest.py b/examples/traceai/typesafe/tests/conftest.py new file mode 100644 index 000000000..98e991265 --- /dev/null +++ b/examples/traceai/typesafe/tests/conftest.py @@ -0,0 +1,94 @@ +"""Fixtures: the System One stand-in and a recording OTLP/HTTP collector, both on 127.0.0.1.""" + +from __future__ import annotations + +import gzip +import os +import sys +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from typing import Any, Dict, List + +import pytest + +HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, os.path.join(HERE, "standin")) + +import systemone_stub # noqa: E402 + + +@pytest.fixture() +def stub(): + server = systemone_stub.serve(0) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + yield "http://127.0.0.1:{0}".format(server.server_address[1]) + server.shutdown() + server.server_close() + + +class Recorder: + def __init__(self) -> None: + self.requests: List[Dict[str, Any]] = [] + + def spans(self) -> List[Dict[str, Any]]: + from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest + + found = [] + for request in self.requests: + message = ExportTraceServiceRequest() + message.ParseFromString(request["body"]) + for resource_spans in message.resource_spans: + resource = {kv.key: _value(kv.value) for kv in resource_spans.resource.attributes} + for scope_spans in resource_spans.scope_spans: + for span in scope_spans.spans: + found.append( + { + "name": span.name, + "trace_id": span.trace_id.hex(), + "span_id": span.span_id.hex(), + "parent": span.parent_span_id.hex(), + "status": span.status.code, + "status_message": span.status.message, + "attributes": {kv.key: _value(kv.value) for kv in span.attributes}, + "events": [[e.name, {kv.key: _value(kv.value) for kv in e.attributes}] for e in span.events], + "resource": resource, + } + ) + return found + + def raw(self) -> bytes: + return b"".join(request["body"] for request in self.requests) + + +def _value(value: Any) -> Any: + kind = value.WhichOneof("value") + return getattr(value, kind) if kind else None + + +@pytest.fixture() +def collector(): + recorder = Recorder() + + class Handler(BaseHTTPRequestHandler): + def log_message(self, *args: Any) -> None: + return + + def do_POST(self) -> None: # noqa: N802 + body = self.rfile.read(int(self.headers.get("content-length") or 0)) + if self.headers.get("content-encoding") == "gzip": + body = gzip.decompress(body) + recorder.gzip = True + recorder.requests.append({"path": self.path, "headers": dict(self.headers), "body": body}) + self.send_response(200) + self.send_header("content-type", "application/x-protobuf") + self.send_header("content-length", "0") + self.end_headers() + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + recorder.url = "http://127.0.0.1:{0}".format(server.server_address[1]) + yield recorder + server.shutdown() + server.server_close() diff --git a/examples/traceai/typesafe/tests/test_page.py b/examples/traceai/typesafe/tests/test_page.py new file mode 100644 index 000000000..dd70d35a8 --- /dev/null +++ b/examples/traceai/typesafe/tests/test_page.py @@ -0,0 +1,79 @@ +"""The docs page and this example agree; no network.""" + +from __future__ import annotations + +import os +import re +import textwrap + +HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +PAGE = os.path.normpath(os.path.join(HERE, "..", "..", "..", "src", "pages", "docs", "integrations", "traceai", "typesafe.mdx")) + + +def _read(path): + with open(path, encoding="utf-8") as handle: + return handle.read() + + +def _blocks(language): + return re.findall(r"```{0}\n(.*?)```".format(language), _read(PAGE), re.S) + + +def _excerpt(name): + match = re.search(r"# --- {0} start ---\n(.*?)\s*# --- {0} end ---".format(name), _read(os.path.join(HERE, "app.py")), re.S) + assert match, name + return textwrap.dedent(match.group(1)).strip() + + +def test_every_page_snippet_from_the_app_is_an_exact_excerpt(): + blocks = [block.strip() for block in _blocks("python")] + for name in ("recipe", "questions", "sync call", "async call"): + assert _excerpt(name) in blocks, name + + +def test_page_install_and_supported_range(): + page = _read(PAGE) + assert "pip install traceAI-typesafe typesafe-sdk" in page + assert "Python 3.10 to 3.13, as required by `traceAI-typesafe` 0.1.0" in page + assert "`typesafe-sdk` 0.7.0 or later, below 1.0" in page + assert "fi-instrumentation-otel does not support" not in page + + +def test_page_snippets_define_what_they_use(): + questions = next(block for block in _blocks("python") if "QUESTIONS = {" in block) + assert questions.lstrip().startswith("TICKET = (") + assert "MARKER" not in _read(PAGE) + + +def test_page_states_privacy_and_scope(): + page = _read(PAGE) + for needle in ("FI_HIDE_INPUTS=true", "FI_HIDE_OUTPUTS=true", "TraceConfig(hide_inputs=True)", + "traceAI-openai", "EVALUATOR", "Not supported", "Troubleshooting", + "never turns answers into Pass/Fail", "models.list()"): + assert needle in page, needle + + +def test_page_has_no_internal_test_material(): + page = _read(PAGE) + for needle in ("STANDIN", "stand-in", "run_e2e.sh", "Live vendor verification", "fi_verify", + "TH-8183", "2026-", "rows_hit", "PASS"): + assert needle not in page, needle + assert not re.search(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", page) + + +def test_page_does_not_send_customers_to_another_platform_by_default(): + page = _read(PAGE) + assert "base_url=\"https://api.typesafe.ai\"" not in page + assert "pip install typesafe-ai" not in page and "pip install jev" not in page + + +def test_run_script_never_prints_keys(): + script = _read(os.path.join(HERE, "run_e2e.sh")) + for name in ("FI_API_KEY", "FI_SECRET_KEY", "TYPESAFE_API_KEY"): + assert not re.search(r"echo[^\n]*\$\{?" + name, script), name + assert "set -euo pipefail" in script + + +def test_stand_in_is_labelled(): + stub = _read(os.path.join(HERE, "standin", "systemone_stub.py")) + assert "NOT TypeSafe" in stub and "127.0.0.1" in stub diff --git a/examples/traceai/typesafe/tests/test_recipe.py b/examples/traceai/typesafe/tests/test_recipe.py new file mode 100644 index 000000000..a00faf705 --- /dev/null +++ b/examples/traceai/typesafe/tests/test_recipe.py @@ -0,0 +1,97 @@ +"""app.py, configured as the page says, exports the expected spans to a recording collector.""" + +from __future__ import annotations + +import os +import subprocess +import sys +import tempfile +import uuid + +import pytest + +pytest.importorskip("traceai_typesafe") +pytest.importorskip("opentelemetry.proto") + +HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +_HIDE = ("FI_HIDE_INPUTS", "FI_HIDE_OUTPUTS", "FI_HIDE_INPUT_TEXT", "FI_HIDE_OUTPUT_TEXT", "FI_PII_REDACTION") + + +def run(scenario, stub, collector, **env): + environment = {k: v for k, v in os.environ.items() if k not in _HIDE and not k.startswith("TYPESAFE_")} + environment.update( + FI_API_KEY="test-fi-key", + FI_SECRET_KEY="test-fi-secret", + FI_BASE_URL=collector.url, + FI_PROJECT_NAME="typesafe-test", + TYPESAFE_BASE_URL=stub, + TYPESAFE_API_KEY="standin-key", + **env, + ) + done = subprocess.run( + [sys.executable, "app.py", "--scenario", scenario], + cwd=HERE, env=environment, capture_output=True, text=True, timeout=120, + ) + assert done.returncode == 0, done.stdout[-2000:] + done.stderr[-2000:] + return done.stdout + + +def test_ok_scenario_spans(stub, collector): + out = run("ok", stub, collector) + assert "RESULT scenario=ok outcome=ok" in out + spans = collector.spans() + by_name = {} + for span in spans: + by_name.setdefault(span["name"], []).append(span) + (root,) = by_name["triage_ticket"] + decisions = by_name["typesafe.system_one"] + assert len(decisions) == 2 # sync + async; models.list makes none + assert len(spans) == 3 + for span in decisions: + a = span["attributes"] + assert span["parent"] == root["span_id"] + assert a["gen_ai.span.kind"] == "EVALUATOR" + assert a["gen_ai.provider.name"] == "typesafe" + assert a["gen_ai.request.model"] == "jev-latest" + assert a["typesafe.response.model"] == "jev-1.13.0" + assert a["session.id"] == root["attributes"]["session.id"] + assert span["resource"]["project_name"] == "typesafe-test" + sync = next(s for s in decisions if s["attributes"]["typesafe.question_count"] == 3) + assert sync["attributes"]["typesafe.question.tone.type"] == "choice" + assert "typesafe.question.billing.confidence" not in sync["attributes"] + headers = {k.lower(): v for k, v in collector.requests[0]["headers"].items()} + assert headers["x-api-key"] == "test-fi-key" and headers["x-secret-key"] == "test-fi-secret" + assert "gzip" not in headers.get("content-encoding", "") + assert collector.requests[0]["path"].endswith("/v1/traces") + + +@pytest.mark.parametrize("hidden", [True, False]) +def test_marker_only_with_content_capture_on(stub, collector, hidden): + marker = "TSTEST-" + uuid.uuid4().hex + env = {"RUN_MARKER": marker} + if hidden: + env.update(FI_HIDE_INPUTS="true", FI_HIDE_OUTPUTS="true") + for scenario in ("ok", "fail-422", "fail-asset"): + run(scenario, stub, collector, **env) + raw = collector.raw() + assert (marker.encode() in raw) is (not hidden) + # Never, in either mode: the stand-in key, the Future AGI keys, the temp path of the missing image. + for secret in ("standin-key", "test-fi-key", "test-fi-secret", tempfile.gettempdir()): + assert secret.encode() not in raw, secret + + +def test_failure_scenarios_are_error_spans(stub, collector): + for scenario in ("fail-422", "fail-auth", "fail-asset"): + run(scenario, stub, collector) + errors = { + s["attributes"]["error.type"]: s + for s in collector.spans() + if s["name"] == "typesafe.system_one" + } + assert set(errors) == {"TypeSafeUnprocessableEntityError", "TypeSafeAuthenticationError", "TypeSafeError"} + for span in errors.values(): + assert span["status"] == 2 # STATUS_CODE_ERROR + assert all("exception.stacktrace" not in attrs for _, attrs in span["events"]) + assert errors["TypeSafeAuthenticationError"]["attributes"]["http.response.status_code"] == 401 + assert errors["TypeSafeUnprocessableEntityError"]["attributes"]["http.response.status_code"] == 422 + assert errors["TypeSafeError"]["status_message"] == "TypeSafeError" diff --git a/examples/traceai/typesafe/tests/test_stub.py b/examples/traceai/typesafe/tests/test_stub.py new file mode 100644 index 000000000..038a9f638 --- /dev/null +++ b/examples/traceai/typesafe/tests/test_stub.py @@ -0,0 +1,71 @@ +"""The stand-in speaks the published System One contract to the real typesafe-sdk client.""" + +from __future__ import annotations + +import json +import urllib.error +import urllib.request + +import pytest + +typesafe_sdk = pytest.importorskip("typesafe_sdk") +from typesafe_sdk import Choice, Noul, RetryPolicy, Score, TypeSafeClient # noqa: E402 + + +def _client(stub, **kwargs): + kwargs.setdefault("api_key", "standin-key") + return TypeSafeClient(base_url=stub, retry=RetryPolicy(max_retries=0), **kwargs) + + +def test_answers_parse_into_the_sdk_types(stub): + with _client(stub) as client: + result = client.system_one( + state="I was charged twice.", + questions={ + "billing": Noul(instructions="Billing?"), + "tone": Choice(instructions="Tone?", criteria={"calm": None, "angry": None}), + "urgency": Score(instructions="Urgent?", criteria=["low", "high"]), + }, + ) + assert result.model == "jev-1.13.0" # jev-latest resolved, as documented for aliases + assert 0 < result.nouls["billing"].noul < 1 + assert result.choices["tone"].choice in {"calm", "angry"} + assert set(result.choices["tone"].probabilities) == {"calm", "angry"} + assert 0 <= result.scores["urgency"].score <= 1 + assert result.scores["urgency"].legend == {0: "low", 1: "high"} + assert result.usage.input_tokens > 0 and result.usage.output_tokens == 6 + assert result.request_id.startswith("req_standin_") + + +def test_noul_has_no_confidence_on_the_wire(stub): + with _client(stub) as client: + result = client.system_one(state="x", questions={"q": Noul(instructions="ok?")}) + raw = json.loads(result.raw_http_response.content) + assert raw["answers"]["q"] == {"type": "noul", "noul": result.nouls["q"].noul} + + +def test_errors_use_the_documented_bodies(stub): + from typesafe_sdk import TypeSafeAuthenticationError, TypeSafeUnprocessableEntityError + + with _client(stub, api_key="ts-invalid-x") as client: + with pytest.raises(TypeSafeAuthenticationError) as raised: + client.system_one(state="x", questions={"q": Noul(instructions="ok?")}) + assert raised.value.body["detail"]["error_type"] == "authentication_error" + with _client(stub) as client: + with pytest.raises(TypeSafeUnprocessableEntityError) as raised: + client.system_one(state="x", questions={"t": {"type": "choice", "criteria": ["a", "b"]}}) + assert raised.value.body["detail"][0]["loc"] == ["body", "questions", "t", "choice", "criteria"] + request = urllib.request.Request(stub + "/v1/systemone", data=b"{}", method="POST") + with pytest.raises(urllib.error.HTTPError) as no_key: + urllib.request.urlopen(request) + assert no_key.value.code == 403 + + +def test_models_list(stub): + with _client(stub) as client: + names = [model.name for model in client.models.list().models] + assert names == ["jev-latest", "jev-1.13.0"] + + +def test_binds_loopback_only(stub): + assert stub.startswith("http://127.0.0.1:") diff --git a/scripts/integrations.test.mjs b/scripts/integrations.test.mjs index 30ff909ed..2bb52647e 100644 --- a/scripts/integrations.test.mjs +++ b/scripts/integrations.test.mjs @@ -3,6 +3,7 @@ import './azure-openai-java.test.mjs'; import './langsmith-page.test.mjs'; import './vercel-platform-page.test.mjs'; import './scalekit-page.test.mjs'; +import './typesafe-page.test.mjs'; import test from 'node:test'; import assert from 'node:assert/strict'; import fs from 'node:fs'; diff --git a/scripts/typesafe-page.test.mjs b/scripts/typesafe-page.test.mjs new file mode 100644 index 000000000..bf9c93272 --- /dev/null +++ b/scripts/typesafe-page.test.mjs @@ -0,0 +1,51 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import { integrations } from '../src/lib/integrations.mjs'; + +const read = file => readFileSync(new URL(`../${file}`, import.meta.url), 'utf8'); +const page = read('src/pages/docs/integrations/traceai/typesafe.mdx'); +const app = read('examples/traceai/typesafe/app.py'); +const python = [...page.matchAll(/```python\n([\s\S]*?)```/g)].map(match => match[1].trim()); + +function excerpt(name) { + const match = app.match(new RegExp(`# --- ${name} start ---\\n([\\s\\S]*?)\\s*# --- ${name} end ---`)); + assert.ok(match, `Missing app.py excerpt: ${name}`); + const lines = match[1].split('\n'); + const indent = Math.min(...lines.filter(line => line.trim()).map(line => line.match(/^ */)[0].length)); + return lines.map(line => line.slice(indent)).join('\n').trim(); +} + +test('TypeSafe snippets are exact excerpts of the runnable example', () => { + for (const name of ['recipe', 'questions', 'sync call', 'async call']) { + assert.ok(python.includes(excerpt(name)), `Page snippet differs from app.py: ${name}`); + } +}); + +test('TypeSafe snippets define the ticket and use no example-only variables', () => { + const questions = python.find(block => block.includes('QUESTIONS = {')); + assert.ok(questions?.startsWith('TICKET = ('), 'Define TICKET with the questions'); + assert.doesNotMatch(page, /\bMARKER\b|RUN_MARKER/); + assert.ok(page.indexOf('TICKET = (') < page.indexOf('state=TICKET')); +}); + +test('TypeSafe states the released package Python range', () => { + assert.match(page, /pip install traceAI-typesafe typesafe-sdk/); + assert.match(page, /Python 3\.10 to 3\.13, as required by `traceAI-typesafe` 0\.1\.0/); + assert.match(page, /`typesafe-sdk` 0\.7\.0 or later, below 1\.0/); + assert.doesNotMatch(page, /fi-instrumentation-otel does not support/); +}); + +test('TypeSafe page keeps engineering evidence out of the setup guide', () => { + assert.doesNotMatch(page, /STANDIN|stand-in|run_e2e\.sh|Live vendor verification|verified|fi_verify|rows_hit|TH-\d+|20\d\d-\d\d-\d\d/i); + assert.doesNotMatch(page, /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/); + assert.match(page, /FI_HIDE_INPUTS=true/); + assert.match(page, /never turns answers into Pass\/Fail/); +}); + +test('TypeSafe is discoverable as a traceAI model-provider guide', () => { + const entry = integrations.find(item => item.id === 'typesafe'); + assert.equal(entry?.href, '/docs/integrations/traceai/typesafe'); + assert.equal(entry.category, 'providers'); + assert.equal(entry.method, 'traceAI SDK'); +}); diff --git a/src/lib/integrations.mjs b/src/lib/integrations.mjs index df0f390a2..062300577 100644 --- a/src/lib/integrations.mjs +++ b/src/lib/integrations.mjs @@ -70,6 +70,7 @@ export const integrations = [ { id: 'togetherai', name: 'Together AI', href: '/docs/integrations/traceai/togetherai', category: 'providers', summary: 'Trace Together AI through its OpenAI-compatible API.', method: 'OpenAI client', logo: 'together' }, { id: 'ollama', name: 'Ollama', href: '/docs/integrations/traceai/ollama', category: 'providers', summary: 'Trace local model calls through the OpenAI client.', method: 'OpenAI client', logo: 'ollama' }, { id: 'cometapi', name: 'CometAPI', href: '/docs/integrations/providers/cometapi', category: 'providers', summary: 'Trace chat completions through the OpenAI client.', method: 'OpenAI client', logo: 'cometapi' }, + { id: 'typesafe', name: 'TypeSafe (Jev)', href: '/docs/integrations/traceai/typesafe', category: 'providers', summary: 'Trace System One decision calls from the Python client.', method: 'traceAI SDK', logo: 'generic-model', aliases: ['Jev', 'System One'] }, { id: 'novita-ai', name: 'Novita AI', href: '/docs/integrations/providers/novita-ai', category: 'providers', summary: 'Trace Novita chat completions from your Python app.', method: 'OpenAI client', logo: 'novita' }, { id: 'litellm-sdk', name: 'LiteLLM SDK', href: '/docs/integrations/traceai/litellm', category: 'frameworks', summary: 'Instrument LiteLLM calls in your Python application.', method: 'traceAI SDK', logo: 'litellm', aliases: ['traceAI-litellm', 'client'] }, diff --git a/src/pages/docs/integrations/traceai/typesafe.mdx b/src/pages/docs/integrations/traceai/typesafe.mdx new file mode 100644 index 000000000..863e80680 --- /dev/null +++ b/src/pages/docs/integrations/traceai/typesafe.mdx @@ -0,0 +1,195 @@ +--- +title: "TypeSafe (Jev) decision tracing with Future AGI" +description: "Trace TypeSafe System One decision calls (Jev) made with the typesafe-sdk Python client: one evaluator span per call with every noul, choice, score and box answer, using traceAI-typesafe." +--- + +Trace every TypeSafe System One decision your Python code makes, with each typed answer recorded on one `EVALUATOR` span in Future AGI. + +[TypeSafe](https://typesafe.ai) serves Jev through the System One decision API: you send some content (`state`) and named questions, and get back a typed answer per question. It is not a chat API and returns no text. `traceAI-typesafe` instruments the official Python client, [`typesafe-sdk`](https://pypi.org/project/typesafe-sdk/): every `TypeSafeClient.system_one(...)` and `AsyncTypeSafeClient.system_one(...)` call becomes one span, however many questions it asks. + +A run of the example on this page is one trace: + +```text +triage_ticket your own span (CHAIN) +├── typesafe.system_one billing (noul), tone (choice), urgency (score) (EVALUATOR) +└── typesafe.system_one escalate (noul), from the async client (EVALUATOR) +``` + +This package only traces calls your own code makes to TypeSafe. It does not add an evaluator model to Future AGI evals, does not call the Future AGI gateway, and never turns answers into Pass/Fail. System One is not OpenAI-compatible, so [`traceAI-openai`](/docs/integrations/traceai/openai) cannot trace it. + +Use Python 3.10 to 3.13, as required by `traceAI-typesafe` 0.1.0, with `typesafe-sdk` 0.7.0 or later, below 1.0. The instrumentor source is in [traceAI `python/frameworks/typesafe`](https://github.com/future-agi/traceAI/tree/main/python/frameworks/typesafe). + +## 1. Install + +```bash +pip install traceAI-typesafe typesafe-sdk +``` + +## 2. Set environment variables + +```bash +export FI_API_KEY="your-futureagi-api-key" +export FI_SECRET_KEY="your-futureagi-secret-key" +export TYPESAFE_API_KEY="your-typesafe-api-key" +``` + +The TypeSafe key goes to the SDK only and never onto a span. `FI_BASE_URL` is optional (set it to your own collector when you self-host). The SDK also reads `TYPESAFE_BASE_URL` and `TYPESAFE_DEFAULT_MODEL` itself; this package reads neither and never sets a host or a model. + +## 3. Register the tracer and instrument the client + +```python +import os + +from fi_instrumentation import register +from fi_instrumentation.fi_types import ProjectType +from traceai_typesafe import TypeSafeInstrumentor +from typesafe_sdk import AsyncTypeSafeClient, Choice, Noul, Score, TypeSafeClient + +trace_provider = register( + project_type=ProjectType.OBSERVE, + project_name=os.environ.get("FI_PROJECT_NAME", "support-triage"), +) +TypeSafeInstrumentor().instrument(tracer_provider=trace_provider) +``` + +Call `instrument()` once, before you make decisions. Clients created before or after are both traced. + +## 4. Make decisions + +Nothing changes in how you call TypeSafe. Define the content you want decided and your questions: + +```python +TICKET = ( + "Order 1042: I was charged twice for the same order and nobody answers my emails. " + "Please refund the duplicate today." +) + +QUESTIONS = { + "billing": Noul(instructions="Is this message about billing?"), + "tone": Choice( + instructions="What is the tone of this message?", + criteria={"calm": None, "angry": "Upset or hostile", "excited": None}, + ), + "urgency": Score( + instructions="How urgent is this message?", + criteria=["Can wait", "Needs attention this week", "Needs attention today"], + ), +} +``` + +Then make the call: + +```python +with TypeSafeClient() as client: + result = client.system_one(state=TICKET, questions=QUESTIONS) +print("billing", result.nouls["billing"].noul) +print("tone", result.choices["tone"].choice, result.choices["tone"].confidence) +print("urgency", result.scores["urgency"].score, result.scores["urgency"].confidence) +``` + +The async client is traced the same way (inside an `async` function): + +```python +async with AsyncTypeSafeClient() as client: + result = await client.system_one( + state={"ticket": TICKET, "customer_tier": "gold"}, + questions={"escalate": Noul(instructions="Should a human agent take this ticket now?")}, + ) +print("escalate", result.nouls["escalate"].noul) +``` + +In a short-lived script, call `trace_provider.force_flush()` before the process exits so the last spans are sent. + +## 5. Open the trace + +Open the `support-triage` project (or the name you set in `FI_PROJECT_NAME`) in Observe. Each decision is a `typesafe.system_one` span with type **Evaluator**, the requested model, token counts, and the answers as span attributes. + +## What a span contains + +One span per `system_one` call, named `typesafe.system_one`, kind `EVALUATOR`. Questions are attributes, not child spans: TypeSafe answers them all in one response. + +
+ +| Answer type | Attributes, under `typesafe.question..` | +| --- | --- | +| `noul` | `type`, `noul`: the probability of yes, from 0 to 1. A number, never a boolean, never Pass/Fail. No `confidence`: the API sends none, so the span has none | +| `choice` | `type`, `choice`: the chosen label, `confidence` | +| `score` | `type`, `score`: the expected score (may fall between levels), `confidence` | +| `box` | `type`, `box.top`, `box.bottom`, `box.left`, `box.right`, and `confidence` only when the API sent a number (it may send `null`) | + +
+ +
+ +| Attribute | Value | +| --- | --- | +| `gen_ai.provider.name` | `typesafe` | +| `gen_ai.request.model` | The model the request asked for, for example `jev-latest`. This is what the **Model** column shows | +| `gen_ai.response.model` | The model that answered. When it differs (an alias resolved to a specific version), `typesafe.response.model` is set too | +| `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.total_tokens` | As the API reported them. No cost is computed | +| `typesafe.question_count` | Number of questions in the request | +| `typesafe.request_id` | The `x-typesafe-request-id` response header, for TypeSafe support | +| `server.address` | The host the client called | +| `input.value` | JSON with the `state` and the questions (instructions and criteria). Images appear as their `` reference, never as bytes or a file path | +| `output.value` | JSON with every decision (the same values as the attributes above) | +| `typesafe.idempotency_key` | Only if you passed an `Idempotency-Key` header in `extra_headers` | +| `session.id`, `user.id`, metadata, tags | From `using_session(...)` and the other Future AGI context helpers, as on every traceAI span | + +
+ +Never on the span: the probability maps (`probabilities`), score legends, image bytes, the API key, and any explanation (the API returns none, so none is invented). + +## Privacy + +`traceAI-typesafe` follows the same `TraceConfig` rules as every traceAI instrumentor: content is captured by default, and the standard flags turn it off. + +
+ +| Setting | What is dropped | +| --- | --- | +| `FI_HIDE_INPUTS=true` or `TraceConfig(hide_inputs=True)` (also `FI_HIDE_INPUT_TEXT`) | `input.value` (the `state` and the questions) and `typesafe.idempotency_key` | +| `FI_HIDE_OUTPUTS=true` or `TraceConfig(hide_outputs=True)` (also `FI_HIDE_OUTPUT_TEXT`) | `output.value` and the chosen `choice` labels. Numeric answers stay | +| `FI_PII_REDACTION=true` | Masks emails, phone numbers and similar patterns in every string attribute and in the error message | + +
+ +Set the flags in the environment before the process starts, or pass a `TraceConfig`: + +```python +from fi_instrumentation import TraceConfig + +TypeSafeInstrumentor().instrument( + tracer_provider=trace_provider, + config=TraceConfig(hide_inputs=True, hide_outputs=True), +) +``` + +Recorded in every mode: question names and answer types, numeric answers (`noul`, `score`, `confidence`, box edges), model, token counts, request id and host. **Question names are attribute keys, so do not put personal data or secrets in them.** Never recorded in any mode: the API key, stack traces, and the text of errors raised inside the SDK before a request is sent (they can name a local file). + +## Errors and retries + +A failed call ends its span with status `ERROR` and these attributes: `error.type` (the SDK exception class, for example `TypeSafeAuthenticationError`), `http.response.status_code`, `typesafe.error_type` (TypeSafe's own code, for example `authentication_error`), `typesafe.request_id`, and `typesafe.retry_after_ms` on a 429. The status message is the class and the HTTP status. While content is captured, TypeSafe's error message is added to it (an API can echo request values, so it is left out when inputs or outputs are hidden). + +The instrumentor adds no retries. The SDK's own `RetryPolicy` (by default up to two retries on 408, 429 and 5xx responses) runs inside the one span, so a 429 followed by a success is a single `OK` span. If you call `system_one` again yourself, that is a second span. + +Errors raised by the SDK before a request is sent (no question, an image file it cannot read) also end the span as `ERROR`, with only the exception class. + +## Not supported + +- **`client.models.list()`** (`GET /v1/models`) is not a decision and creates no span. +- **The TypeSafe JavaScript SDK.** Python only. +- **Probability maps and legends.** Only the chosen value and its confidence are recorded. +- **A custom `response_model`.** The span reads the `answers` field; a model that drops it still gets a span with model and usage, but no per-question attributes. +- **`system-one.dev`.** A separate platform with its own keys and credits; this page does not cover it. If you point the client there, you still get a span, but its credit headers are not read. +- **Idempotency replay.** An `Idempotency-Key` you send is recorded; nothing is cached or replayed. + +## Troubleshooting + +- **No spans.** Call `TypeSafeInstrumentor().instrument(...)` before the first decision, and `trace_provider.force_flush()` before a short script exits. Check that `FI_API_KEY` and `FI_SECRET_KEY` are set in the same process. +- **`pip` cannot find a matching `traceAI-typesafe`.** Use Python 3.10 to 3.13. +- **`TypeSafeError: No API key was provided`.** The SDK needs `TYPESAFE_API_KEY` (or `api_key=`). This is raised when the client is created, before any span. +- **An `ERROR` span with `TypeSafeAuthenticationError (HTTP 401)`.** The key is wrong or revoked. +- **`TypeSafeUnprocessableEntityError (HTTP 422)`.** TypeSafe rejected the request body; with content captured, the message names the field (for example `questions.tone.choice.criteria`). A `choice` question needs a mapping of criteria, not a list. +- **The Model column shows `jev-latest`, not a version.** That is the model you asked for. The resolved version is in `gen_ai.response.model` and `typesafe.response.model`. +- **A `noul` shows no confidence.** Expected: the API returns none for `noul`. +- **The `choice` label is missing.** `FI_HIDE_OUTPUTS` or `FI_HIDE_OUTPUT_TEXT` is on.