From cb1f70ff9df7ae328b15684db2a45f394833d29f Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Fri, 16 Oct 2026 07:30:10 -0700 Subject: [PATCH 1/4] docs(traceai): TypeSafe (Jev) decision tracing (TH-8322) New page /docs/integrations/traceai/typesafe for traceAI-typesafe (traceAI PR #264): install, env vars, register + instrument, sync and async System One calls, span contents (EVALUATOR, per-question noul / choice / score / box attributes, models, usage), privacy (traceAI TraceConfig convention), errors and retries, not supported, troubleshooting, and how it differs from TH-8183 and traceAI-openai. examples/traceai/typesafe: app.py (the page's code), a local System One stand-in shaped like TypeSafe's published OpenAPI document, run_e2e.sh (stand-in, privacy comparison, real-API invalid-key probe) and tests. No navigation.ts change; the nav PR adds the page. Co-Authored-By: Claude Opus 5.5 --- examples/traceai/typesafe/.gitignore | 3 + examples/traceai/typesafe/README.md | 52 ++++ examples/traceai/typesafe/app.py | 163 +++++++++++++ examples/traceai/typesafe/pytest.ini | 3 + examples/traceai/typesafe/requirements.txt | 7 + examples/traceai/typesafe/run_e2e.sh | 93 ++++++++ .../typesafe/standin/systemone_stub.py | 204 ++++++++++++++++ examples/traceai/typesafe/tests/__init__.py | 0 examples/traceai/typesafe/tests/conftest.py | 94 ++++++++ examples/traceai/typesafe/tests/test_page.py | 65 +++++ .../traceai/typesafe/tests/test_recipe.py | 97 ++++++++ examples/traceai/typesafe/tests/test_stub.py | 71 ++++++ .../docs/integrations/traceai/typesafe.mdx | 223 ++++++++++++++++++ 13 files changed, 1075 insertions(+) create mode 100644 examples/traceai/typesafe/.gitignore create mode 100644 examples/traceai/typesafe/README.md create mode 100644 examples/traceai/typesafe/app.py create mode 100644 examples/traceai/typesafe/pytest.ini create mode 100644 examples/traceai/typesafe/requirements.txt create mode 100755 examples/traceai/typesafe/run_e2e.sh create mode 100644 examples/traceai/typesafe/standin/systemone_stub.py create mode 100644 examples/traceai/typesafe/tests/__init__.py create mode 100644 examples/traceai/typesafe/tests/conftest.py create mode 100644 examples/traceai/typesafe/tests/test_page.py create mode 100644 examples/traceai/typesafe/tests/test_recipe.py create mode 100644 examples/traceai/typesafe/tests/test_stub.py create mode 100644 src/pages/docs/integrations/traceai/typesafe.mdx diff --git a/examples/traceai/typesafe/.gitignore b/examples/traceai/typesafe/.gitignore new file mode 100644 index 000000000..77ac75498 --- /dev/null +++ b/examples/traceai/typesafe/.gitignore @@ -0,0 +1,3 @@ +.venv/ +__pycache__/ +*.pyc diff --git a/examples/traceai/typesafe/README.md b/examples/traceai/typesafe/README.md new file mode 100644 index 000000000..0ec07991c --- /dev/null +++ b/examples/traceai/typesafe/README.md @@ -0,0 +1,52 @@ +# TypeSafe (Jev) tracing example + +Runnable companion to the docs page `src/pages/docs/integrations/traceai/typesafe.mdx` +(`/docs/integrations/traceai/typesafe`). It traces TypeSafe System One decision calls made with the +official `typesafe-sdk` Python client using the `traceAI-typesafe` instrumentor +([traceAI PR #264](https://github.com/future-agi/traceAI/pull/264)). + +Status: alpha. **No TypeSafe account or API key was used.** The success path ran against the local +stand-in below; one failure case ran against the real `api.typesafe.ai` with a deliberately invalid key. + +## Contents + +| Path | What it is | +| --- | --- | +| `app.py` | The recipe the page shows (between the `--- ... start/end ---` markers) inside a small support-ticket triage app, with scenarios `ok`, `fail-422`, `fail-auth`, `fail-asset` | +| `run_e2e.sh` | Creates a venv, runs the tests, runs `app.py` against a Future AGI collector, optionally the privacy comparison and the real-API 401 probe | +| `standin/systemone_stub.py` | Local **STAND-IN** shaped like TypeSafe's published OpenAPI document (`POST /v1/systemone`, `GET /v1/models`, the real error bodies). Answers are hash-derived numbers, **not Jev** | +| `tests/` | pytest: the stand-in against the real SDK, `app.py` against a recording OTLP collector (spans, headers, no gzip, privacy markers), and page/example consistency | +| `requirements.txt` | Pinned versions the example was tested with | + +## Run it + +Python 3.10 to 3.13 (fi-instrumentation-otel 1.1.0 does not support 3.14). + +```bash +export FI_API_KEY=... FI_SECRET_KEY=... +export FI_BASE_URL=http://localhost:4318 # your Future AGI collector (run_e2e.sh defaults to this) +./run_e2e.sh # tests, then app.py against the stand-in. No TypeSafe needed +PRIVACY=1 ./run_e2e.sh # also: hide flags on vs default, with a unique marker +REAL_AUTH_PROBE=1 ./run_e2e.sh # also: one invalid-key call to the real api.typesafe.ai (HTTP 401) +``` + +Against the real TypeSafe API (needs your own account and key; a run is billed by TypeSafe): + +```bash +export TYPESAFE_API_KEY=... # goes to the SDK only +./run_e2e.sh +``` + +Until `traceAI-typesafe` is on PyPI, `run_e2e.sh` installs it from the traceAI PR commit +(`TRACEAI_TYPESAFE` overrides the pip requirement, for example a local checkout: +`TRACEAI_TYPESAFE=/path/to/traceAI/python/frameworks/typesafe`). + +## What is real and what is a stand-in + +| Piece | Real or stand-in | +| --- | --- | +| `typesafe-sdk` client (0.7.5), `traceAI-typesafe`, `fi-instrumentation-otel` (`register()`), OTLP export | Real | +| Future AGI collector, storage and Observe UI | Real (a local Future AGI) | +| System One answers (`noul`, `choice`, `score`) | **STAND-IN** (`standin/systemone_stub.py`) | +| HTTP 401 for an invalid key | Real `api.typesafe.ai` with `REAL_AUTH_PROBE=1`; the stand-in otherwise | +| HTTP 422 validation error | STAND-IN, using the FastAPI error shape TypeSafe's OpenAPI document declares | diff --git a/examples/traceai/typesafe/app.py b/examples/traceai/typesafe/app.py new file mode 100644 index 000000000..f220fb9ca --- /dev/null +++ b/examples/traceai/typesafe/app.py @@ -0,0 +1,163 @@ +"""Trace TypeSafe (Jev) System One decisions in Future AGI with traceAI-typesafe. + +The block between "recipe start" and "recipe end" is the code the docs page +shows. The rest is a small support-ticket triage app around it, with failure +scenarios, used by run_e2e.sh. + +Environment: FI_API_KEY, FI_SECRET_KEY (Future AGI), TYPESAFE_API_KEY +(TypeSafe), optional TYPESAFE_BASE_URL (another System One host) and +FI_BASE_URL (a self-hosted Future AGI collector). + + python app.py --scenario ok # triage one ticket: sync + async decisions, a session + python app.py --scenario fail-422 # the API rejects a malformed question (HTTP 422) + python app.py --scenario fail-auth # a wrong API key (HTTP 401) + python app.py --scenario fail-asset # an image file that does not exist (client-side error) + +RUN_MARKER (optional) is added to the ticket text (also as an email address) and the +question wording, so a privacy check can search the stored traces for it. +""" + +from __future__ import annotations + +import argparse +import asyncio +import tempfile +import uuid +from pathlib import Path + +from fi_instrumentation import FITracer, using_session +from fi_instrumentation.fi_types import FiSpanKindValues +from opentelemetry.trace import Status, StatusCode + +# --- recipe start --- +import os + +from fi_instrumentation import register +from fi_instrumentation.fi_types import ProjectType +from traceai_typesafe import TypeSafeInstrumentor +from typesafe_sdk import AsyncTypeSafeClient, Choice, Noul, Score, TypeSafeClient + +trace_provider = register( + project_type=ProjectType.OBSERVE, + project_name=os.environ.get("FI_PROJECT_NAME", "support-triage"), +) +TypeSafeInstrumentor().instrument(tracer_provider=trace_provider) +# --- recipe end --- + +MARKER = os.environ.get("RUN_MARKER", "") +TICKET = ( + "Order 1042: I was charged twice for the same order and nobody answers my emails. " + "Please refund the duplicate today. {0}".format(MARKER) +).strip() +if MARKER: # privacy check: an email address in the content too + TICKET += " Reply to {0}@example.com.".format(MARKER) + +# --- questions start --- +QUESTIONS = { + "billing": Noul(instructions="Is this message about billing? {0}".format(MARKER).strip()), + "tone": Choice( + instructions="What is the tone of this message?", + criteria={"calm": None, "angry": "Upset or hostile {0}".format(MARKER).strip(), "excited": None}, + ), + "urgency": Score( + instructions="How urgent is this message?", + criteria=["Can wait", "Needs attention this week", "Needs attention today"], + ), +} +# --- questions end --- + +tracer = FITracer(trace_provider.get_tracer("support-triage")) + + +def triage_sync() -> None: + # --- sync call start --- + with TypeSafeClient() as client: + result = client.system_one(state=TICKET, questions=QUESTIONS) + print("billing", result.nouls["billing"].noul) + print("tone", result.choices["tone"].choice, result.choices["tone"].confidence) + print("urgency", result.scores["urgency"].score, result.scores["urgency"].confidence) + # --- sync call end --- + print("RESULT model", result.model, "tokens", result.usage.input_tokens, result.usage.output_tokens) + with TypeSafeClient() as client: + models = client.models.list() # GET /v1/models: not a decision, no span + print("RESULT models", [model.name for model in models.models]) + + +async def triage_async() -> None: + # --- async call start --- + async with AsyncTypeSafeClient() as client: + result = await client.system_one( + state={"ticket": TICKET, "customer_tier": "gold"}, + questions={"escalate": Noul(instructions="Should a human agent take this ticket now?")}, + ) + print("escalate", result.nouls["escalate"].noul) + # --- async call end --- + + +def fail_422() -> None: + with TypeSafeClient() as client: + # A choice question needs a mapping of criteria; a list passes the SDK's + # own check and is rejected by the API. + client.system_one( + state=TICKET, + questions={"tone": {"type": "choice", "instructions": "Tone?", "criteria": ["calm", "angry"]}}, + ) + + +def fail_auth() -> None: + with TypeSafeClient(api_key="ts-invalid-e2e-key") as client: + client.system_one(state=TICKET, questions={"billing": QUESTIONS["billing"]}) + + +def fail_asset() -> None: + from typesafe_sdk import Image + + missing = Path(tempfile.gettempdir()) / "typesafe-e2e-{0}".format(MARKER or "x") / "photo.png" + photo = Image(missing) + with TypeSafeClient() as client: + client.system_one( + state="Product photo: {0}".format(photo), + questions={"damaged": Noul(instructions="Is the product damaged?")}, + assets={"photo": photo}, + ) + + +SCENARIOS = { + "ok": lambda: (triage_sync(), asyncio.run(triage_async())), + "fail-422": fail_422, + "fail-auth": fail_auth, + "fail-asset": fail_asset, +} + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--scenario", choices=sorted(SCENARIOS), default="ok") + args = parser.parse_args() + session_id = "triage-" + uuid.uuid4().hex[:12] + outcome = "ok" + with using_session(session_id): + # The app's own parent span. It records no exception text of its own. + with tracer.start_as_current_span( + "triage_ticket", + fi_span_kind=FiSpanKindValues.CHAIN, + record_exception=False, + set_status_on_exception=False, + ) as span: + span.set_input(TICKET) + trace_id = "{0:032x}".format(span.get_span_context().trace_id) + try: + SCENARIOS[args.scenario]() + span.set_output("triaged") + span.set_status(Status(StatusCode.OK)) + except Exception as error: # the failure scenarios end here + outcome = type(error).__name__ + span.set_status(Status(StatusCode.ERROR, outcome)) + trace_provider.force_flush() + print("RESULT scenario={0} outcome={1} trace_id={2} session_id={3}".format(args.scenario, outcome, trace_id, session_id)) + trace_provider.shutdown() + return 0 if (outcome == "ok") == (args.scenario == "ok") else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/traceai/typesafe/pytest.ini b/examples/traceai/typesafe/pytest.ini new file mode 100644 index 000000000..0e02926da --- /dev/null +++ b/examples/traceai/typesafe/pytest.ini @@ -0,0 +1,3 @@ +[pytest] +testpaths = tests +addopts = -p no:cacheprovider diff --git a/examples/traceai/typesafe/requirements.txt b/examples/traceai/typesafe/requirements.txt new file mode 100644 index 000000000..cf6d8bb8b --- /dev/null +++ b/examples/traceai/typesafe/requirements.txt @@ -0,0 +1,7 @@ +# Tested 2026-10-16 on Python 3.12 (fi-instrumentation-otel 1.1.0 requires Python < 3.14). +# traceAI-typesafe itself is installed by run_e2e.sh (TRACEAI_TYPESAFE) until it is on PyPI. +typesafe-sdk==0.7.5 +fi-instrumentation-otel==1.1.0 +# tests +pytest>=8 +opentelemetry-proto>=1.27 diff --git a/examples/traceai/typesafe/run_e2e.sh b/examples/traceai/typesafe/run_e2e.sh new file mode 100755 index 000000000..274460916 --- /dev/null +++ b/examples/traceai/typesafe/run_e2e.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# End-to-end run of the TypeSafe (Jev) tracing example against a Future AGI collector. +# +# FI_API_KEY=... FI_SECRET_KEY=... FI_BASE_URL=http://localhost:4318 ./run_e2e.sh +# +# What it does +# 1. creates ./.venv (Python 3.10 to 3.13) and installs requirements.txt plus traceAI-typesafe +# (TRACEAI_TYPESAFE: a pip requirement; set VENV_DIR to reuse an existing environment) +# 2. runs the tests (local stand-in and a recording collector; no Future AGI, no TypeSafe) +# 3. runs app.py: scenario ok (sync + async decisions, models.list), then the failure +# scenarios fail-422, fail-auth and fail-asset, tracing to FI_BASE_URL +# 4. PRIVACY=1: runs ok, fail-422 and fail-asset again with a unique RUN_MARKER, once with the +# hide flags on (FI_HIDE_INPUTS/FI_HIDE_OUTPUTS=true) into -hidden and once with the +# default into -default, and prints the markers so you can search your store +# 5. REAL_AUTH_PROBE=1: one fail-auth call to the real https://api.typesafe.ai with a +# deliberately invalid key (no account needed, nothing is billed) +# +# Which API it calls +# default the local stand-in in standin/ (NOT TypeSafe) +# TYPESAFE_API_KEY (TYPESAFE_BASE_URL unset) the real TypeSafe API, https://api.typesafe.ai. +# Needs your own account and key; a real run is billed. +# TYPESAFE_API_KEY + TYPESAFE_BASE_URL any System One host you choose +# +# Other knobs: FI_PROJECT_NAME (default typesafe-e2e), SKIP_TESTS=1, SCENARIOS="ok fail-422". +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +cd "$HERE" +: "${FI_API_KEY:?set FI_API_KEY}" "${FI_SECRET_KEY:?set FI_SECRET_KEY}" +export FI_BASE_URL="${FI_BASE_URL:-http://localhost:4318}" +export FI_PROJECT_NAME="${FI_PROJECT_NAME:-typesafe-e2e}" +# Until traceAI-typesafe is on PyPI, install it from the traceAI pull request commit. +TRACEAI_TYPESAFE="${TRACEAI_TYPESAFE:-traceAI-typesafe @ git+https://github.com/future-agi/traceAI@e6dd3c0e16e7bd034739fd2537f13a8ec2018b6e#subdirectory=python/frameworks/typesafe}" + +VENV_DIR="${VENV_DIR:-$HERE/.venv}" +if [ ! -x "$VENV_DIR/bin/python" ]; then + PY="${PYTHON:-$(command -v python3.12 || command -v python3.13 || command -v python3.11 || command -v python3.10 || true)}" + [ -n "$PY" ] || { echo "need python3.10 to python3.13 on PATH (or set PYTHON)"; exit 2; } + "$PY" -m venv "$VENV_DIR" + "$VENV_DIR/bin/python" -m pip install --quiet --upgrade pip + "$VENV_DIR/bin/python" -m pip install --quiet -r requirements.txt "$TRACEAI_TYPESAFE" +fi +PYBIN="$VENV_DIR/bin/python" +"$PYBIN" - <<'PY' +import importlib.metadata as m, sys +names = ["traceai-typesafe", "typesafe-sdk", "fi-instrumentation-otel", "opentelemetry-sdk"] +print("versions: python", sys.version.split()[0], *(f"{n}=={m.version(n)}" for n in names)) +PY + +if [ "${SKIP_TESTS:-0}" != "1" ]; then + "$PYBIN" -m pytest -q +fi + +STUB_PID="" +cleanup() { [ -n "$STUB_PID" ] && kill "$STUB_PID" 2>/dev/null || true; } +trap cleanup EXIT + +if [ -n "${TYPESAFE_API_KEY:-}" ]; then + echo "target: ${TYPESAFE_BASE_URL:-REAL TypeSafe, https://api.typesafe.ai}" +else + PORT="$("$PYBIN" -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1])')" + "$PYBIN" standin/systemone_stub.py --port "$PORT" >/dev/null 2>&1 & + STUB_PID=$! + for _ in $(seq 1 50); do curl -fs "http://127.0.0.1:$PORT/healthz" >/dev/null 2>&1 && break; sleep 0.2; done + export TYPESAFE_BASE_URL="http://127.0.0.1:$PORT" TYPESAFE_API_KEY="standin-key" + echo "target: STAND-IN (not TypeSafe) at $TYPESAFE_BASE_URL" +fi + +echo "== app.py -> System One, traces -> $FI_BASE_URL project=$FI_PROJECT_NAME" +for scenario in ${SCENARIOS:-ok fail-422 fail-auth fail-asset}; do + "$PYBIN" app.py --scenario "$scenario" | grep -E '^(RESULT|billing|tone|urgency|escalate)' +done + +if [ "${PRIVACY:-0}" = "1" ]; then + for mode in hidden default; do + MARK="TSPRIV-$(date +%s)-$RANDOM$RANDOM" + echo "== privacy run: mode=$mode marker=$MARK project=$FI_PROJECT_NAME-$mode" + for scenario in ok fail-422 fail-asset; do + if [ "$mode" = "hidden" ]; then + FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true RUN_MARKER="$MARK" FI_PROJECT_NAME="$FI_PROJECT_NAME-$mode" \ + "$PYBIN" app.py --scenario "$scenario" | grep -E '^RESULT scenario' + else + RUN_MARKER="$MARK" FI_PROJECT_NAME="$FI_PROJECT_NAME-$mode" "$PYBIN" app.py --scenario "$scenario" | grep -E '^RESULT scenario' + fi + done + done + echo "Search your Future AGI project -hidden for the hidden marker: it must not appear anywhere." +fi + +if [ "${REAL_AUTH_PROBE:-0}" = "1" ]; then + echo "== real api.typesafe.ai, invalid key (expects HTTP 401)" + env -u TYPESAFE_BASE_URL "$PYBIN" app.py --scenario fail-auth | grep -E '^RESULT' +fi diff --git a/examples/traceai/typesafe/standin/systemone_stub.py b/examples/traceai/typesafe/standin/systemone_stub.py new file mode 100644 index 000000000..fbbbdd2bc --- /dev/null +++ b/examples/traceai/typesafe/standin/systemone_stub.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python3 +"""STAND-IN for the TypeSafe System One API. This is NOT TypeSafe and NOT Jev. + +A small local server shaped like TypeSafe's published OpenAPI document +(https://api.typesafe.ai/openapi.json, version 0.2.0, read 2026-10-16) so the +real ``typesafe-sdk`` client can run end to end without a TypeSafe key: + +* ``POST /v1/systemone``: ``{model, state, questions[, assets]}`` in, + ``{model, answers, usage}`` out. ``noul`` answers carry only ``noul``; + ``choice`` answers ``choice``, ``confidence`` and ``probabilities``; + ``score`` answers ``score``, ``confidence``, ``legend`` and + ``probabilities``; ``box`` answers ``box`` and ``confidence: null``. +* ``GET /v1/models``: ``{"models": [{name, description, release_date}]}``. +* Errors use the bodies api.typesafe.ai returned on 2026-10-16: 403 with no + key, 401 for a key starting ``ts-invalid``, and FastAPI-style 422 + ``{"detail": [{loc, msg, type}]}`` for a body that breaks the schema. +* Every response has an ``x-typesafe-request-id`` header. + +Answers are deterministic numbers derived from a hash of the request. They +are not model output and say nothing about how Jev decides. ``jev-latest`` +resolves to ``jev-1.13.0`` in the response, as the API documents for aliases. + +Usage: ``python systemone_stub.py --port 8765``. Binds 127.0.0.1 only. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import uuid +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from typing import Any, Dict, List, Optional, Tuple + +MODELS = [ + {"name": "jev-latest", "description": "General-purpose system one model.", "release_date": "2026-09-15"}, + {"name": "jev-1.13.0", "description": "General-purpose system one model.", "release_date": "2026-09-15"}, +] +ALIASES = {"jev-latest": "jev-1.13.0"} +QUESTION_TYPES = ("noul", "choice", "score", "box") +NO_KEY = {"detail": {"error_type": "authentication_error", "message": "Must supply an API key! Check your request and try again."}} +BAD_KEY = {"detail": {"error_type": "authentication_error", "message": "Cannot authenticate with the server. Please check your API key and try again."}} + + +def _unit(*parts: str) -> float: + """A deterministic number in (0, 1) from the request.""" + digest = hashlib.sha256("\x1f".join(parts).encode()).digest() + return (int.from_bytes(digest[:4], "big") % 9973 + 1) / 9975 + + +def _distribution(names: List[str], seed: str) -> Dict[str, float]: + weights = [_unit(seed, name) ** 2 for name in names] + total = sum(weights) + return {name: round(weight / total, 4) for name, weight in zip(names, weights)} + + +def _answer(name: str, question: Dict[str, Any], state_key: str) -> Dict[str, Any]: + kind = question["type"] + seed = "{0}|{1}|{2}".format(state_key, name, json.dumps(question, sort_keys=True)) + if kind == "noul": + return {"type": "noul", "noul": round(_unit(seed), 4)} + if kind == "choice": + probabilities = _distribution(list(question["criteria"]), seed) + choice = max(probabilities, key=probabilities.get) + return {"type": "choice", "choice": choice, "confidence": probabilities[choice], "probabilities": probabilities} + if kind == "score": + levels = [str(index) for index in range(len(question["criteria"]))] + probabilities = _distribution(levels, seed) + score = round(sum(int(level) * p for level, p in probabilities.items()), 4) + return { + "type": "score", + "score": score, + "confidence": max(probabilities.values()), + "legend": dict(zip(levels, question["criteria"])), + "probabilities": probabilities, + } + top, left = round(_unit(seed, "t") / 2, 4), round(_unit(seed, "l") / 2, 4) + return {"type": "box", "box": {"top": top, "bottom": round(top + 0.25, 4), "left": left, "right": round(left + 0.25, 4)}, "confidence": None} + + +def validate(body: Any) -> List[Dict[str, Any]]: + """FastAPI-style validation errors for the SystemOneRequest schema (empty when valid).""" + if not isinstance(body, dict): + return [{"loc": ["body"], "msg": "Input should be a valid dictionary", "type": "dict_type"}] + errors: List[Dict[str, Any]] = [] + for field in ("model", "state", "questions"): + if field not in body: + errors.append({"loc": ["body", field], "msg": "Field required", "type": "missing"}) + if "model" in body and not isinstance(body["model"], str): + errors.append({"loc": ["body", "model"], "msg": "Input should be a valid string", "type": "string_type"}) + if "state" in body and not isinstance(body["state"], (str, dict, list)): + errors.append({"loc": ["body", "state"], "msg": "Input should be a valid string", "type": "string_type"}) + questions = body.get("questions") + if "questions" in body: + if not isinstance(questions, dict): + errors.append({"loc": ["body", "questions"], "msg": "Input should be a valid dictionary", "type": "dict_type"}) + elif not questions: + errors.append({"loc": ["body", "questions"], "msg": "Dictionary should have at least 1 item after validation, not 0", "type": "too_short"}) + else: + for name, question in questions.items(): + kind = question.get("type") if isinstance(question, dict) else None + if kind not in QUESTION_TYPES: + errors.append({"loc": ["body", "questions", name], "msg": "Input tag does not match any of the expected tags", "type": "union_tag_invalid"}) + elif kind == "choice" and "criteria" not in question: + errors.append({"loc": ["body", "questions", name, "choice", "criteria"], "msg": "Field required", "type": "missing"}) + elif kind == "choice" and not isinstance(question["criteria"], dict): + errors.append({"loc": ["body", "questions", name, "choice", "criteria"], "msg": "Input should be a valid dictionary", "type": "dict_type"}) + elif kind == "score" and not (isinstance(question.get("criteria"), list) and question["criteria"]): + errors.append({"loc": ["body", "questions", name, "score", "criteria"], "msg": "List should have at least 1 item after validation, not 0", "type": "too_short"}) + return errors + + +def _decode_body(content: bytes, encoding: str, content_type: str) -> Any: + if encoding == "zstd": + try: + from compression.zstd import decompress # Python 3.14 + except ImportError: + from backports.zstd import decompress # typesafe-sdk[zstd] on Python < 3.14 + content = decompress(content) + if content_type.startswith("multipart/form-data"): + boundary = content_type.split("boundary=", 1)[1].strip('"').encode() + for part in content.split(b"--" + boundary): + head, _, data = part.partition(b"\r\n\r\n") + if b'name="request"' in head: + return json.loads(data.rstrip(b"\r\n")) + raise ValueError("no request part") + return json.loads(content) + + +class Handler(BaseHTTPRequestHandler): + server_version = "systemone-standin/1" + + def log_message(self, fmt: str, *args: Any) -> None: # quiet: never log bodies or keys + return + + def _send(self, status: int, body: Any) -> None: + payload = json.dumps(body).encode() + self.send_response(status) + self.send_header("content-type", "application/json") + self.send_header("content-length", str(len(payload))) + self.send_header("x-typesafe-request-id", "req_standin_" + uuid.uuid4().hex) + self.end_headers() + self.wfile.write(payload) + + def _auth(self) -> Optional[Tuple[int, Any]]: + header = self.headers.get("authorization", "") + if not header.lower().startswith("bearer ") or not header[7:].strip(): + return 403, NO_KEY + if header[7:].strip().startswith("ts-invalid"): + return 401, BAD_KEY + return None + + def do_GET(self) -> None: # noqa: N802 + if self.path == "/healthz": + return self._send(200, {"ok": True, "standin": True}) + if self.path.split("?")[0] != "/v1/models": + return self._send(404, {"detail": "Not Found"}) + failure = self._auth() + if failure: + return self._send(*failure) + return self._send(200, {"models": MODELS}) + + def do_POST(self) -> None: # noqa: N802 + if self.path.split("?")[0] != "/v1/systemone": + return self._send(404, {"detail": "Not Found"}) + failure = self._auth() + if failure: + return self._send(*failure) + length = int(self.headers.get("content-length") or 0) + raw = self.rfile.read(length) + try: + body = _decode_body(raw, self.headers.get("content-encoding", ""), self.headers.get("content-type", "")) + except Exception: + return self._send(422, {"detail": [{"loc": ["body"], "msg": "JSON decode error", "type": "json_invalid"}]}) + errors = validate(body) + if errors: + return self._send(422, {"detail": errors}) + state_key = json.dumps(body["state"], sort_keys=True) + answers = {name: _answer(name, question, state_key) for name, question in body["questions"].items()} + self._send( + 200, + { + "model": ALIASES.get(body["model"], body["model"]), + "answers": answers, + "usage": {"input_tokens": max(1, len(raw) // 4), "output_tokens": 2 * len(answers)}, + }, + ) + + +def serve(port: int) -> ThreadingHTTPServer: + return ThreadingHTTPServer(("127.0.0.1", port), Handler) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--port", type=int, default=8765) + args = parser.parse_args() + server = serve(args.port) + print("STANDIN System One (not TypeSafe) on http://127.0.0.1:{0}".format(server.server_address[1]), flush=True) + server.serve_forever() + + +if __name__ == "__main__": + main() diff --git a/examples/traceai/typesafe/tests/__init__.py b/examples/traceai/typesafe/tests/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/examples/traceai/typesafe/tests/conftest.py b/examples/traceai/typesafe/tests/conftest.py new file mode 100644 index 000000000..98e991265 --- /dev/null +++ b/examples/traceai/typesafe/tests/conftest.py @@ -0,0 +1,94 @@ +"""Fixtures: the System One stand-in and a recording OTLP/HTTP collector, both on 127.0.0.1.""" + +from __future__ import annotations + +import gzip +import os +import sys +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from typing import Any, Dict, List + +import pytest + +HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, os.path.join(HERE, "standin")) + +import systemone_stub # noqa: E402 + + +@pytest.fixture() +def stub(): + server = systemone_stub.serve(0) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + yield "http://127.0.0.1:{0}".format(server.server_address[1]) + server.shutdown() + server.server_close() + + +class Recorder: + def __init__(self) -> None: + self.requests: List[Dict[str, Any]] = [] + + def spans(self) -> List[Dict[str, Any]]: + from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest + + found = [] + for request in self.requests: + message = ExportTraceServiceRequest() + message.ParseFromString(request["body"]) + for resource_spans in message.resource_spans: + resource = {kv.key: _value(kv.value) for kv in resource_spans.resource.attributes} + for scope_spans in resource_spans.scope_spans: + for span in scope_spans.spans: + found.append( + { + "name": span.name, + "trace_id": span.trace_id.hex(), + "span_id": span.span_id.hex(), + "parent": span.parent_span_id.hex(), + "status": span.status.code, + "status_message": span.status.message, + "attributes": {kv.key: _value(kv.value) for kv in span.attributes}, + "events": [[e.name, {kv.key: _value(kv.value) for kv in e.attributes}] for e in span.events], + "resource": resource, + } + ) + return found + + def raw(self) -> bytes: + return b"".join(request["body"] for request in self.requests) + + +def _value(value: Any) -> Any: + kind = value.WhichOneof("value") + return getattr(value, kind) if kind else None + + +@pytest.fixture() +def collector(): + recorder = Recorder() + + class Handler(BaseHTTPRequestHandler): + def log_message(self, *args: Any) -> None: + return + + def do_POST(self) -> None: # noqa: N802 + body = self.rfile.read(int(self.headers.get("content-length") or 0)) + if self.headers.get("content-encoding") == "gzip": + body = gzip.decompress(body) + recorder.gzip = True + recorder.requests.append({"path": self.path, "headers": dict(self.headers), "body": body}) + self.send_response(200) + self.send_header("content-type", "application/x-protobuf") + self.send_header("content-length", "0") + self.end_headers() + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + recorder.url = "http://127.0.0.1:{0}".format(server.server_address[1]) + yield recorder + server.shutdown() + server.server_close() diff --git a/examples/traceai/typesafe/tests/test_page.py b/examples/traceai/typesafe/tests/test_page.py new file mode 100644 index 000000000..db2d087f4 --- /dev/null +++ b/examples/traceai/typesafe/tests/test_page.py @@ -0,0 +1,65 @@ +"""The docs page and this example agree; no network.""" + +from __future__ import annotations + +import os +import re +import textwrap + +HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +PAGE = os.path.normpath(os.path.join(HERE, "..", "..", "..", "src", "pages", "docs", "integrations", "traceai", "typesafe.mdx")) + + +def _read(path): + with open(path, encoding="utf-8") as handle: + return handle.read() + + +def _blocks(language): + return re.findall(r"```{0}\n(.*?)```".format(language), _read(PAGE), re.S) + + +def _excerpt(name): + match = re.search(r"# --- {0} start ---\n(.*?)\s*# --- {0} end ---".format(name), _read(os.path.join(HERE, "app.py")), re.S) + assert match, name + return textwrap.dedent(match.group(1)).strip() + + +def test_every_page_snippet_from_the_app_is_an_exact_excerpt(): + blocks = [block.strip() for block in _blocks("python")] + for name in ("recipe", "questions", "sync call", "async call"): + assert _excerpt(name) in blocks, name + + +def test_page_install_and_tested_versions_match_requirements(): + page = _read(PAGE) + pins = dict(re.findall(r"^([A-Za-z0-9_.-]+)==([0-9.]+)", _read(os.path.join(HERE, "requirements.txt")), re.M)) + assert "typesafe-sdk **{0}**".format(pins["typesafe-sdk"]) in page + assert "fi-instrumentation-otel {0}".format(pins["fi-instrumentation-otel"]) in page + assert "pip install traceAI-typesafe typesafe-sdk" in page + + +def test_page_states_privacy_differences_and_the_stand_in(): + page = _read(PAGE) + for needle in ("FI_HIDE_INPUTS=true", "FI_HIDE_OUTPUTS=true", "TraceConfig(hide_inputs=True)", + "TH-8183", "traceAI-openai", "EVALUATOR", "STANDIN", "Not supported", "Troubleshooting", + "never Pass/Fail", "models.list()"): + assert needle in page, needle + + +def test_page_does_not_send_customers_to_another_platform_by_default(): + page = _read(PAGE) + assert "base_url=\"https://api.typesafe.ai\"" not in page + assert "pip install typesafe-ai" not in page and "pip install jev" not in page + + +def test_run_script_never_prints_keys(): + script = _read(os.path.join(HERE, "run_e2e.sh")) + for name in ("FI_API_KEY", "FI_SECRET_KEY", "TYPESAFE_API_KEY"): + assert not re.search(r"echo[^\n]*\$\{?" + name, script), name + assert "set -euo pipefail" in script + + +def test_stand_in_is_labelled(): + stub = _read(os.path.join(HERE, "standin", "systemone_stub.py")) + assert "NOT TypeSafe" in stub and "127.0.0.1" in stub diff --git a/examples/traceai/typesafe/tests/test_recipe.py b/examples/traceai/typesafe/tests/test_recipe.py new file mode 100644 index 000000000..a00faf705 --- /dev/null +++ b/examples/traceai/typesafe/tests/test_recipe.py @@ -0,0 +1,97 @@ +"""app.py, configured as the page says, exports the expected spans to a recording collector.""" + +from __future__ import annotations + +import os +import subprocess +import sys +import tempfile +import uuid + +import pytest + +pytest.importorskip("traceai_typesafe") +pytest.importorskip("opentelemetry.proto") + +HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +_HIDE = ("FI_HIDE_INPUTS", "FI_HIDE_OUTPUTS", "FI_HIDE_INPUT_TEXT", "FI_HIDE_OUTPUT_TEXT", "FI_PII_REDACTION") + + +def run(scenario, stub, collector, **env): + environment = {k: v for k, v in os.environ.items() if k not in _HIDE and not k.startswith("TYPESAFE_")} + environment.update( + FI_API_KEY="test-fi-key", + FI_SECRET_KEY="test-fi-secret", + FI_BASE_URL=collector.url, + FI_PROJECT_NAME="typesafe-test", + TYPESAFE_BASE_URL=stub, + TYPESAFE_API_KEY="standin-key", + **env, + ) + done = subprocess.run( + [sys.executable, "app.py", "--scenario", scenario], + cwd=HERE, env=environment, capture_output=True, text=True, timeout=120, + ) + assert done.returncode == 0, done.stdout[-2000:] + done.stderr[-2000:] + return done.stdout + + +def test_ok_scenario_spans(stub, collector): + out = run("ok", stub, collector) + assert "RESULT scenario=ok outcome=ok" in out + spans = collector.spans() + by_name = {} + for span in spans: + by_name.setdefault(span["name"], []).append(span) + (root,) = by_name["triage_ticket"] + decisions = by_name["typesafe.system_one"] + assert len(decisions) == 2 # sync + async; models.list makes none + assert len(spans) == 3 + for span in decisions: + a = span["attributes"] + assert span["parent"] == root["span_id"] + assert a["gen_ai.span.kind"] == "EVALUATOR" + assert a["gen_ai.provider.name"] == "typesafe" + assert a["gen_ai.request.model"] == "jev-latest" + assert a["typesafe.response.model"] == "jev-1.13.0" + assert a["session.id"] == root["attributes"]["session.id"] + assert span["resource"]["project_name"] == "typesafe-test" + sync = next(s for s in decisions if s["attributes"]["typesafe.question_count"] == 3) + assert sync["attributes"]["typesafe.question.tone.type"] == "choice" + assert "typesafe.question.billing.confidence" not in sync["attributes"] + headers = {k.lower(): v for k, v in collector.requests[0]["headers"].items()} + assert headers["x-api-key"] == "test-fi-key" and headers["x-secret-key"] == "test-fi-secret" + assert "gzip" not in headers.get("content-encoding", "") + assert collector.requests[0]["path"].endswith("/v1/traces") + + +@pytest.mark.parametrize("hidden", [True, False]) +def test_marker_only_with_content_capture_on(stub, collector, hidden): + marker = "TSTEST-" + uuid.uuid4().hex + env = {"RUN_MARKER": marker} + if hidden: + env.update(FI_HIDE_INPUTS="true", FI_HIDE_OUTPUTS="true") + for scenario in ("ok", "fail-422", "fail-asset"): + run(scenario, stub, collector, **env) + raw = collector.raw() + assert (marker.encode() in raw) is (not hidden) + # Never, in either mode: the stand-in key, the Future AGI keys, the temp path of the missing image. + for secret in ("standin-key", "test-fi-key", "test-fi-secret", tempfile.gettempdir()): + assert secret.encode() not in raw, secret + + +def test_failure_scenarios_are_error_spans(stub, collector): + for scenario in ("fail-422", "fail-auth", "fail-asset"): + run(scenario, stub, collector) + errors = { + s["attributes"]["error.type"]: s + for s in collector.spans() + if s["name"] == "typesafe.system_one" + } + assert set(errors) == {"TypeSafeUnprocessableEntityError", "TypeSafeAuthenticationError", "TypeSafeError"} + for span in errors.values(): + assert span["status"] == 2 # STATUS_CODE_ERROR + assert all("exception.stacktrace" not in attrs for _, attrs in span["events"]) + assert errors["TypeSafeAuthenticationError"]["attributes"]["http.response.status_code"] == 401 + assert errors["TypeSafeUnprocessableEntityError"]["attributes"]["http.response.status_code"] == 422 + assert errors["TypeSafeError"]["status_message"] == "TypeSafeError" diff --git a/examples/traceai/typesafe/tests/test_stub.py b/examples/traceai/typesafe/tests/test_stub.py new file mode 100644 index 000000000..038a9f638 --- /dev/null +++ b/examples/traceai/typesafe/tests/test_stub.py @@ -0,0 +1,71 @@ +"""The stand-in speaks the published System One contract to the real typesafe-sdk client.""" + +from __future__ import annotations + +import json +import urllib.error +import urllib.request + +import pytest + +typesafe_sdk = pytest.importorskip("typesafe_sdk") +from typesafe_sdk import Choice, Noul, RetryPolicy, Score, TypeSafeClient # noqa: E402 + + +def _client(stub, **kwargs): + kwargs.setdefault("api_key", "standin-key") + return TypeSafeClient(base_url=stub, retry=RetryPolicy(max_retries=0), **kwargs) + + +def test_answers_parse_into_the_sdk_types(stub): + with _client(stub) as client: + result = client.system_one( + state="I was charged twice.", + questions={ + "billing": Noul(instructions="Billing?"), + "tone": Choice(instructions="Tone?", criteria={"calm": None, "angry": None}), + "urgency": Score(instructions="Urgent?", criteria=["low", "high"]), + }, + ) + assert result.model == "jev-1.13.0" # jev-latest resolved, as documented for aliases + assert 0 < result.nouls["billing"].noul < 1 + assert result.choices["tone"].choice in {"calm", "angry"} + assert set(result.choices["tone"].probabilities) == {"calm", "angry"} + assert 0 <= result.scores["urgency"].score <= 1 + assert result.scores["urgency"].legend == {0: "low", 1: "high"} + assert result.usage.input_tokens > 0 and result.usage.output_tokens == 6 + assert result.request_id.startswith("req_standin_") + + +def test_noul_has_no_confidence_on_the_wire(stub): + with _client(stub) as client: + result = client.system_one(state="x", questions={"q": Noul(instructions="ok?")}) + raw = json.loads(result.raw_http_response.content) + assert raw["answers"]["q"] == {"type": "noul", "noul": result.nouls["q"].noul} + + +def test_errors_use_the_documented_bodies(stub): + from typesafe_sdk import TypeSafeAuthenticationError, TypeSafeUnprocessableEntityError + + with _client(stub, api_key="ts-invalid-x") as client: + with pytest.raises(TypeSafeAuthenticationError) as raised: + client.system_one(state="x", questions={"q": Noul(instructions="ok?")}) + assert raised.value.body["detail"]["error_type"] == "authentication_error" + with _client(stub) as client: + with pytest.raises(TypeSafeUnprocessableEntityError) as raised: + client.system_one(state="x", questions={"t": {"type": "choice", "criteria": ["a", "b"]}}) + assert raised.value.body["detail"][0]["loc"] == ["body", "questions", "t", "choice", "criteria"] + request = urllib.request.Request(stub + "/v1/systemone", data=b"{}", method="POST") + with pytest.raises(urllib.error.HTTPError) as no_key: + urllib.request.urlopen(request) + assert no_key.value.code == 403 + + +def test_models_list(stub): + with _client(stub) as client: + names = [model.name for model in client.models.list().models] + assert names == ["jev-latest", "jev-1.13.0"] + + +def test_binds_loopback_only(stub): + assert stub.startswith("http://127.0.0.1:") diff --git a/src/pages/docs/integrations/traceai/typesafe.mdx b/src/pages/docs/integrations/traceai/typesafe.mdx new file mode 100644 index 000000000..e46553dde --- /dev/null +++ b/src/pages/docs/integrations/traceai/typesafe.mdx @@ -0,0 +1,223 @@ +--- +title: "TypeSafe (Jev) decision tracing with Future AGI" +description: "Trace TypeSafe System One decision calls (Jev) made with the typesafe-sdk Python client: one evaluator span per call with every noul, choice, score and box answer, using traceAI-typesafe. Alpha." +--- + +## About + +[TypeSafe](https://typesafe.ai) serves Jev through the System One decision API: you send some content (`state`) and named questions, and get back a typed answer per question. It is not a chat API and returns no text. `traceAI-typesafe` instruments the official Python client, [`typesafe-sdk`](https://pypi.org/project/typesafe-sdk/): every `TypeSafeClient.system_one(...)` and `AsyncTypeSafeClient.system_one(...)` call becomes one `EVALUATOR` span in Future AGI, however many questions it asks. + +A run of the example on this page is one trace: + +``` +triage_ticket your own span (CHAIN) +├── typesafe.system_one billing (noul), tone (choice), urgency (score) (EVALUATOR) +└── typesafe.system_one escalate (noul), from the async client (EVALUATOR) +``` + +This is not the same as: + +- **Jev as an evaluator model in Future AGI.** Running Future AGI evals with Jev as the judge is a separate feature (tracked as TH-8183). This package adds no evaluator model, does not call Future AGI's gateway, and does not turn answers into Pass/Fail. It only traces calls your own code makes to TypeSafe. +- **[OpenAI](/docs/integrations/traceai/openai) (`traceAI-openai`).** System One is not OpenAI-compatible. Pointing `openai.OpenAI` at TypeSafe does not work, and `traceAI-openai` would not see the answers. + +
+ +| | | +| --- | --- | +| Tested with | typesafe-sdk **0.7.5** (latest on PyPI, released 2026-10-14) end to end on Python 3.12; unit tests also on typesafe-sdk 0.7.4 (Python 3.13) and 0.7.0 (Python 3.10); fi-instrumentation-otel 1.1.0; traceAI-typesafe 0.1.0 | +| Supported | typesafe-sdk 0.7.0 or later, below 1.0; Python 3.10 to 3.13 (fi-instrumentation-otel does not support 3.14 yet) | +| TypeSafe docs read | [Python SDK](https://docs.typesafe.ai/sdk/python), [System One](https://docs.typesafe.ai/concepts/system-one), [noul](https://docs.typesafe.ai/primitives/noul), [choice](https://docs.typesafe.ai/primitives/choice), [score](https://docs.typesafe.ai/primitives/score), [changelog](https://docs.typesafe.ai/sdk/python/changelog) and the [OpenAPI document](https://api.typesafe.ai/openapi.json) (version 0.2.0), all read 2026-10-16 | +| Source | [traceAI `python/frameworks/typesafe`](https://github.com/future-agi/traceAI/tree/main/python/frameworks/typesafe) and [examples/traceai/typesafe](https://github.com/future-agi/docs/tree/main/examples/traceai/typesafe) | + +
+ + +Alpha. No TypeSafe account or API key was used. What ran end to end: the real `typesafe-sdk` 0.7.5 client with this package, configured as on this page and exporting to a Future AGI collector, against a local stand-in server shaped like TypeSafe's published OpenAPI document (`STANDIN`, numbers from a hash, not Jev), plus one request to the real `api.typesafe.ai` with a deliberately invalid key to check the authentication-error span. The span for a real Jev answer is built from the same SDK response object, but has not been seen with a live key. [Run it yourself](#run-it-yourself) shows how to check it with yours. + + +## 1. Install + +```bash +pip install traceAI-typesafe typesafe-sdk +``` + + +`traceAI-typesafe` ships with [traceAI PR #264](https://github.com/future-agi/traceAI/pull/264). Until it is released to PyPI, install it from the PR commit: + +```bash +pip install "traceAI-typesafe @ git+https://github.com/future-agi/traceAI@e6dd3c0e16e7bd034739fd2537f13a8ec2018b6e#subdirectory=python/frameworks/typesafe" typesafe-sdk +``` + + +## 2. Set environment variables + +```bash +export FI_API_KEY="your-futureagi-api-key" +export FI_SECRET_KEY="your-futureagi-secret-key" +export TYPESAFE_API_KEY="your-typesafe-api-key" +``` + +The TypeSafe key goes to the SDK only and never onto a span. `FI_BASE_URL` is optional (set it to your own collector when you self-host). The SDK also reads `TYPESAFE_BASE_URL` and `TYPESAFE_DEFAULT_MODEL` itself; this package reads neither and never sets a host or a model. + +## 3. Register the tracer and instrument the client + +```python +import os + +from fi_instrumentation import register +from fi_instrumentation.fi_types import ProjectType +from traceai_typesafe import TypeSafeInstrumentor +from typesafe_sdk import AsyncTypeSafeClient, Choice, Noul, Score, TypeSafeClient + +trace_provider = register( + project_type=ProjectType.OBSERVE, + project_name=os.environ.get("FI_PROJECT_NAME", "support-triage"), +) +TypeSafeInstrumentor().instrument(tracer_provider=trace_provider) +``` + +Call `instrument()` once, before you make decisions. Clients created before or after are both traced. + +## 4. Make decisions + +Nothing changes in how you call TypeSafe. Ask your questions (`TICKET` is the text you want decided): + +```python +QUESTIONS = { + "billing": Noul(instructions="Is this message about billing? {0}".format(MARKER).strip()), + "tone": Choice( + instructions="What is the tone of this message?", + criteria={"calm": None, "angry": "Upset or hostile {0}".format(MARKER).strip(), "excited": None}, + ), + "urgency": Score( + instructions="How urgent is this message?", + criteria=["Can wait", "Needs attention this week", "Needs attention today"], + ), +} +``` + +(`MARKER` is empty unless you set `RUN_MARKER`; the example uses it for the privacy check below.) + +```python +with TypeSafeClient() as client: + result = client.system_one(state=TICKET, questions=QUESTIONS) +print("billing", result.nouls["billing"].noul) +print("tone", result.choices["tone"].choice, result.choices["tone"].confidence) +print("urgency", result.scores["urgency"].score, result.scores["urgency"].confidence) +``` + +The async client is traced the same way (inside an `async` function): + +```python +async with AsyncTypeSafeClient() as client: + result = await client.system_one( + state={"ticket": TICKET, "customer_tier": "gold"}, + questions={"escalate": Noul(instructions="Should a human agent take this ticket now?")}, + ) +print("escalate", result.nouls["escalate"].noul) +``` + +In a short-lived script, call `trace_provider.force_flush()` before the process exits so the last spans are sent. + +## What a span contains + +One span per `system_one` call, named `typesafe.system_one`, kind `EVALUATOR`. Questions are attributes, not child spans: TypeSafe answers them all in one response. + +
+ +| Answer type | Attributes, under `typesafe.question..` | +| --- | --- | +| `noul` | `type`, `noul`: the probability of yes, from 0 to 1. A number, never a boolean, never Pass/Fail. No `confidence`: the API sends none, so the span has none | +| `choice` | `type`, `choice`: the chosen label, `confidence` | +| `score` | `type`, `score`: the expected score (may fall between levels), `confidence` | +| `box` | `type`, `box.top`, `box.bottom`, `box.left`, `box.right`, and `confidence` only when the API sent a number (it may send `null`) | + +
+ +
+ +| Attribute | Value | +| --- | --- | +| `gen_ai.provider.name` | `typesafe` | +| `gen_ai.request.model` | The model the request asked for, for example `jev-latest`. This is what the **Model** column shows | +| `gen_ai.response.model` | The model that answered. When it differs (an alias resolved to a version such as `jev-1.13.0`), `typesafe.response.model` is set too | +| `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.total_tokens` | As the API reported them. No cost is computed | +| `typesafe.question_count` | Number of questions in the request | +| `typesafe.request_id` | The `x-typesafe-request-id` response header, for TypeSafe support | +| `server.address` | The host the client called | +| `input.value` | JSON with the `state` and the questions (instructions and criteria). Images appear as their `` reference, never as bytes or a file path | +| `output.value` | JSON with every decision (the same values as the attributes above) | +| `typesafe.idempotency_key` | Only if you passed an `Idempotency-Key` header in `extra_headers` | +| `session.id`, `user.id`, metadata, tags | From `using_session(...)` and the other Future AGI context helpers, as on every traceAI span | + +
+ +Never on the span: the probability maps (`probabilities`), score legends, image bytes, the API key, and any explanation (the API returns none, so none is invented). + +## Privacy + +`traceAI-typesafe` follows the same `TraceConfig` rules as every traceAI instrumentor: content is captured by default, and the standard flags turn it off. + +
+ +| Setting | What is dropped | +| --- | --- | +| `FI_HIDE_INPUTS=true` or `TraceConfig(hide_inputs=True)` (also `FI_HIDE_INPUT_TEXT`) | `input.value` (the `state` and the questions) and `typesafe.idempotency_key` | +| `FI_HIDE_OUTPUTS=true` or `TraceConfig(hide_outputs=True)` (also `FI_HIDE_OUTPUT_TEXT`) | `output.value` and the chosen `choice` labels. Numeric answers stay | +| `FI_PII_REDACTION=true` | Masks emails, phone numbers and similar patterns in every string attribute and in the error message | + +
+ +Set the flags in the environment before the process starts, or pass a `TraceConfig`: + +```python +from fi_instrumentation import TraceConfig + +TypeSafeInstrumentor().instrument( + tracer_provider=trace_provider, + config=TraceConfig(hide_inputs=True, hide_outputs=True), +) +``` + +Recorded in every mode: question names and answer types, numeric answers (`noul`, `score`, `confidence`, box edges), model, token counts, request id and host. **Question names are attribute keys, so do not put personal data or secrets in them.** Never recorded in any mode: the API key, stack traces, and the text of errors raised inside the SDK before a request is sent (they can name a local file). + +## Errors and retries + +A failed call ends its span with status `ERROR` and these attributes: `error.type` (the SDK exception class, for example `TypeSafeAuthenticationError`), `http.response.status_code`, `typesafe.error_type` (TypeSafe's own code, for example `authentication_error`), `typesafe.request_id`, and `typesafe.retry_after_ms` on a 429. The status message is the class and the HTTP status. While content is captured, TypeSafe's error message is added to it (an API can echo request values, so it is left out when inputs or outputs are hidden). + +The instrumentor adds no retries. The SDK's own `RetryPolicy` (by default up to two retries on 408, 429 and 5xx responses) runs inside the one span, so a 429 followed by a success is a single `OK` span. If you call `system_one` again yourself, that is a second span. + +Errors raised by the SDK before a request is sent (no question, an image file it cannot read) also end the span as `ERROR`, with only the exception class. + +## Not supported + +- **`client.models.list()`** (`GET /v1/models`) is not a decision and creates no span. +- **The TypeSafe JavaScript SDK.** Python only. +- **Probability maps and legends.** Only the chosen value and its confidence are recorded. +- **A custom `response_model`.** The span reads the `answers` field; a model that drops it still gets a span with model and usage, but no per-question attributes. +- **`system-one.dev`.** A separate platform with its own keys and credits; this page does not cover it. If you point the client there, you still get a span, but its credit headers are not read. +- **Idempotency replay.** An `Idempotency-Key` you send is recorded; nothing is cached or replayed. + +## Troubleshooting + +- **No spans.** Call `TypeSafeInstrumentor().instrument(...)` before the first decision, and `trace_provider.force_flush()` before a short script exits. Check that `FI_API_KEY` and `FI_SECRET_KEY` are set in the same process. +- **`TypeSafeError: No API key was provided`.** The SDK needs `TYPESAFE_API_KEY` (or `api_key=`). This is raised when the client is created, before any span. +- **An `ERROR` span with `TypeSafeAuthenticationError (HTTP 401)`.** The key is wrong or revoked. +- **`TypeSafeUnprocessableEntityError (HTTP 422)`.** TypeSafe rejected the request body; with content captured, the message names the field (for example `questions.tone.choice.criteria`). A `choice` question needs a mapping of criteria, not a list. +- **The Model column shows `jev-latest`, not a version.** That is the model you asked for. The resolved version is in `gen_ai.response.model` and `typesafe.response.model`. +- **A `noul` shows no confidence.** Expected: the API returns none for `noul`. +- **The `choice` label is missing.** `FI_HIDE_OUTPUTS` or `FI_HIDE_OUTPUT_TEXT` is on. + +## Run it yourself + +[examples/traceai/typesafe](https://github.com/future-agi/docs/tree/main/examples/traceai/typesafe) has the program above (`app.py`), a local System One stand-in, tests, and `run_e2e.sh`: + +```bash +cd examples/traceai/typesafe +export FI_API_KEY=... FI_SECRET_KEY=... +./run_e2e.sh # tests, then app.py against the local stand-in (not TypeSafe) +PRIVACY=1 ./run_e2e.sh # also: a unique marker with the hide flags on, and with the default +TYPESAFE_API_KEY=... ./run_e2e.sh # against the real api.typesafe.ai with your own key +``` + +With your own key, the run makes two System One decisions (a few hundred input tokens), one request the API rejects with 422, one request with a deliberately invalid key, and one `GET /v1/models`. From 888c917d1820f028eecec510ae5b11f96c8436e2 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Fri, 16 Oct 2026 18:43:26 -0700 Subject: [PATCH 2/4] docs(traceai): TypeSafe live vendor verification 2026-10-17 (TH-8322) Real api.typesafe.ai run (typesafe-sdk 0.7.5, jev-1.13.0): trace ids, fi_verify excerpt, privacy scan. Drop the no-key/alpha warning; keep STANDIN for the default local run and the paths not seen live (429, box, assets). Co-Authored-By: Claude Sonnet 5.5 --- examples/traceai/typesafe/README.md | 14 +++--- .../docs/integrations/traceai/typesafe.mdx | 46 ++++++++++++++++--- 2 files changed, 48 insertions(+), 12 deletions(-) diff --git a/examples/traceai/typesafe/README.md b/examples/traceai/typesafe/README.md index 0ec07991c..2cb103dc7 100644 --- a/examples/traceai/typesafe/README.md +++ b/examples/traceai/typesafe/README.md @@ -5,8 +5,10 @@ Runnable companion to the docs page `src/pages/docs/integrations/traceai/typesaf official `typesafe-sdk` Python client using the `traceAI-typesafe` instrumentor ([traceAI PR #264](https://github.com/future-agi/traceAI/pull/264)). -Status: alpha. **No TypeSafe account or API key was used.** The success path ran against the local -stand-in below; one failure case ran against the real `api.typesafe.ai` with a deliberately invalid key. +Status: verified against the live TypeSafe API on 2026-10-17 (typesafe-sdk 0.7.5, model `jev-1.13.0`): two real +decisions (sync and async), a real 422 and a real 401, traced into a local Future AGI. The default `./run_e2e.sh` +(no `TYPESAFE_API_KEY`) still uses the local STANDIN server below, so it needs no account. Not yet seen live: 429 +(`retry_after_ms`), `box` answers, image assets. ## Contents @@ -30,7 +32,7 @@ PRIVACY=1 ./run_e2e.sh # also: hide flags on vs default REAL_AUTH_PROBE=1 ./run_e2e.sh # also: one invalid-key call to the real api.typesafe.ai (HTTP 401) ``` -Against the real TypeSafe API (needs your own account and key; a run is billed by TypeSafe): +Against the real TypeSafe API (needs your own account and key; a run is billed by TypeSafe, two decisions of a few hundred tokens each): ```bash export TYPESAFE_API_KEY=... # goes to the SDK only @@ -47,6 +49,6 @@ Until `traceAI-typesafe` is on PyPI, `run_e2e.sh` installs it from the traceAI P | --- | --- | | `typesafe-sdk` client (0.7.5), `traceAI-typesafe`, `fi-instrumentation-otel` (`register()`), OTLP export | Real | | Future AGI collector, storage and Observe UI | Real (a local Future AGI) | -| System One answers (`noul`, `choice`, `score`) | **STAND-IN** (`standin/systemone_stub.py`) | -| HTTP 401 for an invalid key | Real `api.typesafe.ai` with `REAL_AUTH_PROBE=1`; the stand-in otherwise | -| HTTP 422 validation error | STAND-IN, using the FastAPI error shape TypeSafe's OpenAPI document declares | +| System One answers (`noul`, `choice`, `score`) | **Real** `api.typesafe.ai` when `TYPESAFE_API_KEY` is set (verified 2026-10-17); **STANDIN** (`standin/systemone_stub.py`, hash-derived numbers, not Jev) otherwise, and in `tests/` | +| HTTP 401 for an invalid key | Real `api.typesafe.ai` (`REAL_AUTH_PROBE=1`, or any run with a key set); the stand-in otherwise | +| HTTP 422 validation error | Real `api.typesafe.ai` when a key is set; STANDIN otherwise (FastAPI error shape from TypeSafe's OpenAPI document) | diff --git a/src/pages/docs/integrations/traceai/typesafe.mdx b/src/pages/docs/integrations/traceai/typesafe.mdx index e46553dde..107e48445 100644 --- a/src/pages/docs/integrations/traceai/typesafe.mdx +++ b/src/pages/docs/integrations/traceai/typesafe.mdx @@ -1,6 +1,6 @@ --- title: "TypeSafe (Jev) decision tracing with Future AGI" -description: "Trace TypeSafe System One decision calls (Jev) made with the typesafe-sdk Python client: one evaluator span per call with every noul, choice, score and box answer, using traceAI-typesafe. Alpha." +description: "Trace TypeSafe System One decision calls (Jev) made with the typesafe-sdk Python client: one evaluator span per call with every noul, choice, score and box answer, using traceAI-typesafe. Verified against the live TypeSafe API." --- ## About @@ -24,16 +24,16 @@ This is not the same as: | | | | --- | --- | -| Tested with | typesafe-sdk **0.7.5** (latest on PyPI, released 2026-10-14) end to end on Python 3.12; unit tests also on typesafe-sdk 0.7.4 (Python 3.13) and 0.7.0 (Python 3.10); fi-instrumentation-otel 1.1.0; traceAI-typesafe 0.1.0 | +| Tested with | typesafe-sdk **0.7.5** (latest on PyPI, released 2026-10-14) end to end on Python 3.12 against the live `api.typesafe.ai` (2026-10-17, model `jev-1.13.0`); unit tests also on typesafe-sdk 0.7.4 (Python 3.13) and 0.7.0 (Python 3.10); fi-instrumentation-otel 1.1.0; traceAI-typesafe 0.1.0 | | Supported | typesafe-sdk 0.7.0 or later, below 1.0; Python 3.10 to 3.13 (fi-instrumentation-otel does not support 3.14 yet) | | TypeSafe docs read | [Python SDK](https://docs.typesafe.ai/sdk/python), [System One](https://docs.typesafe.ai/concepts/system-one), [noul](https://docs.typesafe.ai/primitives/noul), [choice](https://docs.typesafe.ai/primitives/choice), [score](https://docs.typesafe.ai/primitives/score), [changelog](https://docs.typesafe.ai/sdk/python/changelog) and the [OpenAPI document](https://api.typesafe.ai/openapi.json) (version 0.2.0), all read 2026-10-16 | | Source | [traceAI `python/frameworks/typesafe`](https://github.com/future-agi/traceAI/tree/main/python/frameworks/typesafe) and [examples/traceai/typesafe](https://github.com/future-agi/docs/tree/main/examples/traceai/typesafe) | - -Alpha. No TypeSafe account or API key was used. What ran end to end: the real `typesafe-sdk` 0.7.5 client with this package, configured as on this page and exporting to a Future AGI collector, against a local stand-in server shaped like TypeSafe's published OpenAPI document (`STANDIN`, numbers from a hash, not Jev), plus one request to the real `api.typesafe.ai` with a deliberately invalid key to check the authentication-error span. The span for a real Jev answer is built from the same SDK response object, but has not been seen with a live key. [Run it yourself](#run-it-yourself) shows how to check it with yours. - + +Verified against the live TypeSafe API on 2026-10-17 (see [Live vendor verification](#live-vendor-verification-2026-10-17)): real Jev answers, real tokens and model, a real 422 and a real 401, exported to Future AGI. Not yet seen live: a 429 with `typesafe.retry_after_ms`, a `box` answer, image assets, and `system_one` retries. Those paths are covered by unit tests and the local `STANDIN` server only. The package itself is still an early (0.1.0) release. + ## 1. Install @@ -215,9 +215,43 @@ Errors raised by the SDK before a request is sent (no question, an image file it ```bash cd examples/traceai/typesafe export FI_API_KEY=... FI_SECRET_KEY=... -./run_e2e.sh # tests, then app.py against the local stand-in (not TypeSafe) +./run_e2e.sh # tests, then app.py against the local STANDIN server (not TypeSafe) PRIVACY=1 ./run_e2e.sh # also: a unique marker with the hide flags on, and with the default TYPESAFE_API_KEY=... ./run_e2e.sh # against the real api.typesafe.ai with your own key ``` With your own key, the run makes two System One decisions (a few hundred input tokens), one request the API rejects with 422, one request with a deliberately invalid key, and one `GET /v1/models`. + +## Live vendor verification (2026-10-17) + +Run exactly as this page says, with `TYPESAFE_API_KEY` set and no `TYPESAFE_BASE_URL`, so the SDK called the real `https://api.typesafe.ai`; traces went to a local Future AGI (project `th8103-rest-typesafe-jev-live`). Versions: typesafe-sdk 0.7.5, traceAI-typesafe 0.1.0 (traceAI PR #264, commit `e6dd3c0e16e7`), fi-instrumentation-otel 1.1.0, opentelemetry-sdk 1.45.1, Python 3.12.10. The request resolved `jev-latest` to `jev-1.13.0`. + +
+ +| Case | Trace id | Result | +| --- | --- | --- | +| `ok`: sync decision (noul, choice, score) and async decision (noul) | `5e0d9155-271c-4183-924f-8f9411b53701` | Two `EVALUATOR` spans under `triage_ticket`, status OK, 397/73 and 318/22 tokens | +| `fail-422`: malformed `choice` question | `1d89de2c-3560-411f-a21f-71c4b7eda3f2` | `ERROR`, `TypeSafeUnprocessableEntityError (HTTP 422)` | +| `fail-auth`: invalid key | `11a1e3da-31e5-49d6-8709-ca1462bfc54b` | `ERROR`, `TypeSafeAuthenticationError (HTTP 401)` | +| `fail-asset`: missing image file | `d45fd1ab-0cb3-4609-948b-aa9413f60937` | `ERROR`, `TypeSafeError`, no path stored | + +
+ +`fi_verify` (hierarchy, model, tokens, errors) excerpt: + +``` +traces=5 spans=11 +trace 5e0d9155-271c-4183-924f-8f9411b53701 (3 spans) +- [chain] triage_ticket in=115B out=7B (616ms) + - [evaluator] typesafe.system_one model=jev-latest tok=397/73 in=534B out=172B (179ms) + - [evaluator] typesafe.system_one model=jev-latest tok=318/22 in=272B out=44B (245ms) +trace 1d89de2c-3560-411f-a21f-71c4b7eda3f2 (2 spans) + - [evaluator] typesafe.system_one status=ERROR:TypeSafeUnprocessableEntityError (HTTP 422) +trace 11a1e3da-31e5-49d6-8709-ca1462bfc54b (2 spans) + - [evaluator] typesafe.system_one status=ERROR:TypeSafeAuthenticationError (HTTP 401) +PASS +``` + +On the success span: `gen_ai.request.model=jev-latest`, `gen_ai.response.model=jev-1.13.0`, `typesafe.response.model=jev-1.13.0`, `gen_ai.usage.*` filled, `server.address=api.typesafe.ai`, a real `typesafe.request_id` (`req_...`), and numeric answers under `typesafe.question..*`. + +Privacy, against the live API: a unique marker in the ticket, an email built from it, the question instructions and a criteria description, with `FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true` (project `...-live-hidden`) and with the defaults (`...-live-default`). Scanning every stored column of every table for the project (ClickHouse and Postgres, including `resource_attrs`, `span_events`, `attributes_extra` and `status_message`): with the hide flags on, the marker, the email, the ticket text, the instruction and criteria text, the `choice` label, the TypeSafe key, the Future AGI keys, `Bearer `, the user name, the host name, the home directory, `Traceback`, `probabilities` and `legend` appear in 0 rows. With the defaults, the marker and ticket text appear in 9 rows (`input`, `attributes_extra`) and the TypeSafe and Future AGI keys in 0. From 2327d8b1b44da42aa9cc748bfbaf970744a7f8a9 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Wed, 21 Oct 2026 16:27:34 -0700 Subject: [PATCH 3/4] docs(typesafe): install traceAI-typesafe from PyPI; drop the PR-commit install (TH-8322) The page goes live only after traceAI-typesafe 0.1.0 is on PyPI. Removes the git+ install Note pinned to traceAI#264's old commit. run_e2e.sh now installs traceAI-typesafe==0.1.0 by default (TRACEAI_TYPESAFE still overrides it); the example README and requirements comment say the same. Example tests: 15 passed with the package from traceAI#264. Co-Authored-By: Claude Opus 5.5 --- examples/traceai/typesafe/README.md | 6 +++--- examples/traceai/typesafe/requirements.txt | 2 +- examples/traceai/typesafe/run_e2e.sh | 5 +++-- src/pages/docs/integrations/traceai/typesafe.mdx | 8 -------- 4 files changed, 7 insertions(+), 14 deletions(-) diff --git a/examples/traceai/typesafe/README.md b/examples/traceai/typesafe/README.md index 2cb103dc7..81ea41a53 100644 --- a/examples/traceai/typesafe/README.md +++ b/examples/traceai/typesafe/README.md @@ -39,9 +39,9 @@ export TYPESAFE_API_KEY=... # goes to the SDK only ./run_e2e.sh ``` -Until `traceAI-typesafe` is on PyPI, `run_e2e.sh` installs it from the traceAI PR commit -(`TRACEAI_TYPESAFE` overrides the pip requirement, for example a local checkout: -`TRACEAI_TYPESAFE=/path/to/traceAI/python/frameworks/typesafe`). +`run_e2e.sh` installs `traceAI-typesafe==0.1.0` from PyPI. `TRACEAI_TYPESAFE` overrides +that pip requirement, for example with a local checkout: +`TRACEAI_TYPESAFE=/path/to/traceAI/python/frameworks/typesafe`. ## What is real and what is a stand-in diff --git a/examples/traceai/typesafe/requirements.txt b/examples/traceai/typesafe/requirements.txt index cf6d8bb8b..d4b2ab3cc 100644 --- a/examples/traceai/typesafe/requirements.txt +++ b/examples/traceai/typesafe/requirements.txt @@ -1,5 +1,5 @@ # Tested 2026-10-16 on Python 3.12 (fi-instrumentation-otel 1.1.0 requires Python < 3.14). -# traceAI-typesafe itself is installed by run_e2e.sh (TRACEAI_TYPESAFE) until it is on PyPI. +# traceAI-typesafe itself is installed by run_e2e.sh (TRACEAI_TYPESAFE, default traceAI-typesafe==0.1.0). typesafe-sdk==0.7.5 fi-instrumentation-otel==1.1.0 # tests diff --git a/examples/traceai/typesafe/run_e2e.sh b/examples/traceai/typesafe/run_e2e.sh index 274460916..d04a579a6 100755 --- a/examples/traceai/typesafe/run_e2e.sh +++ b/examples/traceai/typesafe/run_e2e.sh @@ -29,8 +29,9 @@ cd "$HERE" : "${FI_API_KEY:?set FI_API_KEY}" "${FI_SECRET_KEY:?set FI_SECRET_KEY}" export FI_BASE_URL="${FI_BASE_URL:-http://localhost:4318}" export FI_PROJECT_NAME="${FI_PROJECT_NAME:-typesafe-e2e}" -# Until traceAI-typesafe is on PyPI, install it from the traceAI pull request commit. -TRACEAI_TYPESAFE="${TRACEAI_TYPESAFE:-traceAI-typesafe @ git+https://github.com/future-agi/traceAI@e6dd3c0e16e7bd034739fd2537f13a8ec2018b6e#subdirectory=python/frameworks/typesafe}" +# The released package from PyPI; set TRACEAI_TYPESAFE to another pip requirement (for example a +# local traceAI checkout) to test an unreleased build. +TRACEAI_TYPESAFE="${TRACEAI_TYPESAFE:-traceAI-typesafe==0.1.0}" VENV_DIR="${VENV_DIR:-$HERE/.venv}" if [ ! -x "$VENV_DIR/bin/python" ]; then diff --git a/src/pages/docs/integrations/traceai/typesafe.mdx b/src/pages/docs/integrations/traceai/typesafe.mdx index 107e48445..05ba1fc2d 100644 --- a/src/pages/docs/integrations/traceai/typesafe.mdx +++ b/src/pages/docs/integrations/traceai/typesafe.mdx @@ -41,14 +41,6 @@ Verified against the live TypeSafe API on 2026-10-17 (see [Live vendor verificat pip install traceAI-typesafe typesafe-sdk ``` - -`traceAI-typesafe` ships with [traceAI PR #264](https://github.com/future-agi/traceAI/pull/264). Until it is released to PyPI, install it from the PR commit: - -```bash -pip install "traceAI-typesafe @ git+https://github.com/future-agi/traceAI@e6dd3c0e16e7bd034739fd2537f13a8ec2018b6e#subdirectory=python/frameworks/typesafe" typesafe-sdk -``` - - ## 2. Set environment variables ```bash From 96c26c9786e3c9e8c7de1809f50a786e303fb5cf Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Thu, 5 Nov 2026 21:04:58 -0800 Subject: [PATCH 4/4] docs(traceai): make the TypeSafe (Jev) page a clean setup guide The page used MARKER and TICKET in its snippets without defining them, so the copied example did not run. The questions snippet now defines a concrete ticket, and the privacy-check marker is applied in app.py outside the excerpt the page shows. Live-verification tables, trace IDs, fi_verify output, privacy-scan results and stand-in harness instructions are engineering evidence, not setup steps, so they move out of the customer page (they remain in the example and its tests). The Python limit is now attributed to the released traceAI-typesafe 0.1.0 metadata (>=3.10,<3.14); the old fi-instrumentation-otel explanation no longer holds. The page is registered in src/lib/integrations.mjs as a traceAI model provider so it is discoverable, and a page-contract test runs in test:integrations so CI checks the snippets against app.py. Co-Authored-By: Claude Opus 5.5 --- examples/traceai/typesafe/README.md | 6 +- examples/traceai/typesafe/app.py | 24 +++-- examples/traceai/typesafe/requirements.txt | 2 +- examples/traceai/typesafe/tests/test_page.py | 28 ++++-- scripts/integrations.test.mjs | 1 + scripts/typesafe-page.test.mjs | 51 ++++++++++ src/lib/integrations.mjs | 1 + .../docs/integrations/traceai/typesafe.mdx | 96 ++++--------------- 8 files changed, 115 insertions(+), 94 deletions(-) create mode 100644 scripts/typesafe-page.test.mjs diff --git a/examples/traceai/typesafe/README.md b/examples/traceai/typesafe/README.md index 81ea41a53..f263d4eed 100644 --- a/examples/traceai/typesafe/README.md +++ b/examples/traceai/typesafe/README.md @@ -2,8 +2,8 @@ Runnable companion to the docs page `src/pages/docs/integrations/traceai/typesafe.mdx` (`/docs/integrations/traceai/typesafe`). It traces TypeSafe System One decision calls made with the -official `typesafe-sdk` Python client using the `traceAI-typesafe` instrumentor -([traceAI PR #264](https://github.com/future-agi/traceAI/pull/264)). +official `typesafe-sdk` Python client using the +[`traceAI-typesafe`](https://pypi.org/project/traceAI-typesafe/) instrumentor. Status: verified against the live TypeSafe API on 2026-10-17 (typesafe-sdk 0.7.5, model `jev-1.13.0`): two real decisions (sync and async), a real 422 and a real 401, traced into a local Future AGI. The default `./run_e2e.sh` @@ -22,7 +22,7 @@ decisions (sync and async), a real 422 and a real 401, traced into a local Futur ## Run it -Python 3.10 to 3.13 (fi-instrumentation-otel 1.1.0 does not support 3.14). +Python 3.10 to 3.13 (traceAI-typesafe 0.1.0 requires Python below 3.14). ```bash export FI_API_KEY=... FI_SECRET_KEY=... diff --git a/examples/traceai/typesafe/app.py b/examples/traceai/typesafe/app.py index f220fb9ca..e27731189 100644 --- a/examples/traceai/typesafe/app.py +++ b/examples/traceai/typesafe/app.py @@ -44,20 +44,17 @@ TypeSafeInstrumentor().instrument(tracer_provider=trace_provider) # --- recipe end --- -MARKER = os.environ.get("RUN_MARKER", "") +# --- questions start --- TICKET = ( "Order 1042: I was charged twice for the same order and nobody answers my emails. " - "Please refund the duplicate today. {0}".format(MARKER) -).strip() -if MARKER: # privacy check: an email address in the content too - TICKET += " Reply to {0}@example.com.".format(MARKER) + "Please refund the duplicate today." +) -# --- questions start --- QUESTIONS = { - "billing": Noul(instructions="Is this message about billing? {0}".format(MARKER).strip()), + "billing": Noul(instructions="Is this message about billing?"), "tone": Choice( instructions="What is the tone of this message?", - criteria={"calm": None, "angry": "Upset or hostile {0}".format(MARKER).strip(), "excited": None}, + criteria={"calm": None, "angry": "Upset or hostile", "excited": None}, ), "urgency": Score( instructions="How urgent is this message?", @@ -66,6 +63,17 @@ } # --- questions end --- +# Privacy check only (not part of the page): put a unique marker in the ticket text, +# an email address, a question instruction and a criteria description. +MARKER = os.environ.get("RUN_MARKER", "") +if MARKER: + TICKET += " {0} Reply to {0}@example.com.".format(MARKER) + QUESTIONS["billing"] = Noul(instructions="Is this message about billing? {0}".format(MARKER)) + QUESTIONS["tone"] = Choice( + instructions="What is the tone of this message?", + criteria={"calm": None, "angry": "Upset or hostile {0}".format(MARKER), "excited": None}, + ) + tracer = FITracer(trace_provider.get_tracer("support-triage")) diff --git a/examples/traceai/typesafe/requirements.txt b/examples/traceai/typesafe/requirements.txt index d4b2ab3cc..882b0264a 100644 --- a/examples/traceai/typesafe/requirements.txt +++ b/examples/traceai/typesafe/requirements.txt @@ -1,4 +1,4 @@ -# Tested 2026-10-16 on Python 3.12 (fi-instrumentation-otel 1.1.0 requires Python < 3.14). +# Python 3.10 to 3.13 (traceAI-typesafe 0.1.0 requires Python < 3.14). # traceAI-typesafe itself is installed by run_e2e.sh (TRACEAI_TYPESAFE, default traceAI-typesafe==0.1.0). typesafe-sdk==0.7.5 fi-instrumentation-otel==1.1.0 diff --git a/examples/traceai/typesafe/tests/test_page.py b/examples/traceai/typesafe/tests/test_page.py index db2d087f4..dd70d35a8 100644 --- a/examples/traceai/typesafe/tests/test_page.py +++ b/examples/traceai/typesafe/tests/test_page.py @@ -31,22 +31,36 @@ def test_every_page_snippet_from_the_app_is_an_exact_excerpt(): assert _excerpt(name) in blocks, name -def test_page_install_and_tested_versions_match_requirements(): +def test_page_install_and_supported_range(): page = _read(PAGE) - pins = dict(re.findall(r"^([A-Za-z0-9_.-]+)==([0-9.]+)", _read(os.path.join(HERE, "requirements.txt")), re.M)) - assert "typesafe-sdk **{0}**".format(pins["typesafe-sdk"]) in page - assert "fi-instrumentation-otel {0}".format(pins["fi-instrumentation-otel"]) in page assert "pip install traceAI-typesafe typesafe-sdk" in page + assert "Python 3.10 to 3.13, as required by `traceAI-typesafe` 0.1.0" in page + assert "`typesafe-sdk` 0.7.0 or later, below 1.0" in page + assert "fi-instrumentation-otel does not support" not in page -def test_page_states_privacy_differences_and_the_stand_in(): +def test_page_snippets_define_what_they_use(): + questions = next(block for block in _blocks("python") if "QUESTIONS = {" in block) + assert questions.lstrip().startswith("TICKET = (") + assert "MARKER" not in _read(PAGE) + + +def test_page_states_privacy_and_scope(): page = _read(PAGE) for needle in ("FI_HIDE_INPUTS=true", "FI_HIDE_OUTPUTS=true", "TraceConfig(hide_inputs=True)", - "TH-8183", "traceAI-openai", "EVALUATOR", "STANDIN", "Not supported", "Troubleshooting", - "never Pass/Fail", "models.list()"): + "traceAI-openai", "EVALUATOR", "Not supported", "Troubleshooting", + "never turns answers into Pass/Fail", "models.list()"): assert needle in page, needle +def test_page_has_no_internal_test_material(): + page = _read(PAGE) + for needle in ("STANDIN", "stand-in", "run_e2e.sh", "Live vendor verification", "fi_verify", + "TH-8183", "2026-", "rows_hit", "PASS"): + assert needle not in page, needle + assert not re.search(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", page) + + def test_page_does_not_send_customers_to_another_platform_by_default(): page = _read(PAGE) assert "base_url=\"https://api.typesafe.ai\"" not in page diff --git a/scripts/integrations.test.mjs b/scripts/integrations.test.mjs index 30ff909ed..2bb52647e 100644 --- a/scripts/integrations.test.mjs +++ b/scripts/integrations.test.mjs @@ -3,6 +3,7 @@ import './azure-openai-java.test.mjs'; import './langsmith-page.test.mjs'; import './vercel-platform-page.test.mjs'; import './scalekit-page.test.mjs'; +import './typesafe-page.test.mjs'; import test from 'node:test'; import assert from 'node:assert/strict'; import fs from 'node:fs'; diff --git a/scripts/typesafe-page.test.mjs b/scripts/typesafe-page.test.mjs new file mode 100644 index 000000000..bf9c93272 --- /dev/null +++ b/scripts/typesafe-page.test.mjs @@ -0,0 +1,51 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import { integrations } from '../src/lib/integrations.mjs'; + +const read = file => readFileSync(new URL(`../${file}`, import.meta.url), 'utf8'); +const page = read('src/pages/docs/integrations/traceai/typesafe.mdx'); +const app = read('examples/traceai/typesafe/app.py'); +const python = [...page.matchAll(/```python\n([\s\S]*?)```/g)].map(match => match[1].trim()); + +function excerpt(name) { + const match = app.match(new RegExp(`# --- ${name} start ---\\n([\\s\\S]*?)\\s*# --- ${name} end ---`)); + assert.ok(match, `Missing app.py excerpt: ${name}`); + const lines = match[1].split('\n'); + const indent = Math.min(...lines.filter(line => line.trim()).map(line => line.match(/^ */)[0].length)); + return lines.map(line => line.slice(indent)).join('\n').trim(); +} + +test('TypeSafe snippets are exact excerpts of the runnable example', () => { + for (const name of ['recipe', 'questions', 'sync call', 'async call']) { + assert.ok(python.includes(excerpt(name)), `Page snippet differs from app.py: ${name}`); + } +}); + +test('TypeSafe snippets define the ticket and use no example-only variables', () => { + const questions = python.find(block => block.includes('QUESTIONS = {')); + assert.ok(questions?.startsWith('TICKET = ('), 'Define TICKET with the questions'); + assert.doesNotMatch(page, /\bMARKER\b|RUN_MARKER/); + assert.ok(page.indexOf('TICKET = (') < page.indexOf('state=TICKET')); +}); + +test('TypeSafe states the released package Python range', () => { + assert.match(page, /pip install traceAI-typesafe typesafe-sdk/); + assert.match(page, /Python 3\.10 to 3\.13, as required by `traceAI-typesafe` 0\.1\.0/); + assert.match(page, /`typesafe-sdk` 0\.7\.0 or later, below 1\.0/); + assert.doesNotMatch(page, /fi-instrumentation-otel does not support/); +}); + +test('TypeSafe page keeps engineering evidence out of the setup guide', () => { + assert.doesNotMatch(page, /STANDIN|stand-in|run_e2e\.sh|Live vendor verification|verified|fi_verify|rows_hit|TH-\d+|20\d\d-\d\d-\d\d/i); + assert.doesNotMatch(page, /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/); + assert.match(page, /FI_HIDE_INPUTS=true/); + assert.match(page, /never turns answers into Pass\/Fail/); +}); + +test('TypeSafe is discoverable as a traceAI model-provider guide', () => { + const entry = integrations.find(item => item.id === 'typesafe'); + assert.equal(entry?.href, '/docs/integrations/traceai/typesafe'); + assert.equal(entry.category, 'providers'); + assert.equal(entry.method, 'traceAI SDK'); +}); diff --git a/src/lib/integrations.mjs b/src/lib/integrations.mjs index df0f390a2..062300577 100644 --- a/src/lib/integrations.mjs +++ b/src/lib/integrations.mjs @@ -70,6 +70,7 @@ export const integrations = [ { id: 'togetherai', name: 'Together AI', href: '/docs/integrations/traceai/togetherai', category: 'providers', summary: 'Trace Together AI through its OpenAI-compatible API.', method: 'OpenAI client', logo: 'together' }, { id: 'ollama', name: 'Ollama', href: '/docs/integrations/traceai/ollama', category: 'providers', summary: 'Trace local model calls through the OpenAI client.', method: 'OpenAI client', logo: 'ollama' }, { id: 'cometapi', name: 'CometAPI', href: '/docs/integrations/providers/cometapi', category: 'providers', summary: 'Trace chat completions through the OpenAI client.', method: 'OpenAI client', logo: 'cometapi' }, + { id: 'typesafe', name: 'TypeSafe (Jev)', href: '/docs/integrations/traceai/typesafe', category: 'providers', summary: 'Trace System One decision calls from the Python client.', method: 'traceAI SDK', logo: 'generic-model', aliases: ['Jev', 'System One'] }, { id: 'novita-ai', name: 'Novita AI', href: '/docs/integrations/providers/novita-ai', category: 'providers', summary: 'Trace Novita chat completions from your Python app.', method: 'OpenAI client', logo: 'novita' }, { id: 'litellm-sdk', name: 'LiteLLM SDK', href: '/docs/integrations/traceai/litellm', category: 'frameworks', summary: 'Instrument LiteLLM calls in your Python application.', method: 'traceAI SDK', logo: 'litellm', aliases: ['traceAI-litellm', 'client'] }, diff --git a/src/pages/docs/integrations/traceai/typesafe.mdx b/src/pages/docs/integrations/traceai/typesafe.mdx index 05ba1fc2d..863e80680 100644 --- a/src/pages/docs/integrations/traceai/typesafe.mdx +++ b/src/pages/docs/integrations/traceai/typesafe.mdx @@ -1,39 +1,23 @@ --- title: "TypeSafe (Jev) decision tracing with Future AGI" -description: "Trace TypeSafe System One decision calls (Jev) made with the typesafe-sdk Python client: one evaluator span per call with every noul, choice, score and box answer, using traceAI-typesafe. Verified against the live TypeSafe API." +description: "Trace TypeSafe System One decision calls (Jev) made with the typesafe-sdk Python client: one evaluator span per call with every noul, choice, score and box answer, using traceAI-typesafe." --- -## About +Trace every TypeSafe System One decision your Python code makes, with each typed answer recorded on one `EVALUATOR` span in Future AGI. -[TypeSafe](https://typesafe.ai) serves Jev through the System One decision API: you send some content (`state`) and named questions, and get back a typed answer per question. It is not a chat API and returns no text. `traceAI-typesafe` instruments the official Python client, [`typesafe-sdk`](https://pypi.org/project/typesafe-sdk/): every `TypeSafeClient.system_one(...)` and `AsyncTypeSafeClient.system_one(...)` call becomes one `EVALUATOR` span in Future AGI, however many questions it asks. +[TypeSafe](https://typesafe.ai) serves Jev through the System One decision API: you send some content (`state`) and named questions, and get back a typed answer per question. It is not a chat API and returns no text. `traceAI-typesafe` instruments the official Python client, [`typesafe-sdk`](https://pypi.org/project/typesafe-sdk/): every `TypeSafeClient.system_one(...)` and `AsyncTypeSafeClient.system_one(...)` call becomes one span, however many questions it asks. A run of the example on this page is one trace: -``` +```text triage_ticket your own span (CHAIN) ├── typesafe.system_one billing (noul), tone (choice), urgency (score) (EVALUATOR) └── typesafe.system_one escalate (noul), from the async client (EVALUATOR) ``` -This is not the same as: - -- **Jev as an evaluator model in Future AGI.** Running Future AGI evals with Jev as the judge is a separate feature (tracked as TH-8183). This package adds no evaluator model, does not call Future AGI's gateway, and does not turn answers into Pass/Fail. It only traces calls your own code makes to TypeSafe. -- **[OpenAI](/docs/integrations/traceai/openai) (`traceAI-openai`).** System One is not OpenAI-compatible. Pointing `openai.OpenAI` at TypeSafe does not work, and `traceAI-openai` would not see the answers. - -
- -| | | -| --- | --- | -| Tested with | typesafe-sdk **0.7.5** (latest on PyPI, released 2026-10-14) end to end on Python 3.12 against the live `api.typesafe.ai` (2026-10-17, model `jev-1.13.0`); unit tests also on typesafe-sdk 0.7.4 (Python 3.13) and 0.7.0 (Python 3.10); fi-instrumentation-otel 1.1.0; traceAI-typesafe 0.1.0 | -| Supported | typesafe-sdk 0.7.0 or later, below 1.0; Python 3.10 to 3.13 (fi-instrumentation-otel does not support 3.14 yet) | -| TypeSafe docs read | [Python SDK](https://docs.typesafe.ai/sdk/python), [System One](https://docs.typesafe.ai/concepts/system-one), [noul](https://docs.typesafe.ai/primitives/noul), [choice](https://docs.typesafe.ai/primitives/choice), [score](https://docs.typesafe.ai/primitives/score), [changelog](https://docs.typesafe.ai/sdk/python/changelog) and the [OpenAPI document](https://api.typesafe.ai/openapi.json) (version 0.2.0), all read 2026-10-16 | -| Source | [traceAI `python/frameworks/typesafe`](https://github.com/future-agi/traceAI/tree/main/python/frameworks/typesafe) and [examples/traceai/typesafe](https://github.com/future-agi/docs/tree/main/examples/traceai/typesafe) | - -
+This package only traces calls your own code makes to TypeSafe. It does not add an evaluator model to Future AGI evals, does not call the Future AGI gateway, and never turns answers into Pass/Fail. System One is not OpenAI-compatible, so [`traceAI-openai`](/docs/integrations/traceai/openai) cannot trace it. - -Verified against the live TypeSafe API on 2026-10-17 (see [Live vendor verification](#live-vendor-verification-2026-10-17)): real Jev answers, real tokens and model, a real 422 and a real 401, exported to Future AGI. Not yet seen live: a 429 with `typesafe.retry_after_ms`, a `box` answer, image assets, and `system_one` retries. Those paths are covered by unit tests and the local `STANDIN` server only. The package itself is still an early (0.1.0) release. - +Use Python 3.10 to 3.13, as required by `traceAI-typesafe` 0.1.0, with `typesafe-sdk` 0.7.0 or later, below 1.0. The instrumentor source is in [traceAI `python/frameworks/typesafe`](https://github.com/future-agi/traceAI/tree/main/python/frameworks/typesafe). ## 1. Install @@ -72,14 +56,19 @@ Call `instrument()` once, before you make decisions. Clients created before or a ## 4. Make decisions -Nothing changes in how you call TypeSafe. Ask your questions (`TICKET` is the text you want decided): +Nothing changes in how you call TypeSafe. Define the content you want decided and your questions: ```python +TICKET = ( + "Order 1042: I was charged twice for the same order and nobody answers my emails. " + "Please refund the duplicate today." +) + QUESTIONS = { - "billing": Noul(instructions="Is this message about billing? {0}".format(MARKER).strip()), + "billing": Noul(instructions="Is this message about billing?"), "tone": Choice( instructions="What is the tone of this message?", - criteria={"calm": None, "angry": "Upset or hostile {0}".format(MARKER).strip(), "excited": None}, + criteria={"calm": None, "angry": "Upset or hostile", "excited": None}, ), "urgency": Score( instructions="How urgent is this message?", @@ -88,7 +77,7 @@ QUESTIONS = { } ``` -(`MARKER` is empty unless you set `RUN_MARKER`; the example uses it for the privacy check below.) +Then make the call: ```python with TypeSafeClient() as client: @@ -111,6 +100,10 @@ print("escalate", result.nouls["escalate"].noul) In a short-lived script, call `trace_provider.force_flush()` before the process exits so the last spans are sent. +## 5. Open the trace + +Open the `support-triage` project (or the name you set in `FI_PROJECT_NAME`) in Observe. Each decision is a `typesafe.system_one` span with type **Evaluator**, the requested model, token counts, and the answers as span attributes. + ## What a span contains One span per `system_one` call, named `typesafe.system_one`, kind `EVALUATOR`. Questions are attributes, not child spans: TypeSafe answers them all in one response. @@ -132,7 +125,7 @@ One span per `system_one` call, named `typesafe.system_one`, kind `EVALUATOR`. Q | --- | --- | | `gen_ai.provider.name` | `typesafe` | | `gen_ai.request.model` | The model the request asked for, for example `jev-latest`. This is what the **Model** column shows | -| `gen_ai.response.model` | The model that answered. When it differs (an alias resolved to a version such as `jev-1.13.0`), `typesafe.response.model` is set too | +| `gen_ai.response.model` | The model that answered. When it differs (an alias resolved to a specific version), `typesafe.response.model` is set too | | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.total_tokens` | As the API reported them. No cost is computed | | `typesafe.question_count` | Number of questions in the request | | `typesafe.request_id` | The `x-typesafe-request-id` response header, for TypeSafe support | @@ -193,57 +186,10 @@ Errors raised by the SDK before a request is sent (no question, an image file it ## Troubleshooting - **No spans.** Call `TypeSafeInstrumentor().instrument(...)` before the first decision, and `trace_provider.force_flush()` before a short script exits. Check that `FI_API_KEY` and `FI_SECRET_KEY` are set in the same process. +- **`pip` cannot find a matching `traceAI-typesafe`.** Use Python 3.10 to 3.13. - **`TypeSafeError: No API key was provided`.** The SDK needs `TYPESAFE_API_KEY` (or `api_key=`). This is raised when the client is created, before any span. - **An `ERROR` span with `TypeSafeAuthenticationError (HTTP 401)`.** The key is wrong or revoked. - **`TypeSafeUnprocessableEntityError (HTTP 422)`.** TypeSafe rejected the request body; with content captured, the message names the field (for example `questions.tone.choice.criteria`). A `choice` question needs a mapping of criteria, not a list. - **The Model column shows `jev-latest`, not a version.** That is the model you asked for. The resolved version is in `gen_ai.response.model` and `typesafe.response.model`. - **A `noul` shows no confidence.** Expected: the API returns none for `noul`. - **The `choice` label is missing.** `FI_HIDE_OUTPUTS` or `FI_HIDE_OUTPUT_TEXT` is on. - -## Run it yourself - -[examples/traceai/typesafe](https://github.com/future-agi/docs/tree/main/examples/traceai/typesafe) has the program above (`app.py`), a local System One stand-in, tests, and `run_e2e.sh`: - -```bash -cd examples/traceai/typesafe -export FI_API_KEY=... FI_SECRET_KEY=... -./run_e2e.sh # tests, then app.py against the local STANDIN server (not TypeSafe) -PRIVACY=1 ./run_e2e.sh # also: a unique marker with the hide flags on, and with the default -TYPESAFE_API_KEY=... ./run_e2e.sh # against the real api.typesafe.ai with your own key -``` - -With your own key, the run makes two System One decisions (a few hundred input tokens), one request the API rejects with 422, one request with a deliberately invalid key, and one `GET /v1/models`. - -## Live vendor verification (2026-10-17) - -Run exactly as this page says, with `TYPESAFE_API_KEY` set and no `TYPESAFE_BASE_URL`, so the SDK called the real `https://api.typesafe.ai`; traces went to a local Future AGI (project `th8103-rest-typesafe-jev-live`). Versions: typesafe-sdk 0.7.5, traceAI-typesafe 0.1.0 (traceAI PR #264, commit `e6dd3c0e16e7`), fi-instrumentation-otel 1.1.0, opentelemetry-sdk 1.45.1, Python 3.12.10. The request resolved `jev-latest` to `jev-1.13.0`. - -
- -| Case | Trace id | Result | -| --- | --- | --- | -| `ok`: sync decision (noul, choice, score) and async decision (noul) | `5e0d9155-271c-4183-924f-8f9411b53701` | Two `EVALUATOR` spans under `triage_ticket`, status OK, 397/73 and 318/22 tokens | -| `fail-422`: malformed `choice` question | `1d89de2c-3560-411f-a21f-71c4b7eda3f2` | `ERROR`, `TypeSafeUnprocessableEntityError (HTTP 422)` | -| `fail-auth`: invalid key | `11a1e3da-31e5-49d6-8709-ca1462bfc54b` | `ERROR`, `TypeSafeAuthenticationError (HTTP 401)` | -| `fail-asset`: missing image file | `d45fd1ab-0cb3-4609-948b-aa9413f60937` | `ERROR`, `TypeSafeError`, no path stored | - -
- -`fi_verify` (hierarchy, model, tokens, errors) excerpt: - -``` -traces=5 spans=11 -trace 5e0d9155-271c-4183-924f-8f9411b53701 (3 spans) -- [chain] triage_ticket in=115B out=7B (616ms) - - [evaluator] typesafe.system_one model=jev-latest tok=397/73 in=534B out=172B (179ms) - - [evaluator] typesafe.system_one model=jev-latest tok=318/22 in=272B out=44B (245ms) -trace 1d89de2c-3560-411f-a21f-71c4b7eda3f2 (2 spans) - - [evaluator] typesafe.system_one status=ERROR:TypeSafeUnprocessableEntityError (HTTP 422) -trace 11a1e3da-31e5-49d6-8709-ca1462bfc54b (2 spans) - - [evaluator] typesafe.system_one status=ERROR:TypeSafeAuthenticationError (HTTP 401) -PASS -``` - -On the success span: `gen_ai.request.model=jev-latest`, `gen_ai.response.model=jev-1.13.0`, `typesafe.response.model=jev-1.13.0`, `gen_ai.usage.*` filled, `server.address=api.typesafe.ai`, a real `typesafe.request_id` (`req_...`), and numeric answers under `typesafe.question..*`. - -Privacy, against the live API: a unique marker in the ticket, an email built from it, the question instructions and a criteria description, with `FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true` (project `...-live-hidden`) and with the defaults (`...-live-default`). Scanning every stored column of every table for the project (ClickHouse and Postgres, including `resource_attrs`, `span_events`, `attributes_extra` and `status_message`): with the hide flags on, the marker, the email, the ticket text, the instruction and criteria text, the `choice` label, the TypeSafe key, the Future AGI keys, `Bearer `, the user name, the host name, the home directory, `Traceback`, `probabilities` and `legend` appear in 0 rows. With the defaults, the marker and ticket text appear in 9 rows (`input`, `attributes_extra`) and the TypeSafe and Future AGI keys in 0.