Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions python/frameworks/typesafe/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Changelog

## [0.1.0] - 2026-10-16
### Feature
- First release. `TypeSafeInstrumentor` traces `typesafe_sdk.TypeSafeClient.system_one` and
`AsyncTypeSafeClient.system_one` (typesafe-sdk 0.7.0 to < 1; tested on 0.7.0, 0.7.4 and 0.7.5):
one `EVALUATOR` span per decision call with `typesafe.question.<name>.*` attributes for
`noul`, `choice`, `score` and `box` answers, request and response model, token usage and the
request id. Content follows the traceAI `TraceConfig` convention (captured by default,
`FI_HIDE_INPUTS` / `FI_HIDE_OUTPUTS` turn it off). `client.models.list()` is not traced.
90 changes: 90 additions & 0 deletions python/frameworks/typesafe/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# traceAI-typesafe

OpenTelemetry instrumentation for the [TypeSafe AI](https://typesafe.ai) Python SDK
(`typesafe-sdk`), the client for Jev's System One decision API (`POST /v1/systemone`).

Every `TypeSafeClient.system_one(...)` and `AsyncTypeSafeClient.system_one(...)` call becomes
one `EVALUATOR` span in Future AGI, however many questions it asks. Each question becomes
`typesafe.question.<name>.*` attributes:

| Answer type | Attributes |
| --- | --- |
| `noul` | `type`, `noul` (probability of yes, a number, never Pass/Fail) |
| `choice` | `type`, `choice` (the label), `confidence` |
| `score` | `type`, `score` (expected score, a number), `confidence` |
| `box` | `type`, `box.top`, `box.bottom`, `box.left`, `box.right`, `confidence` when not null |

The span also carries `gen_ai.provider.name=typesafe`, `gen_ai.request.model` (as you asked
for it), `gen_ai.response.model` plus `typesafe.response.model` when the API resolved an
alias, `gen_ai.usage.input_tokens` / `output_tokens`, `typesafe.request_id`, `server.address`
and `typesafe.question_count`. Probability maps, score legends and image bytes are never
recorded. The API returns no explanation, so the span has none.

This package traces calls your code makes to TypeSafe. It does not add Jev as a judge model
for Future AGI evals, and it does not call the API itself.

## Install

```bash
pip install traceAI-typesafe typesafe-sdk
```

## Use

```python
from fi_instrumentation import register
from fi_instrumentation.fi_types import ProjectType
from traceai_typesafe import TypeSafeInstrumentor
from typesafe_sdk import Choice, Noul, TypeSafeClient

trace_provider = register(project_type=ProjectType.OBSERVE, project_name="support-triage")
TypeSafeInstrumentor().instrument(tracer_provider=trace_provider)

with TypeSafeClient() as client: # reads TYPESAFE_API_KEY
result = client.system_one(
state="I was charged twice. Please help.",
questions={
"billing": Noul(instructions="Is this about billing?"),
"tone": Choice(instructions="What is the tone?", criteria={"calm": None, "angry": None}),
},
)
trace_provider.force_flush()
```

The instrumentor never sets `base_url` or a model. Point the client wherever you already do
(`base_url=` or `TYPESAFE_BASE_URL`); `server.address` records the host.

## Privacy

Like every traceAI instrumentor, content is captured by default:

| Setting | Effect |
| --- | --- |
| default | `input.value` = the `state` and the questions (instructions, criteria); `output.value` and `typesafe.question.<name>.choice` = the decisions |
| `FI_HIDE_INPUTS=true` (or `TraceConfig(hide_inputs=True)`, `FI_HIDE_INPUT_TEXT`) | no `input.value`, no `typesafe.idempotency_key` |
| `FI_HIDE_OUTPUTS=true` (or `TraceConfig(hide_outputs=True)`, `FI_HIDE_OUTPUT_TEXT`) | no `output.value`, no `choice` labels; numeric answers stay |
| `FI_PII_REDACTION=true` | regex PII redaction on every string attribute and the error message |

Always recorded: question names and types (they are attribute keys, so do not put secrets in
question names), numeric answers, model, usage, request id. Never recorded: the API key,
stack traces, the text of client-side errors (they can name a local file), image bytes or
paths. The API's own error message is added to the span status only while neither inputs nor
outputs are hidden.

## Errors and retries

An HTTP error (401, 403, 422, 429, 5xx) or a connection error ends the span with `ERROR`,
`error.type`, `http.response.status_code`, the server's `typesafe.error_type` and the request
id. The instrumentor adds no retries. The SDK's own `RetryPolicy` (by default up to 2 retries
for 408, 429 and 5xx) runs inside the one span, so a 429 followed by a success is one `OK`
span. A call you repeat yourself is a second span.

## Tests

```bash
pip install -e . pytest pytest-asyncio
pytest
```

The tests run the real `typesafe-sdk` clients against an in-process fake shaped like
TypeSafe's published OpenAPI document. They never contact `api.typesafe.ai`.
32 changes: 32 additions & 0 deletions python/frameworks/typesafe/examples/system_one.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
"""Trace TypeSafe System One decision calls in Future AGI.

Needs FI_API_KEY, FI_SECRET_KEY and TYPESAFE_API_KEY. TYPESAFE_BASE_URL is
optional (the SDK default is https://api.typesafe.ai).
"""

from fi_instrumentation import register
from fi_instrumentation.fi_types import ProjectType
from traceai_typesafe import TypeSafeInstrumentor
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

trace_provider = register(project_type=ProjectType.OBSERVE, project_name="typesafe-example")
TypeSafeInstrumentor().instrument(tracer_provider=trace_provider)

with TypeSafeClient() as client:
result = client.system_one(
state="I was charged twice for my order and nobody is answering. Please help.",
questions={
"billing": Noul(instructions="Is this message about billing?"),
"tone": Choice(
instructions="What is the tone of this message?",
criteria={"calm": None, "angry": None, "excited": None},
),
"urgency": Score(
instructions="How urgent is this message?",
criteria=["Can wait", "Needs attention this week", "Needs attention today"],
),
},
)
print(result.model, result.nouls["billing"].noul, result.choices["tone"].choice, result.scores["urgency"].score)

trace_provider.force_flush()
35 changes: 35 additions & 0 deletions python/frameworks/typesafe/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
[tool.poetry]
name = "traceAI-typesafe"
version = "0.1.0"
description = "OpenTelemetry instrumentation for TypeSafe AI (Jev) System One decision calls"
authors = ["Future AGI <no-reply@futureagi.com>"]
license = "Apache-2.0"
readme = "README.md"
packages = [
{ include = "traceai_typesafe" }
]
classifiers = [
"Development Status :: 3 - Alpha",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Topic :: Software Development :: Libraries :: Python Modules",
]

[tool.poetry.dependencies]
python = ">=3.10,<3.14"
fi-instrumentation-otel = ">=1.1.0"
typesafe-sdk = ">=0.7.0,<1"
wrapt = ">=1.15.0"

[tool.poetry.group.dev.dependencies]
pytest = ">=7.0.0"
pytest-asyncio = ">=0.23"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
Empty file.
Loading
Loading