Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
249 changes: 106 additions & 143 deletions python/frameworks/voyage/README.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,23 @@
# traceAI-voyage

OpenTelemetry instrumentation for the official
[Voyage AI Python client](https://github.com/voyage-ai/voyageai-python)
(`voyageai`).

`traceAI-openai` does not trace Voyage: it wraps the OpenAI client only and
never sees `voyageai.Client`. Use this package for Voyage embeddings and
reranking.
Trace embeddings and reranking from the official
[Voyage AI Python client](https://github.com/voyage-ai/voyageai-python) in Future AGI.
The same instrumentor supports `voyageai.Client` and `voyageai.AsyncClient`.

## Installation

```bash
pip install traceAI-voyage
```

It accepts `voyageai>=0.3.7,<1` (0.3.7 is the first release that routes Atlas
keys) and `fi-instrumentation-otel>=1.1.0`. The suite runs on Python 3.10,
3.11, 3.12 and 3.13 against `voyageai` 0.3.7 and 0.5.0.
Requires Python 3.10–3.13, `voyageai>=0.3.7,<1` and
`fi-instrumentation-otel>=1.1.0`.

## Usage

Set `FI_API_KEY`, `FI_SECRET_KEY` and `VOYAGE_API_KEY` in your environment, then
register a trace provider and instrument the client:

```python
import voyageai
from fi_instrumentation import register
Expand All @@ -32,165 +30,130 @@ VoyageInstrumentor().instrument(tracer_provider=trace_provider)
client = voyageai.Client() # reads VOYAGE_API_KEY
client.embed(["What is the capital of France?"], model="voyage-3.5", input_type="query")
client.rerank("capital of France", ["Paris is...", "Berlin is..."], model="rerank-2.5", top_k=1)
client.contextualized_embed(
[["Paris is the capital of France.", "The Seine flows through Paris."]],
model="voyage-context-3",
input_type="document",
)
trace_provider.force_flush()
```

`examples/embed_and_rerank.py` is a runnable version of this.

The methods are patched on the client classes, so a client built before
`instrument()` is traced too; only calls made while instrumented are traced.
`uninstrument()` restores the original methods.
Use `client.multimodal_embed(inputs, model="voyage-multimodal-3")` for mixed
text and image inputs. Pass inputs in the format accepted by your Voyage SDK;
the instrumentor does not transform them or invoke chunking callbacks.

## Hosts and `base_url`

The instrumentor never sets or changes `base_url`. The Voyage client picks the
host from the API key, and a `base_url` you pass to the client yourself wins.
In `voyageai` 0.3.7 to 0.5.0, keys that start with `al-` (created in MongoDB
Atlas) go to `https://ai.mongodb.com/v1` and other keys (created on the Voyage
platform) to `https://api.voyageai.com/v1`. MongoDB also documents geography
hosts such as `eu.ai.mongodb.com` and `us.ai.mongodb.com`; to use one, pass it
as `base_url` when you construct the client.

The span records only the host the client is using, as `server.address`. A
key sent to the wrong host fails in the client; the span records that error
and nothing retries against another host.
Methods are wrapped on the client classes, so a client created before
`instrument()` is traced too. `uninstrument()` restores the original methods.
`examples/embed_and_rerank.py` provides a runnable embedding and reranking example.

## What is traced

| Call | Span name | `gen_ai.span.kind` |
The following methods are supported on both clients:

| Method | Span name | `gen_ai.span.kind` |
|---|---|---|
| `Client.embed`, `AsyncClient.embed` | `voyage.embed` | `EMBEDDING` |
| `Client.rerank`, `AsyncClient.rerank` | `voyage.rerank` | `RERANKER` |
| `embed` | `voyage.embed` | `EMBEDDING` |
| `multimodal_embed` | `voyage.multimodal_embed` | `EMBEDDING` |
| `contextualized_embed` | `voyage.contextualized_embed` | `EMBEDDING` |
| `rerank` | `voyage.rerank` | `RERANKER` |

One span per call: a list of texts is one span, and the client's own retries
(`max_retries`) happen inside that one span. The span is a child of the active
span when there is one, and it is current while the SDK sends the request, so
a span from an HTTP client instrumentation nests under it. Local models that
`embed` runs in process (`voyageai` 0.5+, for example `voyage-4-nano`) are
traced the same way, without `server.address`; their token count is the one the
client computes locally.
Each public call produces one span, including an entire nested batch and the
client's own retries. The span is a child of the active span and remains current
while the SDK sends its request, so HTTP instrumentation can nest underneath it.
Local models run by `embed` are traced without `server.address`.

Not traced (no span, not an error): `multimodal_embed`,
`contextualized_embed`, the deprecated module functions
`voyageai.get_embedding(s)` / `aget_embedding(s)`, and the TypeScript client.
The deprecated module functions `voyageai.get_embedding(s)` / `aget_embedding(s)`
and the TypeScript client are not covered. `traceAI-openai` does not instrument
`voyageai.Client`; use this package rather than an OpenAI wrapper for these calls.

## Span attributes

Always recorded when known (never content):

| Attribute | Calls | Value |
|---|---|---|
| `gen_ai.provider.name` | all | `voyage` |
| `gen_ai.operation.name` | all | `embeddings` or `rerank` |
| `gen_ai.request.model` | all | The model; for `embed` without `model`, the SDK default it falls back to. |
| `embedding.model_name` / `reranker.model_name` | embed / rerank | Same model. |
| `server.address` | all | Host from the client's base URL (no path, query or credentials). |
| `voyage.embedding.count` | embed | Number of texts in the call (a bare string is 1). |
| `voyage.input_type`, `voyage.output_dtype`, `voyage.output_dimension` | embed | As passed; absent when not passed. |
| `voyage.embedding.dimension`, `gen_ai.embeddings.dimension.count` | embed, success | Length of the first returned vector; times 8 for the packed `binary` / `ubinary` dtypes. |
| `voyage.rerank.document_count` | rerank | Number of documents sent. |
| `reranker.top_k` | rerank | As passed; absent when not passed. |
| `voyage.rerank.result_count` | rerank, success | Number of results returned. |
| `gen_ai.usage.input_tokens`, `gen_ai.usage.total_tokens` | success | The client's `total_tokens`. Voyage reports one total and every embedding or rerank token is input, so both keys carry it. Never estimated. No cost is set. |
| `voyage.cancelled` | cancelled calls | `true`. |

Usage is written only on the Voyage span, once, on the `gen_ai.usage.*` keys
the Future AGI collector reads. If another instrumentation in your app (for
example a framework that wraps the same embedding call) also records usage for
it, the trace counts those tokens twice.

## Live vendor verification (2026-10-17)

Run against the real Voyage API (`api.voyageai.com`, default host, no `base_url`) with `voyageai` 0.5.0, `voyage-3.5-lite` embeddings and `rerank-2.5-lite`, exported to Future AGI: an `AGENT` span with `voyage.embed` (EMBEDDING, tokens recorded), `voyage.rerank` (RERANKER, tokens recorded) and an invalid-key `voyage.embed` as an `ERROR` span (`AuthenticationError`). Default, `capture_content=True` and `FI_HIDE_*` privacy modes were checked with a marker scan. Not yet run live: the Atlas host (`ai.mongodb.com`), geography hosts and local in-process models.

## Privacy and content capture

By default the rerank span records the query and the relevance scores, as PRD
J2 / AC-03 require, so this default is not a deviation from the spec. Embed
texts and rerank documents are recorded only with `capture_content=True`.
Embedding vectors are never recorded.

| Attribute | Calls | Value | Recorded | Dropped by |
|---|---|---|---|---|
| `reranker.query` | rerank | The query. | By default | `hide_inputs`, `hide_input_text` |
| `input.value` (`input.mime_type` = `text/plain`) | rerank | The query. | By default | `hide_inputs`, `hide_input_text` |
| `output.value` (`output.mime_type` = `application/json`) | rerank | JSON list of `{"index", "relevance_score"}` in the client's order. | By default | `hide_outputs` |
| `input.value` (`application/json`) | rerank | JSON `{"query": ..., "documents": [...]}`, in place of the plain query. | Only with `capture_content=True` | `hide_inputs`, `hide_input_text` |
| `input.value` (`application/json`) | embed | JSON list of the texts. | Only with `capture_content=True` | `hide_inputs`, `hide_input_text` |

Hiding inputs keeps the scores (they are numbers, not input text), and hiding
outputs keeps the query. The counts (`voyage.embedding.count`,
`voyage.rerank.document_count`, `voyage.rerank.result_count`) stay whatever is
hidden. `TraceConfig` reads `FI_HIDE_INPUTS` / `FI_HIDE_OUTPUTS` from the
environment, so `FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true` records no content
at all.

To record embed texts and rerank documents too, opt in:
| Attribute | Value |
|---|---|
| `gen_ai.provider.name` | `voyage` |
| `gen_ai.operation.name` | `embeddings` or `rerank` |
| `gen_ai.request.model` | Requested model, or the SDK default for `embed` without a model |
| `embedding.model_name` / `reranker.model_name` | Model for the operation |
| `server.address` | Host chosen by the client, without URL path, query or credentials |
| `voyage.embedding.count` | Input count for `embed`/`multimodal_embed`; total returned chunk vectors across all documents for successful `contextualized_embed` |
| `voyage.contextualized.document_count` | Number of contextualized input documents, before chunking |
| `voyage.input_type`, `voyage.output_dtype`, `voyage.output_dimension` | Embedding options when supplied |
| `voyage.embedding.dimension`, `gen_ai.embeddings.dimension.count` | First returned vector's dimension; packed `binary` / `ubinary` lengths are multiplied by 8 |
| `voyage.rerank.document_count`, `voyage.rerank.result_count` | Rerank input document and successful result counts |
| `reranker.top_k` | Requested result limit, when supplied |
| `gen_ai.usage.input_tokens`, `gen_ai.usage.total_tokens` | The SDK's reported `total_tokens`, once on the Voyage span; no estimate or cost |
| `voyage.usage.text_tokens`, `voyage.usage.image_pixels`, `voyage.usage.video_pixels` | Multimodal usage as reported by the SDK; pixels are not tokens and are not added to the token total |
| `voyage.cancelled` | `true` on a cancelled call |

Contextualized counts come from the returned nested result, so client-side or
server-side chunking does not confuse document count with embedding count.
Vectors and returned chunk text are never exported.

Disable any other instrumentor wrapping the same Voyage methods to avoid duplicate
spans and usage. Framework-level parent spans may report their own token totals.

## Privacy

| Setting | Captured content |
|---|---|
| Default | Rerank query and relevance scores; no embedding inputs or rerank documents |
| `capture_content=True` | Also embedding text inputs and rerank documents |
| `TraceConfig(hide_inputs=True)` or `FI_HIDE_INPUTS=true` | No input content; rerank scores remain |
| `TraceConfig(hide_input_text=True)` | No input content |
| `TraceConfig(hide_outputs=True)` or `FI_HIDE_OUTPUTS=true` | No rerank scores; input capture is unchanged |
| `TraceConfig(pii_redaction=True)` or `FI_PII_REDACTION=true` | Applies core PII redaction to recorded strings |

```python
from fi_instrumentation import TraceConfig

# Use these options in your instrument() call instead of the default call above.
VoyageInstrumentor().instrument(
tracer_provider=trace_provider,
capture_content=True,
config=TraceConfig(), # optional; honours hide_inputs / hide_outputs / pii_redaction
config=TraceConfig(pii_redaction=True),
)
```

At most 64 texts, 64 documents and 64 scores (the first 64 results, in the
client's order) are recorded, and the query and each text or document are
cut to 2 KB of UTF-8 on a character boundary; `voyage.embedding.count`,
`voyage.rerank.document_count` and `voyage.rerank.result_count` stay exact.

`TraceConfig(pii_redaction=True)` (or `FI_PII_REDACTION=true`) applies
`fi_instrumentation`'s regex PII redaction, which replaces email addresses,
SSNs, card numbers, `sk-`/`pk-` live/test/prod API keys, IPv4 addresses and
phone numbers with tokens such as `<EMAIL_ADDRESS>`. It covers:

- every string span attribute: the query, captured texts and documents, the
scores JSON, and the other attributes above (an IP `server.address` becomes
`<IP_ADDRESS>`);
- the error status description and the `exception` event's
`exception.message`, after the API key is
removed from them.

It does not change span names, event names or `exception.type`. The patterns
match digits, not meaning: a relevance score with 10 or more digits after the
decimal point is rewritten too (for example `0.<CREDIT_CARD>`), which leaves
that `output.value` as invalid JSON.

Embedding vectors are never recorded, with or without capture, and
`hide_embedding_vectors` has nothing to hide here.

The Voyage API key is never recorded. The instrumentor does not read it from
the environment itself; it takes the key the client holds (`client.api_key`,
`client._params["api_key"]`) and a module-level `voyageai.api_key`, and
replaces any occurrence in the recorded query, captured content, error
messages and stack traces with `[redacted]`.

## Errors and cancellation

An error raised by the client (for example `voyageai.error.AuthenticationError`
for HTTP 401, or `voyageai.error.Timeout`) sets the span status to ERROR with
`<ExceptionType>: <message>` and records one `exception` event; the key is
redacted from both, and with `pii_redaction=True` so is PII (see above). The
exception reaches your code unchanged.
Embedding vectors, multimodal media payloads, media URLs and media object
representations are never captured. With capture enabled, multimodal inputs
retain their batch boundaries and text segments; media segments become
`[media omitted]`. Contextualized inputs retain their document/chunk boundaries.
Text supplied explicitly as text is treated as text, even if it contains a URL.

Cancelling an `AsyncClient` call (`asyncio.CancelledError`) or interrupting a
blocking call (`KeyboardInterrupt`) ends the span with status ERROR,
description `cancelled` and `voyage.cancelled` = `true`, with no exception
event. Result attributes and usage are absent on error and cancellation, never
written as 0.
Nested capture is limited to 64 outer groups and 64 segments in total, with each
text capped at 2 KB of UTF-8. Ordinary embedding texts, rerank documents and scores
are limited to 64 items each. Counts and usage remain exact.

The instrumentation never changes what a call returns or raises: a failure
while building attributes or starting the span is logged at debug level and the
Voyage call runs as if uninstrumented.
Known client API keys are redacted from captured content and exported error
messages. PII redaction is pattern-based: review your capture settings for the
sensitivity of your data. The core redactor can also match long numeric scores;
use `hide_outputs=True` if you do not need scores in traces.

## Configuration errors
## Errors and cancellation

`instrument(config=...)` must be a `fi_instrumentation.TraceConfig` and
`capture_content` a `bool`; anything else raises `TypeError` and wraps
nothing. On an installed `voyageai` outside `>=0.3.7,<1`, `instrument()` logs
an error and wraps nothing (OpenTelemetry's dependency check).
Failed calls produce an ERROR span and an exception event, without invented
usage or result dimensions. For `multimodal_embed` and `contextualized_embed`,
only the exception type and a generic message are recorded: SDK validation or
service errors can echo media or entire documents. Original errors still reach
your application unchanged. Other methods redact the client key from the error
message and apply the configured core error-privacy policy. Stack traces are
never exported by this instrumentor.

Cancelling an async call or interrupting a blocking call ends its span with
`voyage.cancelled=true` and no exception event. Instrumentation failures do not
change the SDK's return value or raised exception.

## Hosts and configuration

The instrumentor never changes `base_url`. The SDK selects its host from your
key; an explicit `base_url` on the client takes precedence. Set any required
regional host on the client, not the instrumentor. A failed request does not
cause the instrumentor to retry against another host.

`instrument(config=...)` requires a `TraceConfig`; `capture_content` must be a
boolean. Invalid option types raise `TypeError` before wrapping any method.
OpenTelemetry's dependency check prevents instrumentation outside the supported
SDK version range.

## License

Expand Down
35 changes: 30 additions & 5 deletions python/frameworks/voyage/tests/_support.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

``FakeVoyage`` speaks enough of the Voyage HTTP API for voyageai 0.3.7 to
0.5.x: ``POST /v1/embeddings`` (base64 or list vectors, float and packed
binary dtypes), ``POST /v1/rerank`` and ``POST /v1/multimodalembeddings``.
binary dtypes), ``POST /v1/rerank``, ``POST /v1/multimodalembeddings`` and
``POST /v1/contextualizedembeddings``.
The real ``voyageai.Client`` / ``AsyncClient`` reach it through
``base_url=fake.base_url``. Nothing here leaves 127.0.0.1 and every key is a
placeholder.
Expand Down Expand Up @@ -175,17 +176,41 @@ def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler API
"usage": {"total_tokens": rerank_tokens(len(documents))},
},
)
elif path == "/v1/contextualizedembeddings":
inputs = list(body.get("inputs") or [])
data = []
for i, chunks in enumerate(inputs):
if body.get("enable_auto_chunking"):
chunks = ["SERVER-CHUNK-PRIVATE"] * 2
data.append(
{
"object": "list",
"index": i,
"data": [
{"embedding": _embedding(body), "index": j, "text": text}
for j, text in enumerate(chunks)
],
}
)
self._send(
200,
{
"object": "list",
"data": data,
"usage": {"total_tokens": 7 * sum(len(d["data"]) for d in data)},
},
)
elif path == "/v1/multimodalembeddings":
inputs = list(body.get("inputs") or [])
data = [
{"object": "embedding", "embedding": _vector(DIMENSION), "index": i}
{"object": "embedding", "embedding": _embedding(body), "index": i}
for i in range(len(inputs))
]
usage = {
"text_tokens": 3,
"image_pixels": 0,
"video_pixels": 0,
"total_tokens": 3,
"image_pixels": 128,
"video_pixels": 256,
"total_tokens": 11,
}
self._send(200, {"object": "list", "data": data, "usage": usage})
else:
Expand Down
Loading
Loading