Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions src/pages/docs/integrations/langsmith.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
---
title: "LangSmith Trace Forwarding to Future AGI (OpenTelemetry)"
description: "Send the runs of a LangSmith-traced LangChain, LangGraph or @traceable app to Future AGI live, using LangSmith's own OpenTelemetry mode. No package, no historical import."
---

## About

LangSmith can emit every run as an OpenTelemetry span. This page points those spans at Future AGI while your app runs. It is **live forwarding only**: it does not import runs that already sit in a LangSmith project, and it puts no proxy or interceptor in front of `api.smith.langchain.com`.

There is no traceAI package for LangSmith. The integration is one OpenTelemetry tracer provider that you install before the first traced call.

Tested with `langsmith` 0.14.11 (latest stable on 2026-10-16) and 0.14.4 (the version the example pins), `langchain` 1.4.6, `opentelemetry-sdk` 1.45.x, Python 3.12 against a local Future AGI stack, and in fan-out mode against the real LangSmith API on 2026-10-19. The recipe's contract tests also run on Python 3.10 to 3.13.

## 1. Installation

```bash
pip install "langsmith[otel]" opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```

The `otel` extra installs the OpenTelemetry SDK and OTLP/HTTP exporter that LangSmith needs. Install `langchain` and your model package as usual if you trace LangChain or LangGraph code.

## 2. Choose a mode

| Goal | LangSmith variables | LangSmith key |
|---|---|---|
| Future AGI only | `LANGSMITH_TRACING=true`, `LANGSMITH_OTEL_ENABLED=true`, `LANGSMITH_OTEL_ONLY=true` | Not needed. LangSmith is not called. |
| LangSmith and Future AGI (fan-out) | `LANGSMITH_TRACING=true`, `LANGSMITH_OTEL_ENABLED=true` | `LANGSMITH_API_KEY`, for the LangSmith copy |

`LANGSMITH_TRACING_MODE` (`langsmith`, `otel` or `hybrid`) wins over both flags when set. `LANGSMITH_OTEL_ONLY` is not on LangSmith's own OpenTelemetry page; it exists in the SDK versions above and stops the calls to LangSmith. Fan-out mode was run against the real LangSmith API on 2026-10-19 (see [Live vendor verification](#live-vendor-verification-2026-10-19)).

## 3. Set environment variables

```bash
export LANGSMITH_TRACING=true
export LANGSMITH_OTEL_ENABLED=true
export LANGSMITH_OTEL_ONLY=true # Future AGI only; remove to fan out

export FI_API_KEY="your-futureagi-api-key"
export FI_SECRET_KEY="your-futureagi-secret-key"
export FI_PROJECT_NAME="my-langsmith-app"

# Clear inherited trace-specific overrides, then set the general OTLP values.
unset OTEL_EXPORTER_OTLP_TRACES_ENDPOINT OTEL_EXPORTER_OTLP_TRACES_HEADERS
export OTEL_EXPORTER_OTLP_ENDPOINT="https://YOUR_FI_COLLECTOR_ORIGIN" # origin only, no path
export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=$FI_API_KEY,x-secret-key=$FI_SECRET_KEY"
export OTEL_EXPORTER_OTLP_COMPRESSION=none
```

Future AGI's collector rejects gzip request bodies, so keep compression off.

## 4. Install the tracer provider

Create the provider before the first traced call and before `langsmith.Client()`. LangSmith adopts a global provider if one exists.

```python
import os
import langsmith
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(
resource=Resource.create(
{"project_name": os.environ["FI_PROJECT_NAME"], "project_type": "observe"}
)
)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

client = langsmith.Client()
langsmith.configure(client=client)
```

Why you need your own provider: LangSmith's built-in provider cannot carry the `project_name` resource attribute that Future AGI requires, and unless `OTEL_EXPORTER_OTLP_HEADERS` is set it sends your LangSmith key as `x-api-key` to whatever OTLP endpoint is configured.

## 5. Run your app

Use LangChain, LangGraph or `@traceable` functions as you normally would, then flush before a short script exits:

```python
from langchain.agents import create_agent
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI

@tool
def lookup_policy(topic: str) -> str:
"""Return the refund policy for a topic."""
return "Refunds are accepted within 30 days of purchase."

agent = create_agent(ChatOpenAI(model="gpt-4o-mini"), tools=[lookup_policy])
agent.invoke({"messages": [{"role": "user", "content": "What is the refund window?"}]})

client.flush()
provider.shutdown()
```

A complete runnable example, including a `@traceable` app and a failing-tool run, is in the traceAI repository under `python/examples/langsmith/`.

## What is captured

Verified in the Future AGI trace view and storage:

- Hierarchy: the LangGraph run is the root `chain`; its `model` and `tools` nodes are child `chain` spans; each `ChatOpenAI` call is an `llm` span and each tool call is a `tool` span.
- Model, provider and token counts fill on the `llm` span (for example `gpt-5.6-terra`, 357 prompt and 19 completion tokens).
- A tool that raises produces an `ERROR` status on the tool span and its parent `tools` chain span. The error text is not exported as a message.
- `langsmith.*` metadata: run tags, trace name, `ls_*` keys, LangGraph node keys, and the `LANGSMITH_*` / `LANGCHAIN_*` variables whose names are not secret-like. This filters names, not values.

Not captured, or not shown:

- **Input and output panels stay empty.** Prompts and completions are exported as the single JSON attributes `gen_ai.prompt` and `gen_ai.completion`. They are stored and visible under Attributes, but the trace view does not turn them into Input/Output messages.
- **Retriever runs** are typed as embeddings and parser runs as unknown, because LangSmith's run-type mapping does not match Future AGI's span types.
- No cost value, no session or user id (LangSmith does not export them in these runs).

## Privacy

LangSmith exports prompts, completions, tool arguments and results by default, and this recipe keeps that default: content appears in Future AGI unless you switch it off.

Turn content off with LangSmith's own switches:

```bash
export LANGSMITH_HIDE_INPUTS=true
export LANGSMITH_HIDE_OUTPUTS=true
```

With both set, a unique marker placed in the question was found in 5 of 7 stored spans without the switches and in 0 of 7 with them (all stored columns scanned, including resource attributes and events). Token counts also empty, because LangSmith reads them from the outputs. `LANGSMITH_HIDE_METADATA=true` drops the metadata too, along with the model name.

Local usernames, absolute paths and host names were not found in the stored spans. The `langsmith.metadata.revision_id` attribute carries the output of `git describe` for the working directory of the app (a tag or commit hash, never a path).

## Troubleshooting

- **HTTP 400 from the collector.** The resource has no `project_name`. Create the provider before `langsmith.Client()`.
- **No spans arrive.** `LANGSMITH_TRACING` must be exactly `true`. A short script must call `client.flush()` and `provider.shutdown()` before exit.
- **Spans go to the wrong place or your LangSmith key shows up in an OTLP header.** A trace-specific variable such as `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_HEADERS` is overriding the general ones. Unset them.
- **Gzip errors.** Set `OTEL_EXPORTER_OTLP_COMPRESSION=none`.
- **`Failed to export spans batch due to timeout, max retries or shutdown.`** The OTLP exporter gives up after 10 seconds by default. A busy collector can take longer (on a loaded local stack an empty request took 4 to 13 seconds), and the batch may still be stored after the message. Raise `OTEL_EXPORTER_OTLP_TIMEOUT` (seconds), for example `60`.

## Limitations

- Live forwarding only. There is no import of historical LangSmith projects, and no LangSmith REST client.
- Fan-out was verified against LangSmith's US cloud (`api.smith.langchain.com`) with a personal access token on a single-workspace account. EU/self-hosted LangSmith (`LANGSMITH_ENDPOINT`), service keys and organisation-scoped keys that need `LANGSMITH_WORKSPACE_ID` were not tried.
- LangChain and LangGraph runs, a real model and a failing tool were verified; streaming, retriever and parser runs, the `hide_*` callables and an OpenTelemetry Collector fan-out were not.

## Live vendor verification (2026-10-19)

Run against the real LangSmith API (US region, `api.smith.langchain.com`, a personal access token) in fan-out mode, with the app's OpenTelemetry spans going to a local Future AGI stack. LangSmith project `th8103-e2e`; Future AGI project `th8103-livefill-th8335`.

- **Versions:** `langsmith` 0.14.11, `langchain` 1.4.6, `langchain-openai` 1.7.2, `opentelemetry-sdk` and `opentelemetry-exporter-otlp-proto-http` 1.45.1, Python 3.12. The setup on this page ran unchanged.
- **LangSmith accepted the runs.** The SDK called `GET /info` (200) and `POST /runs/multipart` (202). Reading the project back through `POST /api/v1/runs/query` returned the full run tree: root `LangGraph` chain, `model` and `tools` chains, `ChatOpenAI` llm runs with token totals, and the `lookup_policy` tool run.
- **Success run:** LangSmith trace `01a15682-3f4a-77ae-8845-0786985be886` (792 total tokens). The same run is Future AGI trace `40aac0f3-d1cf-e174-f734-f7e572573276`: 7 spans, `chain LangGraph > chain model > llm ChatOpenAI` (model `gpt-5.6-terra`, 357/19 and 398/18 tokens), `chain tools > tool lookup_policy`.
- **Error run:** LangSmith trace `01a1568c-9cc1-7217-8cfb-0078e9343cfb` (status `error`). The same run is Future AGI trace `f8033fac-dd16-c3e0-6ada-af81b58d49af`: 5 spans, with `broken_lookup` and its `tools` chain in `ERROR` status. LangSmith keeps the error text and traceback; Future AGI stores neither (all stored columns scanned).
- **The two trace ids differ.** The OpenTelemetry trace id is not the LangSmith run id. Both runs have the same start time to the microsecond, and the LangSmith run id appears inside the exported content (`lc_run--<run id>`), which is how the pairs above were matched. With content hidden you can match by start time.
- **Hide switches reach both copies.** With `LANGSMITH_HIDE_INPUTS=true` and `LANGSMITH_HIDE_OUTPUTS=true`, the run read back from LangSmith has `inputs` and `outputs` of `null`, and the unique marker was in 0 of 7 stored Future AGI spans (5 of 7 without the switches).
- **Transient exporter timeouts.** Several runs on the shared local stack printed the timeout message above while the stack was saturated. That is the local collector's latency, not LangSmith's; the batches were stored and `OTEL_EXPORTER_OTLP_TIMEOUT=90` removed the message.

Still unproven: EU or self-hosted LangSmith, org-scoped keys (`LANGSMITH_WORKSPACE_ID`), streaming, retriever and parser runs, and an OpenTelemetry Collector fan-out. Future AGI Cloud was not used; this recipe does not need a public endpoint.

## How this differs from related integrations

- [traceAI LangChain](/docs/integrations/traceai/langchain) instruments LangChain directly and needs no LangSmith. Use it when you do not need LangSmith at the same time.
- [Langfuse import](/docs/integrations/import/langfuse) copies historical data. This page forwards live traffic.
Loading