diff --git a/src/pages/docs/integrations/langsmith.mdx b/src/pages/docs/integrations/langsmith.mdx new file mode 100644 index 000000000..00b215a76 --- /dev/null +++ b/src/pages/docs/integrations/langsmith.mdx @@ -0,0 +1,162 @@ +--- +title: "LangSmith Trace Forwarding to Future AGI (OpenTelemetry)" +description: "Send the runs of a LangSmith-traced LangChain, LangGraph or @traceable app to Future AGI live, using LangSmith's own OpenTelemetry mode. No package, no historical import." +--- + +## About + +LangSmith can emit every run as an OpenTelemetry span. This page points those spans at Future AGI while your app runs. It is **live forwarding only**: it does not import runs that already sit in a LangSmith project, and it puts no proxy or interceptor in front of `api.smith.langchain.com`. + +There is no traceAI package for LangSmith. The integration is one OpenTelemetry tracer provider that you install before the first traced call. + +Tested with `langsmith` 0.14.11 (latest stable on 2026-10-16) and 0.14.4 (the version the example pins), `langchain` 1.4.6, `opentelemetry-sdk` 1.45.x, Python 3.12 against a local Future AGI stack, and in fan-out mode against the real LangSmith API on 2026-10-19. The recipe's contract tests also run on Python 3.10 to 3.13. + +## 1. Installation + +```bash +pip install "langsmith[otel]" opentelemetry-sdk opentelemetry-exporter-otlp-proto-http +``` + +The `otel` extra installs the OpenTelemetry SDK and OTLP/HTTP exporter that LangSmith needs. Install `langchain` and your model package as usual if you trace LangChain or LangGraph code. + +## 2. Choose a mode + +| Goal | LangSmith variables | LangSmith key | +|---|---|---| +| Future AGI only | `LANGSMITH_TRACING=true`, `LANGSMITH_OTEL_ENABLED=true`, `LANGSMITH_OTEL_ONLY=true` | Not needed. LangSmith is not called. | +| LangSmith and Future AGI (fan-out) | `LANGSMITH_TRACING=true`, `LANGSMITH_OTEL_ENABLED=true` | `LANGSMITH_API_KEY`, for the LangSmith copy | + +`LANGSMITH_TRACING_MODE` (`langsmith`, `otel` or `hybrid`) wins over both flags when set. `LANGSMITH_OTEL_ONLY` is not on LangSmith's own OpenTelemetry page; it exists in the SDK versions above and stops the calls to LangSmith. Fan-out mode was run against the real LangSmith API on 2026-10-19 (see [Live vendor verification](#live-vendor-verification-2026-10-19)). + +## 3. Set environment variables + +```bash +export LANGSMITH_TRACING=true +export LANGSMITH_OTEL_ENABLED=true +export LANGSMITH_OTEL_ONLY=true # Future AGI only; remove to fan out + +export FI_API_KEY="your-futureagi-api-key" +export FI_SECRET_KEY="your-futureagi-secret-key" +export FI_PROJECT_NAME="my-langsmith-app" + +# Clear inherited trace-specific overrides, then set the general OTLP values. +unset OTEL_EXPORTER_OTLP_TRACES_ENDPOINT OTEL_EXPORTER_OTLP_TRACES_HEADERS +export OTEL_EXPORTER_OTLP_ENDPOINT="https://YOUR_FI_COLLECTOR_ORIGIN" # origin only, no path +export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=$FI_API_KEY,x-secret-key=$FI_SECRET_KEY" +export OTEL_EXPORTER_OTLP_COMPRESSION=none +``` + +Future AGI's collector rejects gzip request bodies, so keep compression off. + +## 4. Install the tracer provider + +Create the provider before the first traced call and before `langsmith.Client()`. LangSmith adopts a global provider if one exists. + +```python +import os +import langsmith +from opentelemetry import trace +from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter +from opentelemetry.sdk.resources import Resource +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchSpanProcessor + +provider = TracerProvider( + resource=Resource.create( + {"project_name": os.environ["FI_PROJECT_NAME"], "project_type": "observe"} + ) +) +provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) +trace.set_tracer_provider(provider) + +client = langsmith.Client() +langsmith.configure(client=client) +``` + +Why you need your own provider: LangSmith's built-in provider cannot carry the `project_name` resource attribute that Future AGI requires, and unless `OTEL_EXPORTER_OTLP_HEADERS` is set it sends your LangSmith key as `x-api-key` to whatever OTLP endpoint is configured. + +## 5. Run your app + +Use LangChain, LangGraph or `@traceable` functions as you normally would, then flush before a short script exits: + +```python +from langchain.agents import create_agent +from langchain_core.tools import tool +from langchain_openai import ChatOpenAI + +@tool +def lookup_policy(topic: str) -> str: + """Return the refund policy for a topic.""" + return "Refunds are accepted within 30 days of purchase." + +agent = create_agent(ChatOpenAI(model="gpt-4o-mini"), tools=[lookup_policy]) +agent.invoke({"messages": [{"role": "user", "content": "What is the refund window?"}]}) + +client.flush() +provider.shutdown() +``` + +A complete runnable example, including a `@traceable` app and a failing-tool run, is in the traceAI repository under `python/examples/langsmith/`. + +## What is captured + +Verified in the Future AGI trace view and storage: + +- Hierarchy: the LangGraph run is the root `chain`; its `model` and `tools` nodes are child `chain` spans; each `ChatOpenAI` call is an `llm` span and each tool call is a `tool` span. +- Model, provider and token counts fill on the `llm` span (for example `gpt-5.6-terra`, 357 prompt and 19 completion tokens). +- A tool that raises produces an `ERROR` status on the tool span and its parent `tools` chain span. The error text is not exported as a message. +- `langsmith.*` metadata: run tags, trace name, `ls_*` keys, LangGraph node keys, and the `LANGSMITH_*` / `LANGCHAIN_*` variables whose names are not secret-like. This filters names, not values. + +Not captured, or not shown: + +- **Input and output panels stay empty.** Prompts and completions are exported as the single JSON attributes `gen_ai.prompt` and `gen_ai.completion`. They are stored and visible under Attributes, but the trace view does not turn them into Input/Output messages. +- **Retriever runs** are typed as embeddings and parser runs as unknown, because LangSmith's run-type mapping does not match Future AGI's span types. +- No cost value, no session or user id (LangSmith does not export them in these runs). + +## Privacy + +LangSmith exports prompts, completions, tool arguments and results by default, and this recipe keeps that default: content appears in Future AGI unless you switch it off. + +Turn content off with LangSmith's own switches: + +```bash +export LANGSMITH_HIDE_INPUTS=true +export LANGSMITH_HIDE_OUTPUTS=true +``` + +With both set, a unique marker placed in the question was found in 5 of 7 stored spans without the switches and in 0 of 7 with them (all stored columns scanned, including resource attributes and events). Token counts also empty, because LangSmith reads them from the outputs. `LANGSMITH_HIDE_METADATA=true` drops the metadata too, along with the model name. + +Local usernames, absolute paths and host names were not found in the stored spans. The `langsmith.metadata.revision_id` attribute carries the output of `git describe` for the working directory of the app (a tag or commit hash, never a path). + +## Troubleshooting + +- **HTTP 400 from the collector.** The resource has no `project_name`. Create the provider before `langsmith.Client()`. +- **No spans arrive.** `LANGSMITH_TRACING` must be exactly `true`. A short script must call `client.flush()` and `provider.shutdown()` before exit. +- **Spans go to the wrong place or your LangSmith key shows up in an OTLP header.** A trace-specific variable such as `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_HEADERS` is overriding the general ones. Unset them. +- **Gzip errors.** Set `OTEL_EXPORTER_OTLP_COMPRESSION=none`. +- **`Failed to export spans batch due to timeout, max retries or shutdown.`** The OTLP exporter gives up after 10 seconds by default. A busy collector can take longer (on a loaded local stack an empty request took 4 to 13 seconds), and the batch may still be stored after the message. Raise `OTEL_EXPORTER_OTLP_TIMEOUT` (seconds), for example `60`. + +## Limitations + +- Live forwarding only. There is no import of historical LangSmith projects, and no LangSmith REST client. +- Fan-out was verified against LangSmith's US cloud (`api.smith.langchain.com`) with a personal access token on a single-workspace account. EU/self-hosted LangSmith (`LANGSMITH_ENDPOINT`), service keys and organisation-scoped keys that need `LANGSMITH_WORKSPACE_ID` were not tried. +- LangChain and LangGraph runs, a real model and a failing tool were verified; streaming, retriever and parser runs, the `hide_*` callables and an OpenTelemetry Collector fan-out were not. + +## Live vendor verification (2026-10-19) + +Run against the real LangSmith API (US region, `api.smith.langchain.com`, a personal access token) in fan-out mode, with the app's OpenTelemetry spans going to a local Future AGI stack. LangSmith project `th8103-e2e`; Future AGI project `th8103-livefill-th8335`. + +- **Versions:** `langsmith` 0.14.11, `langchain` 1.4.6, `langchain-openai` 1.7.2, `opentelemetry-sdk` and `opentelemetry-exporter-otlp-proto-http` 1.45.1, Python 3.12. The setup on this page ran unchanged. +- **LangSmith accepted the runs.** The SDK called `GET /info` (200) and `POST /runs/multipart` (202). Reading the project back through `POST /api/v1/runs/query` returned the full run tree: root `LangGraph` chain, `model` and `tools` chains, `ChatOpenAI` llm runs with token totals, and the `lookup_policy` tool run. +- **Success run:** LangSmith trace `01a15682-3f4a-77ae-8845-0786985be886` (792 total tokens). The same run is Future AGI trace `40aac0f3-d1cf-e174-f734-f7e572573276`: 7 spans, `chain LangGraph > chain model > llm ChatOpenAI` (model `gpt-5.6-terra`, 357/19 and 398/18 tokens), `chain tools > tool lookup_policy`. +- **Error run:** LangSmith trace `01a1568c-9cc1-7217-8cfb-0078e9343cfb` (status `error`). The same run is Future AGI trace `f8033fac-dd16-c3e0-6ada-af81b58d49af`: 5 spans, with `broken_lookup` and its `tools` chain in `ERROR` status. LangSmith keeps the error text and traceback; Future AGI stores neither (all stored columns scanned). +- **The two trace ids differ.** The OpenTelemetry trace id is not the LangSmith run id. Both runs have the same start time to the microsecond, and the LangSmith run id appears inside the exported content (`lc_run--`), which is how the pairs above were matched. With content hidden you can match by start time. +- **Hide switches reach both copies.** With `LANGSMITH_HIDE_INPUTS=true` and `LANGSMITH_HIDE_OUTPUTS=true`, the run read back from LangSmith has `inputs` and `outputs` of `null`, and the unique marker was in 0 of 7 stored Future AGI spans (5 of 7 without the switches). +- **Transient exporter timeouts.** Several runs on the shared local stack printed the timeout message above while the stack was saturated. That is the local collector's latency, not LangSmith's; the batches were stored and `OTEL_EXPORTER_OTLP_TIMEOUT=90` removed the message. + +Still unproven: EU or self-hosted LangSmith, org-scoped keys (`LANGSMITH_WORKSPACE_ID`), streaming, retriever and parser runs, and an OpenTelemetry Collector fan-out. Future AGI Cloud was not used; this recipe does not need a public endpoint. + +## How this differs from related integrations + +- [traceAI LangChain](/docs/integrations/traceai/langchain) instruments LangChain directly and needs no LangSmith. Use it when you do not need LangSmith at the same time. +- [Langfuse import](/docs/integrations/import/langfuse) copies historical data. This page forwards live traffic.