temporalio · DABH · Sep 11, 2026 · Sep 11, 2026 · Sep 11, 2026
@@ -15,6 +15,7 @@
 
 # The AI SDK team owns the AI integration samples and their tests. We add
 # @temporalio/sdk too, so the SDK team can continue to manage repo-wide concerns.
+/arize_tracing/           @temporalio/sdk @temporalio/ai-sdk
 /deepagents_plugin/       @temporalio/sdk @temporalio/ai-sdk
 /google_adk_agents/       @temporalio/sdk @temporalio/ai-sdk
 /google_genai/            @temporalio/sdk @temporalio/ai-sdk
@@ -24,6 +25,7 @@
 /litellm_activity/        @temporalio/sdk @temporalio/ai-sdk
 /openai_agents/           @temporalio/sdk @temporalio/ai-sdk
 /strands_plugin/          @temporalio/sdk @temporalio/ai-sdk
+/tests/arize_tracing/     @temporalio/sdk @temporalio/ai-sdk
 /tests/deepagents_plugin/ @temporalio/sdk @temporalio/ai-sdk
 /tests/google_adk_agents/ @temporalio/sdk @temporalio/ai-sdk
 /tests/google_genai/      @temporalio/sdk @temporalio/ai-sdk

@@ -60,6 +60,7 @@ Some examples require extra dependencies. See each sample's directory for specif
   * [hello update](hello/hello_update.py) - Send a request to and a response from a client to a workflow execution.
 <!-- Keep this list in alphabetical order -->
 * [activity_worker](activity_worker) - Use Python activities from a workflow in another language.
+* [arize_tracing](arize_tracing) - Trace Temporal workflows and OpenAI Agents in Arize Phoenix or Arize AX with the OpenTelemetry plugin and OpenInference.
 * [batch_sliding_window](batch_sliding_window) - Batch processing with a sliding window of child workflows.
 * [bedrock](bedrock) - Orchestrate a chatbot with Amazon Bedrock.
 * [cloud_export_to_parquet](cloud_export_to_parquet) - Set up schedule workflow to process exported files on an hourly basis

@@ -0,0 +1,30 @@
+# Copy to .env and adjust. Load with: set -a; source arize_tracing/.env; set +a
+
+# Arize Phoenix (default target). Matches arize_tracing/phoenix/docker-compose.yml;
+# the UI, REST API, and OTLP/HTTP collector all live on this base URL.
+PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006
+# Only when Phoenix authentication is on (Phoenix Cloud, or PHOENIX_ENABLE_AUTH):
+# PHOENIX_API_KEY=...
+
+# Arize AX (SaaS). Setting both switches the exporter from Phoenix to Arize AX.
+# ARIZE_SPACE_ID=...
+# ARIZE_API_KEY=...
+# EU region only:
+# ARIZE_OTLP_ENDPOINT=https://otlp.eu-west-1a.arize.com/v1/traces
+
+# Project that receives the spans (Phoenix creates it on first use).
+ARIZE_PROJECT_NAME=temporal-ticket-triage
+# Reported as the OpenInference user.id on each trace by the starters.
+ARIZE_DEMO_USER=demo-user
+
+# LLM. Any OpenAI-compatible endpoint works.
+OPENAI_API_KEY=sk-...
+MODEL_CLASSIFY=gpt-4o-mini
+MODEL_DRAFT=gpt-4o-mini
+MODEL_AGENT=gpt-4o-mini
+# To use a local OpenAI-compatible gateway (for example a LiteLLM proxy) instead:
+# OPENAI_BASE_URL=http://localhost:4000/v1
+# OPENAI_API_KEY=<gateway key>
+# MODEL_CLASSIFY=<gateway model alias>
+# MODEL_DRAFT=<gateway model alias>
+# MODEL_AGENT=<gateway model alias>
@@ -0,0 +1,256 @@
+# Arize Tracing
+
+This sample shows the recommended way to get Temporal workflow traces into
+[Arize](https://arize.com/) — the open-source [Arize Phoenix](https://arize.com/docs/phoenix)
+or the [Arize AX](https://arize.com/docs/ax) platform — using Temporal's
+[`OpenTelemetryPlugin`](https://python.temporal.io/temporalio.contrib.opentelemetry.OpenTelemetryPlugin.html)
+plus a standard OTLP/HTTP exporter and the
+[OpenInference](https://github.com/Arize-ai/openinference) semantic conventions
+that Arize reads. No Arize SDK or Arize-specific plugin is involved, workflow
+code stays deterministic and sandboxed, and traces are correctly nested,
+correctly typed, and duplicate-free across replay and worker restarts.
+
+Contents:
+
+- **[ticket_triage/](ticket_triage/)** — the framework-agnostic pattern: an LLM
+  ticket-triage workflow (two LLM activities, one plain activity, one human
+  approval delivered as a workflow update). LLM calls run in activities and are
+  captured by the OpenInference OpenAI instrumentation.
+- **[ticket_triage_agents/](ticket_triage_agents/)** — the same workflow built
+  with the [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/)
+  through Temporal's `OpenAIAgentsPlugin(use_otel_instrumentation=True)`, so
+  Arize shows native AGENT, LLM, and TOOL spans.
+- **[verify_trace.py](verify_trace.py)** — checks a trace through the Phoenix
+  REST API: whole-tree equality with span kinds, enrichment attributes, LLM
+  token usage, activity attempts, and no duplicates.
+- **[phoenix/docker-compose.yml](phoenix/docker-compose.yml)** — pinned
+  self-hosted Phoenix, one container.
+- **[telemetry.py](telemetry.py)** — the OpenTelemetry wiring: replay-safe
+  tracer provider, Phoenix/Arize AX exporter, the OpenInference enrichment
+  processor, and LLM instrumentation.
+
+## Prerequisites
+
+- Docker (for Phoenix) or `uvx`, a local Temporal server
+  (`temporal server start-dev`), and `uv`.
+- An OpenAI-compatible LLM endpoint: either a real `OPENAI_API_KEY`, or any
+  OpenAI-compatible gateway via `OPENAI_BASE_URL`.
+
+## Run it
+
+```bash
+# 1. Start Phoenix (UI, REST API, and OTLP collector on port 6006)
+docker compose -f arize_tracing/phoenix/docker-compose.yml up -d
+curl -sf http://localhost:6006/healthz && echo ok
+#    ...or without Docker: uvx --from "arize-phoenix==20.9.0" phoenix serve
+
+# 2. Install dependencies and set environment (repo root)
+uv sync --group arize-tracing
+cp arize_tracing/.env.example arize_tracing/.env   # edit the LLM settings
+set -a; source arize_tracing/.env; set +a
+
+# 3. Run the sample (two terminals, same environment)
+uv run python -m arize_tracing.ticket_triage.worker
+uv run python -m arize_tracing.ticket_triage.starter
+
+# 4. Verify the trace through the Phoenix API (uses the printed trace ID)
+uv run python -m arize_tracing.verify_trace --trace-id <printed trace id>
+```
+
+The starter prints a direct link to the trace in Phoenix (project
+`temporal-ticket-triage`). You should see one trace shaped like this, with the
+OpenInference kinds Arize renders:
+
+```
+ticket-triage                                    CHAIN  (root; session = workflow id, user, input, output, tags)
+├─ StartWorkflow:TicketTriageWorkflow            CHAIN
+│  └─ RunWorkflow:TicketTriageWorkflow           CHAIN
+│     ├─ triage                                  CHAIN  (custom span from workflow code)
+│     │  ├─ StartActivity:classify_ticket → RunActivity:classify_ticket   CHAIN, attempt 1
+│     │  │  └─ ChatCompletion                    LLM    (model, tokens, messages)
+│     │  └─ StartActivity:lookup_account  → RunActivity:lookup_account
+│     └─ StartActivity:draft_reply → RunActivity:draft_reply
+│        └─ ChatCompletion                       LLM
+└─ StartWorkflowUpdate:approve                   CHAIN
+   ├─ ValidateUpdate:approve                     CHAIN
+   └─ HandleUpdate:approve                       CHAIN
+```
+
+![Ticket triage trace in Phoenix](phoenix-ticket-triage.png)
+
+The Sessions tab groups traces by Temporal workflow ID (`session.id`), so all
+interactions with one workflow execution appear as one session. Start the
+starter with `--workflow-id <id>` more than once to see several traces in a
+session.
+
+### With the OpenAI Agents SDK
+
+```bash
+# Two worker processes (see "Known issue" below), then the starter
+uv run python -m arize_tracing.ticket_triage_agents.worker --role workflows
+uv run python -m arize_tracing.ticket_triage_agents.worker --role activities
+uv run python -m arize_tracing.ticket_triage_agents.starter
+uv run python -m arize_tracing.verify_trace --trace-id <printed trace id> --scenario agents
+```
+
+Here the Agents SDK trace itself becomes the root, and the plugin's
+OpenTelemetry bridge produces the OpenInference kinds directly:
+
+```
+Ticket triage agents                             AGENT  (root; session, user, input, output)
+├─ temporal:startWorkflow:TicketTriageAgentsWorkflow   CHAIN
+│  └─ temporal:executeWorkflow                   CHAIN
+│     ├─ Agent workflow → Triage agent           AGENT
+│     │  ├─ turn                                 CHAIN
+│     │  │  └─ temporal:startActivity            CHAIN  (model call)
+│     │  │     ├─ temporal:executeActivity → response   LLM
+│     │  │     └─ lookup_account                 TOOL   (a Temporal activity as an agent tool)
+│     │  │        └─ temporal:startActivity → temporal:executeActivity
+│     │  └─ turn → temporal:startActivity → temporal:executeActivity → response   LLM
+│     └─ Agent workflow → Reply agent            AGENT
+│        └─ turn → temporal:startActivity → temporal:executeActivity → response   LLM
+└─ temporal:updateWorkflow                       CHAIN
+```
+
+(The tool span nests under the preceding model call's `temporal:startActivity`
+span rather than directly under the turn because the plugin leaves that span
+current in the Agents SDK scope after it finishes; see
+[temporalio/sdk-python#1855](https://github.com/temporalio/sdk-python/issues/1855).)
+
+![Ticket triage agents trace in Phoenix](phoenix-ticket-triage-agents.png)
+
+**Known issue.** With `use_otel_instrumentation=True`, a single worker process
+that runs both the workflow and its activities exports the
+`temporal:startActivity` spans with a parent that is not in the trace, so the
+model-call and tool subtrees appear detached from the agent turns in Arize
+([temporalio/sdk-python#1852](https://github.com/temporalio/sdk-python/issues/1852)).
+`verify_trace.py` reports this as spans referencing a missing parent. Running
+the workflow and the activities in separate worker processes, as above, avoids
+it; `worker.py` without `--role` runs both in one process, which is fine for
+the framework-agnostic scenario but not for this one until the fix lands.
+
+## How replay, retries, and restarts show up
+
+Durable execution means workflow code re-executes (replays) on worker
+restarts and cache evictions, and activities retry. The rule this sample
+demonstrates: **replay produces no spans, real re-executions do**, which is
+exactly what the Temporal Web UI shows too (Event History records nothing for a
+replay, but it does record every activity attempt).
+
+| What happened | Temporal Web UI | Arize |
+|---|---|---|
+| Workflow replayed (worker restart, cache eviction, `--replay-stress`) | Nothing: Event History is unchanged | Nothing: spans re-created during replay have the same deterministic IDs and are never exported again |
+| Activity retried | Pending Activities shows attempt N and the last failure | One `RunActivity` span per attempt under one `StartActivity`, failed attempts with error status, `temporal.activity.attempt` = 1, 2, ... |
+| Worker died while the workflow waited | Workers tab / Workflow Task timeouts | The `RunWorkflow` span exports once, when the workflow finishes on another worker; its duration covers the outage |
+| Worker died mid-activity | The attempt is not recorded; the next attempt is | The attempt's span was never ended, so it does not appear; the next attempt does |
+| Workflow Task failed (bug, non-determinism) | `WorkflowTaskFailed` events | Spans that ended inside the failed task are exported again with the same span ID; Phoenix deduplicates by span ID and keeps one copy |
+| Reset or retried workflow (new run) | New run under the same Workflow Id | Another `RunWorkflow` span with its own `temporalRunID`, same trace; the session (workflow ID) groups them |
+| Continue-As-New | New run under the same Workflow Id | The new run's `RunWorkflow` span nests under the previous run's, same trace |
+
+Reproduce each case with the flags below; every run must still verify cleanly:
+
+```bash
+# Replay stress: disable the workflow cache so EVERY workflow task replays
+# the workflow from the start of history. The trace must be identical.
+uv run python -m arize_tracing.ticket_triage.worker --replay-stress
+uv run python -m arize_tracing.ticket_triage.starter
+uv run python -m arize_tracing.verify_trace --trace-id <printed trace id>
+
+# Worker restart mid-workflow: the starter waits 20s before sending the
+# approval. Give the triage activities a few seconds to finish, then kill the
+# worker while the workflow durably awaits approval; start a new worker and
+# watch the workflow (and its trace) complete cleanly.
+uv run python -m arize_tracing.ticket_triage.starter --pause-before-approval 20
+#   ... after ~5s, ctrl+c the worker, then start it again in another terminal
+uv run python -m arize_tracing.verify_trace --trace-id <printed trace id>
+
+# Activity retry: classify_ticket fails on its first attempt. Arize shows two
+# RunActivity:classify_ticket spans, the first with an error status.
+uv run python -m arize_tracing.ticket_triage.worker --fail-first-attempt
+uv run python -m arize_tracing.ticket_triage.starter
+uv run python -m arize_tracing.verify_trace --trace-id <id> --expect-attempts classify_ticket=2
+
+# Worker crash mid-activity: classify_ticket heartbeats for 30s first. Kill
+# the worker hard (kill -9) during that time and start a new one. The first
+# attempt's span was never ended, so it is absent; the retry appears with
+# attempt 2 after the heartbeat timeout.
+uv run python -m arize_tracing.ticket_triage.worker --slow-classify 30
+uv run python -m arize_tracing.ticket_triage.starter
+uv run python -m arize_tracing.verify_trace --trace-id <id> --expect-attempt classify_ticket=2
+
+# Reset: rerun a finished workflow from its first workflow task. The reset
+# reapplies the original approval update, and the new run shares the trace as
+# a second RunWorkflow span with its own run ID.
+temporal workflow reset --workflow-id <workflow id> --type FirstWorkflowTask --reason demo
+uv run python -m arize_tracing.verify_trace --workflow-id <workflow id> --expect-runs 2
+```
+
+![Activity retry attempts in Phoenix](phoenix-retry-attempts.png)
+
+## Where spans come from
+
+| Span | Emitted by | Where it runs |
+|---|---|---|
+| `ticket-triage` (root) + OpenInference session/user/input/output | starter code | starter |
+| `StartWorkflow:*`, `StartWorkflowUpdate:*` | `OpenTelemetryPlugin` | starter (client side) |
+| `RunWorkflow:*`, `StartActivity:*`, `ValidateUpdate:*`, `HandleUpdate:*` | `OpenTelemetryPlugin` | worker (workflow) |
+| `triage` | plain OpenTelemetry API in workflow code | worker (workflow) |
+| `RunActivity:*` | `OpenTelemetryPlugin` | worker (activity) |
+| `ChatCompletion` LLM spans | `openinference-instrumentation-openai` | worker (activity) |
+| `openinference.span.kind`, `session.id`, `metadata`, `temporal.activity.attempt` on Temporal spans | `OpenInferenceEnrichmentProcessor` in `telemetry.py` | every process |
+
+The enrichment processor is optional but recommended: without it Phoenix and
+Arize AX show Temporal's spans as UNKNOWN, do not group them into sessions,
+and cannot tell activity attempts apart.
+
+## Where tracing works
+
+| Location | Works? | Notes |
+|---|---|---|
+| Activity bodies | ✅ | Plain OpenTelemetry + any OpenInference instrumentation, no restrictions. This is where LLM calls belong. |
+| Workflow bodies | ✅ | Plain OpenTelemetry APIs are replay-safe under the plugin: deterministic span IDs, no re-export on replay. Spans export when they end; the `RunWorkflow` span exports when the run completes. |
+| Signal/query/update handlers | ✅ | Handled by the plugin automatically (`HandleUpdate:*` etc.). |
+| Client / starter code | ✅ | Standard OpenTelemetry; put the OpenInference trace-level attributes on your root span. |
+
+## Sending to Arize AX instead of Phoenix
+
+Set `ARIZE_SPACE_ID` and `ARIZE_API_KEY` (and `ARIZE_OTLP_ENDPOINT` for the EU
+region) and `telemetry.py` exports to `https://otlp.arize.com/v1/traces` with
+the same spans and attributes; `ARIZE_PROJECT_NAME` selects the project.
+`verify_trace.py` reads Phoenix's REST API and does not apply to Arize AX; use
+the AX UI or the `ax` CLI there.
+
+## Operational notes
+
+- Phoenix and Arize AX both accept OTLP/HTTP; this sample uses
+  `opentelemetry-exporter-otlp-proto-http`.
+- Short-lived processes must flush: the starters and workers call
+  `force_flush()` on exit (see `telemetry.py`).
+- The workflow ID doubles as the Arize session ID. A fresh ID per run is the
+  default; reuse one to group runs.
+- `OTEL_SDK_DISABLED=true` turns off export without code changes.
+- Ingestion is asynchronous; `verify_trace.py` polls until the trace is stable.
+- The OpenAI Agents SDK bridge (`use_otel_instrumentation=True`) is Public
+  Preview in the Temporal SDK. Run the starter, the workflow worker, and the
+  activity worker as separate processes, as the sample does (see the known
+  issue above), and see `quiet_otel_context_detach_errors()` in `telemetry.py`
+  for a known log-noise issue.
+
+## Tests
+
+`tests/arize_tracing/` runs without Arize, Docker, or an LLM: mocked activities
+(and the SDK's `TestModel` for the agents scenario), an in-memory span
+exporter, a worker with the workflow cache disabled, whole-tree span
+assertions, enrichment and retry-attempt assertions, and a `Replayer` pass
+asserting that replaying the finished workflow's history emits zero new spans.
+
+```bash
+uv run --group arize-tracing pytest tests/arize_tracing -v
+```
+
+## Using this outside samples-python
+
+The sample is self-contained: copy the `arize_tracing/` directory, change the
+absolute imports (`arize_tracing.ticket_triage.activities` →
+`ticket_triage.activities` or similar), and install the dependencies listed
+under `arize-tracing` in this repo's `pyproject.toml`.
@@ -0,0 +1,21 @@
+# Self-hosted Arize Phoenix for the arize_tracing samples.
+#
+# One container serves the UI, the REST API, and the OTLP/HTTP collector on
+# port 6006 (the OTLP gRPC listener on 4317 is not published; this sample
+# exports over HTTP). Traces persist in a named volume across restarts.
+#
+# Start:  docker compose -f arize_tracing/phoenix/docker-compose.yml up -d
+# Check:  curl -sf http://localhost:6006/healthz
+# UI:     http://localhost:6006
+name: phoenix-demo
+services:
+  phoenix:
+    image: arizephoenix/phoenix:version-20.9.0
+    ports:
+      - "6006:6006"
+    environment:
+      PHOENIX_WORKING_DIR: /mnt/data
+    volumes:
+      - phoenix_data:/mnt/data
+volumes:
+  phoenix_data: {}