From 9460889706534d5072c63f6efce750acce7b69a7 Mon Sep 17 00:00:00 2001 From: silanhe Date: Mon, 27 Jul 2026 22:09:47 +0000 Subject: [PATCH] docs(otel): expand plugin README Rewrite the otel package README modeled on the JS SDK README: covers both ExecutionOtelPlugin and InvocationOtelPlugin, the ADOT vs community-collector layer options, deployment matrix, export strategies, IAM, env vars, SAM templates, trace structure, log correlation, and API reference. Python code snippets throughout. Accurately reflects that only ExecutionOtelPlugin takes OtelPluginConfig/auto-creates a provider, while InvocationOtelPlugin uses a passed or global provider. --- .../README.md | 732 ++++++++++++++---- 1 file changed, 565 insertions(+), 167 deletions(-) diff --git a/packages/aws-durable-execution-sdk-python-otel/README.md b/packages/aws-durable-execution-sdk-python-otel/README.md index 5a38774a..7e8f46ed 100644 --- a/packages/aws-durable-execution-sdk-python-otel/README.md +++ b/packages/aws-durable-execution-sdk-python-otel/README.md @@ -1,15 +1,45 @@ # AWS Durable Execution SDK - OpenTelemetry Plugin -OpenTelemetry instrumentation plugin for the [AWS Durable Execution SDK for Python](https://github.com/aws/aws-durable-execution-sdk-python). Emits distributed traces that correlate across multiple Lambda invocations of a single durable execution, producing deterministic span and trace IDs so that spans from different invocations are stitched into a single coherent trace. - -## Features - -- **Deterministic Trace IDs**: All invocations of the same durable execution share a single trace, derived from the X-Ray trace header or execution ARN -- **Span-per-Operation**: Each durable operation (step, wait, invoke) gets its own span with accurate timing -- **Continuation Spans**: Operations completing in a different invocation are linked back to the original span -- **Log Correlation**: Enrich application logs with trace ID and span ID for end-to-end observability -- **Configurable Sampling**: Control trace volume via plugin options -- **Self-Contained Setup**: No manual TracerProvider configuration required +> **⚠️ Experimental Beta:** This plugin is currently in experimental beta. Functionality may change without notice between releases. It is not recommended for production workloads at this time. + +OpenTelemetry instrumentation plugin for the AWS Durable Execution SDK for Python. Emits distributed traces that correlate across multiple Lambda invocations of a single durable execution, producing deterministic span and trace IDs so that spans from different invocations are stitched into a single coherent trace. + +This package provides two plugin implementations: + +| Plugin | Trace Structure | +| ---------------------- | --------------------------------------------------------------------------------------- | +| `ExecutionOtelPlugin` | Workflow span as synthetic root; operations parent under it and link to the invocation span | +| `InvocationOtelPlugin` | Invocation span as the trace root; operations parent under the invocation span | + +The two plugins differ in how they obtain a `TracerProvider`: + +- **`ExecutionOtelPlugin`** accepts an `OtelPluginConfig` and supports three provider modes: + 1. **Auto-created** (default) — creates its own `TracerProvider` with OTLP export to `localhost:4318`. + 2. **Custom** — you pass your own `tracer_provider`. + 3. **Global default** — set `use_default_tracer_provider=True` to use the globally registered provider (e.g. from the ADOT layer). +- **`InvocationOtelPlugin`** takes plain constructor arguments (`trace_provider`, `context_extractor`, `instrument_name`, `enrich_logger`). It uses the `trace_provider` you pass, or the globally registered provider (`opentelemetry.trace.get_tracer_provider()`) when omitted — it does **not** auto-create an OTLP provider. For a community-collector deployment you construct an OTLP-exporting `TracerProvider` yourself and pass it in; for an ADOT deployment you omit it and the layer's global provider is used. + +Both plugins can be deployed with either the **ADOT Lambda layer** or the **OpenTelemetry community collector-only layer**. + +## Table of Contents + +- [Installation](#installation) +- [Quick Start](#quick-start) +- [Choosing a Plugin](#choosing-a-plugin) +- [Lambda Layer Options](#lambda-layer-options) +- [Deployment Matrix](#deployment-matrix) +- [Configuration](#configuration) +- [Export Strategies](#export-strategies) +- [Collector Configuration](#collector-configuration) +- [IAM Permissions](#iam-permissions) +- [Environment Variables](#environment-variables) +- [SAM/CloudFormation Templates](#samcloudformation-templates) +- [Trace Structure Comparison](#trace-structure-comparison) +- [Log Correlation](#log-correlation) +- [Additional Python Dependencies](#additional-python-dependencies) +- [API Reference](#api-reference) +- [Verification](#verification) +- [License](#license) ## Installation @@ -17,276 +47,644 @@ OpenTelemetry instrumentation plugin for the [AWS Durable Execution SDK for Pyth pip install aws-durable-execution-sdk-python-otel ``` -## Quick Start using X-Ray/CloudWatch Tracing +When the plugin auto-creates its `TracerProvider` (default mode) and you want AWS SDK / HTTP auto-instrumentation, install the optional `instrumentation` extra: -1. Add the [ADOT Lambda Layer](#1-adot-lambda-layer) to your function and set `AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument` -2. Enable [X-Ray Active Tracing](#2-aws-x-ray-active-tracing) on the function -3. Pass `InvocationOtelPlugin` to your handler's `plugins` list -4. Add X-Ray write permissions +```bash +pip install "aws-durable-execution-sdk-python-otel[instrumentation]" +``` -### 1. ADOT Lambda Layer +--- -This plugin requires the [AWS Distro for OpenTelemetry (ADOT) Lambda layer](https://aws-otel.github.io/docs/getting-started/lambda) to export traces from your Lambda function. +## Quick Start -The layer ARN follows the format: +Both plugins are used the same way — only the import and class name differ: -``` -arn:aws:lambda:::layer:aws-otel-python--ver- -``` +```python +from aws_durable_execution_sdk_python import DurableContext +from aws_durable_execution_sdk_python.config import Duration +from aws_durable_execution_sdk_python.execution import durable_execution +from aws_durable_execution_sdk_python_otel import ExecutionOtelPlugin +# OR: from aws_durable_execution_sdk_python_otel import InvocationOtelPlugin -Refer to the [ADOT Lambda Layer ARNs](https://aws-otel.github.io/docs/getting-started/lambda/lambda-python) page for the latest version number, architecture, and supported regions. -**AWS CLI:** +@durable_execution(plugins=[ExecutionOtelPlugin()]) +# OR: @durable_execution(plugins=[InvocationOtelPlugin()]) +def handler(event: dict, context: DurableContext) -> dict: + result = context.step(lambda _: fetch_data(event["id"]), name="fetch-data") -```bash -aws lambda update-function-configuration \ - --function-name your-function-name \ - --layers "arn:aws:lambda:::layer:aws-otel-python-amd64-ver-" -``` + context.wait(duration=Duration.from_seconds(5), name="cooldown") -You must also set the `AWS_LAMBDA_EXEC_WRAPPER` environment variable: + processed = context.step(lambda _: process(result), name="process") -```bash -aws lambda update-function-configuration \ - --function-name your-function-name \ - --environment "Variables={AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument}" + return processed ``` -> **Note:** Replace `` with your function's region and ``/`` with the latest layer version and architecture from the ADOT docs. +With no configuration, `ExecutionOtelPlugin` auto-creates a `TracerProvider` with: + +- OTLP export to `http://localhost:4318/v1/traces` +- AWS SDK (botocore) and HTTP (urllib3) instrumentations (when the `instrumentation` extra is installed) +- AWS X-Ray + W3C TraceContext propagators +- Deterministic trace and span ID generation + +`InvocationOtelPlugin()` with no arguments uses the globally registered `TracerProvider` (e.g. the one the ADOT layer configures). Outside an ADOT deployment, pass your own `trace_provider` so spans are exported. + +--- + +## Choosing a Plugin + +| Aspect | `ExecutionOtelPlugin` | `InvocationOtelPlugin` | +| ------------------------ | -------------------------------------------- | --------------------------------------------------------- | +| Trace root | Workflow span (synthetic, deterministic) | Invocation span | +| Operation parent | Workflow span | Invocation span | +| Invocation span role | Child of Workflow span; operations link to it | Trace root / parent of operations | +| Export timing | Operations deferred until complete | All spans exported immediately | +| Non-terminal invocations | Workflow span discarded (clean traces) | Invocation span emitted per invocation | +| Trace continuity | Single trace across all invocations | Per-invocation traces, correlated via span links | + +**Use `ExecutionOtelPlugin` when** you want a single unified trace view across all invocations of a durable execution, with the workflow as the logical root. + +**Use `InvocationOtelPlugin` when** you want a lighter-weight, per-invocation view, or want to delegate to the ADOT layer's ambient invocation span (deploy on the ADOT layer and omit `trace_provider`). + +--- + +## Lambda Layer Options + +Both plugins can use either Lambda layer. The layer provides span transport (a collector that listens on `localhost:4318` and forwards to X-Ray/CloudWatch). + +| Layer | What It Provides | ARN Format | +| ---------------------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------- | +| **ADOT Lambda Layer** | OTel SDK auto-instrumentation + collector extension | `arn:aws:lambda:::layer:aws-otel-python--ver-` | +| **Community Collector-Only Layer** | Collector extension only (no SDK auto-instrumentation) | `arn:aws:lambda:::layer:opentelemetry-collector--` | + +Consult the [ADOT Lambda (Python) docs](https://aws-otel.github.io/docs/getting-started/lambda/lambda-python) for the current layer ARN, architecture, and supported regions. Pin the layer version in production. -**CloudFormation / SAM:** +**ADOT Layer:** Registers a global `TracerProvider` with auto-instrumentation. Use `use_default_tracer_provider=True` so the plugin delegates to that provider. Set `AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument` to activate it. + +**Community Collector Layer:** Only runs a collector process at `localhost:4318`. The plugin creates its own `TracerProvider` (default mode) and exports spans to the collector. Requires a `collector.yaml` in your function bundle and `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/collector.yaml`. + +> **Tip:** The community collector layer is smaller and purpose-built for span transport. The ADOT layer is convenient if you want zero-config auto-instrumentation from the layer itself. + +--- + +## Deployment Matrix + +| # | Plugin | Layer | Provider selection | `AWS_LAMBDA_EXEC_WRAPPER` | `collector.yaml` needed? | +| --- | ---------------------- | ------------------------- | ------------------------------------------- | ------------------------- | ------------------------ | +| 1 | `ExecutionOtelPlugin` | ADOT Layer | `use_default_tracer_provider=True` | `/opt/otel-instrument` | No | +| 2 | `ExecutionOtelPlugin` | Community Collector Layer | auto-created (default) | Do NOT set | Yes | +| 3 | `InvocationOtelPlugin` | ADOT Layer | omit `trace_provider` (uses global) | `/opt/otel-instrument` | No | +| 4 | `InvocationOtelPlugin` | Community Collector Layer | pass your own OTLP `trace_provider` | Do NOT set | Yes | + +### 1. ExecutionOtelPlugin + ADOT Layer + +The ADOT layer provides both the collector and a global `TracerProvider`. The plugin uses the global provider and produces a Workflow span as the trace root. + +**Handler code:** + +```python +from aws_durable_execution_sdk_python_otel import ExecutionOtelPlugin + +plugin = ExecutionOtelPlugin(OtelPluginConfig(use_default_tracer_provider=True)) +``` + +**SAM template:** ```yaml MyFunction: Type: AWS::Serverless::Function Properties: + Runtime: python3.12 + Handler: index.handler + CodeUri: ./src Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:aws-otel-python-amd64-ver- + - !Sub arn:aws:lambda:${AWS::Region}::layer:aws-otel-python-amd64-ver- Environment: Variables: AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument + Tracing: Active + DurableConfig: + ExecutionTimeout: 3600 + RetentionPeriodInDays: 7 + Policies: + - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy + - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess + AutoPublishAlias: live ``` -**CDK:** +### 2. ExecutionOtelPlugin + Community Collector Layer -```python -from aws_cdk import aws_lambda as lambda_ +The plugin creates its own `TracerProvider` and exports spans to the collector on `localhost:4318`. Produces a Workflow span as the trace root. -adot_layer = lambda_.LayerVersion.from_layer_version_arn( - self, - "AdotLayer", - f"arn:aws:lambda:::layer:aws-otel-python-amd64-ver-", -) +**Handler code:** -fn = lambda_.Function( - self, - "MyFunction", - runtime=lambda_.Runtime.PYTHON_3_12, - handler="index.handler", - code=lambda_.Code.from_asset("lambda"), - layers=[adot_layer], - environment={"AWS_LAMBDA_EXEC_WRAPPER": "/opt/otel-instrument"}, -) +```python +from aws_durable_execution_sdk_python_otel import ExecutionOtelPlugin + +plugin = ExecutionOtelPlugin() ``` -> **Tip:** Pin the layer version to a specific number in production deployments to avoid unexpected behavior from automatic version changes. +**SAM template:** -### 2. AWS X-Ray Active Tracing +```yaml +MyFunction: + Type: AWS::Serverless::Function + Properties: + Runtime: python3.12 + Handler: index.handler + CodeUri: ./src + Layers: + - !Sub arn:aws:lambda:${AWS::Region}::layer:opentelemetry-collector-amd64- + Environment: + Variables: + OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml + Tracing: Active + DurableConfig: + ExecutionTimeout: 3600 + RetentionPeriodInDays: 7 + Policies: + - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy + - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess + AutoPublishAlias: live +``` -Enable active tracing on your Lambda function so the `_X_AMZN_TRACE_ID` environment variable is populated at invocation time. The plugin uses this header to derive deterministic trace IDs that remain consistent across all invocations of the same durable execution. +### 3. InvocationOtelPlugin + ADOT Layer -**AWS Console:** Lambda → Configuration → Monitoring and operations tools → Active tracing → Enable +The ADOT layer provides both the collector and a global `TracerProvider`. The plugin uses that global provider (omit `trace_provider`); the invocation span parents to the ADOT layer's ambient invocation span, and operations link to it. -**AWS CLI:** +**Handler code:** -```bash -aws lambda update-function-configuration \ - --function-name your-function-name \ - --tracing-config Mode=Active +```python +from aws_durable_execution_sdk_python_otel import InvocationOtelPlugin + +# No trace_provider passed -> uses the ADOT layer's globally registered provider. +plugin = InvocationOtelPlugin() ``` -**CloudFormation / SAM:** +Use the same SAM template as option 1 (ADOT layer + `AWS_LAMBDA_EXEC_WRAPPER`). -```yaml -MyFunction: - Type: AWS::Lambda::Function - Properties: - TracingConfig: - Mode: Active -``` +### 4. InvocationOtelPlugin + Community Collector Layer + +The plugin does not auto-create a provider, so construct an OTLP-exporting `TracerProvider` (targeting the collector on `localhost:4318`) and pass it in. Produces an invocation span as the trace root with operations parented beneath it. -**CDK:** +**Handler code:** ```python -lambda_.Function( - self, - "MyFunction", - tracing=lambda_.Tracing.ACTIVE, +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchSpanProcessor +from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter + +from aws_durable_execution_sdk_python_otel import InvocationOtelPlugin + +provider = TracerProvider() +provider.add_span_processor( + BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")) ) +plugin = InvocationOtelPlugin(trace_provider=provider) ``` -### 3. In your Lambda handler (index.py) +> If you want zero-config auto-creation of the OTLP provider, use `ExecutionOtelPlugin` (option 2) instead. + +Use the same SAM template as option 2 (community collector layer + `collector.yaml`). + +### Which Combination Should I Use? + +| Scenario | Recommendation | +| ------------------------------------------------------ | ----------------------------------------------------- | +| New deployment, want unified trace per execution | ExecutionOtelPlugin + Community Collector (option 2) | +| New deployment, want per-invocation traces | InvocationOtelPlugin + Community Collector (option 4) | +| Already have ADOT layer, want unified execution traces | ExecutionOtelPlugin + ADOT Layer (option 1) | +| Already have ADOT layer, want per-invocation traces | InvocationOtelPlugin + ADOT Layer (option 3) | +| Want smallest layer size | Community Collector (collector-only, no bundled SDK) | +| Want zero-config auto-instrumentation from ADOT | ADOT Layer with `use_default_tracer_provider=True` | + +--- + +## Configuration + +`ExecutionOtelPlugin` accepts an `OtelPluginConfig`: ```python -from aws_durable_execution_sdk_python import DurableContext -from aws_durable_execution_sdk_python.execution import durable_execution -from aws_durable_execution_sdk_python_otel import InvocationOtelPlugin +from dataclasses import dataclass +@dataclass +class OtelPluginConfig: + # Explicit provider to use as-is. Highest priority; skips auto-setup. + tracer_provider: SdkTracerProvider | None = None -@durable_execution(plugins=[InvocationOtelPlugin()]) -def handler(event: dict, context: DurableContext) -> dict: - result = context.step(lambda _: fetch_data(event["id"]), name="fetch-data") + # Use the globally registered TracerProvider (e.g. from ADOT). Defaults to False. + use_default_tracer_provider: bool | None = None - context.wait(duration=Duration.from_seconds(5)) + # Upstream trace-context extractor. Defaults to xray_context_extractor. + context_extractor: ContextExtractor | None = None - context.step(lambda _: process(result), name="process") + # Instrumentation scope name. Defaults to "aws-durable-execution-sdk-python". + instrument_name: str = DEFAULT_INSTRUMENT_NAME - return result -``` + # Whether to register HTTP (urllib3) instrumentation. Defaults to True. + enable_http_instrumentation: bool = True -That's it. The plugin handles TracerProvider setup, deterministic ID generation, and span lifecycle internally. + # OTLP exporter settings for the auto-configured provider (endpoint, headers). + exporter_config: ExporterConfig = ExporterConfig() -### 4. Grant Permissions + # Custom propagators. Replaces the default [AWS X-Ray, W3C TraceContext]. + propagators: Sequence[TextMapPropagator] | None = None -The function's execution role needs the `AWSXRayDaemonWriteAccess` managed policy (or equivalent permissions) if using X-Ray as the tracing backend. + # Custom Workflow span name (ExecutionOtelPlugin). Defaults to "Workflow". + workflow_span_name: str = DEFAULT_WORKFLOW_SPAN_NAME -### Environment Variables for ADOT layer + # Install the root-logger OTel context filter for log correlation. Defaults to True. + enrich_logger: bool = True -| Variable | Description | Default | -| ----------------------------- | --------------------------------------------------------------------------------------------- | ----------------- | -| `OTEL_EXPORTER_OTLP_ENDPOINT` | Endpoint for the OTLP exporter (e.g., `http://localhost:4318` for the ADOT collector sidecar) | Set by ADOT layer | -| `AWS_LAMBDA_EXEC_WRAPPER` | Set to `/opt/otel-instrument` for the ADOT layer to instrument your function | — | -| `OTEL_TRACES_SAMPLER` | Sampler to use (e.g., `traceidratio` for ratio-based sampling) | `always_on` | -| `OTEL_TRACES_SAMPLER_ARG` | Argument for the sampler (e.g., `0.3` to sample 30% of traces) | — | -See the [ADOT sampling configuration](https://aws-otel.github.io/docs/getting-started/lambda#sampling-configuration) for more details. +@dataclass +class ExporterConfig: + endpoint: str | None = None + headers: dict[str, str] | None = None +``` -## Configuration +**TracerProvider precedence:** explicit `tracer_provider` > `use_default_tracer_provider=True` > auto-created. -### Plugin Options +**Usage examples:** ```python from aws_durable_execution_sdk_python_otel import ( + ExecutionOtelPlugin, InvocationOtelPlugin, - xray_context_extractor, + OtelPluginConfig, + ExporterConfig, +) + +# Zero-config (auto-creates TracerProvider with OTLP export) +plugin = ExecutionOtelPlugin() + +# Use the ADOT layer's globally registered TracerProvider (ExecutionOtelPlugin) +plugin = ExecutionOtelPlugin(OtelPluginConfig(use_default_tracer_provider=True)) + +# Custom endpoint and headers (third-party vendor) +plugin = ExecutionOtelPlugin( + OtelPluginConfig( + exporter_config=ExporterConfig( + endpoint="https://api.honeycomb.io/v1/traces", + headers={"x-honeycomb-team": os.environ["HONEYCOMB_API_KEY"]}, + ) + ) ) -plugin = InvocationOtelPlugin( - # Provide your own TracerProvider if you already have one configured. - # Defaults to the globally configured tracer provider. - trace_provider=None, - # Use a custom context extractor (default: xray_context_extractor). - context_extractor=xray_context_extractor, - # Custom instrumentation scope name - # (default: "aws-durable-execution-sdk-python"). - instrument_name="my-service", - # Install a root-logger filter that stamps trace context onto every - # log record (default: True). +# Bring your own TracerProvider (either plugin) +from opentelemetry.sdk.trace import TracerProvider + +provider = TracerProvider() # your config +plugin = ExecutionOtelPlugin(OtelPluginConfig(tracer_provider=provider)) +# InvocationOtelPlugin takes the provider directly (no OtelPluginConfig): +plugin = InvocationOtelPlugin(trace_provider=provider) +``` + +`InvocationOtelPlugin` is constructed with plain keyword arguments rather than an `OtelPluginConfig`: + +```python +InvocationOtelPlugin( + trace_provider=None, # provider to use; falls back to the global provider + context_extractor=None, # defaults to xray_context_extractor + instrument_name="aws-durable-execution-sdk-python", enrich_logger=True, ) ``` -### Context Extractors +--- + +## Export Strategies + +When the plugin auto-creates its `TracerProvider` (default mode), you can configure where spans go: + +### Via a Collector Layer (Recommended) + +``` +Lambda → OTLP (localhost:4318) → Collector Extension → X-Ray/CloudWatch +``` + +No code changes needed — auto-created providers target `localhost:4318` by default. + +### Direct to CloudWatch OTLP Endpoint + +```bash +OTEL_EXPORTER_OTLP_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces +``` + +> **Note:** Direct export requires SigV4-signed requests. -The plugin supports multiple strategies for extracting upstream trace context: +### Via Third-Party OTLP Endpoint ```python -from aws_durable_execution_sdk_python_otel import ( - InvocationOtelPlugin, - w3c_client_context_extractor, - xray_context_extractor, +plugin = ExecutionOtelPlugin( + OtelPluginConfig( + exporter_config=ExporterConfig( + endpoint="https://api.honeycomb.io/v1/traces", + headers={"x-honeycomb-team": os.environ["HONEYCOMB_API_KEY"]}, + ) + ) ) +``` -# Default: X-Ray trace header (recommended for most Lambda deployments) -InvocationOtelPlugin(context_extractor=xray_context_extractor) +Or via environment variables: -# W3C Trace Context via clientContext (requires backend propagation support) -InvocationOtelPlugin(context_extractor=w3c_client_context_extractor) +```bash +OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io/v1/traces +OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=YOUR_API_KEY ``` -### Log Correlation +--- -When `enrich_logger=True` (the default), the plugin installs a logging filter on -the root logger at invocation start. The filter stamps the active OTel trace -context onto every emitted log record using these attributes: +## Collector Configuration -- `traceId`: 32-char hex trace identifier -- `spanId`: 16-char hex span identifier -- `otelTraceSampled`: boolean indicating if the trace is sampled +When using the community collector-only layer, include a `collector.yaml` in your function bundle: -These attributes are only set when a valid span context is active, so any log -formatter or schema must treat the fields as optional. +```yaml +receivers: + otlp: + protocols: + http: + endpoint: "localhost:4318" + +exporters: + awsxray: + region: "${AWS_REGION}" + +service: + pipelines: + traces: + receivers: [otlp] + exporters: [awsxray] +``` -## Verification +Set the environment variable: + +```bash +OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/collector.yaml +``` -After deploying your function with the plugin configured: +### Why Use a Collector? -1. **Invoke your durable function** — trigger at least one execution that includes multiple steps or a wait/resume cycle. +Using the community collector-only layer lets you export traces directly to third-party observability platforms (such as Datadog, Honeycomb, or Grafana) without first sending them to AWS and then re-exporting from CloudWatch or X-Ray. -2. **Check the CloudWatch console** — Navigate to CloudWatch → Traces in the AWS Console. You should see a trace with: - - An "invocation" span per invocation - - Child spans for each durable operation (named after your step names) - - All invocations of the same execution grouped under one trace ID +--- -3. **Check log correlation** — verify that your logs include `traceId` and `spanId` fields matching the spans in X-Ray. +## IAM Permissions -4. **Confirm sampling** — If you set `OTEL_TRACES_SAMPLER=traceidratio` and `OTEL_TRACES_SAMPLER_ARG` to a value less than 1.0, verify that only the expected proportion of traces appear. +### Via Collector Layer (ADOT or Community) -5. **Span links** — For operations that span multiple invocations (e.g., after a wait resumes), though span links are set, they are not visualized within the CloudWatch console. +The function's execution role needs X-Ray write permissions: -### Troubleshooting +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": ["xray:PutTraceSegments", "xray:PutTelemetryRecords"], + "Resource": "*" + } + ] +} +``` + +Or attach: `arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess` + +### Via Third-Party Endpoint + +No AWS IAM permissions required. Authentication is handled via headers in `OTEL_EXPORTER_OTLP_HEADERS` or `exporter_config.headers`. + +--- + +## Environment Variables + +| Variable | Description | Default | +| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP exporter endpoint URL | `http://localhost:4318/v1/traces` | +| `OTEL_EXPORTER_OTLP_HEADERS` | Comma-separated key=value headers for the exporter | — | +| `OTEL_DURABLE_SAMPLING_RATIO` | Trace-ID-based probabilistic sampling ratio (0.0 to 1.0). All invocations of the same execution are sampled/dropped consistently. | `1.0` (all traces sampled) | +| `AWS_LAMBDA_EXEC_WRAPPER` | Set to `/opt/otel-instrument` to activate the ADOT layer's auto-instrumentation | — | +| `OPENTELEMETRY_COLLECTOR_CONFIG_URI` | Path to `collector.yaml` for the community collector layer | — | +| `AWS_LAMBDA_FUNCTION_NAME` | Set by the Lambda runtime. Used to detect the Lambda environment and populate resource attributes. | — | +| `AWS_REGION` | Set by the Lambda runtime. Used for resource attributes and collector configuration. | — | +| `AWS_LAMBDA_FUNCTION_MEMORY_SIZE` | Set by the Lambda runtime. Populates the `faas.max_memory` span attribute (in MB). | — | + +--- + +## SAM/CloudFormation Templates + +See the [Deployment Matrix](#deployment-matrix) section for plugin-specific templates with both layer options. Below are additional templates for alternative export targets. + +### Direct to CloudWatch (No Layer) + +```yaml +MyFunction: + Type: AWS::Serverless::Function + Properties: + Runtime: python3.12 + Handler: index.handler + CodeUri: ./src + DurableConfig: + ExecutionTimeout: 3600 + RetentionPeriodInDays: 7 + Environment: + Variables: + OTEL_EXPORTER_OTLP_ENDPOINT: !Sub "https://xray.${AWS::Region}.amazonaws.com/v1/traces" + Policies: + - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy + - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess + AutoPublishAlias: live +``` + +### Third-Party OTLP Endpoint + +```yaml +MyFunction: + Type: AWS::Serverless::Function + Properties: + Runtime: python3.12 + Handler: index.handler + CodeUri: ./src + DurableConfig: + ExecutionTimeout: 3600 + RetentionPeriodInDays: 7 + Layers: + # Optional: collector layer for reliability (retry/buffering) + - !Sub arn:aws:lambda:${AWS::Region}::layer:opentelemetry-collector-amd64- + Environment: + Variables: + OTEL_EXPORTER_OTLP_ENDPOINT: "https://api.honeycomb.io/v1/traces" + OTEL_EXPORTER_OTLP_HEADERS: "x-honeycomb-team=YOUR_API_KEY" + Policies: + - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy + AutoPublishAlias: live +``` + +--- + +## Trace Structure Comparison + +The Python SDK exposes these durable operations, each of which produces an Operation span: `step` (STEP), `wait` (WAIT), `wait_for_condition` (WAIT_FOR_CONDITION), `wait_for_callback` (CALLBACK), `invoke` (chained invoke), `map`, and `parallel`. `map` and `parallel` run their branches in child contexts, which appear as CONTEXT operation spans. + +### ExecutionOtelPlugin + +Produces a hierarchical trace with the Workflow span as the synthetic root: + +``` +Workflow span (deterministic ID from execution ARN, exported on terminal status only) +├── Invocation span (one per Lambda invocation, always exported) +├── Operation span: "fetch-data" (STEP) +│ ├── Attempt span: "fetch-data attempt 1" +│ │ └── HTTP span: GET https://api.example.com/data +│ └── [link → Invocation span] +├── Operation span: "cooldown" (WAIT) +│ └── [link → Invocation span] +└── Operation span: "process" (STEP) + └── [link → Invocation span] +``` + +> When `use_default_tracer_provider=True`, the plugin's Invocation span is parented to the ambient invocation span from the ADOT layer's context. + +### InvocationOtelPlugin -| Symptom | Likely Cause | -| --------------------------------- | --------------------------------------------------------------- | -| No traces appear | ADOT layer not configured, or `AWS_LAMBDA_EXEC_WRAPPER` not set | -| Traces appear but are fragmented | X-Ray active tracing not enabled on the Lambda function | -| Missing spans for some operations | `OTEL_TRACES_SAMPLER_ARG` set below 1.0 | -| `_X_AMZN_TRACE_ID` not populated | X-Ray active tracing not enabled | +Produces a per-invocation trace with the invocation span as root: + +``` +Invocation span (one per Lambda invocation) +├── Operation span: "fetch-data" (STEP) +│ ├── Attempt span: "fetch-data attempt 1" +│ │ └── HTTP span: GET https://api.example.com/data +│ └── [link → deterministic operation span ID] +├── Operation span: "cooldown" (WAIT) +└── Operation span: "process" (STEP) +``` + +Cross-invocation operations are correlated via span links to deterministic span IDs. + +### Span Attributes + +- **Workflow span** (ExecutionOtelPlugin): `durable.execution.arn`, `durable.execution.status` +- **Invocation span**: `durable.execution.arn`, `durable.invocation.first`, `durable.invocation.status`, and (when the plugin owns the provider) `faas.invocation_id`, `faas.coldstart`, `cloud.provider`, `cloud.platform` +- **Operation span**: `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.operation.subtype`, `durable.operation.status`, and `durable.attempt.number` (STEP / WAIT_FOR_CONDITION, on completion) +- **Attempt span**: all operation attributes plus `durable.attempt.number` and `durable.attempt.outcome` + +**Span status mapping:** + +- Invocation span: `SUCCEEDED`/`PENDING` → `OK`, `RETRY`/`FAILED` → `ERROR` +- Workflow span: `SUCCEEDED` → `OK`, `FAILED` → `ERROR` (non-terminal statuses are never exported, so they stay `UNSET`) + +--- + +## Log Correlation + +When `enrich_logger=True` (the default), the plugin installs an `OtelContextLogFilter` on the root logger's handlers. The filter stamps the active OTel trace context onto every log record: + +- `traceId` — 32-char hex trace ID +- `spanId` — 16-char hex span ID +- `otelTraceSampled` — whether the trace is sampled + +These fields are only added when a valid span is active, so any log formatter must treat them as optional. You can also install the filter manually on a specific logger: + +```python +from aws_durable_execution_sdk_python_otel import install_log_filter + +install_log_filter(plugin, target_logger=my_logger) +``` + +--- + +## Additional Python Dependencies + +The core package depends on `opentelemetry-api`, `opentelemetry-sdk`, `opentelemetry-exporter-otlp`, and `opentelemetry-propagator-aws-xray`. + +When the plugin auto-creates its `TracerProvider` (default mode) and you want AWS SDK / HTTP auto-instrumentation, install the optional `instrumentation` extra, which adds: + +```bash +pip install "aws-durable-execution-sdk-python-otel[instrumentation]" +# adds: opentelemetry-instrumentation-botocore, opentelemetry-instrumentation-urllib3 +``` + +The instrumentation module degrades gracefully: if these packages are not installed, AWS SDK / HTTP calls simply are not auto-traced. When using `use_default_tracer_provider=True` (ADOT layer mode), the ADOT layer provides its own instrumentation. + +--- ## API Reference +### `ExecutionOtelPlugin` + +Plugin that produces a Workflow span as the synthetic trace root. Implements `DurableInstrumentationPlugin`. + +```python +ExecutionOtelPlugin(config: OtelPluginConfig | None = None) +``` + ### `InvocationOtelPlugin` -The main plugin class. Implements `DurableInstrumentationPlugin` from `aws_durable_execution_sdk_python`. +Plugin that produces an invocation span as the trace root. Implements `DurableInstrumentationPlugin`. ```python -InvocationOtelPlugin( - trace_provider=None, - context_extractor=None, - instrument_name="aws-durable-execution-sdk-python", - enrich_logger=True, -) +InvocationOtelPlugin(trace_provider=None, context_extractor=None, enrich_logger=True, ...) ``` +### `OtelPluginConfig` / `ExporterConfig` + +OtelPluginConfig is used by ExecutionOtelPlugin (see [Configuration](#configuration)). + ### `DeterministicIdGenerator` -A custom OpenTelemetry `IdGenerator` that produces reproducible trace and span IDs from execution metadata. Exported for advanced use cases. +Custom OpenTelemetry `IdGenerator` that produces reproducible trace and span IDs from execution metadata. + +### `derive_workflow_span_id(execution_arn: str) -> int` + +Derives a deterministic 64-bit span ID from an execution ARN. + +### `operation_id_to_span_id(execution_arn: str, operation_id: str) -> int` + +Derives a deterministic span ID for a durable operation. ### `xray_context_extractor` -Default context extractor. Reads the `_X_AMZN_TRACE_ID` environment variable to derive trace context. +Default context extractor. Reads the X-Ray trace header to derive trace context. ### `w3c_client_context_extractor` -Alternative context extractor. Reads W3C `traceparent` from `context.clientContext.custom.traceparent`. Requires backend `clientContext` propagation to be enabled. +Alternative context extractor. Reads `traceparent` from the invocation's client context. ### `ContextExtractor` -Type alias for custom context extractor functions. +Type alias for custom context-extractor callables. ### `OtelContextLogFilter` / `install_log_filter` -The logging filter (and its installer) used to stamp trace context onto log -records. Installed automatically when `enrich_logger=True`; exported for manual -setups. +Logging filter and installer for trace/span log correlation (see [Log Correlation](#log-correlation)). + +### `create_tracer_provider` / `ProviderResult` + +Lower-level factory used internally to resolve/auto-create the `TracerProvider`. + +--- -## Requirements +## Verification + +> **Important:** When using the community collector layer, you must enable **CloudWatch Transaction Search** in your AWS account for traces to be visible in X-Ray. Navigate to CloudWatch → Settings → Traces and Logs and turn on Transaction Search. + +After deploying with either plugin and either layer: + +1. **Invoke your durable function** — trigger an execution with multiple steps or a wait/resume cycle. +2. **Check the CloudWatch console** — Navigate to CloudWatch → Traces. You should see spans grouped under one trace ID. +3. **Check log correlation** — With `enrich_logger=True`, verify logs include `traceId` and `spanId`. +4. **Confirm sampling** — Set `OTEL_DURABLE_SAMPLING_RATIO` below 1.0 and verify only the expected proportion of traces appear. + +### Troubleshooting -- Python >= 3.11 -- `aws-durable-execution-sdk-python` >= 1.5.0 -- `opentelemetry-api` >= 1.20.0, <= 1.42.1 -- `opentelemetry-sdk` >= 1.20.0, <= 1.42.1 -- `opentelemetry-exporter-otlp` <= 1.42.1 +| Symptom | Likely Cause | +| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| No traces appear | Collector layer not attached, or config env var not set | +| No traces with ADOT layer | `AWS_LAMBDA_EXEC_WRAPPER` not set (when using `use_default_tracer_provider=True`) | +| Traces fragmented across IDs | X-Ray active tracing not enabled on the function | +| Missing operation spans | Sampling ratio set below 1.0 | +| AWS SDK / HTTP spans missing | The `instrumentation` extra is not installed | +| Collector layer errors | Check `collector.yaml` is in the function bundle at the path specified | +| Duplicate spans with ADOT layer | `AWS_LAMBDA_EXEC_WRAPPER` is set but `use_default_tracer_provider` is `False` — either remove the env var or set `use_default_tracer_provider=True` | ## License