Add optional embedded tracing components

.github/pr-test-selection.json

@@ -11,12 +11,17 @@
     "tests/Unit/V2/CancellationContextTest.php",
     "tests/Unit/V2/CompiledWorkflowDefinitionTest.php",
     "tests/Unit/V2/EmbeddedHttpSnapshotTest.php",
+    "tests/Unit/V2/ExecutionTracingTest.php",
     "tests/Unit/V2/ExternalPayloadStorageTest.php",
     "tests/Unit/V2/PlatformProtocolSpecsTest.php",
     "tests/Unit/V2/ReadmeBranchDestinationsTest.php",
     "tests/Unit/V2/ScopedCancellationContextTest.php",
     "tests/Unit/V2/SqliteTransactionTest.php",
     "tests/Unit/V2/StartOptionsTest.php",
+    "tests/Unit/V2/TaskTraceTimingsTest.php",
+    "tests/Unit/V2/TaskTraceCarrierTest.php",
+    "tests/Unit/V2/TraceCarrierTest.php",
+    "tests/Unit/V2/TracingQueueErrorDiagnosticsTest.php",
     "tests/Unit/V2/VersionMarkerWireFormatTest.php",
     "tests/Unit/V2/WorkflowFiberRunnerTest.php",
     "tests/Unit/V2/WorkflowHistoryPayloadCastTest.php",

.github/tracing-http/bootstrap.php

@@ -0,0 +1,6 @@
+<?php
+
+declare(strict_types=1);
+
+require dirname(__DIR__, 2) . '/vendor/autoload.php';
+require __DIR__ . '/vendor/autoload.php';

.github/tracing-http/composer.json

@@ -0,0 +1,14 @@
+{
+    "require": {
+        "open-telemetry/api": "^1.10.1",
+        "open-telemetry/sdk": "^1.15",
+        "open-telemetry/exporter-otlp": "^1.4",
+        "open-telemetry/context": "^1.5",
+        "open-telemetry/opentelemetry-auto-guzzle": "^1.5",
+        "guzzlehttp/guzzle": "^8.2"
+    },
+    "autoload": {
+        "psr-4": {"Workflow\\": "../../src/"}
+    },
+    "config": {"allow-plugins": false}
+}

.github/tracing-performance/.gitignore

@@ -0,0 +1,4 @@
+/vendor/
+/state/
+/.env
+!/composer.lock

.github/tracing-performance/Dockerfile

@@ -0,0 +1,9 @@
+FROM php:8.3.35-cli-bookworm
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        git libonig-dev libxml2-dev unzip \
+    && docker-php-ext-install pdo_mysql mbstring dom pcntl \
+    && pecl install redis-6.1.0 && docker-php-ext-enable redis \
+    && rm -rf /var/lib/apt/lists/*
+COPY --from=composer:2.9 /usr/bin/composer /usr/bin/composer
+USER 1000:1000
+WORKDIR /app

.github/tracing-performance/RESULTS-2026-10-17.md

@@ -0,0 +1,179 @@
+# Embedded tracing measurement, October 17, 2026
+
+All 20 original cells completed and passed their business, resource and export
+audits. **None of the four CPU-budget criteria qualifies.** This is an unreleased
+embedded candidate measurement, with no stable, service-mode or capacity claim.
+
+## Method and software
+
+The [method was recorded before results](https://github.com/durable-workflow/workflow/issues/773#issuecomment-6228453784).
+The [functional and offered-load prerequisites](https://github.com/durable-workflow/workflow/issues/773#issuecomment-6228327954)
+are separate observations. This batch ran three rotating matched repetitions
+of published Workflow 2.5.28, candidate tracing disabled, and candidate tracing
+enabled. Two final published-baseline cells checked drift. Every cell used fresh
+application/database state and the same four workers, logging call, dependency
+references, resource limits and local accepting collector.
+
+Measured candidate: `316316a827fadc7b5aa545842451ca9c8e7a9db5`, tree
+`c5ac80029a831e2664200e8b58aa301e657053dd`. Published source matched all 606
+files at tag `592dee`; other dependency references matched the candidate.
+This compares the complete candidate with the published package. It does not
+isolate every candidate change as a causal tracing cost.
+
+The dedicated amd64 host had eight logical CPUs, about 16 GiB RAM, AMD EPYC-Rome
+as observed, Microsoft hypervisor, Linux 6.8.0-146 and cgroup v2. Docker was
+29.1.3, Compose 2.40.3, PHP 8.3.35 with pcntl, Redis extension 6.1.0 and CLI
+OPcache disabled. Builds and dependency installation finished before measurement.
+Administrative SSH/status reads and one bounded first-cell reader-validation
+copy occurred during the batch. No concurrent build or unrelated application
+workload was admitted. This is not a matched hardware performance claim against
+another product or an estimate of maximum capacity.
+
+| Role | CPU cap | Memory cap |
+| --- | ---: | ---: |
+| HTTP application | 0.5 | 512 MiB |
+| Each of four workers | 1 | 384 MiB |
+| MySQL | 1 | 2 GiB |
+| Redis | 0.1 | 128 MiB |
+| Collector | 0.2 | 256 MiB |
+| Probe | 0.2 | 256 MiB |
+
+Swap was disabled. The primary quantity is the sum of exact CPU counter deltas
+from all nine containers, divided by measured completed workflows. It includes
+application, all workers, database, cache, collector and probe through child
+completion/provider drain. Setup, warmup, audit queries, host sampling tools and
+teardown are outside those deltas. Every container identity, cap and counter
+interval was checked before/after. No OOM, restart or nonzero child exit occurred.
+
+Ordinary cells offered one one-activity workflow per second for 120 seconds,
+120 measured completions. Fan-out cells offered one 64-activity workflow per
+minute for 120 seconds, two measured completions. Separate warmups completed
+30 ordinary workflows or one fan-out workflow before measurement. Original
+90-second observer, 100-second run and 120-second execution bounds were retained.
+Providers were installed in every cell. The application registered the log tap
+only in the enabled profile, as described in the [recipe](README.md).
+
+## CPU acceptance
+
+Each estimate is the mean of three matched candidate/baseline ratios, with a
+two-sided 95% paired-t interval, n=3, df=2. The frozen acceptance requires both
+the upper interval endpoint and absolute final baseline drift to fit the target.
+Negative point estimates do not demonstrate a speed improvement.
+
+| Workload | Profile | Mean CPU change | 95% interval | Baseline drift | Target | Qualified |
+| --- | --- | ---: | --- | ---: | ---: | --- |
+| Ordinary | Disabled | -1.28% | [-6.15%, +3.60%] | +0.30% | 1% | No |
+| Ordinary | Enabled | +6.26% | [+3.16%, +9.36%] | +0.30% | 5% | No |
+| 64-member fan-out | Disabled | -1.28% | [-8.27%, +5.70%] | +4.10% | 1% | No |
+| 64-member fan-out | Enabled | +0.43% | [-12.20%, +13.07%] | +4.10% | 5% | No |
+
+All CPU observations below are seconds per completed measured workflow. The
+three values are original repetitions, with no discarded cell or extra sample.
+
+| Workload | Profile | Repetition 1 | Repetition 2 | Repetition 3 | Final baseline drift cell |
+| --- | --- | ---: | ---: | ---: | ---: |
+| Ordinary | Published | 0.605717 | 0.596319 | 0.636958 | 0.614822 |
+| Ordinary | Disabled | 0.595531 | 0.601429 | 0.617848 | |
+| Ordinary | Enabled | 0.647830 | 0.638079 | 0.667641 | |
+| Fan-out | Published | 84.775585 | 90.481546 | 92.066782 | 92.759297 |
+| Fan-out | Disabled | 85.815143 | 89.805320 | 88.088379 | |
+| Fan-out | Enabled | 90.109925 | 88.509247 | 89.471048 | |
+
+### Component CPU
+
+Arithmetic means across the three repetitions, seconds per completed workflow.
+Worker means sum all four worker containers.
+
+| Workload/profile | HTTP | Workers | MySQL | Redis | Collector | Probe |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| Ordinary/published | 0.101366 | 0.244040 | 0.220823 | 0.023822 | 0.000072 | 0.022874 |
+| Ordinary/disabled | 0.099768 | 0.240792 | 0.218220 | 0.023648 | 0.000074 | 0.022434 |
+| Ordinary/enabled | 0.109262 | 0.256221 | 0.220766 | 0.023732 | 0.018796 | 0.022405 |
+| Fan-out/published | 2.725616 | 66.573568 | 17.606948 | 1.236480 | 0.004065 | 0.961294 |
+| Fan-out/disabled | 2.695288 | 65.571619 | 17.429256 | 1.236462 | 0.004066 | 0.966255 |
+| Fan-out/enabled | 2.719067 | 66.332326 | 17.513380 | 1.235851 | 0.586929 | 0.975852 |
+
+Ordinary enabled CPU is higher in HTTP, workers and the collector. The collector
+accounts for about half the difference between the raw enabled and published
+means. That identifies a useful investigation, without establishing which
+source change or exporter setting causes the full observed difference.
+
+## Latency, memory and correctness
+
+All declared offers were accepted and completed, with no missed offer or HTTP
+error. Per-cell observed completion rates were about 0.9960–1.00001 workflows/s
+for ordinary work and 0.0166665–0.0166668 for fan-out. These are offered-load
+results. Fan-out has only two measured observations per cell, so its high
+percentiles describe those few observations, not a reliable tail distribution.
+
+Each latency entry is the arithmetic mean of three per-cell percentiles, in
+milliseconds. These are not pooled percentiles.
+
+| Workload/profile | Mean p50 | Mean p95 | Mean p99 |
+| --- | ---: | ---: | ---: |
+| Ordinary/published | 887.50 | 1524.28 | 1688.58 |
+| Ordinary/disabled | 891.34 | 1434.45 | 1698.46 |
+| Ordinary/enabled | 915.68 | 1457.40 | 1629.62 |
+| Fan-out/published | 54319.66 | 55497.18 | 55497.18 |
+| Fan-out/disabled | 53638.97 | 55474.13 | 55474.13 |
+| Fan-out/enabled | 53690.74 | 55916.89 | 55916.89 |
+
+Memory below is sampled cgroup memory in MiB: repetition-mean / largest sampled
+sum of individual container peaks. Worker values sum four containers. Peaks
+are not simultaneous aggregate peaks or PHP RSS. Requested sampling was one
+second; observed intervals over all 20 cells were 1.288–1.461 seconds. Samples
+can miss short peaks. Container lifetime peaks also include startup/warmup and
+are excluded from this table.
+
+| Workload/profile | HTTP | Workers | MySQL | Redis | Collector | Probe |
+| --- | --- | --- | --- | --- | --- | --- |
+| Ordinary/published | 34.20 / 34.50 | 194.53 / 206.57 | 565.99 / 579.05 | 3.46 / 5.05 | 8.32 / 8.32 | 21.53 / 30.57 |
+| Ordinary/disabled | 34.32 / 34.63 | 195.36 / 207.56 | 566.83 / 579.21 | 3.56 / 5.94 | 8.32 / 8.32 | 21.52 / 26.44 |
+| Ordinary/enabled | 36.90 / 37.69 | 207.73 / 220.63 | 565.98 / 580.74 | 3.53 / 5.14 | 13.76 / 14.02 | 21.09 / 30.49 |
+| Fan-out/published | 34.15 / 34.66 | 221.51 / 233.35 | 577.20 / 591.85 | 3.49 / 5.21 | 8.33 / 8.33 | 19.49 / 30.63 |
+| Fan-out/disabled | 34.27 / 34.54 | 222.89 / 234.96 | 578.72 / 592.33 | 3.50 / 6.04 | 8.33 / 8.34 | 19.68 / 29.73 |
+| Fan-out/enabled | 36.84 / 37.38 | 235.36 / 247.54 | 579.47 / 592.66 | 3.48 / 5.03 | 13.79 / 13.80 | 19.28 / 24.55 |
+
+Including warmup and drift cells, the audit counts 1,530 runs, 3,420 activities
+and callbacks, 6,480 first-attempt tasks and 14,850 business events. Typed results,
+history counts and identities match. No unfinished run/task/attempt/activity,
+non-first task/attempt, residual queue or failed job remains. The summed task
+`repair_count` is **375**, all in fan-out cells; do not describe this batch as
+zero repair/redispatch work. Fan-out repetition counters are published
+32/45/44, disabled 35/36/34 and enabled 41/30/38, plus 40 in the final baseline.
+Every ordinary cell has zero. This work remains included in measured CPU.
+
+All enabled spans ended and were received: 4,456 / 4,456, including warmup.
+Ordinary enabled cells each have 1,050 spans, 21,750 bytes of run metadata and
+143,400 bytes of task metadata. Fan-out enabled cells have 440/429/437 spans,
+435 bytes of run metadata and 102,813 bytes of task metadata each. These are
+stored observability-field byte counts, not total database size or network
+traffic. Activity-execution metadata is zero. Disabled/published cells produce
+no spans or stored carrier metadata. No durable business payload/history field
+was replaced by telemetry.
+
+## Reader correction and remaining limits
+
+The original analyzer rejected three actual callback elapsed values:
+119.999219339, 119.999356686 and 119.999713308 seconds. Their original drivers
+completed with all declared offers and results. Node's integer timer precision
+does not guarantee an exact callback time. The corrected reader retains those
+actual values, verifies the observation covers the final declared offer, and
+requires **every cgroup counter interval to cover the full 120-second window**.
+It neither rounds elapsed data nor rescales CPU. The new controls failed on the
+prior reader and pass in the corrected 24-test analysis/probe suite. Original
+samples, workloads, CPU targets and statistical method are unchanged. No cell
+was rerun or relabeled as another original.
+
+An earlier functional baseline completed business work but failed probe teardown;
+it remains unsuccessful. A separate corrected lifecycle control passed before
+this batch. This batch's original drivers all completed successfully. The
+experiment host, key and raw data are disposable; the code and this readable
+result are the retained outcome.
+
+The encoded-baggage unit failure remains selected against the released OTel
+dependency. Full tracing, migration/catalog integration and consumer acceptance
+remain unfinished. These measurements do not authorize a release. Next source
+work should investigate the enabled ordinary execution/export cost and improve
+measurement sensitivity before considering a new, separately declared study.
+No adaptive extra repetitions or relaxed targets are part of this result.

.github/tracing-performance/RESULTS-2026-10-20.md

@@ -0,0 +1,113 @@
+# Embedded tracing measurement, October 20, 2026
+
+All 20 original cells passed their workload, resource, exit and official
+Collector drain gates. **None of the four CPU budgets qualifies.** These results
+do not qualify a release or maximum service capacity. The failed October 17
+study remains unchanged in [its original report](RESULTS-2026-10-17.md).
+
+## Fixed method
+
+The [method was declared before admission](https://github.com/durable-workflow/workflow/issues/773#issuecomment-6294532491).
+Candidate `b2d394c5c3e16484c798f3e7a7293bfb001b07eb`, tree
+`37651d791f918a80fab9d19cc31d408a9504599e`, was compared with normally installed
+published Core **2.7.3**, `dda951d93f58b82447348f1b3b4e4e17dba149fb`.
+All other Composer package references matched. No later diagnostic correction
+was inserted into this source or measurement.
+
+The isolated amd64 host had eight logical AMD EPYC-Genoa CPUs, approximately
+16 GiB RAM, Microsoft virtualization, Linux 6.8.0-146, cgroup v2 and no swap.
+Docker was 29.1.3, Compose 2.40.3, PHP 8.3.35 with CLI OPcache disabled,
+Laravel 13.37, OpenTelemetry API 1.10.1, SDK 1.15.0 and OTLP exporter 1.4.0.
+Builds and installation completed before admission. No unrelated application
+workload or build was run during the study. Bounded administrative SSH reads
+continued. The host and all experiment state were retired after results.
+
+The accepting official Collector was
+`otel/opentelemetry-collector:0.163.1@sha256:f0e23036dce3db3f9cb80d907c75a0d034cd32e93511a1943482d46129b9ae9c`.
+The application used the existing four-span queue, two-span batch and 100 ms
+transport timeout; Collector counters independently matched ended spans after
+complete provider drain. Export failure observations did not change, and
+refused/receiver-failed counters remained zero. No span payloads were retained.
+
+Each cell used a fresh database/application, four workers and identical caps:
+HTTP application 0.5 CPU / 512 MiB; each worker 1 CPU / 384 MiB; MySQL 1 CPU /
+2 GiB; Redis 0.1 CPU / 128 MiB; Collector and probe each 0.2 CPU / 256 MiB.
+The primary quantity sums exact cgroup CPU deltas for these nine containers
+through business completion and export drain. Setup, warmup, audit queries,
+sampling tools and teardown are outside those deltas. The one-second requested
+sampling interval was approximately 1.26 seconds in practice; exact CPU counters
+do not depend on that interval. Memory observations are sampled, not allocation
+or simultaneous-peak measurements.
+
+Ordinary cells offered one one-activity workflow per second for 120 seconds
+(120 measured completions), following 30 warmups. Fan-out cells offered one
+64-activity workflow per minute for 120 seconds (two measured completions),
+following one warmup. Original observer/run/execution bounds remained
+90/100/120 seconds. Repetitions rotated base/off/on, on/base/off, off/on/base,
+then repeated the baseline for drift, separately for each workload. No extra
+repetition was selected after results.
+
+## CPU acceptance
+
+Estimates use the mean of three paired candidate/baseline ratios, two-sided
+95% paired-t intervals, n=3, df=2. Both the upper endpoint and absolute final
+baseline drift must fit the declared budget. Negative estimates do not prove
+an improvement. The budgets remain unchanged for this study.
+
+| Workload | Profile | Mean | 95% interval | Baseline drift | Budget | Qualified |
+| --- | --- | ---: | --- | ---: | ---: | --- |
+| Ordinary | Disabled | +1.11% | [-7.40%, +9.62%] | +1.62% | 1% | No |
+| Ordinary | Enabled | +9.46% | [+1.35%, +17.56%] | +1.62% | 5% | No |
+| 64-member fan-out | Disabled | -0.59% | [-13.91%, +12.72%] | +2.04% | 1% | No |
+| 64-member fan-out | Enabled | +3.18% | [-7.85%, +14.21%] | +2.04% | 5% | No |
+
+Every original CPU observation below is seconds per completed measured workflow.
+
+| Workload | Repetition | Published baseline | Disabled | Enabled |
+| --- | --- | ---: | ---: | ---: |
+| Ordinary | 1 | 0.440138 | 0.450680 | 0.487149 |
+| Ordinary | 2 | 0.433282 | 0.421278 | 0.484975 |
+| Ordinary | 3 | 0.439790 | 0.456113 | 0.465107 |
+| Ordinary | Drift | 0.444807 | — | — |
+| Fan-out | 1 | 48.130522 | 47.228357 | 50.872936 |
+| Fan-out | 2 | 47.496107 | 45.026346 | 46.572337 |
+| Fan-out | 3 | 44.848782 | 47.221256 | 47.447672 |
+| Fan-out | Drift | 47.780681 | — | — |
+
+## Other observations
+
+All offered workflows were accepted and completed. The independent reader
+validated exact callback/history counts and results, no missed offers, extra
+attempts, unfinished work, failed jobs or residual queue entries. No container
+OOM/restart or nonzero worker child exit occurred. MySQL, Redis and the official
+Collector exited zero; HTTP and retaining shells used their declared 143 stop.
+Repairs remained observable and were not substituted for extra attempts.
+
+| Workload/profile | Observed p95 range | Observed p99 range | Sum of sampled container peaks |
+| --- | --- | --- | --- |
+| Ordinary/base | 1.131–1.371 s | 1.386–1.655 s | 919,670,784–924,098,560 bytes |
+| Ordinary/off | 1.134–1.379 s | 1.374–1.401 s | 922,849,280–924,983,296 bytes |
+| Ordinary/on | 1.388–1.393 s | 1.399–1.458 s | 944,701,440–949,186,560 bytes |
+| Fan-out/base | 28.756–29.613 s | 28.756–29.613 s | 964,190,208–969,908,224 bytes |
+| Fan-out/off | 28.866–29.791 s | 28.866–29.791 s | 963,702,784–971,960,320 bytes |
+| Fan-out/on | 28.771–31.492 s | 28.771–31.492 s | 982,581,248–988,864,512 bytes |
+
+Fan-out tails have only two measured observations per cell. These ranges are
+descriptive, without a throughput or latency regression claim. Maximum offer
+lag across cells was 74.66 ms, below the original 100 ms admission gate.
+
+Disabled and baseline cells persisted zero tracing bytes and ended/exported
+zero spans. Enabled ordinary cells, including warmup, each persisted 21,750 run
+bytes and 143,400 task bytes and ended/accepted/exported exactly 1,050 spans.
+Enabled fan-out cells persisted 435 run bytes and 102,813–102,814 task bytes;
+their ended/accepted/exported counts were respectively 413, 403 and 401.
+No activity-execution carrier bytes were stored. Different task processing and
+repair observations account for differing physical span counts; the reader
+verified business cardinality separately.
+
+The prerequisites also passed: functional baseline/off/on (two runs, 65
+activities each) and the published-baseline offered-load check (120 ordinary
+plus two fan-out workflows). These are workload checks, not maximum-capacity
+estimates. The next qualification must name its exact changed source and
+published baseline, declare its method and budgets before admission, and retain
+these original failures. Changing a target cannot turn this study into a pass.