diff --git a/.github/tracing-performance/README.md b/.github/tracing-performance/README.md index 4d016b2e..25fca9cf 100644 --- a/.github/tracing-performance/README.md +++ b/.github/tracing-performance/README.md @@ -1,14 +1,19 @@ # Embedded tracing overhead workload -This is the qualification recipe for draft Workflow #788. The +This is the qualification recipe for Workflow #788. The [October 17 measurement](RESULTS-2026-10-17.md) and [October 20 official-Collector measurement](RESULTS-2026-10-20.md) completed; -neither qualified any of its four CPU budgets. The current recipe uses the stable +neither qualified any of its four CPU budgets. The complete +[October 21 study](RESULTS-2026-10-21.md) qualified three of its four original +5%/15% budgets. Disabled fan-out remains failed against 5% (95% upper +6.43%). +The report separately records the maintainer's post-study acceptance of a 7.5% +disabled release allowance. The original analyzer and results are unchanged. +The current recipe uses the stable OpenTelemetry API baggage fix. The remaining acceptance requirements of [#773](https://github.com/durable-workflow/workflow/issues/773) are separate release gates. -## Prospective corrected-source method +## Recorded corrected-source method The [prospective decision](https://github.com/durable-workflow/workflow/issues/773#issuecomment-6296360973) defines `embedded-300s-5pairs-v2`: five fixed matched repetitions, 300-second @@ -98,9 +103,15 @@ a separate Docker-hostname label. The analyzer requires that container label to match the measured CPU role's container-ID prefix. A same-container process restart still changes the process identifier and refuses the old counter proof. -The prospective package baseline is published Workflow 2.7.3 with OpenTelemetry -API 1.10.1. Both baseline -and candidate include the released bounded history-payload read optimization. +The original official-Collector preparation pinned published Workflow 2.7.3 +with OpenTelemetry API 1.10.1. The corrected 32-cell study declared in +[the owning record](https://github.com/durable-workflow/workflow/issues/773#issuecomment-6301992685) +uses normal published **Workflow 2.8.0 / +`2edaa6b1c6f45fcec4570813c2e741aadde92b3b`** with a separately recorded normal +Composer lock and unchanged non-Workflow dependencies. Both installations +include the released bounded history-payload read optimization. Preserve each +study's actual lock and source identity; the older committed preparation lock +is not a substitute for the admitted 2.8.0 baseline. This source/baseline update precedes admission of any official-Collector measurement. The earlier prospective 2.6.1, 2.6.2 and 2.7.0 preparations and the October 17 study's 2.5.28 lock remain available at their recorded sources. Their results diff --git a/.github/tracing-performance/RESULTS-2026-10-21.md b/.github/tracing-performance/RESULTS-2026-10-21.md new file mode 100644 index 00000000..c607894f --- /dev/null +++ b/.github/tracing-performance/RESULTS-2026-10-21.md @@ -0,0 +1,165 @@ +# Embedded tracing measurement, October 21, 2026 + +All 32 original cells passed their workload, resource, exit and official Collector drain gates. +**Three of four predeclared CPU budgets qualify. The original 5% disabled fan-out target +remains unproven.** This is a bounded embedded workload measurement, without a maximum-capacity +or cross-SDK performance claim. The failed [October 17](RESULTS-2026-10-17.md) and [October +20](RESULTS-2026-10-20.md) studies remain unchanged. + +## Fixed method + +The [method was declared before +admission](https://github.com/durable-workflow/workflow/issues/773#issuecomment-6301992685). +Candidate `ffe793c46fad8febf90766f5f5f30217e4627a6e`, complete tree +`51255c7651841fa503b79d27e8d314993c0bc6be`, was compared with normally installed published Core +**2.8.0**, `2edaa6b1c6f45fcec4570813c2e741aadde92b3b`. The implementation merge +`9776f4a4d1ad14fd0ebdd78a706af66b00c44ffd` has that identical complete tree. All other Composer +package objects match, and installed production source equality was checked before admission. +Later release documentation changes no measured production file. + +The fixed batch ran from 2026-10-20T21:52:06Z to 2026-10-21T01:17:20Z. It used a separate amd64 +host with eight logical AMD EPYC-Rome CPUs (four cores, two threads per core), approximately 16 +GiB RAM, Microsoft virtualization, Linux 6.8.0-146, cgroup v2 and no swap. Docker was 29.1.3 +and Compose 2.40.3. PHP was 8.3.35 with CLI OPcache disabled, Laravel v13.37.0, OpenTelemetry +API 1.10.1, SDK 1.15.0 and OTLP exporter 1.4.0. Builds and installation completed before +admission. No unrelated application build or workload was admitted on the guest. Bounded +administrative SSH reads continued. The VM and experiment state were retired after results. + +The official Collector was +`otel/opentelemetry-collector:0.163.1@sha256:f0e23036dce3db3f9cb80d907c75a0d034cd32e93511a1943482d46129b9ae9c`. +The application retained its four-span queue, two-span batch and 100 ms transport timeout. +Ended spans equaled accepted and exported Collector deltas after complete provider drain. +Failure observations did not change; refused and receiver-failed counters stayed zero. No span +payloads were retained. + +Each cell used a fresh database/application and identical nine-role limits: HTTP application +0.5 CPU / 512 MiB; four workers each 1 CPU / 384 MiB; MySQL 1 CPU / 2 GiB; Redis 0.1 CPU / 128 +MiB; Collector and probe each 0.2 CPU / 256 MiB, all without swap. The primary quantity sums +exact cgroup CPU deltas after warmup through business completion and full provider/exporter +drain. Setup, warmup, later audit and teardown are outside those deltas. Memory values are +sampled observations, not allocation or simultaneous-peak measurements. + +Ordinary cells offered one one-activity workflow per second for 300 seconds (300 measured +completions), following 30 warmups. Fan-out cells offered one 64-activity workflow per minute +for 300 seconds (five measured completions), following one warmup. Original +observer/run/execution bounds stayed 90/100/120 seconds. Five fixed repetitions rotated +base/off/on; on/base/off; off/on/base; base/on/off; off/base/on, then repeated the baseline for +drift, separately for each workload. No extra repetitions or automatic study reruns were +selected. + +## CPU acceptance + +The predeclared `embedded-300s-5pairs-v2` method uses the mean of five matched +candidate/baseline ratios and two-sided 95% paired-t intervals, n=5, df=4, critical value +2.7764451052. Both the upper endpoint and absolute final baseline drift must meet the +applicable target. The agent-selected limits are 5% disabled / 15% enabled. These revised +limits acknowledge observed cost and earlier uncertainty; they do not relabel any previous 1% / +5% failure. They are qualification limits for this profile, not an operational SLA. Negative +estimates do not prove an improvement. + +| Workload | Profile | Mean | 95% interval | Baseline drift | Budget | Qualified | +| --- | --- | ---: | --- | ---: | ---: | --- | +| Ordinary | Disabled | +0.02% | [-3.71%, +3.76%] | -0.75% | 5% | Yes | +| Ordinary | Enabled | +5.41% | [+3.22%, +7.59%] | -0.75% | 15% | Yes | +| 64-member fan-out | Disabled | +1.41% | [-3.61%, +6.43%] | +4.53% | 5% | No | +| 64-member fan-out | Enabled | -0.22% | [-4.21%, +3.76%] | +4.53% | 15% | Yes | + +Every original CPU observation below is seconds per completed measured workflow. + +| Workload | Repetition | Published baseline | Disabled | Enabled | +| --- | --- | ---: | ---: | ---: | +| Ordinary | 1 | 0.716193 | 0.731170 | 0.754242 | +| Ordinary | 2 | 0.722159 | 0.725754 | 0.750391 | +| Ordinary | 3 | 0.737987 | 0.699521 | 0.763601 | +| Ordinary | 4 | 0.721790 | 0.727623 | 0.771572 | +| Ordinary | 5 | 0.704823 | 0.718455 | 0.757247 | +| Ordinary | Drift | 0.715209 | — | — | +| Fan-out | 1 | 86.932660 | 84.792693 | 88.565797 | +| Fan-out | 2 | 87.705031 | 91.995293 | 90.053616 | +| Fan-out | 3 | 93.705998 | 92.845650 | 88.947765 | +| Fan-out | 4 | 88.116971 | 93.914688 | 86.483126 | +| Fan-out | 5 | 89.654106 | 88.699468 | 90.781325 | +| Fan-out | Drift | 93.262946 | — | — | + +## Other observations + +All offered workflows were accepted and completed. The independent reader checked exact +callback/history counts and results, with no missed offers, extra attempts, unfinished work, +failed jobs or residual queue entries. No container OOM/restart or nonzero worker child exit +occurred. MySQL, Redis and Collector exited zero; HTTP and retaining shells used their declared +143 stop. Repair counts remained visible and were not substituted for extra attempts. + +| Workload/profile | Observed p95 range | Observed p99 range | Sum of sampled container peaks | +| --- | --- | --- | --- | +| Ordinary/base | 1.481–1.508 s | 1.731–1.844 s | 992,096,256–994,549,760 bytes | +| Ordinary/off | 1.470–1.499 s | 1.657–1.755 s | 992,276,480–997,048,320 bytes | +| Ordinary/on | 1.504–1.521 s | 1.751–1.821 s | 1,010,745,344–1,020,710,912 bytes | +| Fan-out/base | 53.415–61.759 s | 53.415–61.759 s | 1,021,923,328–1,028,579,328 bytes | +| Fan-out/off | 54.186–59.937 s | 54.186–59.937 s | 1,026,633,728–1,031,483,392 bytes | +| Fan-out/on | 53.981–55.608 s | 53.981–55.608 s | 1,037,586,432–1,044,594,688 bytes | + +Fan-out tails have only five measured observations per cell. These ranges are descriptive. +Maximum offer lag was 89.77 ms, below the original 100 ms gate. The one-second requested +sampling interval averaged 1.383–1.408 seconds per cell. Exact cumulative CPU deltas do not +depend on that sampling interval. + +Enabled ordinary cells ended, accepted and exported 2,310–2,310 spans each, including warmup. +Disabled and baseline cells ended/exported zero spans and persisted zero carrier bytes. No +activity-execution carrier bytes were stored. Business cardinality was checked separately from +physical span counts. + +Enabled fan-out cells ended, accepted and exported 876–890 spans each, including warmup. +Disabled and baseline cells ended/exported zero spans and persisted zero carrier bytes. No +activity-execution carrier bytes were stored. Business cardinality was checked separately from +physical span counts. + +Whole-study host CPU counters show 0.0072% steal time. This includes between-cell setup and is +context for the guest, not a replacement for the nine-role measurement. + +All four prerequisites passed before measurement: functional baseline/off/on, actual enabled +export (150 ended/accepted/exported spans), and published-baseline capacity at the declared +ordinary/fan-out offered rates. These checks do not estimate maximum sustainable capacity. The +complete fixed batch and its original decisions remain separate from subsequent source, +installed-consumer and release qualification. + +## Resolved images + +| Component | Recorded immutable identity | +| --- | --- | +| Qualification PHP image built from the recorded recipe | `workflow773-php@sha256:6d6e3255b9d61a2df7e30598c1a17bdd928f45eb374b70d7c8dff881bc0df296` | +| php:8.3.35-cli-bookworm | `php@sha256:c24b55afdf860c874b9ec6267ced132ff52de97d308ab08e6cf41e8596e509f7` | +| composer:2.9 | `composer@sha256:b09bccd91a78fe8a9ab4b33d707b862e8fe54fec17782e32683ad2a69c46867d` | +| mysql:8.4.5 | `mysql@sha256:679e7e924f38a3cbb62a3d7df32924b83f7321a602d3f9f967c01b3df18495d6` | +| redis:7.2.16 | `redis@sha256:46f15cf64fdef63e483dcda99e74e3d51c017b1175547b6bb874152ee2391aad` | +| node:22.23.3-alpine | `node@sha256:0a7108bf6c7bf5de370ffb1a3ed6be93d405b43ff159f681a8d18c0e2bc2e402` | +| otel/opentelemetry-collector@sha256:f0e23036dce3db3f9cb80d907c75a0d034cd32e93511a1943482d46129b9ae9c | `otel/opentelemetry-collector@sha256:f0e23036dce3db3f9cb80d907c75a0d034cd32e93511a1943482d46129b9ae9c` | + +## Component release decision + +The complete 32-cell study passes every workload, resource, exit and official-Collector drain +gate. Three of its four predeclared 5% disabled / 15% enabled CPU budgets qualify. Disabled +64-member fan-out remains **failed against its original 5% target**: mean +1.41%, 95% interval +[-3.61%, +6.43%], baseline drift +4.53%. Negative enabled fan-out estimates do not prove an +improvement. The original 1% / 5% studies remain failed too. + +As the component maintainer, I accept this measured uncertainty for the optional embedded minor +and revise the release's disabled CPU allowance to **7.5%**, retaining **15% enabled**. This is +an evidence-based, **post-study release decision**, not a predeclared passing result. The +complete observations fall within that revised allowance, including the drift check. The +analyzer, original targets, samples and three-of-four qualification decision remain unchanged. +No new run, selected repetitions, trimmed samples or performance SLA is implied. Future +measurements must declare their own actual targets before admission. + +The reason is concrete: ordinary disabled overhead is +0.02% (95% upper +3.76%); ordinary +enabled overhead is +5.41% (upper +7.59%); fan-out enabled's upper bound is +3.76%; all +business outcomes, attempts, histories and exports passed. The remaining disabled fan-out +observation does not establish overhead below 5%, and its interval includes zero. It cannot +establish an improvement. Its uncertainty is small enough to accept for this documented, +optional component rather than spend on another study. Applications must measure their own +workload and provider profile. + +The [public +decision](https://github.com/durable-workflow/workflow/issues/773#issuecomment-6305626003) +separates this post-study acceptance from the original budget results. Registry, migration, +upgrade and consumer qualification are recorded separately with the actual release. Full +Server/SDK delivery remains open in Workflow #773. diff --git a/README.md b/README.md index 7ce59aae..5b91dca4 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,14 @@ For the default database queue on SQLite, failed reservation writes preserve unclaimed work after a proven rollback. See [SQLite queue reservations](docs/sqlite-queue-reservations.md) for the supported framework versions and custom-connector boundary. +## Optional tracing and log correlation + +Core 2.9.0 can recover bounded W3C context across embedded queue workers and +add run, task, activity, attempt, trace and span identifiers to Laravel logs. +Tracing is disabled by default and uses the application's OpenTelemetry +provider. See [embedded tracing](docs/embedded-tracing.md) for migrations, +configuration, the log tap, exporter limits and measured overhead. + ## Your First Workflow ```php diff --git a/docs/embedded-tracing.md b/docs/embedded-tracing.md index 5959bce1..f802f5df 100644 --- a/docs/embedded-tracing.md +++ b/docs/embedded-tracing.md @@ -1,11 +1,11 @@ -# Embedded tracing preparation +# Embedded tracing -This document describes the optional embedded tracing implementation being -qualified in the [tracing proposal](https://github.com/durable-workflow/workflow/issues/773). -It is not a published-package support claim. OpenTelemetry API 1.10.1 fixes the -encoded-value baggage regression and is the minimum supported API version. -Full qualification and a compatible Durable Workflow release are required -before enabling this in production. +Core 2.9.0 adds optional OpenTelemetry tracing and Laravel log correlation for +embedded Durable Workflow. This guide covers that embedded component; the +[full tracing issue](https://github.com/durable-workflow/workflow/issues/773) +also tracks separately released Server and PHP, Python and Rust SDK support. +OpenTelemetry API 1.10.1 fixes the encoded-value baggage regression and is the +minimum supported API version. The application owns its OpenTelemetry providers, instrumentation, sampling, exporters and process shutdown. Durable Workflow uses those providers when @@ -32,7 +32,7 @@ Invalid queue-dispatch link metadata uses the separate bounded attribute `durable.queue_context.disposition`; dropping that link never replaces the run's durable parent or task authority. -## Configure the source integration +## Configure the integration In a qualification application, use one application-owned environment flag for tracing and its optional log tap. Set `DW_TRACING_ENABLED=true` in that @@ -133,22 +133,34 @@ the access controls appropriate to their contents. ## Runtime and SDK qualification -This matrix describes this draft's durable propagation and correlation +This matrix describes Core 2.9.0's embedded propagation and correlation contract. Ordinary application HTTP instrumentation has a separate scope. -| Runtime or client | Scope established by this draft | Support status | +| Runtime or client | Scope of this component | Support status | | --- | --- | --- | -| Embedded Laravel | Source controls cover incoming HTTP context, separate queue workers, retries, recovery and the transitions described below. | Unreleased. Remaining full gates and overhead acceptance still block stable support. | -| PHP Server | This embedded draft adds no service transport contract. | Separate integration and consumer qualification required. | +| Embedded Laravel | Incoming HTTP context, separate queue workers, retries, recovery and the transitions described below. | Core 2.9.0 embedded component. Use the qualified application/exporter profile and review its measured overhead. | +| PHP Server | This package adds no service transport contract. | Requires a separately qualified Server release and SDK tuple. | | Rust Server draft | Proposed observability columns and carriers have not been adopted by its catalog. | Unqualified for this contract. | -| Cloud | No deployment of this draft is claimed. | Separate release and deployment qualification required. | +| Cloud | No deployment is included in this component. | Separate release and deployment qualification required. | | PHP service SDK | No durable carrier API or service propagation is added here. | Unqualified for this contract. | | Python service SDK | No durable carrier API or service propagation is added here. | Unqualified for this contract. | | Rust service SDK | No durable carrier API or service propagation is added here. | Unqualified for this contract. | +The [complete overhead study](../.github/tracing-performance/RESULTS-2026-10-21.md) +compares this production source with published Core 2.8.0 using 32 fixed cells +and an accepting official Collector. Ordinary disabled overhead averaged ++0.02% (95% upper +3.76%); ordinary enabled overhead averaged +5.41% (upper ++7.59%). Disabled 64-member fan-out averaged +1.41%, with an upper bound of ++6.43%, so its original 5% target remains unproven. Enabled fan-out's interval +includes zero and does not establish an improvement. The maintainer accepted a +7.5% disabled / 15% enabled release allowance after reviewing those complete +results. That post-study decision does not turn the original failed target +into a pass. These observations apply to the recorded workload, resource and +provider profile; applications must measure their own configuration. + The package's optional tracing API can be absent, and tracing is disabled by default. In those modes the execution callback still runs with its ordinary -business outcome. Enabling this source requires review of its proposed nullable +business outcome. Upgrading to this version adds nullable `observability_context` columns on runs, commands and tasks. Existing rows keep null metadata. Older executions use independent processing segments instead of borrowing an unrelated worker's parent. Canonical histories, command receipt @@ -161,6 +173,11 @@ connection. They preserve existing rows and can be reapplied. Applications that create replacement tables later must include these nullable JSON columns in their schema. This is a schema upgrade even when tracing is disabled. +Tracing-enabled workflow processing uses cold replay. It clears retained +continuation frames and does not publish new frames while tracing is enabled. +Changing the flag cannot extend an old frame's lifetime or generation; workers +use the ordinary history path. This behavior contributes to enabled overhead. + Worker processes can handle successive jobs, but this integration scopes context to the current operation. It does not keep one execution span open for the lifetime of a worker session. Concurrent Fiber and cold-process controls @@ -302,9 +319,9 @@ The local callback precedes the parallel group. The existing `all()` API does not admit local-activity descriptors as group members. Simultaneous-worker load and large-fan-out overhead remain separate qualification gates. -### Outcome links (draft) +### Outcome links -The draft adds a nullable, internal `workflow_tasks.observability_context` +The package adds a nullable, internal `workflow_tasks.observability_context` column. A recording task can retain its actual outcome span context alongside its existing task update. Entries bind the physical history event ID, sequence and type to the original task, run, namespace and activity attempt or timer. @@ -351,8 +368,8 @@ disabled/legacy modes and wrong-task/namespace metadata controls. Transaction rollback, duplicate decisions and legacy migration are exercised separately. Timeout/cancellation causality, concurrent/large fan-out, multi-backend acceptance and overhead remain separate qualification gates. -The new column remains unreleased and is not adopted by the service wire or -native catalogs. +The task column belongs to the embedded schema upgrade. Service wire and +native catalog adoption require separate qualification. Queue dispatch emits a short `durable.task.dispatch` producer span outside the database transaction. It covers the queue dispatcher call. A returned call is @@ -378,7 +395,7 @@ request followed by its cleanup activity, and immediate cancellation or termination. Existing carrier metadata and deadlines remain unchanged. Cleanup uses the original run's parent, and task scopes restore the worker's context. An accepted direct cooperative cancellation request also captures its caller -in the proposed command column. Its first actual delivery links that caller +in the command column. Its first actual delivery links that caller to the workflow-processing span. Duplicate requests keep the first carrier. Cleanup activities and later replay keep their original parent and add no new delivery link. Caller baggage never replaces the run's baggage. Internal @@ -441,13 +458,13 @@ history and deadlines. A later occurrence gets its own caller without changing the earlier completed run. Disabled controls keep the same business outcomes without capturing a carrier or emitting spans. -These schedule controls add no schedule column or service wire field. The -internal buffered metadata has no stable or native adoption claim. Full tracing -acceptance, broader gates and overhead measurements remain unfinished. +These schedule controls add no schedule column or service wire field. Buffered +metadata remains internal; it is not an application API or native catalog +contract. Full Server/SDK tracing acceptance remains separate. ## Signal caller links -This draft also proposes a nullable `workflow_commands.observability_context` +Core 2.9.0 includes a nullable `workflow_commands.observability_context` column. New accepted embedded signals and signal-with-start commands capture their own bounded carrier before the admission transaction. Existing command metadata, including null, stays unchanged. The column is separate from command @@ -484,7 +501,7 @@ fan-in tracing. ## Update caller links New accepted embedded updates capture their independent caller in the same -proposed command column before the admission transaction. Idempotent duplicate +command column before the admission transaction. Idempotent duplicate requests keep the first carrier. Rejected commands do not retain it. The processing span links to the captured caller immediately before invoking @@ -502,9 +519,9 @@ link bounds also apply to updates. Service/native update propagation and wider concurrent update cases remain unqualified. The run carrier is separate observability metadata. It is not command, history, -payload or lease authority. The proposed nullable run column has not been -adopted by the native Server. Other admission paths, broader trigger links, -runtime coverage and performance qualification remain unfinished. +payload or lease authority. The nullable run column has not been adopted by the native Server. The +qualification described here does not establish service transport or native +catalog support. ## Processing duration and durable waits