diff --git a/tests/fixtures/worker_startup_reconstruction/README.md b/tests/fixtures/worker_startup_reconstruction/README.md new file mode 100644 index 0000000..a5fefe2 --- /dev/null +++ b/tests/fixtures/worker_startup_reconstruction/README.md @@ -0,0 +1,147 @@ +# Historical worker startup reconstruction proposal + +This is the unexecuted source/vector proposal for [issue +123](https://github.com/durable-workflow/sdk-python/issues/123), following +[General's historical reconstruction decision](https://github.com/durable-workflow/sdk-python/issues/123#issuecomment-6644072344). +It does not recover the missing original trace or provide an executable +experiment packet. The original failures remain failures. A new successful +reconstruction would establish behavior in its declared environment and leave +the original unexplained delay open. + +`proposal.json` pins the available historical identities, existing observer +sources, six cases, budgets and remaining preparation gates. `vectors.json` +describes synthetic evidence and the limited conclusion each combination can +support. The vectors are review inputs, not captured observations or an +implemented classifier. Neither file is an execution manifest. + +## Reuse the existing driver + +Use `tests/integration/test_worker_task_families.py` and +`tests/worker_observation.py` from the exact source in `proposal.json`. Do not +copy the SDK source into the future test environment, install it editable, +subclass `Client`, or replace Worker registration, matching, retry or timeout +behavior. In the historical SDK, the query-poller thread requires +`type(client) is Client` to use its cloned client. HTTP observation patches +`httpx.AsyncClient.send` globally, so it also sees additional clients. + +The current driver declares all six split/mixed/timer by run/run_until cases. +Keep the result and run_until budgets at 30 seconds, poll timeout at 0.5 seconds, +result polling at 0.1 seconds, run_until polling at 0.05 seconds, and both the +worker shutdown setting and runner drain wait at 10 seconds. The timer remains +two seconds. Run each case once, sequentially. Preserve an observed failure and +perform bounded cleanup without a retry or a replacement worker. Delayed replies +and recovery were separate historical experiments and are outside this proposal. + +Static inspection of release source found the Client/Worker arguments, +interceptor types, task context fields and handle result API used by these +sources. This is source compatibility evidence only. Before any later SDK import, +the complete normally installed distribution must match the pinned wheel and +release source, with installed metadata and RECORD proof. A missing API or an +incompatible harness is a setup finding, not permission to patch the SDK. + +## Preserve the actual observation boundaries + +- The HTTP recorder counts discovery, registration, polls, submissions and + deregistration. It records only allowlisted task identity and typed response + metadata. It omits bodies over its size bound and does not consume a streamed + response to inspect it. Missing metadata remains unknown. +- Its send duration includes response headers and ordinary body consumption. + It cannot independently identify a header wait, body wait or backend delay. + Any proposed extra boundary instrumentation needs its own reviewed source and + controls before execution. Do not insert a proxy, reply delay or hidden retry. +- A present task in a poll response proves the observed response contained a + task. It does not by itself prove handler admission or publication. +- `workflow_task_enter` is before `_load_workflow_claim_history` and input decode + in the historical Worker. `workflow_handler_enter` is later. The intervening + interval includes history loading, decode and replay. It is not a unique + hydration-cause measurement. +- The activity interceptor runs after argument decode. An absent activity + handler milestone does not by itself identify a decode failure. +- Submission metadata such as `recorded=false` must remain false. A returned + handler, a sent request or a response without an authoritative recorded result + does not prove durable publication. Match the original task/attempt/run against + the Server comparator and terminal history. +- Existing primary HTTP clients have explicit labels; all other clients share + the label `additional`. Capture their actual discovery/query-poll coverage, + while retaining that limit on per-client attribution. A missing cloned-client + trace cannot count as full coverage. + +The current driver emits a full before-shutdown observation on failure only. +The next observer-source revision must add the same bounded snapshot on success +before calling `stop_and_drain`, bind API/database snapshots to that boundary, +and preserve the existing primary exception if observation or shutdown fails. +Do not run the current sources and claim those additions were present. Return +the revised source hashes for review rather than carrying these source pins. + +## Server comparator review + +Server's [canonical comparator response](https://github.com/durable-workflow/sdk-python/issues/123#issuecomment-6646942397) +establishes that claim commits before history fetch, and deregistration can +expire/repair the original lease and then delete its registration. Capture +read-only SQL for the exact original namespace/instance/run/task, including +task type/status, queue/compatibility, availability, lease times, attempt/repair +counters, dispatch times and error-present booleans; join run identity/status +and execution/run deadlines. Do not substitute the latest run or replacement +task. Compare private poll/claim/registration bindings internally and publish +only match/missing/invalid booleans. A registration row must also have current +incarnation, routing/capability compatibility and heartbeat freshness evidence. + +Use only existing run-scoped GET describe/history for added API observation. +Do not add a history POST, heartbeat, poll, repair or other mutation as an +observer probe. Instrumentation can observe requests the driver already makes. +Require snapshots before teardown on success and failure, then a distinct +post-shutdown snapshot. Bracket reads with monotonic times and a DB/server time +sample; do not treat API and SQL views as globally atomic or clock-equivalent. +Snapshot reads have a proposed five-second bound inside the original case +window, never an extra five seconds of worker execution. If the read cannot +finish before the case/teardown boundary, mark it incomplete and clean up on +time. The future owner must enforce the ten-second overall cleanup boundary +around the existing stop/drain calls, not add their internal waits to the budget. + +Report complete/missing/truncated/invalid for every required phase and snapshot. +An omitted task body, evicted observation, unmatched task/run/owner/attempt or +missing clone coverage cannot count as a completed phase. A cleanup repair +requires the original attempt binding, command/event identity and independently +bracketed cleanup path; nearby timestamps or RepairRequested alone do not prove +its cause. Attempt zero supports only no committed claim visible in that +snapshot; a committed claim without history delivery is a claim/delivery gap. + +The original timer had attempt 1 and an observed `RepairRequested`. The source +shows that cleanup can create such a repair; the missing original +command/event/attempt/path proof does not establish that cleanup caused this +one. Post-shutdown ready state cannot prove it was never claimed. The original mixed +trace places its timeout in registration while waiting for response headers; +the missing wire trace does not establish why the reply was delayed. Do not +rewrite these facts as two unclaimed tasks or a single demonstrated cause. + +API/history and database snapshots contain only this experiment's synthetic +identities and state. Exclude fixture authentication, request headers, payloads, +exception messages/stacks, arbitrary error text and SQL dumps. Review temporary +evidence before publishing bounded outcomes or hashes. + +## Finish preparation before requesting execution + +The original MySQL/Redis/runner image digests and host details were not retained. +Future compatible pins are explicitly reconstructed choices, never recovered +original identities. `proposal.json` leaves those pins and the dependency lock +unfilled. Do not resolve or download dependencies, retrieve images, import the +SDK or start a runtime from this source-authoring proposal. + +The complete successor packet needs exact published image/platform digests, +an exact offline wheel closure, installed-package proof, revised observer and +snapshot source, a Server-reviewed comparator, and a reviewed bounded owner. +Proposed simultaneous caps are Server 1 CPU/768 MiB, MySQL 1 CPU/1 GiB, Redis +0.25 CPU/128 MiB and Python 1 CPU/512 MiB. These are planning ceilings, not a +capacity reservation. The owner must check fresh host/engine capacity and other +owners, use fixture-only credentials, isolate the database network, expose only +loopback if needed, bound PIDs/logs/state, and clean up all owned processes and +resources on exit, stop or deadline. Capture the proposed 900-second total +ceiling before setup and leave cleanup time inside it. No extension or automatic +retry is proposed. + +Next owners: sdk-python prepares the revised sources and exact input packet; +Server reviews these vectors and the implementation of its comparator; +Security reviews the complete source, trust and +cleanup boundary; General decides any later exact invocation. The frozen 2.15 +qualification, package/release decisions and all other stopped work remain +separate. diff --git a/tests/fixtures/worker_startup_reconstruction/proposal.json b/tests/fixtures/worker_startup_reconstruction/proposal.json new file mode 100644 index 0000000..a690375 --- /dev/null +++ b/tests/fixtures/worker_startup_reconstruction/proposal.json @@ -0,0 +1,112 @@ +{ + "status": "unexecuted-source-proposal-not-an-execution-manifest", + "issue": "https://github.com/durable-workflow/sdk-python/issues/123", + "source_authoring_decision": "https://github.com/durable-workflow/sdk-python/issues/123#issuecomment-6646875083", + "server_comparator": "https://github.com/durable-workflow/sdk-python/issues/123#issuecomment-6646942397", + "owner": "sdk-python", + "historical_facts": { + "sdk": { + "version": "2.5.6", + "wheel_sha256": "8bfa26ac97840bd1d93eb6dbed872281e1ee28ad50f4d285a2cbaaca37502496", + "source": "7da889ce010b92c00d2b998d1b45dccf4ee4f4f0" + }, + "server": { + "version": "2.6.0", + "published_index": "sha256:8cc64fd0b577ebe02ac4dc49856585f1d6295363d2c16083d008e699dfc4b0b6", + "source": "8709feb28f1b85853de1d9ad15152d961860926f" + }, + "core": {"version": "2.5.12", "source": "19a7a1cf33604c66f1f838e8b3db22563041f6b2"}, + "other_versions": {"python": "3.12", "httpx": "0.28.1", "fastavro": "1.13.1", "mysql": "8.0", "redis": "7"}, + "original_backend_runner_platform_host_digests": null, + "original_failed_phase_trace": null, + "original_timer": {"attempt_after_shutdown": 1, "repair_observed": true, "cause_established": false}, + "original_mixed": {"attempt_after_shutdown": 0, "registration_header_wait_observed": true, "cause_established": false}, + "later_passes_explain_original_failure": false + }, + "reused_sources": { + "commit": "5f51d0c8e1bb86c8ce7cac021c3b81c5bf607132", + "files": [ + { + "path": "tests/integration/test_worker_task_families.py", + "blob": "9e6a82c102d687673efc70ef8d0ff750c61239ee", + "sha256": "b5a0c80d1a0c2ef78ce6620585bbcde73442ddde27fa5f90b1c4bc975dc55b6f" + }, + { + "path": "tests/worker_observation.py", + "blob": "8d38a4e04c785ff5f371df74e341717d9614de97", + "sha256": "754227f1871d17b1f61e8e9a00440880106ce4c85e4cea4141b18c7bf5172c30" + } + ], + "current_sources_have_complete_proposed_observation": false, + "required_reconstruction_adapter_changes": [ + "retain successful and failed discovery, registration, poll and existing history-request phases", + "link original task/run/owner/attempt through claim delivery, hydration, handler and publication", + "identify cloned query-client coverage without subclassing Client or changing Worker behavior", + "add complete/missing/truncated/invalid flags and refuse conclusions from incomplete evidence", + "emit bounded before-teardown snapshots on success and failure, preserving the primary exception", + "bracket teardown and independently compare the same original rows after cleanup" + ], + "source_compatibility_is_runtime_proof": false, + "sdk_overlay_or_editable_install": false + }, + "cases": [ + {"mode": "split", "entry_point": "run"}, + {"mode": "mixed", "entry_point": "run"}, + {"mode": "timer", "entry_point": "run"}, + {"mode": "split", "entry_point": "run_until"}, + {"mode": "mixed", "entry_point": "run_until"}, + {"mode": "timer", "entry_point": "run_until"} + ], + "bounds": { + "case_invocations": 1, + "case_order": "sequential", + "result_timeout_seconds": 30, + "run_until_timeout_seconds": 30, + "poll_timeout_seconds": 0.5, + "result_poll_interval_seconds": 0.1, + "run_until_poll_interval_seconds": 0.05, + "timer_seconds": 2, + "shutdown_setting_seconds": 10, + "runner_drain_wait_seconds": 10, + "proposed_outer_case_cleanup_seconds": 10, + "proposed_snapshot_seconds_inside_case_window": 5, + "proposed_total_seconds_including_setup_cleanup": 900, + "retry_or_extension": false + }, + "snapshot_contract": { + "identity": ["namespace", "workflow_instance_id", "original_run_id", "original_task_id", "original_attempt"], + "task": ["type", "status", "queue_compatibility", "available_at", "leased_at", "lease_expiry", "attempt_count", "repair_count", "dispatch_times", "repair_availability", "error_present"], + "run": ["namespace", "instance", "status", "execution_deadline_at", "run_deadline_at"], + "registration": ["status", "routing_build_type_capability_compatibility", "heartbeat_times", "freshness_boundary", "current_incarnation_match"], + "private_binding_output": ["match", "missing", "invalid"], + "api": "existing run-scoped GET describe/history only", + "additional_observer_mutations": false, + "clock_brackets": ["monotonic_before", "monotonic_after", "database_or_server_time_sample"], + "complete_states": ["complete", "missing", "truncated", "invalid"], + "globally_atomic_or_shared_clock_assumption": false, + "before_teardown_required_on_success_and_failure": true, + "after_teardown_same_original_rows_required": true + }, + "future_execution_inputs_not_selected": { + "backend_platform_digests": null, + "runner_platform_digest": null, + "server_platform_digest": null, + "complete_offline_dependency_closure": null, + "revised_observer_comparator_source_hashes": null, + "bounded_owner_source_hash": null, + "host_capacity_and_overlap_receipt": null, + "security_exact_packet_review": null, + "general_exact_invocation_admission": null + }, + "future_requirements": [ + "label new backend/platform/host pins reconstruction, never recovered original provenance", + "prove every normally installed SDK file, metadata and RECORD against the public wheel before import", + "fail setup on missing historical API without patching the package or increasing timeouts", + "fixture-only credentials/state, bounded logs/PIDs, no production credentials or endpoints", + "one owned handle, independent stop/deadline cleanup and final process/container/network/volume absence", + "return revised exact source and input hashes for Server, Security and General review before execution" + ], + "execution_admitted": false, + "original_root_cause_established": false, + "frozen_215_qualification_or_publication_carry": false +} diff --git a/tests/fixtures/worker_startup_reconstruction/vectors.json b/tests/fixtures/worker_startup_reconstruction/vectors.json new file mode 100644 index 0000000..7677083 --- /dev/null +++ b/tests/fixtures/worker_startup_reconstruction/vectors.json @@ -0,0 +1,92 @@ +{ + "status": "synthetic-review-vectors-not-captured-observations-or-an-executable-classifier", + "all_vectors_explain_the_original_execution": false, + "default_missing_evidence_conclusion": "unknown; do not assign a product or runner cause", + "vectors": [ + { + "id": "attempt-zero-before-teardown", + "given": ["complete original-row snapshot before teardown", "attempt_count remains zero", "no matching delivered task"], + "conclusion": "no committed claim visible in this snapshot", + "must_not_conclude": ["no claim ever happened", "SDK handler stalled", "original failure explained"] + }, + { + "id": "committed-claim-without-history-delivery", + "given": ["same original task/run has incremented attempt and matching lease owner", "current registration and poll binding match", "complete relevant HTTP coverage including cloned client, body/stream completeness and non-truncated observation buffers for the bracketed interval", "no completed history-bearing response within that observed interval"], + "conclusion": "claim/delivery gap within the completely observed interval; hydration or transport cause still unknown", + "missing_coverage_conclusion": "incomplete/unknown observational gap only, not a completed phase verdict", + "must_not_conclude": ["task was never claimed", "handler admitted", "history fetch succeeded"] + }, + { + "id": "registration-latency", + "given": ["discovery completion recorded", "registration send and response boundaries recorded", "case deadline expires before registration completes"], + "conclusion": "registration interval exceeded the case budget", + "must_not_conclude": ["backend admission refusal", "specific network or database cause from aggregate send time"] + }, + { + "id": "explicit-routing-refusal", + "given": ["complete current registration/freshness comparator", "existing poll response records a typed matching refusal", "same queue/build/type/capability identities linked"], + "conclusion": "observed matching refusal for this poll and registration", + "must_not_conclude": ["transport stall", "original failure explained"] + }, + { + "id": "stale-or-replaced-registration", + "given": ["registration row exists and says active", "freshness or current-incarnation comparison is stale, missing or invalid"], + "conclusion": "registration eligibility/current incarnation is not established", + "must_not_conclude": ["active row proves routable fresh worker", "empty poll proves no committed claim"] + }, + { + "id": "complete-task-with-missing-hydration-link", + "given": ["complete poll task body matches original task/run/attempt", "workflow_task_enter observed", "no exact history-load/decode boundary evidence"], + "conclusion": "task entered the interceptor; hydration/handler interval remains unresolved", + "must_not_conclude": ["hydration completed", "handler stalled", "decode failed"] + }, + { + "id": "hydration-failure", + "given": ["original task/attempt linked", "existing history request start and failed completion observed", "handler entry absent", "observer completeness established"], + "conclusion": "observed history-hydration failure before handler entry", + "must_not_conclude": ["specific transport/backend cause without further phase evidence", "original failure explained"] + }, + { + "id": "handler-open-at-deadline", + "given": ["exact task/attempt hydration completed", "handler entry recorded", "handler return/error absent at unchanged deadline", "phase buffer complete"], + "conclusion": "handler interval remained open at the deadline", + "must_not_conclude": ["publication failure", "network stall", "performance regression from this single case"] + }, + { + "id": "publication-refused", + "given": ["handler returned for exact original task/attempt", "existing submission response has recorded=false", "authoritative task/run/history comparator matches"], + "conclusion": "observed publication refusal for that attempt", + "must_not_conclude": ["handler return proves durable completion", "recorded=false is success"] + }, + { + "id": "complete-publication", + "given": ["task/run/attempt linked across hydration, handler and submission", "recorded=true", "same-run terminal API/history and SQL state reconcile before teardown"], + "conclusion": "this reconstructed case completed and published", + "must_not_conclude": ["historical untraced failure explained", "current supported tuple qualified", "performance acceptance"] + }, + { + "id": "cleanup-induced-repair", + "given": ["pre-teardown original attempt/lease and registration snapshot complete", "teardown start and deregistration recovery boundaries recorded", "same original attempt binding, repair command/event identity and cleanup path reconcile", "post-teardown original task ready and registration absent"], + "conclusion": "that exact repair is attributable to the observed cleanup path", + "must_not_conclude": ["ready after shutdown proves no claim", "registration was absent before teardown"] + }, + { + "id": "repair-near-cleanup-only", + "given": ["RepairRequested timestamp near shutdown", "original attempt or command/event/path binding missing"], + "conclusion": "repair origin unknown", + "must_not_conclude": ["cleanup caused repair", "worker lease was never acquired"] + }, + { + "id": "incomplete-observation", + "given": ["HTTP success may exist", "one or more task bodies, phase links, clone coverage, buffer entries or original-row snapshots are missing/truncated/invalid"], + "conclusion": "incomplete; no end-to-end phase verdict", + "must_not_conclude": ["complete observation", "no handler because no milestone survived", "substitute latest task/run", "retry or extend the case"] + }, + { + "id": "snapshot-clock-and-teardown-gap", + "given": ["API and SQL snapshots have separate monotonic brackets", "server/DB time differs or cleanup overlaps either bracket"], + "conclusion": "non-atomic views; retain each interval and mark causal linkage unknown", + "must_not_conclude": ["globally atomic state", "cross-clock ordering from raw timestamps", "extra worker execution to finish observation"] + } + ] +}