Verify example catalog against conductor-oss 3.32.0-rc.8; fix broken examples + worker-tool registry lookup

examples/README.md

@@ -89,6 +89,12 @@ Durable agent authoring (`Agent`, `AgentRuntime`, tools, guardrails, handoffs, m
 strategies) — a separate, more extensive catalog of 270+ examples, requiring
 `pip install 'conductor-python[agents]'`. See [agents/README.md](agents/README.md).
 
+> **Server version:** the agent examples require a Conductor server with the agent
+> runtime — conductor-oss **`3.32.0-rc.8` or newer**. The stable `latest` installed by
+> `conductor server start` (3.30.x) does not expose `/api/agent/*` and every agent
+> example fails with `AgentNotFoundError: HTTP 404`. Use
+> `conductor server start --version 3.32.0-rc.8`.
+
 ---
 
 ### Monitoring

examples/VERIFICATION_REPORT.md

@@ -0,0 +1,162 @@
+# Examples Verification Report
+
+**Date:** 2026-07-18 · **Branch:** `examples/verification-and-fixes` · **Base:** `7481aed2`
+
+Full verification of the SDK example catalog against a live Conductor server, with
+fixes for every reproducible example bug found. This report tracks the status of each
+suite so it can be re-checked as the SDK and server evolve.
+
+## Environment
+
+| Component | Value |
+|-----------|-------|
+| Server | conductor-oss **3.32.0-rc.8** boot JAR (Maven Central), `java -jar --server.port=8080` |
+| Python | 3.12.13 (uv venv), `pip install -e '.[agents]'` |
+| Platform | macOS (darwin arm64) — worker processes use the **spawn** start method |
+| LLM | `AGENTSPAN_LLM_MODEL=openai/gpt-4o-mini`, `OPENAI_API_KEY` + `ANTHROPIC_API_KEY` set |
+| Harness | `examples/agents/run_all_examples.py --jobs 8 --timeout 360` + curated core-example runner |
+
+### ⚠️ Server version requirement (root cause of the most common failure)
+
+The agent examples need the **agent runtime**, on by default from conductor-oss
+**3.32.0-rc.8** (the version pinned in `.github/workflows/agent-e2e.yml`). The `latest`
+JAR installed by `conductor server start` is the 3.30.x stable line, which has **no
+`/api/agent/*` endpoints** — every agent example fails with
+`AgentNotFoundError: HTTP 404 ... api/agent/start`. On 3.30.2 the suite scored
+3 PASS / 103 ERROR; on 3.32.0-rc.8 (same code) it scored 96 PASS / 2 ERROR.
+
+Workaround (works today, `conductor-server-3.32.0-rc.8.jar` is already in the CLI's S3
+bucket): `conductor server start --version 3.32.0-rc.8`. Decision: do **not** repoint
+the CLI's floating `latest` at an rc — this resolves itself when 3.32.0 GA ships.
+
+## Results — agents suite (`examples/agents/`, 146 files)
+
+`run_all_examples.py` after the fixes in this PR:
+
+| Status | Count | Notes |
+|--------|------:|-------|
+| PASS | 96 | end-to-end against the live server, real LLM calls |
+| SKIP | 46 | interactive / daemon-by-design / external infra (MCP, Docker, Kafka, Slack, OCG, skills repo, messages API) |
+| ERROR | 2 | see "Remaining failures" |
+| HUNG | 2 | see "Remaining failures" |
+
+Before fixes (same server): 91 PASS / 12 ERROR / 3 HUNG / 40 SKIP.
+
+## Results — core suite (curated, 18 examples)
+
+| Example | Status | Notes |
+|---------|--------|-------|
+| workers_e2e.py | ✅ fixed | workers run clean; e2e workflow COMPLETED server-side; blocks on `join_processes()` by design |
+| worker_example.py | ✅ fixed | runs clean as a worker daemon |
+| task_listener_example.py | ✅ fixed | all workers start, listeners fire, no spawn errors |
+| helloworld/helloworld.py | ✅ fixed | exits 0 |
+| worker_configuration_example.py | ✅ | |
+| task_workers.py | ✅ | |
+| dynamic_workflow.py | ✅ | |
+| workflow_ops.py | ✅ | |
+| workflow_status_listner.py | ✅ | |
+| kitchensink.py | ✅ | |
+| task_configure.py | ✅ | |
+| metadata_journey_oss.py | ✅ | |
+| schedule_journey.py | ✅ | |
+| event_listener_examples.py | ✅ | |
+| lease_extension_example.py | ✅ | ~62s |
+| agentic_workflows/llm_chat.py | ✅ fixed | previously a **false pass** — reported success while the workflow FAILED server-side |
+| task_context_example.py | ➖ daemon | runs until Ctrl+C by design; workers run clean |
+| agentic_workflows/function_calling_example.py | ➖ interactive | reads stdin; not runnable headless |
+
+Static check: every `.py` under `examples/` compiles (`compileall`). Unit tests:
+`tests/unit/ai` — 1744 passed.
+
+## Bugs found and fixed in this PR
+
+1. **`examples/helloworld/greetings_worker.py` — duplicate `def greet`.** The module
+   defined `greet` twice (tasks `greet` and `greet_sync`); the second shadowed the
+   first, so spawn-pickling the first failed with
+   `PicklingError: not the same object as helloworld.greetings_worker.greet`. Broke
+   `workers_e2e.py`, `helloworld.py`, and `task_listener_example.py` on macOS/Windows.
+   Renamed the second function `greet_sync`.
+
+2. **`src/conductor/ai/agents/tool.py` — SDK bug: worker-task tool lookup broken after
+   re-registration.** `ToolRegistry.register_tool_workers` overwrites a
+   `@worker_task` tool's `_decorated_functions` entry with a spawn-safe
+   `ToolWorkerEntry` wrapper; the identity check in `_try_worker_task` then failed for
+   the same tool later in the same run
+   (`TypeError: Expected a @tool-decorated function ...`). Broke
+   `14_existing_workers.py`. The lookup now walks `__wrapped__` chains and the
+   wrapper's `fn_direct`/`fn_ref` carriers. Two regression tests added.
+
+3. **`examples/worker_example.py` — hardcoded `/Users/viren/` metrics path.**
+   `PermissionError` on any other machine. Now uses `tempfile.gettempdir()`.
+
+4. **`examples/user_example/user_workers.py` — inconsistent package identity.**
+   Imported `examples.user_example.models` while every consumer loads the module as
+   `user_example.user_workers`; spawn children (which only inherit `PYTHONPATH`, not
+   runtime `sys.path` edits) crashed. Import is now package-relative-consistent.
+
+5. **`examples/agents/kitchen_sink.py` — `@agent` classifiers declared a required
+   `prompt` arg.** `@agent`-decorated functions are invoked with zero args at compile
+   time for dynamic instructions → `TypeError`. Signatures fixed.
+
+6. **`examples/agents/16i/16j` — tools defined inside factory functions.**
+   `<locals>` callables can't be re-imported by spawn workers →
+   `SpawnSafetyError`. Tools hoisted to module level; both now pass.
+
+7. **`examples/agents/94_openai_runner_tools.py` — `@function_tool` rebinding.** The
+   decorator rebinds the module global to a `FunctionTool`, so the extracted original
+   couldn't be pickled by reference. The example now keeps the plain function at module
+   level and applies `function_tool()` at `Agent(...)` construction; passes end-to-end.
+
+8. **`examples/agentic_workflows/llm_chat.py` — two bugs.** (a) Two LLM tasks sent
+   **system-only** message lists; the server requires a non-empty user message and
+   failed the workflow before the first task ran. (b) The example treated any terminal
+   state as success (`is_completed()`), printing "Conversation complete." over a FAILED
+   workflow and exiting 0. Both fixed; a full 3-turn conversation now completes.
+
+9. **`examples/agents/run_all_examples.py` — skip-list updates.** Six examples that
+   need infra this environment (and CI's pinned server) don't provide are now
+   classified SKIP with reasons: `30`/`32` (need the `dg` skill cloned into
+   `~/.claude/skills/dg`), `75`/`82`/`83`/`84` (need the workflow *messages* API,
+   `GET /api/workflow/{id}/messages`, which is not in conductor-oss 3.32.0-rc.8).
+
+10. **`examples/agents/README.md`** — added the server-version requirement + start
+    commands; fixed the provider/model table (OpenAI row showed the Anthropic default
+    string); renumbered setup steps. **`examples/README.md`** — added the same server
+    note to the AI Agents section.
+
+## Remaining failures (known, not fixed here)
+
+| Example | Status | Diagnosis |
+|---------|--------|-----------|
+| `agents/kitchen_sink.py` | ERROR | Fixed classifier TypeError, now fails server-side: `HTTP 500 — The Task translation_swarm defined as a sub-workflow has no workflow definition available`. Sub-workflow (SWARM strategy) definitions aren't registered before `/agent/start` on conductor-oss rc.8. Needs SDK/server investigation. |
+| `agents/74_cli_error_output.py` | FLAKY | LLM-behavior assertion: the agent paraphrases stderr instead of quoting it verbatim; passes/fails depending on model output. Consider loosening the assertion or pinning a stricter prompt. |
+| `agents/86_coding_agent.py` | HUNG | Does not finish within 420s even running solo. Needs investigation (or reclassification if it is expected to be long-running). |
+| `agents/68_context_condensation.py` | SLOW | Passes solo in ~346s; exceeds the suite timeout under 8-way contention. Consider `--timeout 480` for suite runs or trimming the example. |
+
+## SDK observations (follow-ups, out of scope for this PR)
+
+- **`AgentNotFoundError` should name the server-version requirement.** When
+  `/api/agent/start` 404s ("No static resource"), the error in
+  `orkes_agent_client.py` should suggest conductor-oss ≥ 3.32.0-rc.8 and the
+  `conductor server start --version` command. This would have made the most common
+  failure self-explanatory.
+- **`run_all_examples.py` preflight**: check `GET /api/version` + probe an agent
+  endpoint before launching 100+ subprocesses against a server that can't run them.
+- **openai-agents `@function_tool` at module level is spawn-unsafe by construction**
+  (the global is rebound to a `FunctionTool`; the original function is only reachable
+  through a closure, which `FunctionRef` can't express). Teaching the spawn-safety
+  layer a closure hop — or documenting the `function_tool(fn)`-at-construction pattern
+  — would let users keep the upstream OpenAI sample shape.
+
+## How to reproduce
+
+```bash
+# server
+curl -fL -o conductor-server.jar "https://repo1.maven.org/maven2/org/conductoross/conductor-server/3.32.0-rc.8/conductor-server-3.32.0-rc.8-boot.jar"
+OPENAI_API_KEY=... ANTHROPIC_API_KEY=... java -jar conductor-server.jar --server.port=8080
+
+# SDK + suite
+uv venv --python 3.12 .venv && VIRTUAL_ENV=$PWD/.venv uv pip install -e '.[agents]'
+export CONDUCTOR_SERVER_URL=http://localhost:8080/api AGENTSPAN_LLM_MODEL=openai/gpt-4o-mini
+python examples/agents/run_all_examples.py --jobs 8 --timeout 360
+```

examples/agentic_workflows/llm_chat.py

@@ -87,8 +87,14 @@ def create_chat_workflow(executor) -> ConductorWorkflow:
         messages=[
             ChatMessage(
                 role="system",
-                message="You are an expert in science. Think of a random scientific "
-                        "discovery and create a short, interesting question about it.",
+                message="You are an expert in science.",
+            ),
+            # The server requires a non-empty user message — a system-only
+            # conversation fails validation before the first task runs.
+            ChatMessage(
+                role="user",
+                message="Think of a random scientific discovery and create a "
+                        "short, interesting question about it.",
             ),
         ],
         temperature=0.7,
@@ -121,9 +127,16 @@ def create_chat_workflow(executor) -> ConductorWorkflow:
             ChatMessage(
                 role="system",
                 message=(
-                    "You are an expert in science. Given the context below, "
+                    "You are an expert in science. Given the context provided, "
                     "generate a follow-up question to dive deeper into the topic. "
-                    "Do not repeat previous questions.\n\n"
+                    "Do not repeat previous questions."
+                ),
+            ),
+            # The server requires a non-empty user message — a system-only
+            # conversation fails validation before the task runs.
+            ChatMessage(
+                role="user",
+                message=(
                     "Context:\n${chat_complete_ref.output.result}\n\n"
                     "Previous questions:\n"
                     "${collect_history_ref.input.history}"
@@ -198,6 +211,16 @@ def main():
                         printed_tasks.add(ref)
             time.sleep(2)
 
+        # is_completed() is true for any terminal state — verify the workflow
+        # actually COMPLETED rather than FAILED/TERMINATED/TIMED_OUT.
+        result = workflow_client.get_workflow(workflow_id=workflow_id, include_tasks=False)
+        if result.status != "COMPLETED":
+            print("=" * 70)
+            print(f"Workflow ended {result.status}: {result.reason_for_incompletion}")
+            print(f"Full execution: {api_config.ui_host}/execution/{workflow_id}")
+            print("=" * 70)
+            raise SystemExit(1)
+
         print("=" * 70)
         print("Conversation complete.")
         print(f"Full execution: {api_config.ui_host}/execution/{workflow_id}")

examples/agents/16i_credentials_langchain.py

@@ -23,19 +23,22 @@
 from settings import settings
 
 
+# Tool callables must live at module level: worker processes are spawned on
+# macOS/Windows and re-import this module, so a tool defined inside a factory
+# function ("<locals>") cannot be resolved by qualified name (SpawnSafetyError).
+def check_github_token() -> str:
+    """Check if GitHub token is available in the environment."""
+    token = os.environ.get("GITHUB_TOKEN", "")
+    if token:
+        return f"GitHub token available (starts with {token[:4]}...)"
+    return "GitHub token is NOT available"
+
+
 def create_langchain_agent():
     """Create a LangChain agent with a tool that uses GITHUB_TOKEN."""
     from langchain.agents import create_agent
     from langchain_core.tools import tool as lc_tool
 
-    @lc_tool
-    def check_github_token() -> str:
-        """Check if GitHub token is available in the environment."""
-        token = os.environ.get("GITHUB_TOKEN", "")
-        if token:
-            return f"GitHub token available (starts with {token[:4]}...)"
-        return "GitHub token is NOT available"
-
     model_str = settings.llm_model
     # create_agent accepts "provider:model" format (e.g. "openai:gpt-4o")
     if "/" in model_str:
@@ -44,7 +47,7 @@ def check_github_token() -> str:
 
     agent = create_agent(
         model_str,
-        tools=[check_github_token],
+        tools=[lc_tool(check_github_token)],
         system_prompt="You are a helpful assistant. Use tools when asked.",
     )
     return agent

examples/agents/16j_credentials_openai_sdk.py

@@ -22,22 +22,27 @@
 from conductor.ai.agents import AgentRuntime
 
 
+# Tool callables must live at module level: worker processes are spawned on
+# macOS/Windows and re-import this module, so a tool defined inside a factory
+# function ("<locals>") cannot be resolved by qualified name (SpawnSafetyError).
+# Keep the plain function importable and apply @function_tool at agent
+# construction, so the module global is not rebound to a FunctionTool.
+def check_github_auth() -> str:
+    """Check if GitHub authentication is available."""
+    token = os.environ.get("GITHUB_TOKEN", "")
+    if token:
+        return f"GitHub token is set (starts with {token[:4]}...)"
+    return "GitHub token is NOT set"
+
+
 def create_openai_agent():
     """Create an OpenAI Agent SDK agent with a credential-aware tool."""
     from agents import Agent, function_tool
 
-    @function_tool
-    def check_github_auth() -> str:
-        """Check if GitHub authentication is available."""
-        token = os.environ.get("GITHUB_TOKEN", "")
-        if token:
-            return f"GitHub token is set (starts with {token[:4]}...)"
-        return "GitHub token is NOT set"
-
     agent = Agent(
         name="github_checker",
         instructions="You check GitHub authentication status. Use the tool when asked.",
-        tools=[check_github_auth],
+        tools=[function_tool(check_github_auth)],
     )
     return agent
 

examples/agents/94_openai_runner_tools.py

@@ -49,7 +49,10 @@ class Weather(BaseModel):
     conditions: str = Field(description="The weather conditions")
 
 
-@function_tool
+# Keep the plain function importable at module level and apply
+# @function_tool at Agent construction: worker processes are spawned on
+# macOS/Windows and re-import this module by qualified name, which fails
+# when the decorator has rebound the module global to a FunctionTool.
 def get_weather(city: Annotated[str, "The city to get the weather for"]) -> Weather:
     """Get the current weather information for a specified city."""
     print("[debug] get_weather called")
@@ -59,7 +62,7 @@ def get_weather(city: Annotated[str, "The city to get the weather for"]) -> Weat
 agent = Agent(
     name="weather_agent",
     instructions="You are a helpful agent.",
-    tools=[get_weather],
+    tools=[function_tool(get_weather)],
 )
 
 

examples/agents/README.md

@@ -120,7 +120,33 @@ To install all framework dependencies at once:
 uv pip install langchain langchain-core langchain-openai langgraph openai-agents google-adk
 ```
 
-### 2. Configure your environment
+### 2. Start a Conductor server
+
+The agent examples need a Conductor server with the **agent runtime**, which is on by
+default from **conductor-oss `3.32.0-rc.8`** onward (the same version pinned by this
+repo's agent-e2e CI). Older servers — including the `latest` stable line installed by
+`conductor server start` (3.30.x at the time of writing) — do not expose the
+`/api/agent/*` endpoints, and every example fails with
+`AgentNotFoundError: HTTP 404 ... api/agent/start`.
+
+Start a known-good version with the [Conductor CLI](https://github.com/conductor-oss/conductor-cli):
+
+```bash
+conductor server start --version 3.32.0-rc.8
+```
+
+Or run the boot JAR from Maven Central directly:
+
+```bash
+curl -fL -o conductor-server.jar \
+  "https://repo1.maven.org/maven2/org/conductoross/conductor-server/3.32.0-rc.8/conductor-server-3.32.0-rc.8-boot.jar"
+java -jar conductor-server.jar --server.port=8080
+```
+
+Export your LLM provider API keys (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) in the
+shell that starts the server — the server auto-enables the matching providers.
+
+### 3. Configure your environment
 
 Export environment variables:
 
@@ -131,14 +157,14 @@ export AGENTSPAN_SERVER_URL=http://localhost:8080/api
 # export AGENTSPAN_AUTH_SECRET=<secret>
 ```
 
-#### 2.1. Choose a model
+#### 3.1. Choose a model
 
 The `AGENTSPAN_LLM_MODEL` variable uses the `provider/model-name` format. Examples:
 
 | Provider | Model string | API key env var |
 |----------|-------------|-----------------|
-| OpenAI | `anthropic/claude-sonnet-4-6` (default) | `OPENAI_API_KEY` |
-| Anthropic | `anthropic/claude-sonnet-4-20250514` | `ANTHROPIC_API_KEY` |
+| OpenAI | `openai/gpt-4o-mini` | `OPENAI_API_KEY` |
+| Anthropic | `anthropic/claude-sonnet-4-6` (default) | `ANTHROPIC_API_KEY` |
 | Google Gemini | `google_gemini/gemini-2.0-flash` | `GOOGLE_GEMINI_API_KEY` |
 | AWS Bedrock | `aws_bedrock/...` | AWS credentials |
 | Azure OpenAI | `azure_openai/...` | Azure credentials |
@@ -147,7 +173,7 @@ All supported providers: `openai`, `anthropic`, `google_gemini`, `google_vertex_
 `azure_openai`, `aws_bedrock`, `cohere`, `mistral`, `groq`, `perplexity`,
 `hugging_face`, `deepseek`.
 
-### 3. Run an example
+### 4. Run an example
 
 ```bash
 # Core SDK examples