[mykytanetipa]

In cluster mode nothing finishes a task whose replica dies mid-turn, and nothing stops two replicas from running the same task at once.

Fixes #1324

Example:

sequenceDiagram
    participant C as Client
    participant A as Replica A (runs agent)
    participant S as Shared store
    participant B as Replica B
    C->>A: SendMessage(T)
    A->>S: T is WORKING
    C->>B: SendMessage(T, follow-up)
    Note over B: accepted, starts a second executor on T
    Note over A: killed (deploy, OOM)
    C->>B: GetTask(T)
    B-->>C: WORKING, and stays WORKING for good
    Note over C: a client subscribed through B waits forever

Loading

Fix: a replica now holds a lease on a task while it runs a turn. It renews the lease on a heartbeat and releases it when the turn ends, including when the task stops to wait for input. AbandonedTaskSweeper, which every replica can run, claims leases that have expired and marks their tasks FAILED. Each expired lease is claimed by exactly one replica, and the FAILED write goes to the event stream and push notifications like any other. A task that was already waiting for input had finished its turn, so the sweeper only drops its lease.

A follow-up that arrives while another replica holds the lease is rejected with UnsupportedOperationError. If the task is already waiting for input, the other replica is about to release the lease, so the follow-up waits for it for up to one heartbeat interval instead of being rejected. A replica that rejected a follow-up does not keep a local copy of the task, so a later SubscribeToTask there follows the shared event stream. A replica that loses its lease, e.g. cut off from the database for longer than the TTL, stops its executor and its late events are dropped. DefaultRequestHandlerV2.aclose() also marks turns still running on the closing replica FAILED, which is on by default when an event stream is configured.

In the example, the follow-up sent to B is rejected while A holds the lease, and after A dies B marks T FAILED one TTL later and the subscribed client receives it.

The lease store is optional; without it, only aclose() behaves differently. It works alongside interrupted_task_timeout (#1338). Alembic revision b0f01b04ff26 (revises b5e3d1c8a2f7) adds the task_leases table and does not alter existing ones. The clustermode sample wires up the lease store and the sweeper, and its README shows how to try a crash.

Tested: multi-replica and DatabaseTaskLeaseStore tests (SQLite, Postgres, MySQL), migration tests, and kill -9 of a replica in the sample, where the task goes FAILED instead of staying WORKING.

[github-actions]

🧪 Code Coverage (vs multiserver/fix-push-for-sdk-writes)

⬇️ Download Full Report

	Base	PR	Delta
src/a2a/server/agent_execution/active_task.py	95.80%	95.67%	🔴 -0.13%
src/a2a/server/cluster/database_event_stream.py	96.94%	92.86%	🔴 -4.08%
src/a2a/server/cluster/database_task_store.py	99.31%	99.32%	🟢 +0.01%
src/a2a/server/events/event_queue_v2.py	92.27%	91.79%	🔴 -0.48%
src/a2a/server/models.py	84.72%	87.06%	🟢 +2.34%
src/a2a/server/request_handlers/default_request_handler_v2.py	88.57%	89.30%	🟢 +0.73%
src/a2a/utils/proto_utils.py	95.78%	95.85%	🟢 +0.07%
src/a2a/utils/telemetry.py	91.47%	90.70%	🔴 -0.78%
src/a2a/migrations/versions/b0f01b04ff26_add_task_leases.py (new)	—	77.27%	—
src/a2a/server/cluster/database_lease_store.py (new)	—	97.22%	—
src/a2a/server/cluster/lease.py (new)	—	100.00%	—
src/a2a/server/cluster/recovery.py (new)	—	100.00%	—
Total	93.24%	93.29%	🟢 +0.05%

Generated by coverage-comment.yml