Optionally retry work when an agent process crashes

[highbyte]

Problem

When a worker-launched agent process exits unexpectedly, Wrighty records the run as failed, stops lease renewal, releases the claim, and preserves any recorded session and workspace. Recovery then requires operator action even when the process crash was transient and the existing session could be resumed safely.

An unexpected process crash is different from an agent-reported task failure. Wrighty should be able to distinguish those outcomes and optionally schedule a bounded retry without hiding the fact that the previous attempt failed.

Desired behavior

Add an opt-in worker policy for unexpected agent-process crashes.

• Keep the ended run's durable outcome as failed.
• Set the item dispatch state to retry-scheduled when the configured policy permits retry.
• Preserve and resume the same vendor session and workspace when a complete local resume address exists.
• Record the deferred dispatch before releasing the fenced claim.
• Reacquire the item under a new claim generation when the retry becomes due.
• Move the item to needs-attention when the retry limit is reached or safe resumption is unavailable.

A possible configuration shape is:

{
  "worker": {
    "processCrash": {
      "action": "retry",
      "initialRetryMinutes": 2,
      "backoffMultiplier": 2,
      "maxRetryMinutes": 15,
      "maxAttempts": 2
    }
  }
}

The default should remain needs-attention so process-crash retry is explicitly enabled.

Safety constraints

• Classify an unexpected process crash separately from ordinary agent-failure and unknown outcomes; retry selection must not rely on a broad nonzero-exit test alone.
• Do not automatically retry fencing, operator cancellation, item timeout, authentication, permission, billing, context-limit, or explicit agent-reported failures.
• Do not silently start a fresh session when no complete resumable session exists.
• Use bounded delay, backoff, and attempt limits; never retry immediately or indefinitely.
• Preserve partial workspace changes and the previous run diagnostics.
• A scheduled retry is not authorization: normal claim fencing and pre-launch validation still apply.

Acceptance criteria

• Wrighty can authoritatively distinguish a supported unexpected process-crash outcome from ordinary agent failure.
• With retry disabled, current failure and release behavior remains unchanged.
• With retry enabled and a complete resume address, the failed run is retained and the item becomes retry-scheduled before its claim is released.
• A due retry reacquires the item with a new claim generation and resumes the recorded vendor session and workspace.
• Missing resume state or exhausted attempts moves the item to needs-attention with an actionable reason.
• The policy is validated, documented, and surfaced through existing structured worker/status output.
• Deterministic tests cover scheduling, release ordering, due resumption, crash-loop limits, missing resume state, and excluded failure categories.