Leasing calls lease-end participants before it reclaims a device #469

Part of #466.

Scope

After this PR, an optional module can hand leasing a lease-end participant. For every simlock lease that ends, leasing calls each participant after the end is committed and before the device's reclaim starts. It waits until they finish or lease.endWaitMs passes (default 10 minutes), then reclaims. Graceful daemon stop and simlock nuke cut the wait short. Nothing registers a participant yet: the agent-device host does in #474. So with the daemon as shipped, nothing a user sees changes, apart from the new config key.

In short

Today a release commits (lease gone, device reclaiming), answers the caller, and starts the wipe straight away. After this PR, leasing first calls the registered participants and waits for them, still in the background. The caller is answered as fast as today.

A lease-end participant is code an optional module gives leasing because it must act on a device after its simlock lease ends and before the device is wiped. Example: #474's agent-device host ends the agent-device session on that device, letting a running command finish first.

flowchart LR
  R[release / expiry / device lost / startup] --> C[commit: lease gone, device reclaiming]
  C --> A[caller answered]
  C --> P["call participants, wait up to lease.endWaitMs (new)"]:::changed
  P --> W[reclaim: shut down, erase]
  W --> F[device ready for the next lease]
  classDef changed stroke-width:3px

If this goes wrong, a released device stays reclaiming for up to 10 minutes for no reason, or the wipe starts while a participant still works on the device.

Technical spec

Modules touched

flowchart LR
  CL[leasing: createLeasing]:::changed --> RC[leasing: LeaseReleaseCoordinator]:::changed
  RC --> LP[leasing: LeaseEndParticipant port]:::changed
  RC --> RCO[core: ReclaimCoordinator.reclaim]
  RC --> CLK[ports: Clock]
  CFG[core: config lease.endWaitMs]:::changed --> CL
  SCH[contract: configSchema]:::changed
  classDef changed stroke-width:3px

• src/leasing/lease-ports.ts declares the port. The core imports nothing from it, and leasing imports no participant.

  export interface EndedLease {
    readonly leaseId: string;
    readonly ownerId: string;
    readonly deviceId: string; // simlock's device id, as on lease.released
    readonly platform: Platform; // the device's spec.platform
    readonly reason: "explicit" | "killed" | "expired" | "device-lost";
  }
  export interface LeaseEndParticipant {
    leaseEnded(ended: EndedLease, signal: AbortSignal): Promise<void>;
  }

  reason is the reason the lease ended with, the one lease.released or lease.expired carries.
• src/leasing/index.ts exports both types.
• src/leasing/create-leasing.ts: LeasingOptions gains leaseEndParticipants?: readonly LeaseEndParticipant[] (default none). It passes them, options.clock and config.lease.endWaitMs to the coordinator.
• src/leasing/lease-release-coordinator.ts does the calls:
  • When. After the commit, and after lease.released or lease.expired is emitted. Never inside decisions.run: a participant may take minutes, and every other lease operation must go on meanwhile.
  • The three reclaim modes of #release (ReclaimMode, :65), each with its participant rule:
    • "background" (release, releaseAll, releaseDeviceLost, expire, and endAtStartup with reclaim). When at least one participant is registered, the commit writes the device's deferred-reclaim marker (beginRelease(..., { deferReclaim: true }), the marker the none path writes today), so a daemon that dies during the wait leaves a device the next start wipes (see Other code). The reclaim's transition out of reclaiming clears the marker, as it does today (transition, src/core/domain.ts:473). So the marker is still set during the wipe that follows the wait: a daemon that dies during that wipe also gets a full wipe at the next start, where today it would only shut the device down. Wiping twice is safe and costs one more wipe; it is the reading of the marker ADR 0019 §2 gives a postponed wipe. Then #reclaimInBackground (:275) takes the reclaim claim, calls every participant at once, and starts reclaim.reclaim when all have settled, the ceiling passes or the wait is cut short, whichever comes first. The whole chain stays in #backgroundReclaims, so the claim is released only when the reclaim ends, as today.
    • "await" (releaseAllDuringMaintenance, used by nuke). Maintenance is active, so each participant is called with an already aborted signal. Leasing does not await the participant's promise and arms no ceiling: it awaits reclaim.reclaim straight after the call, as today. A participant that ignores the signal cannot hang nuke.
    • "none" (endAtStartup with wait), and the endings outside #release that start no reclaim (endAtStartup with missing, endForAbsentDevice). The participants are called the same way, with a signal that aborts at the ceiling. Nothing waits for them.
  • The ceiling. One budget per ended lease, shared by all its participants, measured from the moment they are called. It never restarts. Armed with Clock.setTimer. It is cancelled as soon as the wait ends for any reason: the participants settle, the wait is cut short by maintenance or settle(), or dispose() runs. So no ceiling timer outlives its wait, and a cut-short wait never logs a ceiling warning or starts a second reclaim. When the ceiling passes, leasing aborts the signal, logs a warning with the lease id and device id, and starts the reclaim. It does not wait for the participant to return after that.
  • A participant that rejects or throws is logged as an error with the lease id, device id and error message. The other participants are still awaited, and the reclaim goes ahead. Leasing attaches its rejection handler when it makes the call, so a rejection that comes after leasing stopped waiting (after the ceiling, a cut-short, settle(), or a call nothing waits for) is logged the same way and is never an unhandled rejection.
  • Maintenance (simlock nuke). beginMaintenance aborts every signal handed out, so each background reclaim stops waiting and starts at once; then it waits for them as today. While maintenance is active, every participant call gets an already aborted signal.
  • Graceful stop. Leasing.settle() aborts every signal handed out before it waits for the background reclaims. Every ending after that, in this process, passes an already aborted signal.
  • Leasing.dispose() cancels every ceiling timer still armed and aborts every signal handed out. A participant that returns after dispose() starts no reclaim: a disposed engine runs no more driver work, and the device's deferred-reclaim marker leaves the wipe to the next start.
  • No participant. The reclaim starts exactly as today: no marker, no timer, no extra step before reclaim.reclaim.
  • A release that commits nothing (unknown lease, or an expiry whose deadline moved) calls no participant.
• src/leasing/testing.ts: TestEngineOptions gains leaseEndParticipants, passed to createLeasing.
• src/core/config.ts: Config.lease.endWaitMs, default 600_000 (the same as exec.timeoutMs), validated as a positive number no larger than 2_147_483_647 (the largest delay a Node timer holds; above it setTimeout fires after about 1 ms, see MAXIMUM_RENEW_DELAY_MS in src/lease-policy/index.ts:53). The message names the key.
• src/contract/schemas.ts: configSchema.lease gains endWaitMs: z.number().
• Every test Config literal gains lease.endWaitMs (about 19 test files; mechanical).

Contract and event changes

• configSchema (the config.get output) gains lease.endWaitMs: protocol +1 at merge. Rerun the tests of every surface that publishes config.get.
• No event changes. lease.released and lease.expired still fire when the end is committed. No new event.

Other code on the same state

The state: an ended lease's device in reclaiming, its reclaim claim, the background reclaim set, the maintenance depth, and the new ceiling timers.

• StartupConverger#recoverInterruptedReclaims (src/core/startup-converger.ts:55) skips claimed devices (:86). The claim is taken before the wait, so a device whose participants are still running is never recovered in-process. Startup endings with reclaim run in leasing.reconcile, before core.converge (the order src/daemon/main.ts and convergeRunningCapacity in src/leasing/testing.ts:117-121 use), so they hold the claim first. The waiting release wins.
• A daemon that dies mid-wait leaves the device reclaiming with no claim and with its deferred-reclaim marker. The next start's interrupted-reclaim recovery (ReclaimCoordinator.recoverInterrupted, src/core/reclaim-coordinator.ts:134-150) sees the marker and runs the full wipe under a claim, as it does today for a wait ending. Without the marker it would only shut the device down (completeReclaimWithoutPurge) and the next holder would get the last holder's data. It calls no participant; the participants' own restart handles their side. A physical device is not recovered there (virtual devices only); its interrupted reclaim stays PhysicalDeviceWatcher's, which wipes it when it returns, unchanged.
• Running capacity (src/core/capacity/limits.ts:20, RUNNING_STATES includes reclaiming) counts the waiting device for the whole wait, as it counts a wipe today. While a participant waits, a request for another spec that needs that slot queues behind it (up to lease.endWaitMs plus the wipe), or gets NO_CAPACITY with noWait. The waiting device wins. Accepted: the wait is bounded and the device is still running.
• isStalledTransition (src/core/doctor.ts:939-946) excludes claimed devices, so a long wait is never a stall.
• The warm pool (src/core/warm-pool/policy.ts:135-144) counts a reclaiming device's running slot and treats a claimed device as not idle. It does so for the whole wait, as it does for a wipe today.
• The reclaim stage told to a queued waiter (src/leasing/lease-acquisition-coordinator.ts:1074-1086) comes from the driver's reclaim estimate and does not include the wait. Unchanged: a waiter may wait longer than told.
• NukeService#runNuke (src/core/nuke-service.ts:72-78) waits for background reclaims in beginMaintenance. With the cut-short, it no longer waits for a participant. Nuke wins.
• DaemonServer#stop (src/daemon/server.ts:600-618) runs stopAuxiliary, then settle (src/daemon/main.ts:445, which calls leasing.settle()), then dispose. The cut-short in settle makes stop wait only for the wipes. Stop wins.
• Physical-device unenrollment (src/core/physical-enrollment.ts:186-190) and re-enrolling an enrolled device (PhysicalEnrollment.add, :93, the same #requireFree) refuse a device in reclaiming or holding a reclaim claim. During the wait, either one on a physical device that just left a lease fails with DeviceBusyError("reclaiming"), as it does during a wipe today, for up to lease.endWaitMs longer. The wait wins; the admin retries once the device is ready.
• PhysicalDeviceWatcher (src/leasing/physical-device-watcher.ts) ends a lease through endForAbsentDevice (no wait). It acts on a reclaiming device only when a killed daemon left it so (recordInterrupted), never on one this process is waiting on. reclaimReturned ends no lease and calls no participant.
• LeaseHealthMonitor (releaseDeviceLost) and LeaseExpiryScheduler (expire) end leases through the paths above and wait like any release.
• A gateway runs no leasing (startGatewayDaemon), so this changes nothing there.

Rules in play

• architecture.md: no workflow step in an event handler, which is why this is a port; every wait has a bound that does not reset; every exit leaves the claim released and the timer cancelled; dependencies point from the optional module to leasing, and leasing behaves the same with no participant.
• events.md: no event changes, and lease.released stays post-commit.
• testing.md: each "does not wait" test must fail when the code waits, and each "waits" test must fail when it does not.

Tests

Seam: createTestEngine (src/leasing/testing.ts) with a test participant in leaseEndParticipants, FakeClock and FakeDriver. The new tests go in a new file, src/leasing/lease-end-participants.test.ts, modelled on src/leasing/create-leasing.test.ts. Config lines in src/core/config.test.ts.

• a release answers its caller while the participant has not returned, and the participant gets the lease id, owner, simlock device id, platform and reason explicit
• the participant is called after lease.released is emitted, while the device is reclaiming and has no lease
• the driver's reclaim does not start until the participant returns; when it returns, the reclaim runs and the device becomes ready
• while the participant has not returned, a request for the same spec is not granted that device, and it is granted it after the reclaim
• the device is claimed (claimReader.isClaimed) from before the participant is called until the reclaim ends
• while the participant has not returned, a renew of another lease and a grant of another device both complete
• a participant that never returns: 1 ms before lease.endWaitMs the reclaim has not started; at lease.endWaitMs its signal is aborted, a warning naming the lease and device is logged, and the reclaim starts; when that participant resolves later, the driver's reclaim has still run once
• two participants are both called before either returns; the reclaim waits for the slower one; one ceiling, counted from the call, covers both and is not reset when the first returns
• a participant that rejects, and one that throws, are each logged as an error with the lease id, device id and error message; the other participant is still awaited; the device becomes ready
• a participant that rejects after the ceiling passed, and one that rejects after nuke cut its wait short, are each logged as an error with the lease id; the run reports no unhandled rejection
• every ending that starts a reclaim calls the participant before the reclaim, with its reason: release (explicit, killed), releaseAll (one call per lease), expiry (expired), the health monitor's device-lost release (device-lost), and a startup ending with reclaim (expired, device-lost)
• a startup ending with reclaim whose participant has not returned, followed by core.converge (convergeRunningCapacity): converge leaves the device alone (no driver shutdown, no device.shutdown with initiator startup-interrupted-reclaim), and the device is wiped once the participant returns
• a startup ending with wait calls the participant with its reason, starts no reclaim, and leaves the device reclaiming with its deferred wipe recorded, as today
• a startup ending with missing calls the participant and marks the device missing in the same write as today, without waiting for it
• a physical device that goes absent (through PhysicalDeviceWatcher) calls the participant with device-lost, and the device becomes absent without waiting for it
• a call nothing waits for, still running at the ceiling, has its signal aborted then; one that rejects is logged
• a release of an unknown lease, and an expiry whose deadline was moved by a renew, call no participant
• nuke while a participant has not returned aborts its signal, starts the reclaim at once, and completes without the clock advancing; advancing the clock past lease.endWaitMs afterwards, and the participant then resolving, logs no ceiling warning and starts no second reclaim (the driver's reclaim ran once)
• nuke with leased devices calls each participant with an already aborted signal, and completes without the clock advancing even when the participant never returns; advancing the clock past lease.endWaitMs afterwards logs no ceiling warning
• after nuke ends, a release waits for its participant again
• settle() while a participant has not returned aborts its signal and returns once the reclaim settles, without the clock advancing; when the participant then resolves, the driver's reclaim has run once, and advancing past lease.endWaitMs logs no ceiling warning
• a release after settle() began calls the participant with an already aborted signal and reclaims at once
• a participant that returns before the ceiling: advancing the clock past lease.endWaitMs afterwards logs no ceiling warning and aborts no signal
• dispose() with a participant still running aborts its signal; advancing the clock past lease.endWaitMs afterwards, and the participant then resolving, logs no ceiling warning and starts no reclaim
• with no participant, the reclaim starts without the clock advancing and the device record carries no deferred-reclaim marker; the existing tests at create-leasing.test.ts:260 and in "createLeasing startup reclaim backgrounding (Take the startup device reclaim off the convergence critical path #43)" (:1586) pass unchanged
• with a participant, the device is reclaiming with its deferred-reclaim marker while the participant runs, and the marker is gone once the device is ready
• with running capacity full, while a participant waits, a lease request for another spec queues, and a noWait request gets NO_CAPACITY; the queued one is granted once the wait and the wipe end
• a daemon stopped while a participant has not returned: a new engine on the same registry runs the full wipe (driver reclaim with clean: "standard", not only shutdown) through interrupted-reclaim recovery, the device becomes ready, and no participant registered in the new engine is called; a daemon stopped during the wipe that follows the wait also gets a full wipe at the next start
• a release with a participant emits the same events in the same order as one without (lease.released, then device.reclaimed)
• lease.endWaitMs defaults to 600000, accepts a positive number and 2147483647, refuses 0, a negative number, 2147483648 and a string with a message naming the key, and a loaded config parsed through configSchema keeps it

Done when

• The tests above pass.
• Removing the wait before reclaim.reclaim makes "the driver's reclaim does not start until the participant returns" fail.
• Removing the cut-short from beginMaintenance makes the first nuke test fail.
• The diff of src/leasing/create-leasing.test.ts and src/leasing/lease-release-coordinator.test.ts adds only the new config key and the coordinator's new options; the participant tests are all in src/leasing/lease-end-participants.test.ts.
• Dropping the deferred-reclaim marker from the background release makes the "daemon stopped while a participant has not returned" test fail.

Out of scope

• Registering a participant, and the agent-device host: simlock ending a simlock lease ends the agent-device session first #474.
• Showing the wait in status or the console: the device stays reclaiming, and no new state is added.
• Adding the wait to the reclaim estimate a queued waiter is told.

Depends on

Approval

• Approved for delivery

Written by an agent.