Heartbeats keep the simlock lease alive #473

Part of #466.

Scope

After this PR, an agent keeps its device while it sends commands. Each heartbeat the agent-device CLI sends renews the simlock lease, and a long command keeps it renewed until it ends. An agent that stops sending heartbeats loses the device one TTL later, 5 minutes with today's CLI. When a lease is gone for a reason the agent did not cause (simlock ended it, the agent-device daemon restarted, or the simlock daemon restarted), the agent's next command fails with lease ended by simlock; run disconnect, then open instead of quietly leasing a new device. A lease the agent ended through agent-device (close or disconnect, which release it through the front) answers "lease not found", so close then open still works. A lease ended any other way, the holder's own simlock release <id> and simlock nuke included, counts as ended by simlock. The agent's release of a lease simlock already ended answers released. After a simlock daemon restart, the agent's release also frees its old simlock lease. An operator can cap the TTL of these leases with agentDevice.leaseTtlCapMs, which the end-to-end suite uses to make expiry take seconds.

In short

The CLI sends a heartbeat (agent_device.lease.heartbeat) before every command except devices. The front sends it to agent-device first. If agent-device answers, the front renews the simlock lease, unless the lease already runs longer (the agent renewed it itself). If the lease is gone, the front answers by who ended it: the agent itself (a release through the front, remembered for one TTL), or anything else. While a command for a lease runs, the front renews the simlock lease every third of its TTL, and once more when the command ends, because agent-device keeps its own lease alive during a command and simlock would otherwise expire it mid-command.

sequenceDiagram
  actor A as agent-device CLI
  participant F as front
  participant D as agent-device daemon (via host)
  participant L as simlock leasing
  A->>F: heartbeat(id, ttlMs)
  F->>D: heartbeat(id, ttlMs = ceiling) first
  alt agent-device answers the lease
    F->>L: lease.renew(id, TTL), only if that moves the deadline later (new)
    alt renewed or already later
      F-->>A: agent-device's answer
    else simlock lease gone
      F-->>A: LEASE_ENDED_BY_HOST (new)
    end
  else lease gone in agent-device
    alt the agent released it within one TTL
      F-->>A: LEASE_NOT_FOUND, as agent-device sent it (new)
    else anything else
      F->>L: lease.release(id) as the token's principal (new)
      F-->>A: LEASE_ENDED_BY_HOST (new)
    end
  end

What the agent sees:

# before (#472): heartbeats do not renew the simlock lease
$ agent-device click @e3          # 6 minutes after open, a command every minute
(the simlock lease expired at 5 minutes and simlock wiped the device under the session)

# after
$ agent-device click @e3          # keeps working while heartbeats arrive
$ simlock release --all --yes     # operator, on the worker
$ agent-device click @e3
error: lease ended by simlock; run disconnect, then open
$ agent-device close              # the agent's own close and release
$ agent-device open MyApp --platform ios   # allocates a new lease

How the TTL is chosen (cap is agentDevice.leaseTtlCapMs):

LEASE_ENDED_BY_HOST is a JSON-RPC error with data: { code: "UNAUTHORIZED", message: "lease ended by simlock; run disconnect, then open", details: { reason: "LEASE_ENDED_BY_HOST" } }. The CLI allocates a new lease only for the reasons LEASE_NOT_FOUND, LEASE_EXPIRED and LEASE_REVOKED (isInactiveLeaseError in agent-device's packages/contracts/src/lease-scope.ts), so for this one it fails the command and prints the message.

If this task goes wrong, a user notices a device wiped under a running command, a lost device that the CLI silently replaced with a clean one, a lease the agent extended with simlock lease renew cut short, or close then open failing.

Technical spec

Modules touched

flowchart LR
  FR["agent-device/front: heartbeat, renewals, who ended *"] --> DSP["daemon dispatcher: lease.renew / lease.release / lease.list"]
  FR --> FM["front map: TTL, who ended *"]
  FR --> HL["host link"]
  FR --> SH["agent-device/shared: LEASE_ENDED_BY_HOST *"]
  MAIN["daemon/main.ts: cap, edge disposal *"] --> FR
  CFG["core/config.ts: agentDevice.leaseTtlCapMs *"] --> MAIN
  K["contract/schemas.ts: configSchema *"]
  FK["agent-device/testing.ts: FakeAgentDevice lease-gone reasons *"]

• src/core/config.ts: agentDevice.leaseTtlCapMs, optional, an integer number of milliseconds from 5000 (agent-device's shortest lease window) to lease.maxTtlMs, unset by default. A value outside that range is refused at load, naming the key, as lease.defaultTtlMs is today.
• src/contract/schemas.ts: configSchema (:776) carries agentDevice.leaseTtlCapMs beside simlock runs the agent-device daemon #470's agentDevice keys.
• src/daemon/main.ts: passes the cap to the edge. In stopAuxiliary (:476) it calls the edge's dispose before it stops the HTTP gateway.
• src/agent-device/shared/: reads a heartbeat's leaseId and ttlMs with An agent opens and closes a device through simlock #472's lease-call reader, and a command's lease ID with An agent opens and closes a device through simlock #472's command reader; it adds nothing to either. It builds the LEASE_ENDED_BY_HOST reply above, and reads agent-device's lease-gone reasons (UNAUTHORIZED with details.reason LEASE_NOT_FOUND, LEASE_EXPIRED or LEASE_REVOKED, the same test as isInactiveLeaseError).
• src/agent-device/front/:
  • The TTL, by the table above: the CLI's ttlMs lowered to the cap and raised to 5 s; the cap when the CLI sent none; and when neither is set, no ttlMs in the allocate and the granted lease's own TTL after it. An agent opens and closes a device through simlock #472's allocate sends this TTL in lease.request. The front map stores it per lease. A heartbeat that carries ttlMs replaces the stored TTL by the same rule, but only once simlock accepted a renewal with it (or the never-shorten rule skipped one); a heartbeat whose renewal simlock refused keeps the old TTL. One that carries none keeps it. A ttlMs that is not a positive finite number is refused with An agent opens and closes a device through simlock #472's message before anything is sent.
  • The ceiling. Every heartbeat the front forwards carries ttlMs equal to An agent opens and closes a device through simlock #472's ceiling (HostLink.status().leaseMaxTtlMs), as An agent opens and closes a device through simlock #472's allocate does, so the agent-device lease never expires before the simlock lease.
  • Renewing never shortens. Before each renewal the front dispatches lease.list with the entry's identity and reads the lease's ttlDeadline. It dispatches lease.renew with the stored TTL only when now + TTL is later than that deadline. A lease the list does not show counts as UNKNOWN_LEASE.
  • Heartbeat. The front sends it through the host link first. On a success reply (An agent opens and closes a device through simlock #472's shared rule) it renews as above, as the map entry's identity. A renewal or list that answers UNKNOWN_LEASE answers LEASE_ENDED_BY_HOST. Any other error from simlock (for example BAD_REQUEST, or a daemon that is stopping) goes back to the CLI as the JSON-RPC error COMMAND_FAILED with simlock's message, and marks nothing. When agent-device answers a lease-gone reason: an entry ended by the agent passes agent-device's answer through; anything else dispatches lease.release for that ID with the requesting token's session, ignores UNKNOWN_LEASE and FORBIDDEN (simlock's own ownership check decides), and answers LEASE_ENDED_BY_HOST. Any other error from that release goes back as COMMAND_FAILED with simlock's message and marks nothing, so the next heartbeat tries again. Any other agent-device answer, including an unreachable daemon, goes back unchanged and renews nothing. A heartbeat naming a lease ID the map does not hold for this principal: a lease-gone reason from agent-device takes the "anything else" path above; a success reply means agent-device still holds that lease (as after a simlock daemon restart), so the front first releases it through the host link with the scope the heartbeat itself carries (runId, leaseProvider, clientId, deviceKey), and only on a success reply releases the simlock lease and answers LEASE_ENDED_BY_HOST; an error or no reply goes back unchanged and touches no lease.
  • Who ended it. An agent opens and closes a device through simlock #472's map entry marks when its lease ended. This task adds who: ended by the agent, or ended by simlock. An agent opens and closes a device through simlock #472's release marks the entry "release in progress" before it dispatches lease.release, and ended by the agent once that returns. While "release in progress" is set, a renewal or heartbeat that finds the lease gone marks nothing, so the agent's own release always ends as ended by the agent. "Release in progress" is a state, not a mark: when that lease.release fails with anything but UNKNOWN_LEASE (An agent opens and closes a device through simlock #472 then keeps the simlock lease), the state is cleared and the entry is not ended, as before the release. The front marks it ended by simlock when a renewal (heartbeat or timer) or its list finds the lease gone, when the "anything else" release above runs, when An agent opens and closes a device through simlock #472's release gets UNKNOWN_LEASE, and when An agent opens and closes a device through simlock #472's allocate check finds the lease unlisted. (So after release --all, the agent's own release gets UNKNOWN_LEASE and marks it ended by simlock; the CLI's close clears its stored lease, so its next open allocates without a heartbeat for the old ID.) A mark, once set, never changes. One function in front/ decides who ended a lease; the heartbeat, the release, the renewal timer and An agent opens and closes a device through simlock #472's allocate check call it. Nothing outside the front writes a mark.
  • A heartbeat for an entry ended by simlock answers LEASE_ENDED_BY_HOST at once. Nothing goes to agent-device or to simlock.
  • A release for an entry ended by simlock. The front sends the release to agent-device and answers its reply (released true or false). It dispatches no lease.release: the simlock lease is already gone.
  • A release naming a lease ID the map does not hold for this principal (as after a simlock daemon restart, when the map is empty). This replaces An agent opens and closes a device through simlock #472's "agent-device only" rule. The front sends the release to agent-device first. On a success reply (released true or false) it dispatches lease.release for that ID with the requesting token's session, and answers agent-device's reply. simlock's own owner check refuses another principal's lease; the front ignores that FORBIDDEN and UNKNOWN_LEASE, which counts as released. On anything else from agent-device it touches no simlock lease and returns agent-device's reply.
  • Renewals while a command runs. A command (An agent opens and closes a device through simlock #472's classifier) whose lease ID the map holds for this principal, in an entry not ended, counts as running from when the front forwards it until its response body ends, errors, or the client hangs up. A command naming an ended entry's lease starts no renewals. A call counts as running for at most lease.maxTtlMs (4 hours by default); after that it renews nothing more, and the lease lives on heartbeats. While a lease has one or more running calls, a Clock timer renews it every TTL/3 (at least 1.67 s, since the TTL is at least 5 s), by the never-shorten rule above. When the stored TTL changes, the timer is re-armed at the new TTL/3 from that moment. When the last call ends, the front cancels the timer and renews once more. A renewal that finds the lease gone (UNKNOWN_LEASE) cancels the timer; any other renewal error logs a warning and keeps the timer, so the next tick tries again. The call's own reply still goes back. A release through the front, and dispose, cancel the timer too.
  • dispose() cancels every renewal timer. From then on the front arms no timer and sends no renewal, final ones included. Everything else about the request path stays as An agent opens and closes a device through simlock #472 left it.
• src/agent-device/testing.ts: FakeAgentDevice can answer a heartbeat with each lease-gone reason, and records each heartbeat's ttlMs.
• docs/CONFIGURATION.md: the new key. docs/HTTP-API.md: the heartbeat answers and the message.

Assumption: an ended entry, either mark, is remembered for one TTL of that lease after it ended (#472's pruning). The notes say "for a while"; one TTL covers a CLI that heartbeats the old ID right after close. After that, a heartbeat takes the "anything else" path and a release the no-entry path. Their answers stay the same, with one exception: a heartbeat for an agent-ended lease then answers LEASE_ENDED_BY_HOST.

Contract and event changes

• configSchema gains agentDevice.leaseTtlCapMs, optional. Protocol +1 at merge.
• No event changes. Each renewal emits lease.renewed, and each release lease.released with reason explicit, as for any client.

Other code on the same state

The state: each front lease's deadline and expiry timer, the front map, the renewal timers, and the agent-device daemon's leases.

• LeaseLifecycle#renew (src/leasing/lease-lifecycle.ts:128-145) sets the deadline to now + the TTL it is given and stores that TTL. The agent's own POST /v1/leases/:id/renew or socket lease.renew reaches it too. A longer renewal by the agent wins: the front skips its renewal while the deadline is later than now + TTL. A shorter one by the agent also stands until the front's next renewal; if that deadline passes first, the lease expires as the agent asked, and simlock ending a simlock lease ends the agent-device session first #474 makes the wipe wait for a running command. When the agent renews between the front's lease.list and its lease.renew, the later renewal sets the deadline; the window is the gap between two in-process dispatches.
• The expiry timer, release --all, simlock nuke (releaseAllDuringMaintenance), a lost device, startup reconcile, and a direct lease.release (the holder's own simlock release <id>) end the simlock lease; simlock wins. A renewal timer that fires after that finds the lease gone, marks the entry ended by simlock and cancels itself. The next heartbeat answers LEASE_ENDED_BY_HOST. The agent-device session stays until simlock ending a simlock lease ends the agent-device session first #474's participant ends it, or until the agent runs disconnect, whose release answers as above.
• simlock ending a simlock lease ends the agent-device session first #474's participant releases the agent-device lease when simlock ends a lease. It writes nothing in the front map. The next heartbeat gets a lease-gone reason from agent-device, or the front's renewal finds the simlock lease gone; either marks the entry ended by simlock here.
• Startup reconcile after a simlock daemon restart (ADR 0019) ends leases whose device is not running and restores the expiry timers of the rest. The front map is empty after a restart, so the first heartbeat for a surviving lease releases it; a lease nobody heartbeats expires at its deadline.
• simlock runs the agent-device daemon #470's supervisor restarts a dead agent-device daemon, and An agent opens and closes a device through simlock #472's host builds a new proxy for it. The restarted daemon has no leases, so each agent's next heartbeat gets a lease-gone reason and takes the "anything else" path.
• The agent-device daemon renews its own lease on each heartbeat with the ttlMs the front sends (the ceiling), and keeps it alive while a command runs (agent-device's heartbeatLease and protected leases in src/daemon/lease-registry.ts). So it outlives the simlock lease.
• Daemon stop: stopAuxiliary disposes the edge, then stops the HTTP gateway, which destroys every socket (src/http/server.ts:203-215). The calls that end then send no final renewal, so no renewal races leasing teardown.

Rules in play

• architecture.md rule 5: renewal is a direct call from the front's request path and its timer, never an event handler.
• architecture.md rule 10: one function decides who ended a lease; each wire rule is read through An agent opens and closes a device through simlock #472's shared/ readers.
• architecture.md rule 12: every exit of a running call (end, error, hang-up, release, failed renewal, dispose) leaves its lease's renewal timer either armed for a running call or cancelled.
• safety.md rule 10: a lease ID from the wire is acted on only through simlock's own ownership checks, and only agent tokens reach the front (An agent opens and closes a device through simlock #472), so no admin bypass applies.
• testing.md rule 2: assert on the lease's deadline and on what FakeAgentDevice and simlock recorded, not on counts of timers.

Tests

Seam: the whole worker daemon in-process with a FakeClock, a FakeDriver, HTTP on loopback and a FakeAgentDevice, as #472 set up in src/agent-device/edge.test.ts; requests sent with fetch. Model: that file and src/daemon/main.test.ts (:1018-1090). The config key: src/core/config.test.ts.

• a heartbeat reaches the agent-device daemon before simlock, carrying ttlMs equal to the ceiling, and once it answers, the simlock lease's deadline is now plus the CLI's ttlMs
• each row of the TTL table gives the simlock lease, its heartbeat renewal and its renewal interval that TTL (one test per row)
• a heartbeat with a longer ttlMs than the allocate's moves the deadline to now plus that TTL, and a following heartbeat with no ttlMs renews with it too
• after the agent's own simlock lease renew with a longer TTL, a heartbeat and a running command's renewals leave that later deadline in place
• a heartbeat the agent-device daemon cannot answer returns that error and leaves the simlock lease's deadline where it was
• with no cap set, a heartbeat whose ttlMs is above lease.maxTtlMs answers COMMAND_FAILED with simlock's BAD_REQUEST message and leaves the deadline where it was, and a following heartbeat with a valid ttlMs renews the lease
• a heartbeat for a lease the agent released through the front answers LEASE_NOT_FOUND, and a following allocate gets a new lease
• one TTL after the agent's release, a heartbeat for that lease answers LEASE_ENDED_BY_HOST
• after simlock release --all, after expiry, and after the device is lost, a heartbeat for the lease answers LEASE_ENDED_BY_HOST with the message and reason above, and makes no new simlock lease
• a heartbeat the agent-device daemon answers LEASE_NOT_FOUND, LEASE_EXPIRED or LEASE_REVOKED, for a lease simlock holds and the agent did not release, releases the simlock lease and answers LEASE_ENDED_BY_HOST
• a heartbeat naming a lease ID the front map lacks, which simlock holds for this token's principal: when agent-device answers a lease-gone reason, the simlock lease is released; when agent-device still holds the lease, it is released with the heartbeat's scope first, then the simlock lease; when agent-device is unreachable, neither lease is touched; each answers as in the table
• a heartbeat from a second token naming the first token's lease ID answers LEASE_ENDED_BY_HOST, and the first token's lease is neither released nor renewed
• after a heartbeat answered LEASE_ENDED_BY_HOST because simlock ended the lease, the next heartbeat answers it again and the agent-device daemon receives nothing for it
• a lease released through the front while a command runs stays ended by the agent: a renewal that fires between the simlock release and the end of the agent's release, and the command's final renewal, mark nothing, and the next heartbeat still answers LEASE_NOT_FOUND
• after simlock release --all, the agent's release answers success, a heartbeat for the old ID then answers LEASE_ENDED_BY_HOST (ended by simlock, not LEASE_NOT_FOUND), and the next allocate gets a new lease; an entry found unlisted by another allocate's check answers its next heartbeat LEASE_ENDED_BY_HOST without reaching the agent-device daemon
• a release for a lease simlock ended answers the agent-device daemon's reply as a success, and the agent-device lease is released
• while a command for the lease runs longer than its TTL, the simlock lease is renewed every TTL/3 and does not expire
• when the command ends, the lease is renewed once more, and its deadline does not move again over the next TTL
• a command that fails, and one whose CLI hangs up, stop the renewals the same way
• with two overlapping commands on one lease, renewals go on until the second one ends
• a renewal that fails because simlock ended the lease mid-command stops the renewals, and the command's reply still reaches the CLI; a renewal refused for another reason (the test makes the daemon's lease.renew dispatch reject once with INTERNAL through a wrapped dispatch) is tried again at the next tick
• an agent release whose lease.release simlock refuses with another error clears "release in progress": the simlock lease stays, and a later renewal that finds it gone marks it ended by simlock
• a heartbeat lowering the TTL from 300 s to 5 s while a command runs re-arms the renewals, and the lease does not expire before the command ends
• a command naming a lease ID the front map does not hold for its token, or the lease of an ended entry, starts no renewals
• a release through the front while a command runs stops that lease's renewals
• a lease whose CLI stops sending heartbeats expires one TTL after its last heartbeat, and its device can be leased again
• after the agent-device daemon restarts, the next heartbeat answers LEASE_ENDED_BY_HOST and releases the simlock lease (the proxy's 409 Proxy instance changed for a request stamped by the old proxy is the proxy's own behaviour, which the CLI retries; not tested here)
• after a simlock daemon restart on the same data directory and the same FakeDriver, so the device keeps running and the test first sees the lease still listed, a heartbeat for a lease made before the restart answers LEASE_ENDED_BY_HOST and releases that simlock lease
• after the same restart (same FakeDriver, lease still listed), a release for a lease made before the restart answers the agent-device daemon's reply as a success and releases that simlock lease
• a release with no map entry from a second token, naming the first token's lease ID, answers the agent-device daemon's reply and leaves the first token's simlock lease in place
• stopping the daemon while a command runs renews nothing once the stop began: the lease's deadline, read after a restart on the same data directory and the same FakeDriver (the lease still listed), is the one it had before the stop
• (config) agentDevice.leaseTtlCapMs is unset by default, accepts 5000 and lease.maxTtlMs, and refuses 4999, a fraction and a value above lease.maxTtlMs, naming the key
• (config) config.get returns agentDevice.leaseTtlCapMs when it is set

Done when

• Every line under Tests is a test, and each fails when the behaviour it names is removed.
• A heartbeat after simlock release --all answers lease ended by simlock; run disconnect, then open, and a heartbeat after the agent's own release answers LEASE_NOT_FOUND.
• A command longer than the lease's TTL keeps its simlock lease, shown with a fake clock.
• docs/CONFIGURATION.md lists agentDevice.leaseTtlCapMs, and docs/HTTP-API.md the heartbeat answers.
• No line needs real hardware: End-to-end tests: agent-device through a simlock worker on real iOS and Android devices #477 runs the expiry, agent-device restart and simlock restart scenarios on iOS and Android.

Out of scope

• Ending the agent-device session when simlock ends a lease (simlock ending a simlock lease ends the agent-device session first #474). Until then a session simlock ended stays until the agent's disconnect.
• Releasing simlock leases whose agents never heartbeat again after a restart; they expire at their TTL.
• Restarting the agent-device daemon (simlock runs the agent-device daemon #470), and the host's new proxy after a restart (An agent opens and closes a device through simlock #472).

Depends on

• An agent opens and closes a device through simlock #472

Approval

• Approved for delivery

Written by an agent.