End-to-end tests: agent-device through a simlock worker on real iOS and Android devices #477

Part of #466.

Scope

After this PR, one slow-lane test file proves every Completion condition of
#466 on a real simulator and a real emulator. It runs the real simlock
daemon, the real agent-device daemon under simlock, and the real
agent-device CLI. The CLI comes from a local build of agent-device's
integration branch. The file is e2e/slow-agent-device.test.ts.

The idle-expiry tests set #473's agentDevice.leaseTtlCapMs, so they take
seconds instead of 5 minutes. This task adds no config key and no product
code.

In short

Each platform gets one simlock daemon for the whole file, serving HTTP on
loopback with the agent-device edge on. Each test creates its own agents.
An agent in these tests is one agent-device CLI with its own HOME and
its own simlock token, connected with
agent-device connect proxy --daemon-base-url http://127.0.0.1:<port>/v1/agent-device.
Tests check the CLI's exit code and JSON output, and simlock's own
list, events and doctor.

flowchart LR
  T["slow-agent-device.test.ts (new)"] --> CLI[agent-device CLI]
  CLI --> F[simlock front]
  F --> L[simlock leasing]
  F --> D[agent-device daemon] --> Dev[simulator / emulator]
  T --> S[simlock CLI: list, events, doctor, release --all]

If this goes wrong: no simlock user notices at once. A break in the
agent-device flow on real devices goes unseen until a user hits it.

Technical spec

Modules touched

• e2e/slow-agent-device.test.ts (new): one describe per platform.
• e2e/helpers/agent-device.ts (new): makes an agent (temp HOME, a new
  agent token from token create --role agent, connect proxy); runs a CLI
  command with --json; starts one in the background with a handle to
  kill it; cleanupAgents() kills the agents' background CLIs and removes
  their homes. Every agent CLI runs with an environment built from the test
  process's environment with every AGENT_DEVICE_* key removed, then
  HOME = the agent's home and AGENT_DEVICE_DAEMON_AUTH_TOKEN = the
  agent's simlock token, so a developer's own agent-device settings never
  point an agent at their own daemon or devices. A module-level afterEach
  calls cleanupAgents() as a backstop, as env.ts does.
• e2e/helpers/env.ts: withSharedDaemon(options). Same options and
  TestEnv as withDaemon, plus teardown() and restart().
  teardown() first reads simlock runs the agent-device daemon #470's pid record (before the existing
  teardown() deletes the home), then runs the existing teardown() in a
  try, then, whatever it threw, if that pid is alive and ps -ww -o args= -p <pid> ends with the
  --simlock-home= marker for this home, sends SIGKILL, so a hung stop
  leaves no agent-device daemon behind. restart() stops the daemon, waits
  up to 30 s for the old simlock daemon's pid to be gone (simlock runs the agent-device daemon #470's stop takes
  up to 12 s) and fails the test if it is not, then starts it again; tests
  use it instead of env.restartDaemon(), so an old daemon never removes
  the new daemon's pid record. withSharedDaemon also gives the shared env
  its own withConfig(config, body), which writes the config and calls
  restart() before and after the body, in place of env.ts's
  withConfig, which uses restartDaemon().
• e2e/helpers/agent-device.ts also has waitForAgentDevice(env, token):
  it polls GET /v1/agent-device/health with an agent token every 500 ms,
  up to 60 s, until the answer is the proxy's health with the agent-device
  daemon's under upstream. The suite calls it after every daemon start or
  restart and after killing the agent-device daemon, so no command meets
  "agent-device is not available on this simlock: starting". It never enters activeEnvs. The test file calls it in each
  describe's beforeAll and calls teardown() in that describe's
  afterAll; a module-level afterAll tears down any shared env still up,
  as a backstop.
• e2e/helpers/index.ts: exports the two helpers.
• docs/internal/agent-rules/toolchain.md: a machine-check row for this
  file (below).
• e2e/README.md: its claim that only two files touch a real emulator.

flowchart LR
  T[slow-agent-device.test.ts] --> H[helpers/agent-device.ts]
  T --> E["helpers/env.ts (withSharedDaemon)"]
  H --> CLI[agent-device CLI]
  E --> D[simlock daemon]

One request through the processes, with the failure reply:

sequenceDiagram
  participant T as test
  participant A as agent-device CLI
  participant F as simlock front
  participant D as agent-device daemon
  participant S as simlock CLI
  A->>F: open --platform ios (simlock token)
  F->>F: lease.request(ttlMs = min(CLI ttl, cap))
  F->>D: allocate, open
  F-->>A: opened
  T->>S: simlock release --all --yes
  S-->>T: released
  A->>F: snapshot
  F-->>A: error: lease ended by simlock
  A-->>T: exit non-zero

Every shared daemon starts with:

• http.enabled: true, http.host: "127.0.0.1", a port from
  freeLoopbackPort();
• agentDevice.enabled: true, agentDevice.command = the path in
  SIMLOCK_E2E_AGENT_DEVICE. The same path is every agent's CLI. It is
  bin/agent-device.mjs in a build of the agent-device integration branch,
  at the commit simlock's @agent-device/proxy dependency pins (An agent opens and closes a device through simlock #472). That
  build reports 0.21.25-dev, which simlock runs the agent-device daemon #470 accepts;
• capacity.config.limits.<platform>.maxDevices: 1 and maxRunning: 1, so
  "the only device" is one known device;
• driver: "real", so it has its own device root and adb server port.

One platform at a time. The iOS describe runs first, then the
Android one (the file runs only through scripts/slow-e2e.sh, with no file
or test parallelism). Each describe starts its shared daemon in
beforeAll and tears it down, with its devices and its agent-device daemon,
in its own afterAll. So the iOS daemon and its simulator are gone before
the Android daemon starts, and each daemon's RAM budget sees the whole
machine with no other simlock device on it.

Warm-up. In beforeAll (hook time limit 420 s), it leases one device
with plain simlock lease --detach --platform <p> --agent-id warmup-1, releases it, and waits
up to 240 s for it to be ready. No open then pays a cold create inside
the CLI's 5.5-minute allocate deadline. The tests that need the device
limit at 2 run last, in a nested describe. Its beforeAll (420 s) writes
maxDevices: 2 and maxRunning: 2 and calls restart(), then does the same warm-up with two detached leases
held at once, with --agent-id warmup-1 and --agent-id warmup-2 (one
lease per requester, and under the hardware verifier every call would
otherwise share one requester), so both devices are ready before the first of those tests.
Its afterAll restores the limits and calls restart().

No lease between tests. Each test closes its agents and releases the
plain leases it took. Then the describe's afterEach (time limit 300 s),
in this order: cleanupAgents(); simlock release --all --yes; and a wait
of up to 240 s until simlock list --leases and simlock list --requests
are empty and every device in simlock list --devices is ready. A test
that fails part way therefore leaves nothing that makes the next one queue.

Free in these tests means: the device is ready in
simlock list --devices and no lease in simlock list --leases names it.

Every wait has a time limit, in the test or hook timeout and in the
wait itself: a CLI command 330 s (open) or 60 s (others); a daemon
restart (restart()) 90 s; a wait for an event, a state or a pid 60 s
unless the test names another; a sample loop 240 s. Each test sets its own
timeout: 900 s for a test that restarts a daemon or opens twice, 600 s for
the others.

Tests that need other config wrap their body in the shared env's
withConfig(...), which restarts the daemon with restart() and restores
the config after. After each start or restart the test calls
waitForAgentDevice, except the wrong-version test, where agent-device
never becomes healthy: it polls simlock doctor --json every 500 ms, up
to 60 s, until it reports agent-device-unavailable, and only then runs
open.

The fixture apps come from #476: e2e/fixtures/appduct/ios/AppductPlayground.app
and e2e/fixtures/appduct/android/AppductPlayground.apk.

The new row in toolchain.md's machine checks:

Each describe is skipped when SIMLOCK_E2E_AGENT_DEVICE is unset or its
platform check fails. The platform checks, run when the file loads:

• iOS: process.platform is darwin, process.arch is arm64, and
  xcrun simctl list runtimes lists an iOS runtime (toolchain.md's iOS
  check plus Apple silicon). On an Intel Mac the iOS describe is skipped.
• Android: command -v emulator adb finds both (toolchain.md's Android
  check).

A set path that does not exist fails the describe's beforeAll with a
message naming the path; it is not skipped.

#470 reads the version with <agentDevice.command> --version, after it checks
the install's layout. So the wrong-version test builds a temp package that
passes that check: package.json naming agent-device, an empty
dist/src/daemon.js, and an executable bin/agent-device.mjs that prints
0.9.2 for --version and exits 0. agentDevice.command points at that
bin/agent-device.mjs.

connect proxy sends no request: it saves the profile and prints that
access is checked by the first remote command (agent-device
src/cli/connection/connect-provider-adapters.ts:69, branch
feat/proxy-authenticate). So while agent-device is unavailable the
helper's connect proxy succeeds, and the first request, open, gets the
"not available" answer.

#470's pid record is <SIMLOCK_HOME>/agent-device-daemon.json, with a pid
field. The test reads the agent-device daemon's pid from it. The test that
kills the simlock daemon registers, right after it reads that pid, an
onTestFinished cleanup: if the pid is still alive and ps -o args= -p <pid>
shows the record's entry, it sends SIGKILL. So a failure between the kill
and the restart leaves no agent-device daemon behind.

The release-all test reads the agent's sessions through simlock, as the
agent: agent-device session list --json from that agent's CLI.
session_list needs no lease in agent-device, and the proxy scopes it to
the agent's tenant (its token's principal, the token id), so it keeps
working after simlock ended the lease. It reads when the wipe began from
the device.reclaimed event (its timestamp minus its duration), as the
running-command test does; this works the same on iOS, where the wipe
shuts the simulator down, and on Android, where it restores a snapshot on
the running emulator.

The shard test writes a one-file suite to a temp dir: a .ad script
holding one snapshot step (agent-device's .ad line grammar).

wait <ms> runs under the session's execution lock: agent-device's
command registry does not mark wait exempt from it
(packages/command-registry/src/registry.ts, the wait descriptor), and
PR #3480 makes a release wait for that lock.

Assumption: every agent-device CLI command is its own process, and the
agent-device session lives in the daemon. So "an agent whose CLI is killed
mid-session" is tested as an agent that sends no more commands.

The tests use agent-device's command names click and press, never
tap: click 'id="tab-status"'.

Tests assert the messages #472 to #475 set: lease ended by simlock; run disconnect, then open, agent-device is not available on this simlock: <reason>, and the refused: … texts. For boot and shutdown it is
agent-device's policy denial: UNAUTHORIZED with details.reason
DAEMON_POLICY_DENIED.

Contract and event changes

None. The task touches no file under src/. No event changes.

Other code on the same state

• e2e/helpers/env.ts activeEnvs and its afterEach: shared envs are kept
  in their own set, so the per-test teardown never stops a shared daemon.
  The shared env's withConfig and restart() replace env.ts's
  withConfig and restartDaemon() for it.
• scripts/slow-e2e.sh: one slow run per machine; this file runs only
  through it, one file at a time, so the iOS and Android shared daemons
  never run at once.
• simlock's capacity limits and RAM budget: the shared daemon pins
  maxDevices/maxRunning to 1, then 2 inside withConfig. The RAM
  budget still applies; the toolchain row's RAM check keeps a machine
  without room for two devices from running the line.
• e2e/helpers/ios-device-set.ts emptyDeviceSet: the shared teardown
  empties the iOS device set as the per-test one does.
• The agent-device daemon's pid record (simlock runs the agent-device daemon #470): the test only reads it.

Rules in play

• testing.md: each title is a claim the body proves. Each refusal starts
  from the same request accepted, then changes one thing. The
  running-command test must go red when the wipe starts before the command
  ends, and the release-all sampling must go red when the device is shut
  down while the session still exists. Rule 2: nothing a test leaves
  behind decides the next test's result, hence the afterEach.
• safety.md: no implicit downloads (no test asks for one); tests never
  touch the user's devices (own SIMLOCK_HOME, adb port, agent HOMEs).
• toolchain.md: the slow lane runs only through scripts/slow-e2e.sh.
• architecture.md: every wait on another process has a deadline.

Tests

Seam: the slow lane, real processes, through scripts/slow-e2e.sh; model
e2e/slow-android-smoke.test.ts and e2e/slow-ios-slim.test.ts.

Slow tests. Each runs in the iOS describe (tags slow, ios) and in the
Android describe (tags slow, android), in this order:

• an agent opens a device, installs the Appduct fixture app as an upload,
  opens it by its bundle id (com.callstack.appduct.playgroundnative on
  iOS, com.callstack.appduct.playground on Android, so a fixture refresh
  that changed the id fails here), takes a snapshot that shows tab-status, clicks tab-status,
  saves a PNG screenshot and closes; within 240 s the device is free
  (ready, and no lease names it)
• with this working agent-device, simlock doctor --json reports no
  agent-device-unavailable finding and no finding whose message names
  agent-device
• close releases the simlock lease before the CLI exits (one read of
  simlock list --leases, no polling), and a following open works
• after an accepted upload install and snapshot, install from a
  remote: path, boot, shutdown and snapshot --ios-simulator-device-set <dir>
  (on Android also --android-device-allowlist <serial>) are refused, and the
  device is still leased and answers snapshot
• simlock release --all ends the agent's agent-device session before the
  device is wiped: sampling session list every 250 ms until the device is
  ready, the first sample with no session comes before the wipe began
  (device.reclaimed timestamp minus its duration), and the last sample
  finds no session. Another agent's open then
  gets that same device (the same deviceId in simlock list --leases),
  where opening the fixture app fails because the wipe removed it; the
  first agent's next command fails with "lease ended by simlock"
• a wait 20000 running when simlock release --all is called exits 0 after
  at least 20 s, and the wipe starts after it exits (the device.reclaimed
  timestamp minus its duration)
• with the only device leased, a second agent's open waits in
  simlock list --requests; killing its CLI empties the queue, and once the
  device is free no lease.granted names that request's requester while a
  third agent's open gets the device
• when the agent-device daemon is killed, a new one with a new pid runs; the
  agent's next command fails with "lease ended by simlock", and its simlock
  lease is released
• after the simlock daemon is killed and started again (then waitForAgentDevice), the old agent-device
  daemon's pid is gone within 60 s; the agent's next command fails with "lease ended by
  simlock", and its old simlock lease is released
• with agentDevice.leaseTtlCapMs 20000, an agent that sends no more
  commands gets lease.expired at most 25 s after its last command, and
  another agent's open then gets the device
• with agentDevice.leaseTtlCapMs 20000, after a graceful simlock daemon
  restart (90 s limit) an old simlock lease whose agent sends nothing gets
  lease.expired at most 5 s after the later of (its last renewal + 20 s)
  and the moment the restarted daemon answers simlock status
• with agentDevice.command pointing at an agent-device that reports
  0.9.2, the daemon starts and grants a plain simlock lease --detach; an
  agent's open fails with agent-device is not available on this simlock: found 0.9.2, need 0.21.25 or later; simlock doctor reports
  agent-device-unavailable with found 0.9.2, need 0.21.25 or later
• with the device limit raised to 2 and both devices ready, devices lists nothing before open, and after open only the device it bound, not the other ready device
• with the device limit raised to 2, two agents on two tokens get two
  different devices (two deviceIds in simlock list --leases); B's snapshot naming its own device is accepted and
  naming A's (--udid, Android --serial) is refused; after B's close,
  A's snapshot still works
• with the device limit raised to 2, both devices booted, agent A holding
  one and agent B (another token) the other: A's test <suite> --shard-all 1
  with no --device passes, and A's test <suite> --shard-all 2 with no
  --device fails with agent-device's test sharding requires 2 devices, but only 1 matched (without simlock's filter both booted devices would
  match and it would run)

Completion conditions of #466, each to its test:

Done when

• On a Mac with an iOS runtime, Apple silicon, at least 7 GiB of RAM and
  SIMLOCK_E2E_AGENT_DEVICE set as the toolchain row says (a build at the
  pinned commit): scripts/slow-e2e.sh e2e/slow-agent-device.test.ts
  passes every test in the iOS describe of e2e/slow-agent-device.test.ts.
• On a machine where command -v emulator adb finds both, with at least
  12 GiB of RAM and SIMLOCK_E2E_AGENT_DEVICE set the same way: the same
  command passes every test in the Android describe.
• With SIMLOCK_E2E_AGENT_DEVICE unset, scripts/slow-e2e.sh e2e/slow-agent-device.test.ts
  writes <log>.exit 0 and its log shows both describes skipped, not failed.
• With SIMLOCK_E2E_AGENT_DEVICE=/nonexistent, on a machine that passes at
  least one platform check, the same command fails, and its log shows that
  describe's beforeAll failing with a message naming /nonexistent.

Out of scope

• Gateway mode and its tests.
• Physical devices.
• Running the slow lane in CI.
• Pinning agent-device as a dev dependency; that waits for its release.
• New error messages: the tests use the ones simlock runs the agent-device daemon #470 to The agent-device front refuses requests that reach past the agent's device, and devices lists only its own #475 merged.

Depends on

• The agent-device front refuses requests that reach past the agent's device, and devices lists only its own #475
• Commit prebuilt Appduct playground apps as end-to-end fixtures #476

Approval

• Approved for delivery

Written by an agent.