Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

<!-- Intro rule: category line, job pitch with the platform list, works-with/proof lines, nothing else. Per-platform transports, tool names, vendor lists, and support caveats belong in "How it works" and website/docs, never here. Every name in the proof lines must link to public evidence. -->

Let your coding agent verify its changes in the running app. `agent-device` lets agents inspect, control, debug, and verify apps on iOS, Android, and HarmonyOS (simulators, emulators, and physical devices), plus tvOS, Android TV, Amazon Vega OS TV (Vega Virtual Device), web, macOS, and Linux. Agents read token-efficient accessibility snapshots instead of reasoning over screenshots alone, act through refs and selectors, and save evidence for review. It also coordinates device access across parallel agent worktrees and connects to remote device clouds.
Let your coding agent verify its changes in the running app. `agent-device` lets agents inspect, control, debug, and verify apps on iOS, Android, and HarmonyOS (simulators, emulators, and physical devices), plus tvOS, Android TV, Fire TV (Fire OS), Amazon Vega OS TV (Vega Virtual Device), web, macOS, and Linux. Agents read token-efficient accessibility snapshots instead of reasoning over screenshots alone, act through refs and selectors, and save evidence for review. It also coordinates device access across parallel agent worktrees and connects to remote device clouds.

Works with Claude Code, Codex, Cursor, Windsurf, Cline, Goose, and any agent that can run a CLI or connect over MCP, or as the runtime under agents you build with the [AI SDK](https://oss.callstack.com/agent-device/docs/ai-sdk) or [Eve](https://oss.callstack.com/agent-device/docs/eve). Developers at [Expensify](https://www.callstack.com/blog/how-expensify-uses-agent-device-for-mobile-bug-evidence-and-profiling), [Shopify](https://x.com/mustafa01ali/status/2036577353178943826), and [others](#who-uses-agent-device) use it to verify their apps.

Expand Down Expand Up @@ -155,7 +155,7 @@ The same session and evidence model works at every step: the agent explores the

## How it works

`agent-device` keeps device state in sessions. It uses a local accessibility bridge for iOS Simulator snapshots and XCTest for iOS interactions, physical iOS, and tvOS; ADB and the snapshot helper on Android; HDC and ArkUI `uitest` on HarmonyOS; Vega CLI/VDA on the Vega Virtual Device; a local helper on macOS, including Electron and other Chromium-based apps through the accessibility tree; and AT-SPI on Linux.
`agent-device` keeps device state in sessions. It uses a local accessibility bridge for iOS Simulator snapshots and XCTest for iOS interactions, physical iOS, and tvOS; ADB and the snapshot helper on Android, Android TV, and Fire OS; HDC and ArkUI `uitest` on HarmonyOS; Vega CLI/VDA on the Vega Virtual Device; a local helper on macOS, including Electron and other Chromium-based apps through the accessibility tree; and AT-SPI on Linux.

Support depth varies by target. Some backends, such as HarmonyOS and Vega OS, cover a subset of commands; run `agent-device capabilities --platform <platform>` to see what a target supports.

Expand Down
26 changes: 26 additions & 0 deletions packages/platform-android/src/__tests__/app-helpers.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,32 @@ async function withMockedAdbScript(script: string, run: () => Promise<void>): Pr
}
}

test('listAndroidAppsWithAdb queries both launcher categories for a TV target', async () => {
// Fire OS apps register under either launcher category, so a TV listing merges both queries.
const calls: (readonly string[])[] = [];
const adb: AndroidAdbExecutor = async (args) => {
calls.push(args);
if (args.includes('query-activities')) {
return {
exitCode: 0,
stdout: args.includes('android.intent.category.LAUNCHER')
? 'com.example.fireosapp/.MainActivity\n'
: '',
stderr: '',
};
}
return { exitCode: 0, stdout: '', stderr: '' };
};

const apps = await listAndroidAppsWithAdb(adb, { filter: 'all', target: 'tv' });

assert.deepEqual(apps, [{ package: 'com.example.fireosapp', name: 'Fireosapp' }]);
assert.deepEqual(calls.map((call) => call.at(-1)).sort(), [
'android.intent.category.LAUNCHER',
'android.intent.category.LEANBACK_LAUNCHER',
]);
});

test('listAndroidAppsWithAdb uses an injected executor', async () => {
const calls: (readonly string[])[] = [];
const adb: AndroidAdbExecutor = async (args) => {
Expand Down
118 changes: 117 additions & 1 deletion packages/platform-android/src/__tests__/app-lifecycle-open.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { test } from 'vitest';
import assert from 'node:assert/strict';
import { openAndroidApp } from '../app-lifecycle.ts';
import { listAndroidApps, openAndroidApp } from '../app-lifecycle.ts';
import { withAndroidAdbProvider } from '../adb-executor.ts';
import type { DeviceInfo } from '@agent-device/kernel/device';
import { AppError } from '@agent-device/kernel/errors';
Expand Down Expand Up @@ -755,6 +755,122 @@ test('openAndroidApp fallback resolve-activity includes MAIN/LAUNCHER flags', as
);
});

test('openAndroidApp TV fallback carries the resolved LAUNCHER category into the component launch', async () => {
// A Fire OS app that declares no LEANBACK_LAUNCHER activity resolves under LAUNCHER only; the
// fallback launch must carry that category, not the target's LEANBACK_LAUNCHER.
await withFakeAdb(
(args) => {
if (args[0] === 'shell' && args[1] === 'am' && args[2] === 'start') {
if (args.includes('-p')) {
return [
'Starting: Intent { act=android.intent.action.MAIN cat=[android.intent.category.DEFAULT,android.intent.category.LEANBACK_LAUNCHER] pkg=com.example.tvapp }',
'Error: Activity not started, unable to resolve Intent { act=android.intent.action.MAIN cat=[android.intent.category.DEFAULT,android.intent.category.LEANBACK_LAUNCHER] flg=0x10000000 pkg=com.example.tvapp }',
].join('\n');
}
return 'Status: ok';
}
if (
args[0] === 'shell' &&
args[1] === 'cmd' &&
args[2] === 'package' &&
args[3] === 'resolve-activity'
) {
if (args.includes('android.intent.category.LEANBACK_LAUNCHER')) {
return { stdout: '', stderr: 'No activity found', exitCode: 1 };
}
return 'com.example.tvapp/com.example.tvapp.MainActivity';
}
return undefined;
},
async ({ calls, device }) => {
await openAndroidApp(device, 'com.example.tvapp');
assert.deepEqual(calls, [
[
'shell',
'am',
'start',
'-W',
'-a',
'android.intent.action.MAIN',
'-c',
'android.intent.category.DEFAULT',
'-c',
'android.intent.category.LEANBACK_LAUNCHER',
'-p',
'com.example.tvapp',
],
// resolve-activity tries LEANBACK_LAUNCHER first on a TV target...
[
'shell',
'cmd',
'package',
'resolve-activity',
'--brief',
'-a',
'android.intent.action.MAIN',
'-c',
'android.intent.category.LEANBACK_LAUNCHER',
'com.example.tvapp',
],
// ...then LAUNCHER, which resolves the component.
[
'shell',
'cmd',
'package',
'resolve-activity',
'--brief',
'-a',
'android.intent.action.MAIN',
'-c',
'android.intent.category.LAUNCHER',
'com.example.tvapp',
],
// The fallback launch uses the category the component resolved under.
[
'shell',
'am',
'start',
'-W',
'-a',
'android.intent.action.MAIN',
'-c',
'android.intent.category.DEFAULT',
'-c',
'android.intent.category.LAUNCHER',
'-n',
'com.example.tvapp/com.example.tvapp.MainActivity',
],
]);
},
{ device: { ...ANDROID_EMULATOR, target: 'tv' } },
);
});

test('listAndroidApps on a TV target queries LEANBACK_LAUNCHER then LAUNCHER', async () => {
await withFakeAdb(
(args) => {
if (args.includes('query-activities')) {
return args.includes('android.intent.category.LAUNCHER')
? 'com.example.fireosapp/.MainActivity\n'
: '';
}
return undefined;
},
async ({ calls, device }) => {
const apps = await listAndroidApps(device, 'all');
assert.deepEqual(apps, [{ package: 'com.example.fireosapp', name: 'Fireosapp' }]);
const queriedCategories = calls
.filter((call) => call.includes('query-activities'))
.map((call) => call.at(-1));
assert.deepEqual(queriedCategories, [
'android.intent.category.LEANBACK_LAUNCHER',
'android.intent.category.LAUNCHER',
]);
},
{ device: { ...ANDROID_EMULATOR, target: 'tv' } },
);
});

// In-process replacement for the old PATH-stub default adb script: any
// `am start` succeeds with the "Status: ok" marker (so the primary launch
// never falls back to resolve-activity), everything else is empty success.
Expand Down
29 changes: 9 additions & 20 deletions packages/platform-android/src/app-helpers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,7 @@ import {
parseAndroidUserInstalledPackages,
} from './app-parsers.ts';
import { inferAndroidAppName } from './app-deployment-resolution.ts';

const ANDROID_LAUNCHER_CATEGORY = 'android.intent.category.LAUNCHER';
const ANDROID_LEANBACK_CATEGORY = 'android.intent.category.LEANBACK_LAUNCHER';
import { androidLauncherCategories } from './launcher-categories.ts';

export type AndroidAppListFilter = AppsFilter;
export type AndroidAppListTarget = 'mobile' | 'tv' | 'auto';
Expand Down Expand Up @@ -38,28 +36,19 @@ async function listAndroidLaunchablePackagesWithAdb(
): Promise<Set<string>> {
const discoveredPackages = (
await Promise.all(
resolveAndroidLaunchCategoriesForAdb(target).map(async (category) => {
const result = await runAdbShell(adb, buildAndroidQueryActivitiesArgs(category), {
allowFailure: true,
});
return result.exitCode === 0 ? parseAndroidLaunchablePackageOutput(result.stdout) : [];
}),
androidLauncherCategories(target, { includeFallbackCategories: true }).map(
async (category) => {
const result = await runAdbShell(adb, buildAndroidQueryActivitiesArgs(category), {
allowFailure: true,
});
return result.exitCode === 0 ? parseAndroidLaunchablePackageOutput(result.stdout) : [];
},
),
)
).flat();
return new Set(discoveredPackages);
}

function resolveAndroidLaunchCategoriesForAdb(target: AndroidAppListTarget): string[] {
switch (target) {
case 'mobile':
return [ANDROID_LAUNCHER_CATEGORY];
case 'tv':
return [ANDROID_LEANBACK_CATEGORY];
default:
return [ANDROID_LAUNCHER_CATEGORY, ANDROID_LEANBACK_CATEGORY];
}
}

function buildAndroidQueryActivitiesArgs(category: string): string[] {
return [
'cmd',
Expand Down
51 changes: 18 additions & 33 deletions packages/platform-android/src/app-lifecycle.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,9 @@ import {
parseAndroidLaunchablePackages,
parseAndroidUserInstalledPackages,
} from './app-parsers.ts';
import { androidLaunchCategory, androidLauncherCategories } from './launcher-categories.ts';
import { getAndroidAppState, getAndroidResumedActivity } from './window-state.ts';

const ANDROID_LAUNCHER_CATEGORY = 'android.intent.category.LAUNCHER';
const ANDROID_LEANBACK_CATEGORY = 'android.intent.category.LEANBACK_LAUNCHER';
const ANDROID_DEFAULT_CATEGORY = 'android.intent.category.DEFAULT';
const ANDROID_LOCALHOST_HOSTNAMES = new Set(['localhost', '127.0.0.1', '::1', '[::1]']);
/** Owns localhost URL reverses, so a teardown that removes only owned mappings removes them too. */
Expand All @@ -53,8 +52,8 @@ export async function listAndroidApps(

async function listAndroidLaunchablePackages(device: DeviceInfo): Promise<Set<string>> {
const packages = new Set<string>();
for (const category of resolveAndroidLaunchCategories(device, {
includeFallbackWhenUnknown: true,
for (const category of androidLauncherCategories(device.target, {
includeFallbackCategories: true,
})) {
const result = await runAndroidShell(
device,
Expand All @@ -80,26 +79,6 @@ async function listAndroidLaunchablePackages(device: DeviceInfo): Promise<Set<st
return packages;
}

function resolveAndroidLauncherCategory(device: DeviceInfo): string {
return resolveAndroidLaunchCategories(device)[0] ?? ANDROID_LAUNCHER_CATEGORY;
}

function resolveAndroidLaunchCategories(
device: DeviceInfo,
options: { includeFallbackWhenUnknown?: boolean } = {},
): string[] {
if (device.target === 'tv') {
return [ANDROID_LEANBACK_CATEGORY];
}
if (device.target === 'mobile') {
return [ANDROID_LAUNCHER_CATEGORY];
}
if (options.includeFallbackWhenUnknown) {
return [ANDROID_LAUNCHER_CATEGORY, ANDROID_LEANBACK_CATEGORY];
}
return [ANDROID_LAUNCHER_CATEGORY];
}

async function listAndroidUserInstalledPackages(device: DeviceInfo): Promise<string[]> {
const result = await runAndroidShell(device, ['pm', 'list', 'packages', '-3']);
return parseAndroidUserInstalledPackages(result.stdout);
Expand Down Expand Up @@ -225,7 +204,7 @@ async function startAndroidApp(
const helperLaunch = await tryAndroidHelperLaunch(device, app, options);
if (helperLaunch) return helperLaunch;
const resolved = await resolveAndroidApp(device, app);
const launchCategory = resolveAndroidLauncherCategory(device);
const launchCategory = androidLaunchCategory(device.target);
if (resolved.type === 'intent') {
return await openAndroidIntent(device, resolved.value, options);
}
Expand Down Expand Up @@ -405,16 +384,22 @@ async function openAndroidPackage(
if (primaryResult.exitCode === 0 && !isAmStartError(primaryResult.stdout, primaryResult.stderr)) {
return primaryResult;
}
const component = await resolveAndroidLaunchComponent(device, packageName);
if (!component) {
const resolvedComponent = await resolveAndroidLaunchComponent(device, packageName);
if (!resolvedComponent) {
if (!(await isAndroidPackageInstalled(device, packageName))) {
throw buildAndroidPackageNotInstalledError(packageName);
}
throw androidAdbResultError(`Failed to launch ${packageName}`, primaryResult);
}
// The fallback launches under the category the component actually resolved under, which on a TV
// target can be LAUNCHER for an app that declares no LEANBACK_LAUNCHER activity.
return await runAndroidShell(
device,
buildAndroidActivityLaunchArgs(component, launchCategory, options),
buildAndroidActivityLaunchArgs(
resolvedComponent.component,
resolvedComponent.category,
options,
),
);
}

Expand Down Expand Up @@ -516,10 +501,10 @@ function looksLikeMissingAndroidPackageOutput(output: string): boolean {
async function resolveAndroidLaunchComponent(
device: DeviceInfo,
packageName: string,
): Promise<string | null> {
const categories = Array.from(
new Set(resolveAndroidLaunchCategories(device, { includeFallbackWhenUnknown: true })),
);
): Promise<{ component: string; category: string } | null> {
const categories = androidLauncherCategories(device.target, {
includeFallbackCategories: true,
});
for (const category of categories) {
const result = await runAndroidShell(
device,
Expand All @@ -540,7 +525,7 @@ async function resolveAndroidLaunchComponent(
continue;
}
const component = parseAndroidLaunchComponent(result.stdout);
if (component) return component;
if (component) return { component, category };
}
return null;
}
Expand Down
31 changes: 31 additions & 0 deletions packages/platform-android/src/inventory.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ test.each([
test.each([
['feature:android.software.leanback_only\nfeature:android.hardware.wifi\n', 'tv'],
['feature:android.hardware.type.television=1\n', 'tv'],
['feature:android.hardware.wifi\nfeature:amazon.hardware.fire_tv\n', 'tv'],
['feature:android.software.leanback_extra\nfeature:android.hardware.wifi\n', 'mobile'],
] as const)(
'Android inventory reads the feature list %j as a %s target',
Expand All @@ -168,6 +169,36 @@ test.each([
},
);

test('Android inventory classifies a device by has-feature amazon.hardware.fire_tv before the feature list', async () => {
// The characteristics probe says nothing, no AOSP leanback/television feature answers, and
// only the Amazon Fire TV feature answers true: the device must be a tv target from the
// has-feature probe, without the pm list features fallback deciding it.
const featureListProbeCalls: string[] = [];
const host = createHost(async (request) => {
const key = request.args.join('\0');
if (key === 'devices\0-l')
return result('List of devices attached\nAFTKMST12 device model:AFTKMST12\n');
if (key.includes('pm\0list\0features')) {
featureListProbeCalls.push(key);
return result('feature:android.hardware.wifi\n');
}
if (key.includes('has-feature')) {
return result(key.includes('amazon.hardware.fire_tv') ? 'true\n' : 'false\n');
}
const probe = RUNNING_EMULATOR_PROBES.find(([fragment]) => key.includes(fragment));
if (probe) return result(probe[1]);
throw new Error(`Unexpected command: ${request.executable} ${request.args.join(' ')}`);
});

const [device] = await createAndroidInventory(host).discover(
{ androidAvdSelection: 'running-only' },
scope,
);

assert.equal(device?.target, 'tv');
assert.deepEqual(featureListProbeCalls, []);
});

test('Android inventory fails closed when adb is unavailable', async () => {
const host = createHost(async () => result(''), { adb: undefined });
await assert.rejects(
Expand Down
4 changes: 4 additions & 0 deletions packages/platform-android/src/inventory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,14 @@ const DEVICE_DESCRIPTION_TIMEOUT_MS = 2_000;
const DEVICE_DESCRIPTION_PROBE = shellFragment(
'getprop ro.product.model; getprop ro.build.version.release',
);
// Fire OS devices declare amazon.hardware.fire_tv and may not declare the AOSP leanback
// features, so TV detection must read it as a TV signal too (Amazon: "Identify Fire TV
// Devices", adb shell pm list features).
const TV_FEATURES = [
'android.software.leanback',
'android.software.leanback_only',
'android.hardware.type.television',
'amazon.hardware.fire_tv',
] as const;

type AndroidInventoryContext = Readonly<{
Expand Down
Loading
Loading