# Read and drive the screen

Canonical URL: https://autonom.yatsuk.me/docs/screen/

Your agent reads the screen as a compact tree of controls and acts on a control by what it says. The same verbs work on an Android emulator and an iOS Simulator.

**Try it**

```sh
autonom ui tree
```

**Needs** [a session on one target](https://autonom.yatsuk.me/docs/sessions/index.md)

## How it works

The tree comes from UI Automator on Android and from idb on iOS. Autonom turns both into one compact node: a ref, a role, text, desc, an id, bounds and a few flags. Every read belongs to the [session that owns the target](https://autonom.yatsuk.me/docs/sessions/index.md), and is saved with it.

![Four boxes in a loop: Read with ui tree, Act with ui tap, type or swipe, Settle with ui wait --settled, and Check with a tree or a screenshot, then back to Read.](https://autonom.yatsuk.me/docs/screen/loop.svg)

Every action ends with a fresh look at the screen, and refs from an old tree are never reused.

Selectors come first: text, desc, resource id, class, package or role, matched exactly, by substring or by regular expression. Coordinates are the fallback. A [flow](https://autonom.yatsuk.me/docs/flows/index.md) uses the same selectors, so what works here replays there.

One difference between the platforms matters more than any other: where the visible label lives. An Android control keeps it in `text`, an iOS control in `desc`, and Flutter on Android in `desc` too.

A tap that exits 0 has not proven anything yet. Compare the tree or [a screenshot before and after](https://autonom.yatsuk.me/docs/evidence/index.md), which is why the loop ends in a check.

## Examples

Every output here comes from Autonom’s test device, with a fixture standing in for each screen, so you can rerun it without an emulator. Reproduce these outputs, at the end of this section, has the exact commands.

### See what is on screen

**Goal:** one line per control, then every field of the one you want.

1. **Read the outline.** Only the controls your agent can act on, one line each.

   ```sh
   autonom ui tree --format outline --interactable
   ```

2. **Read the full nodes.** The JSON form of the same screen carries every field.

   ```sh
   autonom ui tree
   ```

You should see five controls in the outline field, printed here one per line, then the Sign in with Email node with its label in `desc`.

*Output of autonom ui tree --format outline --interactable, then autonom ui tree*

```
$ autonom ui tree --format outline --interactable
n3 button "Back" @21,163 105x105 [clickable]
n7 view "P Sign in with Partner ID" @95,1495 890x126 [clickable]
n8 view "Sign in with Email" @95,1631 890x126 [clickable]
n9 checkbox @95,1936 63x63 [clickable]
n11 button "Privacy Policy" @344,1962 205x63 [clickable]
$ autonom ui tree
{
  "ok": true,
  …
  "nodes": […, {"ref": "n8", "role": "view", "text": null, "desc": "Sign in with Email", …, "bounds": [95, 1631, 985, 1757], "clickable": true, …}, …],
  …
}
```

**What to look at.** Each outline line is ref, role, label, position, size and flags. On this Flutter screen `n8` has `text` null and its label in `desc`, so select it with `--desc`.

**Show full output**

```
{
  "ok": true,
  "source": "device",
  "target": "emulator-5554",
  "count": 5,
  "platform": "android",
  "target_id": "emulator-5554",
  "serial": "emulator-5554",
  "screen": [
    1080,
    1920
  ],
  "saved": "/tmp/autonom-screen/sessions/s_ff8096586d/trees/0001_214241.json",
  "format": "outline",
  "outline": "n3 button \"Back\" @21,163 105x105 [clickable]\nn7 view \"P Sign in with Partner ID\" @95,1495 890x126 [clickable]\nn8 view \"Sign in with Email\" @95,1631 890x126 [clickable]\nn9 checkbox @95,1936 63x63 [clickable]\nn11 button \"Privacy Policy\" @344,1962 205x63 [clickable]"
}
```

![Two node cards. The Android card shows text Settings, desc Open settings, select with --text. The iOS card shows text null, desc General, select with --desc.](https://autonom.yatsuk.me/docs/screen/label.svg)

An Android control keeps its label in text; an iOS control keeps it in desc.

### Find and tap a Flutter control

**Goal:** tap a Flutter Log In button on Android, whose label lives in `desc`.

1. **Find it by its label.** An exact match makes sure only this button answers.

   ```sh
   autonom ui find --desc "Log In" --mode exact
   ```

2. **Tap it.** The same label selects the same node.

   ```sh
   autonom ui tap --desc "Log In"
   ```

You should see one match at index 0, then a tap on the same node.

*Output of autonom ui find --desc, then autonom ui tap --desc*

```
$ autonom ui find --desc "Log In" --mode exact
{
  "ok": true,
  "source": "device",
  …
  "count": 1,
  "matches": [{"ref": "n3", "role": "view", "text": null, "desc": "Log In", …, "bounds": [100, 300, 500, 420], …, "index": 0}],
  …
}
$ autonom ui tap --desc "Log In"
{"ok": true, "x": 300, "y": 360, "ref": "n3", "backend": "adb", …}
```

**What to look at.** `text` is null and `desc` holds the label, so `--desc` finds it. The tap acts on the same `ref`, `n3`, at the center of its bounds, and names the tool that touched the screen.

### Read where iOS keeps a label

**Goal:** see which field holds the label on an iOS Settings screen before you select.

1. **Read the tree.** On iOS it comes from the accessibility hierarchy.

   ```sh
   autonom ui tree
   ```

You should see the General row with `text` null and its label in `desc`.

*Output of autonom ui tree on an iOS Simulator*

```
{
  "ok": true,
  "source": "device",
  …
  "nodes": [
    …
    {"ref": "n5", "role": "button", "text": null, "desc": "General", "resource_id": "com.apple.settings.general", "class": "Button", "package": null, "bounds": [16, 380, 386, 432], …},
    …
  ],
  "platform": "ios",
  …
}
```

**What to look at.** `desc` holds General and `text` is null, so select it with `--desc`, or by `resource_id` when the app sets accessibility identifiers. The bounds are points, not pixels.

### Tap one of several identical controls

**Goal:** tap the Irina Weaver button you can see on an iOS catalog screen, where a second button with the same label sits off screen.

1. **Find every match.** Each match shows its index, and one off screen says so.

   ```sh
   autonom ui find --desc "Irina Weaver" --mode exact --all
   ```

2. **Tap the one on screen.** `--index` counts only on-screen matches, from 0.

   ```sh
   autonom ui tap --desc "Irina Weaver" --mode exact --index 0
   ```

You should see two matches, `n4` on screen at index 0 and `n12` off screen with no index, then a tap on `n4`.

*Output of autonom ui find --all, then autonom ui tap --index 0*

```
$ autonom ui find --desc "Irina Weaver" --mode exact --all
{
  "ok": true,
  "source": "device",
  …
  "count": 2,
  "matches": [
    {"ref": "n4", "role": "button", "text": null, "desc": "Irina Weaver", …, "bounds": [16, 314, 216, 358], …, "index": 0},
    {"ref": "n12", "role": "button", "text": null, "desc": "Irina Weaver", …, "bounds": [0, 0, 0, 0], …, "visible": false, "index": null}
  ],
  …
}
$ autonom ui tap --desc "Irina Weaver" --mode exact --index 0
{"ok": true, "x": 116, "y": 336, "ref": "n4", "backend": "idb", …}
```

**What to look at.** `n12` has zero bounds and `visible` false, so its index is null and `--index` never reaches it. `--index 0` picks `n4`, and the tap lands at the center of its bounds. The tap exits 0, but only a fresh tree or a screenshot proves the screen changed.

**Show full output**

```
{
  "ok": true,
  "source": "device",
  "target": "AAAAAAAA-1111-2222-3333-BBBBBBBBBBBB",
  "count": 2,
  "matches": [
    {
      "ref": "n4",
      "role": "button",
      "text": null,
      "desc": "Irina Weaver",
      "resource_id": null,
      "class": "Button",
      "package": null,
      "bounds": [
        16,
        314,
        216,
        358
      ],
      "clickable": true,
      "long_clickable": false,
      "checkable": false,
      "enabled": true,
      "focusable": false,
      "focused": false,
      "scrollable": false,
      "selected": false,
      "checked": false,
      "depth": 0,
      "parent": null,
      "index": 0
    },
    {
      "ref": "n12",
      "role": "button",
      "text": null,
      "desc": "Irina Weaver",
      "resource_id": null,
      "class": "Button",
      "package": null,
      "bounds": [
        0,
        0,
        0,
        0
      ],
      "clickable": true,
      "long_clickable": false,
      "checkable": false,
      "enabled": true,
      "focusable": false,
      "focused": false,
      "scrollable": false,
      "selected": false,
      "checked": false,
      "depth": 0,
      "parent": null,
      "visible": false,
      "index": null
    }
  ],
  "platform": "ios",
  "target_id": "AAAAAAAA-1111-2222-3333-BBBBBBBBBBBB",
  "detail": "actions/0001_find.json"
}
{
  "ok": true,
  "x": 116,
  "y": 336,
  "ref": "n4",
  "backend": "idb",
  "detail": "actions/0002_tap.json",
  "platform": "ios",
  "target_id": "AAAAAAAA-1111-2222-3333-BBBBBBBBBBBB"
}
```

### Reproduce these outputs

From the root of an Autonom checkout at tag v0.36.1 (commit ca2a280), in bash or zsh. The `autonom` function runs that checkout against its test doubles, so no device is needed. Ids and times differ on every run.

*Script that reproduces the outputs*

```sh
# autonom below runs this checkout against Autonom's test doubles; each screen is a fixture.
export AUTONOM_HOME=/tmp/autonom-screen AUTONOM_FAKE_STATE=/tmp/autonom-screen.json
rm -rf "$AUTONOM_HOME"
# Android emulator: the Yarn Shop sign-in screen
autonom() { python3 scripts/autonom.py --adb tests/fakes/fake_adb.py "$@"; }
echo '{"ui_dump": "tests/fixtures/woolbox/android_sign_in.xml"}' > "$AUTONOM_FAKE_STATE"
autonom session start --serial emulator-5554 --app-id com.example.yarnshop
autonom ui tree --format outline --interactable
autonom ui tree --format outline --interactable | python3 -c 'import json, sys; print(json.load(sys.stdin)["outline"])'
autonom ui tree
# Android emulator: a Flutter screen, its button renamed Log In
sed 's/Flutter Save Button/Log In/' tests/fixtures/ui_dump.xml > /tmp/autonom-screen-login.xml
echo '{"ui_dump": "/tmp/autonom-screen-login.xml"}' > "$AUTONOM_FAKE_STATE"
autonom ui find --desc "Log In" --mode exact
autonom ui tap --desc "Log In"
autonom session stop
# iOS Simulator: Settings, then the Yarn Shop catalog
autonom() { python3 scripts/autonom.py --simctl tests/fakes/fake_simctl.py --idb tests/fakes/fake_idb.py "$@"; }
echo '{"idb_describe_all": "tests/fixtures/idb_describe_all_sample.json"}' > "$AUTONOM_FAKE_STATE"
autonom session start --udid AAAAAAAA-1111-2222-3333-BBBBBBBBBBBB
autonom ui tree
echo '{"idb_describe_all": "tests/fixtures/woolbox/ios_catalog_offscreen.json"}' > "$AUTONOM_FAKE_STATE"
autonom ui find --desc "Irina Weaver" --mode exact --all
autonom ui tap --desc "Irina Weaver" --mode exact --index 0
autonom session stop
```

## Good to know

- **Exit 0 does not prove a tap changed the screen.** Compare trees or screenshots before and after, and pin the status bar first so the difference shows only what the app changed.
- **On iOS the label is in `desc`.** A control labeled General has `text: null`, so `--text` finds nothing there. On Android, a `--text` search whose label sits in `desc` returns a [`label_is_in_desc`](https://autonom.yatsuk.me/docs/safety/index.md#label_is_in_desc) warning whose hint is the `--desc` retry.
- **An off-screen copy is never picked first.** On-screen matches are counted first. If a tap answers [`element_offscreen`](https://autonom.yatsuk.me/docs/safety/index.md#element_offscreen), swipe the control into view and select it again, rather than tapping coordinates.
- **Typing needs focus.** With no focused field the characters vanish while `ui type` still exits 0. Read `focused` in the response or the [`no_focused_field`](https://autonom.yatsuk.me/docs/safety/index.md#no_focused_field) warning, and confirm with `ui find`.
- **iOS coordinates are points.** Never multiply by the display scale. A tap outside the reported screen is refused with [`coordinate_space_mismatch`](https://autonom.yatsuk.me/docs/safety/index.md#coordinate_space_mismatch), while an iOS screenshot is measured in pixels.
- **Re-read after every navigation.** Refs belong to one tree. Run `ui wait --settled`, then read again before the next action.
- **A sparse tree means the app exposes little.** [`sparse_accessibility_tree`](https://autonom.yatsuk.me/docs/safety/index.md#sparse_accessibility_tree) does not mean the screen is empty. Add labels in the app, or fall back to a screenshot.
- **An empty iOS tree has a fallback.** With `--ui-backend auto`, the default, an app session whose idb tree is empty or broken switches to Autonom's bundled XCUITest runner for both tree and input. It needs full Xcode and builds on first use.
- **An uncertain tap is never repeated.** [`ui_action_uncertain`](https://autonom.yatsuk.me/docs/safety/index.md#ui_action_uncertain) means the input may have happened. Look at the screen before you do anything else.

## Reference

The `ui` verbs and the evidence around them. Three iOS switches go before the verb: `--ui-backend` picks idb or the XCUITest runner, `--ios-hid` picks idb or AXe for input, and `--axe` names the AXe binary.

| Command | What it does |
| --- | --- |
| `autonom ui tree [--dump FILE]` | Reads the compact tree, or a saved dump. `--format outline` prints one line per node; `--interactable` keeps only what an agent can act on. |
| `autonom ui wait --settled` | Waits until two trees in a row match, up to `--timeout-ms`. Exits 1 if the screen never settled. |
| `autonom ui find` | Finds nodes by `--text`, `--desc`, `--resource-id`, `--class-name`, `--package` or `--role`, on-screen ones first, each with the `index` that selects it. `--mode` is exact, contains or regex. |
| `autonom ui tap [selector flags]` | Taps a node by selector, or a point with `--x` and `--y` as a fallback. |
| `autonom ui swipe --from X,Y --to X,Y` | Drags from one point to another. `--duration` is in seconds, such as 0.4; over 60 is refused. |
| `autonom ui type <text> [--sensitive]` | Types into the focused field. `--sensitive` keeps only its length in the response and the saved action. |
| `autonom ui key <keycode>` | Presses a hardware key. |
| `autonom ui pinch --at X,Y [--scale F]` | Refused on both platforms with [`unsupported_on_platform`](https://autonom.yatsuk.me/docs/safety/index.md#unsupported_on_platform), like `ui rotate` and `ui shake`. |
| `autonom screenshot [--label L]` | Saves a screenshot with its width, height and where it was taken. |
| `autonom simulator status-bar <action>` | Pins the status bar so before and after screenshots differ only where the app changed. |

## Android and iOS differences

The iOS tree is the accessibility hierarchy, not the SwiftUI or UIKit view tree, so its quality depends on the app’s labels.

| Capability | Android | iOS Simulator |
| --- | --- | --- |
| Tree | UI Automator | idb, or the XCUITest runner when idb's tree is empty |
| Visible label | `text`, or `desc` in Flutter | `desc` |
| Input | adb | idb, AXe on Xcode 27, or the XCUITest runner |
| Keys | `KEYCODE_*` | HOME, LOCK, SIDE_BUTTON, SIRI, APPLE_PAY or a HID code |
| Back | `KEYCODE_BACK` | None: tap the navigation bar’s back control |
| Focus before typing | Verified | Unverified: a text field is on screen |
| Pinch, rotate, shake | Refused | Refused |

## Next steps

You can read the screen and act on a control by what it says.

- [Device state](https://autonom.yatsuk.me/docs/device-state/index.md): Open a deep link, grant a permission and set a location.
- [Evidence](https://autonom.yatsuk.me/docs/evidence/index.md): Prove a tap changed the screen.
- [Flows](https://autonom.yatsuk.me/docs/flows/index.md): Save the steps and replay them.
- [Network](https://autonom.yatsuk.me/docs/network/index.md): Mock a backend answer and watch the screen react.