# Mobile Canvas

Canonical URL: https://autonom.yatsuk.me/docs/canvas/

Watch and drive the devices your agent uses, live in a browser tab. Your taps there land in the same journal as your agent's commands.

**Try it**

```sh
autonom canvas serve --platform android --serial emulator-5554 --transport auto
```

**Needs** [a session on one target](https://autonom.yatsuk.me/docs/sessions/index.md) · [Autonom installed, with Node](https://autonom.yatsuk.me/docs/install/index.md)

## How it works

`autonom canvas serve` with a target starts a small web server for that device. It listens on 127.0.0.1 only. The URL it prints carries a random token after the `#`. The page trades that token for a cookie once, then removes it from the address.

Canvas shows the device and passes your input on. Each action lands in the same [session journal](https://autonom.yatsuk.me/docs/sessions/index.md) as your agent's commands, marked with where it came from: human, agent, replay or system. On the fast transports Canvas sends taps and keys to the device itself, because a hop per pointer move is too slow, and records each finished action in the journal.

![Two boxes at the top, the Canvas page used by a person and the autonom ui commands used by the agent, both point into one action pipeline. It points down to one target and to the session journal.](https://autonom.yatsuk.me/docs/canvas/how.svg)

Your taps and your agent's commands reach one target and land in one journal.

Frames come by the fastest path the target has:

- **Android.** `auto` prefers scrcpy: the device's own H.264 video, decoded in the page at up to 60 frames a second, with touch, keys and the clipboard sent as they happen. It needs a scrcpy 4.1 server on your machine; [Install](https://autonom.yatsuk.me/docs/install/index.md) says how to add one. Without it, Canvas falls back to `screenrecord` with ffmpeg, then to screenshots, and says why.
- **iOS Simulator.** `auto` streams the Simulator's own video at up to 60 frames a second and full resolution, through an `idb_companion` that Canvas starts and owns. Without one, it falls back to screenshots.

On Android the video size follows the device's encoder: 2048 pixels on the long side with a software encoder, which every emulator has, and native size on a phone with a hardware one. `--max-size` sets it yourself; 0 means native.

### What the page shows

- **Toolbar.** The target with a live dot, the transport, the Size menu on Android, who holds control, Tools and the inspector toggle.
- **Device.** The picture in a frame that follows the video, the frame rate under it, and a pill of device buttons: Back, Home, Recent apps, Rotate, Volume down and up, and Power. On iOS the pill holds Home, the volume buttons and Power.
- **Inspector.** A text box, keys, device actions such as Copy device clipboard, control, and the stream with Reconnect and a Diagnostics panel.
- **Tools.** A drawer with five tabs for this target: App for permissions and location, Simulate for push, biometrics, battery, network and appearance, Network for capture and Mock this, Mocks, and Logs. Its device changes are journaled like input.
- **Size.** On Android, presets switch the screen size and density live: Small, Pixel 11, Pixel Fold, Tablet or the device's own. Canvas puts the original back when it stops.
- **Shared control.** A person’s first input on a device nobody holds takes its control lease, with no click. Agents and flows on that device are then refused from their next device action, and the page reads “You have control” with Release. When an agent holds the device, the page names it and input is off until you press Take over; after a Take over the page names the holder you replaced and the button reads Hand back.
- **Letting go.** Release or Hand back frees the device. Canvas also lets go after `--control-idle` seconds without input, 120 by default, or about 30 seconds after the page closes. Hand back never restarts a run that was interrupted.
- **Inside one page.** Pause input and Take control in the inspector still sort out input between this Canvas’s own clients. They never refuse `autonom ui` commands; only the lease does.

Light and dark follow your system. At phone width the inspector moves under the device.

## Examples

Canvas needs a real emulator or Simulator, so most outputs here are examples. The process listing comes from Autonom's test device; the commands are in Reproduce these outputs, after the examples.

### Watch an Android emulator from your agent's browser

**Goal:** see the app your agent drives, live, in the browser panel beside it.

1. **Own the target and launch the app.** The session names the emulator, and the app is running before you watch it.

   ```sh
   autonom session start --platform android --serial emulator-5554 --launch --app-id com.example
   ```

2. **Serve Canvas.** Name the platform and the emulator's serial from `autonom devices`, here `emulator-5554`. `auto` picks the best stream this machine has. Leave it running.

   ```sh
   autonom canvas serve --platform android --serial emulator-5554 --transport auto
   ```

3. **Open the printed URL.** Open that exact URL in the browser panel, then check the frame rate under the device while you scroll.

You should see a ready line, the transport it chose and a local URL with a token after the `#`.

*Example output of autonom canvas serve; your values will differ*

```
autonom Canvas ready for android:emulator-5554
Transport preference: auto
Transport: scrcpy
Preview at http://127.0.0.1:3277/#token=<token>
…
Stream size: 2048 on the long side (software H.264 encoder c2.android.avc.encoder)
```

**What to look at.** `Transport: scrcpy` means the device's own video. If it names `screenrecord` or `screencap`, a `Fallback reason` line says why, most often a missing scrcpy 4.1 server. The page trades the token for a cookie and removes it from the address, so a link you share later carries no token.

![A drawing of the Canvas page in a browser at 127.0.0.1:3277: a toolbar with the target emulator-5554, the scrcpy transport, the Size menu, the control status and Tools; the device in a frame with a tap ringed on a button, 60 fps under it and a pill of device buttons; and an inspector with Type, Keys, Control and Stream sections.](https://autonom.yatsuk.me/docs/canvas/page.svg)

A drawing of the page for an Android emulator, with the token already gone from the address.

### Mirror an iOS Simulator at 60 frames a second

**Goal:** watch and drive a booted Simulator the same way.

1. **Serve Canvas for the Simulator.** Put your Simulator's UDID from `autonom devices` after `--target`. With `idb_companion` installed, `auto` streams video.

   ```sh
   autonom canvas serve --platform ios --target AAAAAAAA-1111-2222-3333-BBBBBBBBBBBB --transport auto
   ```

You should see the same lines, naming the Simulator and the idb transport.

*Example output of autonom canvas serve for iOS; your values will differ*

```
autonom Canvas ready for ios:AAAAAAAA-1111-2222-3333-BBBBBBBBBBBB
Transport preference: auto
Transport: idb
Preview at http://127.0.0.1:3277/#token=<token>
…
```

**What to look at.** `Transport: idb` is the Simulator's own video. One finger drags live, and Home, Power and the volume buttons press the Simulator's own buttons, not your Mac's. A two-finger pinch is sent once, when a finger lifts. A paste is typed when it is plain ASCII of up to 300 bytes; other text is only put on the Simulator's clipboard, and you paste it with a long press on the field.

### Work on several devices in one workspace

**Goal:** watch an emulator and a Simulator side by side, then shut everything down.

1. **Start a workspace.** With no target, Canvas starts with no device and runs until you stop it.

   ```sh
   autonom canvas
   ```

2. **Attach a device.** It lands in the active tab; the page's empty tiles offer the same, with Attach on each running target.

   ```sh
   autonom canvas attach --serial emulator-5554
   ```

3. **See what runs.** It lists each Canvas with its tabs and devices.

   ```sh
   autonom canvas list
   ```

4. **Stop it.** Detaching ends only the sessions the Canvas started.

   ```sh
   autonom canvas stop
   ```

You should see a workspace URL, then one tab, then the stopped Canvas.

*Example output of a workspace Canvas; your values will differ*

```
$ autonom canvas
autonom Canvas workspace default ready
Preview at http://127.0.0.1:3277/#token=<token>

$ autonom canvas list
{"ok": true, "canvases": [{"port": 3277, …, "mode": "workspace", "workspace": "default", "tabs": [{"id": "t_e2e1a5d4", "name": "Canvas 1", "devices": […]}]}]}

$ autonom canvas stop
{"ok": true, "stopped": [{"port": 3277, "pid": 92494, "workspace": "default"}]}
```

**What to look at.** A workspace holds up to eight tabs, each a split of one to four tiles, and at most eight devices. Each device gets its own Autonom session, or reuses the live one on that target, and your current session never changes. Only the focused tile streams at full rate. Tabs and devices are saved per workspace, so a restart brings them back.

### Make sure Canvas stops with the session

**Goal:** leave nothing running when the work is done.

1. **See what is running.** A running Canvas appears under `background`.

   ```sh
   autonom processes
   ```

2. **Stop the session.** It stops every process the session started, a Canvas served for its target included. A workspace has no session owner; stop it with `autonom canvas stop`.

   ```sh
   autonom session stop
   ```

3. **Clean up after a crash.** If a process was killed hard, `cleanup` stops what it left behind.

   ```sh
   autonom cleanup
   ```

With no Canvas running, you should see every list empty.

*Output of autonom processes*

```
{
  "ok": true,
  "live": [],
  "orphans": [],
  "stale_entries": [],
  "harness": [],
  "background": [],
  "group_remnants": []
}
```

**What to look at.** `background` is where a running Canvas pair shows up: the supervisor and its node process. A group whose leader is gone is only reported in `group_remnants`, never stopped.

### Reproduce these outputs

From the root of an Autonom checkout at tag v0.36.1 (commit ca2a280), in bash or zsh. Autonom's test device stands in for the emulator. Serving a device needs a real emulator or Simulator, so those lines above are an example of what Canvas prints. The workspace lines come from the run below, on a free port and a workspace named demo; plain `autonom canvas` uses port 3277 and the workspace default.

*Script that reproduces the outputs*

```sh
S=$PWD W=/tmp/autonom-canvas
rm -rf $W && mkdir -p $W
export AUTONOM_HOME=$W/.autonom
autonom() { python3 $S/scripts/autonom.py --adb $S/tests/fakes/fake_adb.py "$@"; }
autonom session start --platform android --serial emulator-5554 --launch --app-id com.example
autonom processes
autonom session stop
autonom canvas serve --port 0 --workspace demo --ephemeral > $W/canvas.log &
until grep -q '^Preview at' $W/canvas.log; do sleep 1; done; head -3 $W/canvas.log
autonom canvas list
autonom canvas stop --workspace demo
```

## Good to know

- **Canvas is not a second runner.** Its input reaches the device as the command line's does, and lands in the same journal. A person's tap is recorded with the origin `human`.
- **It records only next to a session.** Canvas writes to the journal only when a session is on its target. Without one, every action still runs and nothing is recorded.
- **An open page is not a working stream.** Check that the frame rate under the device moves before you call the setup done.
- **Choose what to tap with ui, not pixels.** Finding elements belongs to the [screen commands](https://autonom.yatsuk.me/docs/screen/index.md). Canvas is for watching and hands-on control.
- **It is not a measurement tool.** For a careful manual review, use scrcpy itself. For [performance claims](https://autonom.yatsuk.me/docs/performance/index.md), use Macrobenchmark, Perfetto or Flutter profile mode. Start an emulator with `-gpu host` for a usable frame rate.
- **A hard kill leaves the size changed.** Canvas puts the screen size, density and rotation back when it stops. A Canvas killed with SIGKILL cannot; reset the size and density with `wm size reset` and `wm density reset`.
- **The lease is coordination, not security.** It keeps a person and cooperating agents of one user out of each other’s way. Any process of that user can take over, so do not rely on it to keep a misbehaving agent off a device. [Sessions and targets](https://autonom.yatsuk.me/docs/sessions/index.md#share-a-device) shows the lease from the command line.
- **A script without an origin counts as a person.** Its first input takes the lease and shuts out your agents. A script that drives Canvas should connect with `origin=agent`.
- **Mocks are shared.** A mock added in Tools applies to every Autonom session on this Mac, and the drawer says so.
- **Never expose it.** Anyone with a forwarded port and the token can drive the device. Treat any tunnel to it as full device control.
- **The token stays out of listings.** `processes` and `doctor` never show a Canvas token or its command line. Clipboard text and log lines are never journaled.
- **Record what you watched.** Keep a screenshot of the panel, the platform and target, the transport and stream size, the app, the build, who held control and the flow you replayed.

## Reference

One command starts Canvas for one device or for a workspace. Its flags pick the target, the stream and how the page is protected.

| Command | What it does |
| --- | --- |
| `autonom canvas serve --platform android --serial <adb-serial> --transport auto` | Mirrors one Android target, supervised and registered to the session. |
| `autonom canvas serve --platform ios --target <simulator-udid> --transport auto` | Mirrors one booted iOS Simulator. |
| `autonom canvas` | Starts a workspace with no device, on port 3277, until `canvas stop`. A second run prints the running one's URL. |
| `autonom canvas attach --serial <adb-serial> [--tab <tab id or name>]` | Adds a device to a running workspace; `--udid` for a Simulator. |
| `autonom canvas detach --serial <adb-serial>` | Removes it, and ends its session only if the Canvas started it. |
| `autonom canvas list` | The running Canvases, their tabs and devices. |
| `autonom canvas stop [--port N \| --workspace NAME \| --all]` | Stops a Canvas, which puts back the screen sizes and rotations it changed. |
| `--transport auto\|scrcpy\|idb\|screenrecord\|screencap` | `auto` prefers scrcpy on Android and idb on a Simulator, and falls back. `scrcpy` and `idb` refuse to start without their tool. `screencap` streams screenshots and needs nothing else. |
| `--max-size PX` | The long side of the scrcpy video; 0 is native. By default 2048 with a software encoder and native with a hardware one. |
| `--fps N` | A frame rate cap: 15 by default for screenshots, up to 60 on scrcpy and idb. |
| `--bit-rate BPS` | The video bit rate. The default is 12 Mbit/s. |
| `--port N` | The local port. The default is 3277; 0 picks a free one. |
| `--device PLATFORM:ID` | A device to show from the start, such as `android:emulator-5554`; repeatable. One alone is a single-device Canvas, unless `--split`. |
| `--workspace NAME` | A named workspace whose tabs persist. `--ephemeral` neither reads nor saves them. |
| `--bootable SPEC` | An emulator or Simulator the page may boot; `--shutdown-booted` shuts it down again at stop. |
| `--install-root DIR` | A folder the page may install builds from. `--captures-dir` moves screenshots and recordings from `~/Downloads/Autonom`. |
| `--control-idle SECONDS` | Releases a person’s control lease after this long without input: 120 by default, 15 to 3600. |
| `--token TOKEN` | Uses your token instead of a random one. |
| `--no-auth` | Turns the token off. Only for local debugging on a machine no one else can reach. |
| `autonom processes` | Lists what Autonom started. A running Canvas is under `background`. |
| `autonom session stop` | Ends the session and every process it owns, a Canvas served for its target included. |

## Android and iOS differences

Both platforms stream video at up to 60 frames a second. What differs is input and what the device lets Canvas change.

| Capability | Android | iOS Simulator |
| --- | --- | --- |
| Frames | scrcpy video, else `screenrecord` with ffmpeg, else screenshots | Video through `idb_companion`, else screenshots |
| Fingers | Up to 10, live; Ctrl or Alt drag pinches | One live finger; a pinch is sent once, at release |
| Back, Recent apps, Rotate and notifications | Yes | Refused, and not shown |
| Home, Power and volume | Yes | Yes, the Simulator's own buttons; volume on idb only |
| Pasting text | Any text, through the device clipboard | Typed when plain ASCII up to 300 bytes; only on the Simulator's own Mac |
| Size presets | Yes | No |
| Logs in Tools | The live device log | Only for a session started with `--log-stream` |

## Next steps

You can watch the devices your agent drives and step in when you need to.

- [Safety and errors](https://autonom.yatsuk.me/docs/safety/index.md): Read every failure by its code, and see what consent protects.
- [Read and drive the screen](https://autonom.yatsuk.me/docs/screen/index.md): Find and tap by what an element says.
- [Ticket pre-test](https://autonom.yatsuk.me/docs/pretest/index.md): Check a ticket on the device before QA.
- [Sessions and targets](https://autonom.yatsuk.me/docs/sessions/index.md): Name one target and read its journal.