Mobile Canvas

Watch and drive the devices your agent uses, live in a browser tab. Your taps there land in the same journal as your agent's commands.

Try it autonom canvas serve --platform android --serial emulator-5554 --transport auto

Needs a session on one target · Autonom installed, with Node

How it works

autonom canvas serve with a target starts a small web server for that device. It listens on 127.0.0.1 only. The URL it prints carries a random token after the #. The page trades that token for a cookie once, then removes it from the address.

Canvas shows the device and passes your input on. Each action lands in the same session journal as your agent's commands, marked with where it came from: human, agent, replay or system. On the fast transports Canvas sends taps and keys to the device itself, because a hop per pointer move is too slow, and records each finished action in the journal.

Two boxes at the top, the Canvas page used by a person and the autonom ui commands used by the agent, both point into one action pipeline. It points down to one target and to the session journal. Two boxes at the top, the Canvas page used by a person and the autonom ui commands used by the agent, both point into one action pipeline. It points down to one target and to the session journal.
Your taps and your agent's commands reach one target and land in one journal.

Frames come by the fastest path the target has:

  • Android. auto prefers scrcpy: the device's own H.264 video, decoded in the page at up to 60 frames a second, with touch, keys and the clipboard sent as they happen. It needs a scrcpy 4.1 server on your machine; Install says how to add one. Without it, Canvas falls back to screenrecord with ffmpeg, then to screenshots, and says why.

  • iOS Simulator. auto streams the Simulator's own video at up to 60 frames a second and full resolution, through an idb_companion that Canvas starts and owns. Without one, it falls back to screenshots.

On Android the video size follows the device's encoder: 2048 pixels on the long side with a software encoder, which every emulator has, and native size on a phone with a hardware one. --max-size sets it yourself; 0 means native.

What the page shows

  • Toolbar. The target with a live dot, the transport, the Size menu on Android, who holds control, Tools and the inspector toggle.

  • Device. The picture in a frame that follows the video, the frame rate under it, and a pill of device buttons: Back, Home, Recent apps, Rotate, Volume down and up, and Power. On iOS the pill holds Home, the volume buttons and Power.

  • Inspector. A text box, keys, device actions such as Copy device clipboard, control, and the stream with Reconnect and a Diagnostics panel.

  • Tools. A drawer with five tabs for this target: App for permissions and location, Simulate for push, biometrics, battery, network and appearance, Network for capture and Mock this, Mocks, and Logs. Its device changes are journaled like input.

  • Size. On Android, presets switch the screen size and density live: Small, Pixel 11, Pixel Fold, Tablet or the device's own. Canvas puts the original back when it stops.

  • Shared control. A person’s first input on a device nobody holds takes its control lease, with no click. Agents and flows on that device are then refused from their next device action, and the page reads “You have control” with Release. When an agent holds the device, the page names it and input is off until you press Take over; after a Take over the page names the holder you replaced and the button reads Hand back.

  • Letting go. Release or Hand back frees the device. Canvas also lets go after --control-idle seconds without input, 120 by default, or about 30 seconds after the page closes. Hand back never restarts a run that was interrupted.

  • Inside one page. Pause input and Take control in the inspector still sort out input between this Canvas’s own clients. They never refuse autonom ui commands; only the lease does.

Light and dark follow your system. At phone width the inspector moves under the device.

Examples

Canvas needs a real emulator or Simulator, so most outputs here are examples. The process listing comes from Autonom's test device; the commands are in Reproduce these outputs, after the examples.

Watch an Android emulator from your agent's browser

Goal: see the app your agent drives, live, in the browser panel beside it.

  1. Own the target and launch the app.

    The session names the emulator, and the app is running before you watch it.

    autonom session start --platform android --serial emulator-5554 --launch --app-id com.example
  2. Serve Canvas.

    Name the platform and the emulator's serial from autonom devices, here emulator-5554. auto picks the best stream this machine has. Leave it running.

    autonom canvas serve --platform android --serial emulator-5554 --transport auto
  3. Open the printed URL.

    Open that exact URL in the browser panel, then check the frame rate under the device while you scroll.

You should see a ready line, the transport it chose and a local URL with a token after the #.

autonom Canvas ready for android:emulator-5554Transport preference: autoTransport: scrcpyPreview at http://127.0.0.1:3277/#token=<token>…Stream size: 2048 on the long side (software H.264 encoder c2.android.avc.encoder)

What to look at. Transport: scrcpy means the device's own video. If it names screenrecord or screencap, a Fallback reason line says why, most often a missing scrcpy 4.1 server. The page trades the token for a cookie and removes it from the address, so a link you share later carries no token.

A drawing of the Canvas page in a browser at 127.0.0.1:3277: a toolbar with the target emulator-5554, the scrcpy transport, the Size menu, the control status and Tools; the device in a frame with a tap ringed on a button, 60 fps under it and a pill of device buttons; and an inspector with Type, Keys, Control and Stream sections. A drawing of the Canvas page on a phone: the target emulator-5554 and Tools at the top, the device with a tap ringed on a button, 60 fps under it and a pill of device buttons, and below it the inspector with Type, Control and Stream sections.
A drawing of the page for an Android emulator, with the token already gone from the address.

Mirror an iOS Simulator at 60 frames a second

Goal: watch and drive a booted Simulator the same way.

  1. Serve Canvas for the Simulator.

    Put your Simulator's UDID from autonom devices after --target. With idb_companion installed, auto streams video.

    autonom canvas serve --platform ios --target AAAAAAAA-1111-2222-3333-BBBBBBBBBBBB --transport auto

You should see the same lines, naming the Simulator and the idb transport.

autonom Canvas ready for ios:AAAAAAAA-1111-2222-3333-BBBBBBBBBBBBTransport preference: autoTransport: idbPreview at http://127.0.0.1:3277/#token=<token>…

What to look at. Transport: idb is the Simulator's own video. One finger drags live, and Home, Power and the volume buttons press the Simulator's own buttons, not your Mac's. A two-finger pinch is sent once, when a finger lifts. A paste is typed when it is plain ASCII of up to 300 bytes; other text is only put on the Simulator's clipboard, and you paste it with a long press on the field.

Work on several devices in one workspace

Goal: watch an emulator and a Simulator side by side, then shut everything down.

  1. Start a workspace.

    With no target, Canvas starts with no device and runs until you stop it.

    autonom canvas
  2. Attach a device.

    It lands in the active tab; the page's empty tiles offer the same, with Attach on each running target.

    autonom canvas attach --serial emulator-5554
  3. See what runs.

    It lists each Canvas with its tabs and devices.

    autonom canvas list
  4. Stop it.

    Detaching ends only the sessions the Canvas started.

    autonom canvas stop

You should see a workspace URL, then one tab, then the stopped Canvas.

autonom canvasautonom Canvas workspace default readyPreview at http://127.0.0.1:3277/#token=<token>autonom canvas list{"ok": true, "canvases": [{"port": 3277, …, "mode": "workspace", "workspace": "default", "tabs": [{"id": "t_e2e1a5d4", "name": "Canvas 1", "devices": […]}]}]}autonom canvas stop{"ok": true, "stopped": [{"port": 3277, "pid": 92494, "workspace": "default"}]}

What to look at. A workspace holds up to eight tabs, each a split of one to four tiles, and at most eight devices. Each device gets its own Autonom session, or reuses the live one on that target, and your current session never changes. Only the focused tile streams at full rate. Tabs and devices are saved per workspace, so a restart brings them back.

Make sure Canvas stops with the session

Goal: leave nothing running when the work is done.

  1. See what is running.

    A running Canvas appears under background.

    autonom processes
  2. Stop the session.

    It stops every process the session started, a Canvas served for its target included. A workspace has no session owner; stop it with autonom canvas stop.

    autonom session stop
  3. Clean up after a crash.

    If a process was killed hard, cleanup stops what it left behind.

    autonom cleanup

With no Canvas running, you should see every list empty.

{ "ok": true, "live": [], "orphans": [], "stale_entries": [], "harness": [], "background": [], "group_remnants": []}

What to look at. background is where a running Canvas pair shows up: the supervisor and its node process. A group whose leader is gone is only reported in group_remnants, never stopped.

Reproduce these outputs

From the root of an Autonom checkout at tag v0.36.1 (commit ca2a280), in bash or zsh. Autonom's test device stands in for the emulator. Serving a device needs a real emulator or Simulator, so those lines above are an example of what Canvas prints. The workspace lines come from the run below, on a free port and a workspace named demo; plain autonom canvas uses port 3277 and the workspace default.

S=$PWD W=/tmp/autonom-canvasrm -rf $W && mkdir -p $Wexport AUTONOM_HOME=$W/.autonomautonom() { python3 $S/scripts/autonom.py --adb $S/tests/fakes/fake_adb.py "$@"; }autonom session start --platform android --serial emulator-5554 --launch --app-id com.exampleautonom processesautonom session stopautonom canvas serve --port 0 --workspace demo --ephemeral > $W/canvas.log &until grep -q '^Preview at' $W/canvas.log; do sleep 1; done; head -3 $W/canvas.logautonom canvas listautonom canvas stop --workspace demo

Good to know

  • Canvas is not a second runner. Its input reaches the device as the command line's does, and lands in the same journal. A person's tap is recorded with the origin human.

  • It records only next to a session. Canvas writes to the journal only when a session is on its target. Without one, every action still runs and nothing is recorded.

  • An open page is not a working stream. Check that the frame rate under the device moves before you call the setup done.

  • Choose what to tap with ui, not pixels. Finding elements belongs to the screen commands. Canvas is for watching and hands-on control.

  • It is not a measurement tool. For a careful manual review, use scrcpy itself. For performance claims, use Macrobenchmark, Perfetto or Flutter profile mode. Start an emulator with -gpu host for a usable frame rate.

  • A hard kill leaves the size changed. Canvas puts the screen size, density and rotation back when it stops. A Canvas killed with SIGKILL cannot; reset the size and density with wm size reset and wm density reset.

  • The lease is coordination, not security. It keeps a person and cooperating agents of one user out of each other’s way. Any process of that user can take over, so do not rely on it to keep a misbehaving agent off a device. Sessions and targets shows the lease from the command line.

  • A script without an origin counts as a person. Its first input takes the lease and shuts out your agents. A script that drives Canvas should connect with origin=agent.

  • Mocks are shared. A mock added in Tools applies to every Autonom session on this Mac, and the drawer says so.

  • Never expose it. Anyone with a forwarded port and the token can drive the device. Treat any tunnel to it as full device control.

  • The token stays out of listings. processes and doctor never show a Canvas token or its command line. Clipboard text and log lines are never journaled.

  • Record what you watched. Keep a screenshot of the panel, the platform and target, the transport and stream size, the app, the build, who held control and the flow you replayed.

Reference

One command starts Canvas for one device or for a workspace. Its flags pick the target, the stream and how the page is protected.

CommandWhat it does
autonom canvas serve --platform android --serial <adb-serial> --transport autoMirrors one Android target, supervised and registered to the session.
autonom canvas serve --platform ios --target <simulator-udid> --transport autoMirrors one booted iOS Simulator.
autonom canvasStarts a workspace with no device, on port 3277, until canvas stop. A second run prints the running one's URL.
autonom canvas attach --serial <adb-serial> [--tab <tab id or name>]Adds a device to a running workspace; --udid for a Simulator.
autonom canvas detach --serial <adb-serial>Removes it, and ends its session only if the Canvas started it.
autonom canvas listThe running Canvases, their tabs and devices.
autonom canvas stop [--port N | --workspace NAME | --all]Stops a Canvas, which puts back the screen sizes and rotations it changed.
--transport auto|scrcpy|idb|screenrecord|screencapauto prefers scrcpy on Android and idb on a Simulator, and falls back. scrcpy and idb refuse to start without their tool. screencap streams screenshots and needs nothing else.
--max-size PXThe long side of the scrcpy video; 0 is native. By default 2048 with a software encoder and native with a hardware one.
--fps NA frame rate cap: 15 by default for screenshots, up to 60 on scrcpy and idb.
--bit-rate BPSThe video bit rate. The default is 12 Mbit/s.
--port NThe local port. The default is 3277; 0 picks a free one.
--device PLATFORM:IDA device to show from the start, such as android:emulator-5554; repeatable. One alone is a single-device Canvas, unless --split.
--workspace NAMEA named workspace whose tabs persist. --ephemeral neither reads nor saves them.
--bootable SPECAn emulator or Simulator the page may boot; --shutdown-booted shuts it down again at stop.
--install-root DIRA folder the page may install builds from. --captures-dir moves screenshots and recordings from ~/Downloads/Autonom.
--control-idle SECONDSReleases a person’s control lease after this long without input: 120 by default, 15 to 3600.
--token TOKENUses your token instead of a random one.
--no-authTurns the token off. Only for local debugging on a machine no one else can reach.
autonom processesLists what Autonom started. A running Canvas is under background.
autonom session stopEnds the session and every process it owns, a Canvas served for its target included.

Android and iOS differences

Both platforms stream video at up to 60 frames a second. What differs is input and what the device lets Canvas change.

CapabilityAndroidiOS Simulator
Framesscrcpy video, else screenrecord with ffmpeg, else screenshotsVideo through idb_companion, else screenshots
FingersUp to 10, live; Ctrl or Alt drag pinchesOne live finger; a pinch is sent once, at release
Back, Recent apps, Rotate and notificationsYesRefused, and not shown
Home, Power and volumeYesYes, the Simulator's own buttons; volume on idb only
Pasting textAny text, through the device clipboardTyped when plain ASCII up to 300 bytes; only on the Simulator's own Mac
Size presetsYesNo
Logs in ToolsThe live device logOnly for a session started with --log-stream

Next steps

You can watch the devices your agent drives and step in when you need to.

Search documentation

Type to find a guide.

↑ ↓ to move · Enter to open · Esc to close