Mobile Canvas
Watch and drive the devices your agent uses, live in a browser tab. Your taps there land in the same journal as your agent's commands.
Try it autonom canvas serve --platform android --serial emulator-5554 --transport auto
Needs a session on one target · Autonom installed, with Node
How it works
autonom canvas serve with a target starts a small web server for that device. It listens on 127.0.0.1 only. The URL it prints carries a random token after the #. The page trades that token for a cookie once, then removes it from the address.
Canvas shows the device and passes your input on. Each action lands in the same session journal as your agent's commands, marked with where it came from: human, agent, replay or system. On the fast transports Canvas sends taps and keys to the device itself, because a hop per pointer move is too slow, and records each finished action in the journal.
Frames come by the fastest path the target has:
Android.
autoprefers scrcpy: the device's own H.264 video, decoded in the page at up to 60 frames a second, with touch, keys and the clipboard sent as they happen. It needs a scrcpy 4.1 server on your machine; Install says how to add one. Without it, Canvas falls back toscreenrecordwith ffmpeg, then to screenshots, and says why.iOS Simulator.
autostreams the Simulator's own video at up to 60 frames a second and full resolution, through anidb_companionthat Canvas starts and owns. Without one, it falls back to screenshots.
On Android the video size follows the device's encoder: 2048 pixels on the long side with a software encoder, which every emulator has, and native size on a phone with a hardware one. --max-size sets it yourself; 0 means native.
What the page shows
Toolbar. The target with a live dot, the transport, the Size menu on Android, who holds control, Tools and the inspector toggle.
Device. The picture in a frame that follows the video, the frame rate under it, and a pill of device buttons: Back, Home, Recent apps, Rotate, Volume down and up, and Power. On iOS the pill holds Home, the volume buttons and Power.
Inspector. A text box, keys, device actions such as Copy device clipboard, control, and the stream with Reconnect and a Diagnostics panel.
Tools. A drawer with five tabs for this target: App for permissions and location, Simulate for push, biometrics, battery, network and appearance, Network for capture and Mock this, Mocks, and Logs. Its device changes are journaled like input.
Size. On Android, presets switch the screen size and density live: Small, Pixel 11, Pixel Fold, Tablet or the device's own. Canvas puts the original back when it stops.
Shared control. A person’s first input on a device nobody holds takes its control lease, with no click. Agents and flows on that device are then refused from their next device action, and the page reads “You have control” with Release. When an agent holds the device, the page names it and input is off until you press Take over; after a Take over the page names the holder you replaced and the button reads Hand back.
Letting go. Release or Hand back frees the device. Canvas also lets go after
--control-idleseconds without input, 120 by default, or about 30 seconds after the page closes. Hand back never restarts a run that was interrupted.Inside one page. Pause input and Take control in the inspector still sort out input between this Canvas’s own clients. They never refuse
autonom uicommands; only the lease does.
Light and dark follow your system. At phone width the inspector moves under the device.
Examples
Canvas needs a real emulator or Simulator, so most outputs here are examples. The process listing comes from Autonom's test device; the commands are in Reproduce these outputs, after the examples.
Watch an Android emulator from your agent's browser
Goal: see the app your agent drives, live, in the browser panel beside it.
- Own the target and launch the app.
The session names the emulator, and the app is running before you watch it.
autonom session start --platform android --serial emulator-5554 --launch --app-id com.example - Serve Canvas.
Name the platform and the emulator's serial from
autonom devices, hereemulator-5554.autopicks the best stream this machine has. Leave it running.autonom canvas serve --platform android --serial emulator-5554 --transport auto - Open the printed URL.
Open that exact URL in the browser panel, then check the frame rate under the device while you scroll.
You should see a ready line, the transport it chose and a local URL with a token after the #.
What to look at. Transport: scrcpy means the device's own video. If it names screenrecord or screencap, a Fallback reason line says why, most often a missing scrcpy 4.1 server. The page trades the token for a cookie and removes it from the address, so a link you share later carries no token.
Mirror an iOS Simulator at 60 frames a second
Goal: watch and drive a booted Simulator the same way.
- Serve Canvas for the Simulator.
Put your Simulator's UDID from
autonom devicesafter--target. Withidb_companioninstalled,autostreams video.autonom canvas serve --platform ios --target AAAAAAAA-1111-2222-3333-BBBBBBBBBBBB --transport auto
You should see the same lines, naming the Simulator and the idb transport.
What to look at. Transport: idb is the Simulator's own video. One finger drags live, and Home, Power and the volume buttons press the Simulator's own buttons, not your Mac's. A two-finger pinch is sent once, when a finger lifts. A paste is typed when it is plain ASCII of up to 300 bytes; other text is only put on the Simulator's clipboard, and you paste it with a long press on the field.
Work on several devices in one workspace
Goal: watch an emulator and a Simulator side by side, then shut everything down.
- Start a workspace.
With no target, Canvas starts with no device and runs until you stop it.
autonom canvas - Attach a device.
It lands in the active tab; the page's empty tiles offer the same, with Attach on each running target.
autonom canvas attach --serial emulator-5554 - See what runs.
It lists each Canvas with its tabs and devices.
autonom canvas list - Stop it.
Detaching ends only the sessions the Canvas started.
autonom canvas stop
You should see a workspace URL, then one tab, then the stopped Canvas.
What to look at. A workspace holds up to eight tabs, each a split of one to four tiles, and at most eight devices. Each device gets its own Autonom session, or reuses the live one on that target, and your current session never changes. Only the focused tile streams at full rate. Tabs and devices are saved per workspace, so a restart brings them back.
Make sure Canvas stops with the session
Goal: leave nothing running when the work is done.
- See what is running.
A running Canvas appears under
background.autonom processes - Stop the session.
It stops every process the session started, a Canvas served for its target included. A workspace has no session owner; stop it with
autonom canvas stop.autonom session stop - Clean up after a crash.
If a process was killed hard,
cleanupstops what it left behind.autonom cleanup
With no Canvas running, you should see every list empty.
What to look at. background is where a running Canvas pair shows up: the supervisor and its node process. A group whose leader is gone is only reported in group_remnants, never stopped.
Reproduce these outputs
From the root of an Autonom checkout at tag v0.36.1 (commit ca2a280), in bash or zsh. Autonom's test device stands in for the emulator. Serving a device needs a real emulator or Simulator, so those lines above are an example of what Canvas prints. The workspace lines come from the run below, on a free port and a workspace named demo; plain autonom canvas uses port 3277 and the workspace default.
Good to know
Canvas is not a second runner. Its input reaches the device as the command line's does, and lands in the same journal. A person's tap is recorded with the origin
human.It records only next to a session. Canvas writes to the journal only when a session is on its target. Without one, every action still runs and nothing is recorded.
An open page is not a working stream. Check that the frame rate under the device moves before you call the setup done.
Choose what to tap with ui, not pixels. Finding elements belongs to the screen commands. Canvas is for watching and hands-on control.
It is not a measurement tool. For a careful manual review, use scrcpy itself. For performance claims, use Macrobenchmark, Perfetto or Flutter profile mode. Start an emulator with
-gpu hostfor a usable frame rate.A hard kill leaves the size changed. Canvas puts the screen size, density and rotation back when it stops. A Canvas killed with SIGKILL cannot; reset the size and density with
wm size resetandwm density reset.The lease is coordination, not security. It keeps a person and cooperating agents of one user out of each other’s way. Any process of that user can take over, so do not rely on it to keep a misbehaving agent off a device. Sessions and targets shows the lease from the command line.
A script without an origin counts as a person. Its first input takes the lease and shuts out your agents. A script that drives Canvas should connect with
origin=agent.Mocks are shared. A mock added in Tools applies to every Autonom session on this Mac, and the drawer says so.
Never expose it. Anyone with a forwarded port and the token can drive the device. Treat any tunnel to it as full device control.
The token stays out of listings.
processesanddoctornever show a Canvas token or its command line. Clipboard text and log lines are never journaled.Record what you watched. Keep a screenshot of the panel, the platform and target, the transport and stream size, the app, the build, who held control and the flow you replayed.
Reference
One command starts Canvas for one device or for a workspace. Its flags pick the target, the stream and how the page is protected.
| Command | What it does |
|---|---|
autonom canvas serve --platform android --serial <adb-serial> --transport auto | Mirrors one Android target, supervised and registered to the session. |
autonom canvas serve --platform ios --target <simulator-udid> --transport auto | Mirrors one booted iOS Simulator. |
autonom canvas | Starts a workspace with no device, on port 3277, until canvas stop. A second run prints the running one's URL. |
autonom canvas attach --serial <adb-serial> [--tab <tab id or name>] | Adds a device to a running workspace; --udid for a Simulator. |
autonom canvas detach --serial <adb-serial> | Removes it, and ends its session only if the Canvas started it. |
autonom canvas list | The running Canvases, their tabs and devices. |
autonom canvas stop [--port N | --workspace NAME | --all] | Stops a Canvas, which puts back the screen sizes and rotations it changed. |
--transport auto|scrcpy|idb|screenrecord|screencap | auto prefers scrcpy on Android and idb on a Simulator, and falls back. scrcpy and idb refuse to start without their tool. screencap streams screenshots and needs nothing else. |
--max-size PX | The long side of the scrcpy video; 0 is native. By default 2048 with a software encoder and native with a hardware one. |
--fps N | A frame rate cap: 15 by default for screenshots, up to 60 on scrcpy and idb. |
--bit-rate BPS | The video bit rate. The default is 12 Mbit/s. |
--port N | The local port. The default is 3277; 0 picks a free one. |
--device PLATFORM:ID | A device to show from the start, such as android:emulator-5554; repeatable. One alone is a single-device Canvas, unless --split. |
--workspace NAME | A named workspace whose tabs persist. --ephemeral neither reads nor saves them. |
--bootable SPEC | An emulator or Simulator the page may boot; --shutdown-booted shuts it down again at stop. |
--install-root DIR | A folder the page may install builds from. --captures-dir moves screenshots and recordings from ~/Downloads/Autonom. |
--control-idle SECONDS | Releases a person’s control lease after this long without input: 120 by default, 15 to 3600. |
--token TOKEN | Uses your token instead of a random one. |
--no-auth | Turns the token off. Only for local debugging on a machine no one else can reach. |
autonom processes | Lists what Autonom started. A running Canvas is under background. |
autonom session stop | Ends the session and every process it owns, a Canvas served for its target included. |
Android and iOS differences
Both platforms stream video at up to 60 frames a second. What differs is input and what the device lets Canvas change.
| Capability | Android | iOS Simulator |
|---|---|---|
| Frames | scrcpy video, else screenrecord with ffmpeg, else screenshots | Video through idb_companion, else screenshots |
| Fingers | Up to 10, live; Ctrl or Alt drag pinches | One live finger; a pinch is sent once, at release |
| Back, Recent apps, Rotate and notifications | Yes | Refused, and not shown |
| Home, Power and volume | Yes | Yes, the Simulator's own buttons; volume on idb only |
| Pasting text | Any text, through the device clipboard | Typed when plain ASCII up to 300 bytes; only on the Simulator's own Mac |
| Size presets | Yes | No |
| Logs in Tools | The live device log | Only for a session started with --log-stream |
Next steps
You can watch the devices your agent drives and step in when you need to.