Ticket pre-test
Check one ticket and the change behind it on an emulator or Simulator before it reaches QA. Your agent plans the checks, drives them under a budget and hands back a verified pack with a plain verdict.
Try it autonom pretest start --ticket tickets/WB-1.md --base main --ticket-ref WB-1
Needs Autonom installed · a session with the app running · flows for the regression checks, if you have them
How it works
A pre-test checks one ticket and its code change on a device, the way a careful QA engineer would. It runs inside the current session and adds no new way to drive the device. Your agent still uses the screen commands, screenshots, network mocks and flows.
What the pre-test adds is a record. While a check is open, every journal entry, screenshot and saved tree is filed under that check. The pack at the end shows every planned check, run or not, with its evidence.
pretest start reads the ticket and compares your working tree with a base branch. It notes the changed files and lines, the flows that cover them, and screens that may show the change. It also takes a baseline of crashes, error logs, requests and memory. The ticket is masked for credentials before it is written.
Checks are planned in a fixed order: one per acceptance point in the ticket, then edge cases from the kit that the diff could break, then the regression flows. Each check ends with one of five verdicts: pass, fail, blocked, not_covered or inconclusive.
How the ticket verdict is set
The pack sets the ticket's verdict by rule. The first rule that matches wins:
fail: a check failed, or a confirmed finding is a blocker or major.blocked: a check could not be carried out.not_covered: a planned check never began, or there are no checks.inconclusive: a check could not decide, or was left open.pass: none of the above.
A not_covered or inconclusive ticket is never rounded up to a pass. Passing flows do not make a ticket done when its acceptance checks never ran.
Guard rails
While a pre-test is active on the target, three limits hold. Each refusal happens before any input reaches the device, and lands in the journal with its code.
A budget. Taps, swipes, typing, keys, opening links, app launches and stops, and each flow input step count against
--max-actionsand--max-minutes: 300 actions and 60 minutes by default. Reads such asui tree,ui findandscreenshotnever count.Forbidden taps. A tap on a control whose text matches
--forbid-textis refused. The default list holds buy, pay, purchase, delete and remove account, plus the Russian words for buy, pay and delete.Allowed apps. Opening a link or launching an app outside
--allow-appis refused. By default only the session's app is allowed.
Examples
The summary in the first example comes from a real pre-test on an Android emulator; its ticket here is shortened. The other four examples run on Autonom's test device with a catalog screen the script writes, so you can reproduce them without an emulator. Open Reproduce these outputs at the end of this section for the exact commands.
Ask your agent to pre-test a ticket
Goal: hand one ticket to your agent and get back a verdict you can trust.
- Write the ticket down.
Number the acceptance criteria, so each one becomes a check of its own.
# WB-1 Catalog: show the number of resultsAcceptance criteria:1. After the catalog loads with N >= 1 items, a line "N items found" shows above the grid.2. With one item it reads "1 item found".3. When loading fails and nothing is cached, only the error banner shows, with no count line.4. The line is aligned with the grid and scrolls with it. - Give your agent one paragraph.
Name the ticket, the target, the app and the base branch. The skill carries the rest of the playbook.
Pre-test the ticket in tickets/WB-1.md on emulator-5600, app com.example.app, base branch main. Follow the ticket-pretest skill. Pack the result and tell me the verdict, the findings and the pack path.
You should see the verdict first, then what set it, what was blocked and which edge cases were covered.
What to look at. The verdict names the three checks that set it. The ticket in that run had seven criteria, and the agent planned 16 checks. It covered four of the 18 kit cases and named the 14 it skipped. One check was blocked because the app's server did not answer from the test machine, and the pack says so instead of calling it a pass.
Plan the checks before the first tap
Goal: turn the ticket into planned checks, so nothing is skipped quietly.
- Own the device and launch the app.
The start takes its memory and CPU baseline from the running app.
autonom session start --serial emulator-5554 --app-id com.example.app --launch - Start the pre-test.
Run it from the app's repository;
--basenames the branch the change is compared with.autonom pretest start --ticket tickets/WB-1.md --base main --ticket-ref WB-1 - Read what changed.
Read it next to the ticket before touching the device.
autonom pretest context - Add one check per acceptance point.
Say why, so a reviewer sees the link to the ticket.
autonom pretest check add "Results show a count line above the grid" --kind ticket --why "acceptance point 1" - Mark a criterion that is also a kit case.
Then the kit counts it, and you do not plan it twice.
autonom pretest check add "A failed load shows only the error banner" --kind ticket --case server-error --why "acceptance point 3 (also the kit case)"
You should see the one changed file, its changed lines, and no flow that covers it.
What to look at. selected is empty and the changed file is in uncovered_files: no flow covers this change, so hand checks are the only proof. A third check, for acceptance point 4, is added the same way.
Show full output
Run a check and record a finding
Goal: check acceptance point 4, that the count line lines up with the grid.
- Open the check.
From here on, every screenshot and action is filed under it.
autonom pretest check begin chk-2 - Find where the line starts.
A find reads the screen once and never taps.
autonom ui find --desc "7 items found" - Find where the cards start.
The first card is the one to compare with.
autonom ui find --desc "Merino yarn" - Keep a screenshot.
It is the evidence the finding will cite.
autonom screenshot --label alignment - Close the check with its verdict.
The note says why, in plain words.
autonom pretest check end chk-2 --verdict fail --note "count line starts 22 px right of the cards" - Record the finding.
Expected against actual, with the screenshot as evidence.
autonom pretest finding add --check chk-2 --severity minor --title "Count line is indented relative to the grid" --expected "the line lines up with the cards" --actual "the line starts at x=66 px, the cards at x=44 px" --evidence shot:2
You should see two left edges that differ, then a finding tied to the check and the screenshot.
What to look at. The line's left edge is 66 and the card's is 44, so the check fails on a measured fact. shot:2 is the second row of autonom shots list. A reference that names nothing is refused with evidence_ref_unknown, and nothing is written.
Show full output
See a risky tap refused
Goal: confirm a pre-test never buys, pays or deletes by accident.
- Try to tap Buy now.
The default forbidden list holds buy.
autonom ui tap --desc "Buy now" - Read the budget.
It shows what was counted and what was refused.
autonom pretest status
You should see the tap refused, with only the earlier Catalog tap counted.
What to look at. forbidden_action says no input was sent. Only the Catalog tap counted; the finds and screenshots did not. Never work around a refusal with a nearby coordinate tap or a direct device command. Only the person who asked for the pre-test widens --forbid-text.
Pack the result and send it back
Goal: seal the evidence, then record QA's decision beside it.
- Pack the pre-test.
It ends the pre-test, sets the verdict and writes the pack.
autonom pretest pack --out packs/WB-1 - Verify the pack.
Every file is checked against its digest.
autonom report verify packs/WB-1 - Send it back to the developer.
A return names the finding to fix.
autonom pretest review packs/WB-1 return-to-dev --finding f-1 --note "count line not aligned with the grid"
You should see a failed verdict, a verified pack and a review written next to it.
What to look at. The verdict is fail because chk-2 failed. The third check never began, so summary.md lists it under Not covered. The review lands in annotations/, outside the sealed files, so the pack still verifies. A packed pre-test refuses changes with pretest_finalized.
Show full output
Reproduce these outputs
From the root of an Autonom checkout at tag v0.36.1 (commit ca2a280), in bash or zsh. Autonom's test device stands in for the emulator, and the catalog screen is a fixture the script writes. Ids, times and digests differ on every run. The summary.md excerpt in the first example comes from a real pre-test on an Android emulator and is not reproduced here.
Good to know
Plan before you drive. A check planned and never run shows up as not covered, instead of quietly vanishing.
Drive by meaning. Tap by text or description and end each check with a
ui findon the state you expect. A draft flow can be compiled from that; a coordinate tap cannot be.Let the screen settle.
ui findreads once and never waits. After an action that loads, runui wait --settledfirst.Separate facts from guesses.
--actualsays what was seen. A guess about the cause belongs in your report, marked as a guess.Mocked data is marked. A screenshot taken while a mock is active says "(mocked data)" in
summary.md. Never present it as the real server's behavior.Automatic findings are leads.
pretest health,a11yandeventswrite unconfirmed findings. They never change a verdict until someone confirms one.Repeated taps are slow. Every guarded tap reads a fresh tree first, about 2.5 s each. A double-tap check passes only with an observed request count; otherwise it is inconclusive.
A missing environment is a finding, not a pass. When a flow fails because the test machine lacks data or a server, record an info finding on that check and say so.
One pre-test per session. A second start is refused with
pretest_already_active. An ended check cannot be ended again (no_open_check); add a finding instead.No request that never answers, yet. Shaping slows every request, and a mock cannot hold its answer. An unfinished request is not recorded either. Run the shaped version, judge the loading screen, and end the check inconclusive.
A Canvas tap with no origin counts as a person's. In Mobile Canvas, input marked
humanis filed under the open check but never counted against the budget or refused by it. A person's first input also takes the device's control lease, so your agent's own commands are refused withcontrol_helduntil Hand back or idle release. A script that drives Canvas without naming its origin is treated as a person on both counts, so it should name its origin asagent.The control lease comes first. It is checked before the pre-test guard, so a refused lease never uses budget.
pretest flows,repro,rerunandwarmuprun under the lease likeflow run, and a run interrupted by it makes its checkblocked.
After the pack
QA reads summary.md, or opens index.html from disk, and records one decision beside the pack: approve, more checks, or return to the developer. A return names at least one finding. QA can also confirm or dismiss an automatic finding. More checks starts a linked child pre-test with the pack's ticket and context. Its app, budget and safety rules come from the current session or the review's own flags, never from the pack.
A finding can become a regression check:
pretest flow-draftcompiles the finding's check into a draft flow, taggeddraft, that covers the changed files and passesflow check. Read its last assertion before you trust it.pretest reproreplays the draft 1 to 10 times and records how often the finding came back: stable, intermittent or not reproduced. A run that ends without a verdict stops it and records nothing.pretest rerunruns a pack's checks again on a new build, in a linked child pre-test. Its pack shows each check before and after; the first pack is only read.
A finding or check with nothing to replay is refused with draft_unavailable.
App profile and extra checks
An app profile tells the pre-test how to sign in and warm the app up. It lives in the app's repository at .autonom/apps/<app-id>/profile.yaml, or in your home folder at ~/.autonom/apps/<app-id>/profile.yaml. pretest context says which one it used.
No secrets in a profile. Accounts name environment variables, never values. A value that looks like a password or token refuses the whole profile with
profile_invalid, naming the key, never the value.Warm-up.
pretest warmupruns the login and warm-up flows, then walks recorded paths to the screens the diff may show. Each screen is reported as reached, unreplayable, known or unknown. It counts against the budget.Accessibility. With an
a11ycheck open,pretest a11yflags controls without a label, tap targets under 48 dp on Android or 44 pt on iOS, and controls that share a label. It does not check contrast or reading order.Analytics. With traffic captured and an
analyticscheck open,pretest eventsmatches the profile's event rules against the requests. It reports matched, missing and unexpected events, and never copies a request body value. Without capture the result is unavailable; end that check blocked.
Reference
Every verb answers in JSON. An expected refusal exits 2 with an error_code and a hint.
| Command | What it does |
|---|---|
autonom pretest start --ticket FILE|- --base REF | Starts one pre-test in the current session. --max-actions, --max-minutes, --forbid-text and --allow-app set the guard rails; --forbid-text replaces the default list. |
autonom pretest context | The changed files and lines, the selected flows, the files no flow covers, screen hints and the app profile. |
autonom pretest kit | The 18 edge cases, and their coverage once checks exist. |
autonom pretest check add <title> --kind KIND [--case KIT-ID] [--why TEXT] | Plans a check. Kinds: ticket, edge, regression, network, a11y, analytics. |
autonom pretest check begin <check-id> | Opens a check. One check is open at a time. |
autonom pretest check end <check-id> --verdict V [--note TEXT] | Closes it with one of the five verdicts. An ended check never reopens. |
autonom pretest flows | Runs each selected flow as a regression check: passed is pass, failed is fail, broken is blocked. |
autonom pretest finding add --check ID --severity S --title T --actual T [--expected T] | Records a finding: blocker, major, minor or info. --evidence REF (repeatable) cites a shot, tree, action, request, log range, run or crash. |
autonom pretest status | Actions and minutes used and left, the open check, counts and refusals. |
autonom pretest health | New crashes, error log groups, failed requests and metric leads since the start, as automatic findings. |
autonom pretest pack [--out DIR] | Ends the pre-test, sets the verdict and writes summary.md, summary.json and an offline index.html, sealed. Packing again returns the same pack. |
autonom report verify <pack> | Checks every file of the pack, nested flow bundles included. |
autonom pretest review <pack> <approve|more-checks|return-to-dev> | Records QA's decision beside the pack. review-status reads it back. |
autonom pretest flow-draft <finding-id> | Compiles the finding's check into a draft flow under .autonom/flows/drafts/. |
autonom pretest repro <finding-id> --times N | Replays the draft N times, 1 to 10, on the active pre-test. |
autonom pretest rerun <pack> [--check ID]... | Reruns checks on the current build in a linked child pre-test. |
autonom pretest warmup | The profile's login and warm-up flows, then the hinted screens. |
autonom pretest a11y [--check ID] | Accessibility rules on the current screen. |
autonom pretest events [--check ID] | The profile's analytics rules against the captured requests. |
autonom pretest stop | Closes the pre-test without a pack; pretest pack can still pack it later. pretest list shows the session's pre-tests. |
The edge-case kit
--case takes one of these ids. Undo each setup before the next check. When a case cannot be set up on the target, end its check not_covered with the reason.
| Case | How it is set up |
|---|---|
offline | No network: simulator network offline, then online. |
slow | A slow network: simulator network shape at edge speed, then full. |
server-error | A 5xx: capture on, then a network mock add with status 500. |
timeout | A request that never answers: capture on, the network shaped to GSM speed. |
background-resume | Home, then session launch of the app. |
process-death | session force-stop, then session launch. |
rotation | A flow step setOrientation: landscape, then portrait. |
dark-mode | simulator appearance dark, then light. |
large-text | simulator text-size: 1.3 on Android, accessibility-extra-large on iOS. |
locale | Another locale: simulator keyboard pin, then reset. |
permission-denied | permissions revoke, then reset. |
empty-input | Submit a form with the field left empty. |
long-input | Type 500 characters into the field. |
emoji-input | Type text with emoji. |
rtl-input | Type right-to-left text. |
double-tap | Tap the same button twice, quickly. |
fast-back | Start an action, then Back while it loads. |
deep-link | open the app link, then again after a force-stop. |
Android and iOS differences
The playbook, verdicts and pack are the same on both. What differs is which edge cases a target can set up, and what the guard rails can check.
| Capability | Android | iOS |
|---|---|---|
| Offline, slow and timeout cases | Yes, on an emulator | Refused with unsupported_capability |
| Locale case | No | Yes |
| Rotation and fast Back cases | Yes | No |
| A link checked against allowed apps | Yes; a link that would open the app chooser is refused | Counted, not checked: iOS cannot say which app takes a link |
Smallest tap target in a11y | 48 dp | 44 pt |
Next steps
Your agent can pre-test a ticket and hand QA a pack it can check.