Agent report

Give your agent a run as compact JSON instead of an HTML report. The agent report names the failure, the steps and the evidence each one kept, and one step can be inspected on its own.

Try it autonom agent export --bundle <dir> --out agent.json

Needs a flow run

How it works

agent export writes one autonom.agent-report/v1 file. agent inspect prints a single step, with the run's attempt and only that step's attachments. Both read the same report, from one of two sources:

  • A report bundle, with --bundle. Autonom verifies every file of the bundle against its digest before it reads anything, so an edited bundle is refused instead of reported on.

  • A run of a session, with --run, from the current session or the one --session names. A stopped session works too. With no --run, the session's latest run is read.

Both commands only read. They change nothing in the bundle or the session, and need no device; agent export writes only its --out file. A run records a secret's name, not its value, in the flow, the command line and the steps' arguments. Evidence of the screen is kept as it was, though: a typed secret that the screen shows is in the screen trees, the screenshots and the steps' target text, and so in the bundle and the report.

agent inspect --step takes the step's runtime index, such as 6, or its stable step id, such as src_4d88cd3b8ce7.1. The stable id is the safer choice for anything your agent stores, because it names the same step after a flow gains a conditional branch.

Examples

Every output here comes from Autonom's test emulator showing a PIN screen, so you can rerun it without a device. The run types a PIN from the APP_PIN secret, taps Continue, then waits for a Welcome screen that never comes, so step 6 fails. Reproduce these outputs, at the end of this section, has the exact commands.

Export the report of a failed run

Goal: hand your agent the failure and its evidence in one file.

  1. Export the report from the run's bundle.

    autonom report bundle --run ID --out DIR wrote the bundle when the run ended.

    autonom agent export --bundle bundle --out agent.json
  2. Read it.

    The file is compact JSON; python3 -m json.tool indents it.

    python3 -m json.tool agent.json

You should see where the file went, then the report: the failure, the steps and a summary.

{ "ok": true, "format": "agent", "out": "/tmp/autonom-agent/agent.json"}
{ "attachments": [ … ], "capabilities": { … "device_class": "simulator", "provider": "local.android", "schema": "autonom.provider/v1", "target_id": "emulator-5554" }, "failure": { "command": "assertVisible", "error": "{'text': 'Welcome', 'match': 'exact'} was not visible within 1000 ms", "error_code": "flow_assertion_timeout", "failure_class": "test_failure", "flow": "/tmp/autonom-agent/pin.yaml", "line": 15, "step_index": 6 }, "replay": { "mode": "baseline", "portable_restore": "replay-from-flow-start", "source_run_id": null }, "schema": "autonom.agent-report/v1", "setup": { … }, "steps": [ … { "attachment_ids": [ "att_…", "att_…" ], "command": "assertVisible", "delta": { … }, "error": { "class": "test_failure", "code": "flow_assertion_timeout", "message": "{'text': 'Welcome', 'match': 'exact'} was not visible within 1000 ms" }, "index": 6, "name": "assertVisible", "status": "failed", "step_id": "src_4d88cd3b8ce7.1" } ], "summary": { "attachments": 5, "attempt_id": "attempt_…", "case_id": "case_daaab23de6ec5430", "first_causal_failure": "src_4d88cd3b8ce7.1", "history_id": "hist_9d9270ba34dd43aa", "proof_verdict": "fail", "run_id": "fr_…", "schema": "autonom.summary/v1", "status": "failed", "step_status": { "broken": 0, "failed": 1, "passed": 5, "skipped": 0, "unknown": 0 }, "steps": 6 }}

What to look at. failure gives the code, the file and the line, so your agent can branch on flow_assertion_timeout without parsing text. summary.first_causal_failure is the stable id of the step to inspect next. The failing step lists the attachments it kept, a screenshot and a screen tree, by id.

Inspect the failing step

Goal: read one step in full, with only the evidence it kept.

  1. Inspect it by its index.autonom agent inspect --bundle bundle --step 6
  2. Inspect it by its stable id.

    The answer is byte for byte the same.

    autonom agent inspect --bundle bundle --step src_4d88cd3b8ce7.1

You should see the run's attempt, the step and its two attachments.

{ "ok": true, "attempt": { "attempt_id": "attempt_…", "execution_command": "autonom flow run /tmp/autonom-agent/pin.yaml --secret APP_PIN", … "proof_verdict": "fail", "reproduction": "autonom replay --bundle <bundle>", "retry_of": null, "run_id": "fr_…", … "status": "failed" }, "step": { … "command": "assertVisible", … "error": { "class": "test_failure", "code": "flow_assertion_timeout", "message": "{'text': 'Welcome', 'match': 'exact'} was not visible within 1000 ms" }, … "index": 6, "name": "assertVisible", … "source_id": "src_4d88cd3b8ce7", … "status": "failed", "step_id": "src_4d88cd3b8ce7.1" }, "attachments": [ { … "blob": "blobs/sha256/da/da41…", "capture_kind": "hierarchy", "mime_type": "application/json", "name": "failure-step-6-hierarchy.json", "path": "flows/fr_…/failure-step-6-hierarchy.json", … "type": "ui_tree" }, { … "blob": "blobs/sha256/…", "capture_kind": "screenshot", "mime_type": "image/png", … "path": "shots/fr_…/0002_…_failure-step-6.png", … "type": "screenshot" } ]}
index 6 and src_4d88cd3b8ce7.1 give the same answer

What to look at. execution_command is the command that made the run; the secret appears only as --secret APP_PIN. reproduction is how to rebuild it, with Replay. To open an attachment from a bundle, join the bundle directory and its blob, such as bundle/blobs/sha256/da/da41…. path is where the file was in the session's directory, flows/ or shots/ under the session, and the bundle has no such folder. A report read from a session with --run has no blob; there, path is relative to the session's directory.

Check that nothing changed and nothing leaked

Goal: confirm the report left the bundle as it was, and that the PIN was not recorded.

  1. Verify the bundle again.autonom report verify bundle
  2. Search the bundle and the report for the PIN.grep -rl 271828 bundle agent.json

You should see the bundle verified, and no file holding the secret, because the PIN field of this screen never shows what was typed.

{ "ok": true, "bundle": "/tmp/autonom-agent/bundle", "files": 18, "run_id": "fr_…"}the secret is in neither the bundle nor the report

What to look at. Every file still matches its digest after the export and both inspections. The second line comes from the script: grep found no file. The run kept only the name APP_PIN; the next example shows what changes when the screen shows the value.

Know that screen evidence keeps what the screen showed

Goal: see where a secret goes when the app displays it, before you share a bundle or a report.

  1. Run the flow again on a screen that shows the PIN.

    The script now shows 271828 in the PIN field, as a plain text field shows what was typed into it. It runs the flow, bundles the run into shown and exports its report to shown-agent.json.

    autonom flow run pin.yaml --secret APP_PIN
  2. Search the new bundle and report for the PIN.grep -rl 271828 shown shown-agent.json

You should see the PIN in the report, in the bundle's screen trees and event stream, and in the files built from them.

shown-agent.jsonshown/blobs/sha256/…shown/blobs/sha256/…shown/blobs/sha256/…shown/model/report.jsonshown/run.jsonshown/steps/src_f8b360411243.1.jsonshown/streams/events.ndjson
the shown bundle still verifies

What to look at. The three blobs are the two screen trees and the event stream; the step file is step 2, whose target text is the field's text. report verify still passes, because nothing was changed after the run: the value was recorded as screen content, not as the secret. On a real device the screenshots show it too. Treat a bundle or an agent report as sensitive when a secret can appear on screen: have the app mask the field, use a test credential, and do not publish such a bundle as a public artifact.

Read the refusals

Goal: tell why a report was not produced, from the error code alone.

Each refusal is one error object on stderr, with exit code 2.

{"ok": false, "error_code": "flow_replay_step_not_reached", "error": "report has no step '7'"}
{"ok": false, "error_code": "no_active_session", "error": "no active session", "hint": "Start one with 'autonom session start', or pass a session id."}
{"ok": false, "error_code": "report_integrity_failed", "error": "report bundle integrity verification failed", "failures": [{"path": "steps/src_4d88cd3b8ce7.1.json", "reason": "digest-mismatch"}]}

What to look at. flow_replay_step_not_reached means the step index or id is not in this run. no_active_session asks for a source: pass --bundle, or name the stopped session with --session and its run with --run, as the script does after session stop. report_integrity_failed lists each file whose digest no longer matches; nothing is read from that bundle.

Reproduce these outputs

From the root of an Autonom checkout at tag v0.36.1 (commit ca2a280), in bash or zsh. The autonom function runs that checkout against its test emulator, and the script writes the PIN screen it shows, then a copy whose PIN field shows the PIN. Run ids, attempt ids, attachment ids, session ids, times and sizes differ on every run, and ids are shortened to … on the page; step, case and history ids stay the same, because the flow has an id. On macOS the paths inside the report start with /private/tmp.

# autonom below runs this checkout against Autonom's test emulator, showing a PIN screen the script writes.S=$PWD W=/tmp/autonom-agentrm -rf $W && mkdir -p $Wexport AUTONOM_HOME=$W/.autonom AUTONOM_FAKE_STATE=$W/device.jsonautonom() { python3 $S/scripts/autonom.py --adb $S/tests/fakes/fake_adb.py "$@"; }cat > $W/pin.xml <<'XML'<?xml version="1.0" encoding="UTF-8" standalone="yes" ?><hierarchy rotation="0"> <node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.example.app" content-desc="" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" bounds="[0,0][1080,2400]"> <node index="0" text="Enter your PIN" resource-id="com.example.app:id/title" class="android.widget.TextView" package="com.example.app" content-desc="" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" bounds="[80,200][1000,300]" /> <node index="1" text="" resource-id="com.example.app:id/pin" class="android.widget.EditText" package="com.example.app" content-desc="PIN" clickable="true" enabled="true" focusable="true" focused="true" scrollable="false" bounds="[80,400][1000,520]" /> <node index="2" text="Continue" resource-id="com.example.app:id/continue" class="android.widget.Button" package="com.example.app" content-desc="" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" bounds="[80,600][1000,720]" /> </node></hierarchy>XMLecho "{\"ui_dump\": \"$W/pin.xml\"}" > $AUTONOM_FAKE_STATEcat > $W/pin.yaml <<'YAML'schema: autonom.dev/flow/v1appId: com.example.appname: PIN sign-inid: pin-sign-in---- launchApp- tapOn: selector: description: PIN- inputText: ${APP_PIN}- checkpoint: pin-entered- tapOn: selector: text: Continue- assertVisible: selector: text: Welcome timeoutMs: 1000YAMLautonom session start --serial emulator-5554 --app-id com.example.app > $W/session.jsonSESSION=$(python3 -c 'import json, sys; print(json.load(open(sys.argv[1]))["session"]["session_id"])' $W/session.json)export APP_PIN=271828# A run that fails at step 6, kept as a report bundleautonom flow run $W/pin.yaml --secret APP_PIN > $W/run.jsonRUN=$(python3 -c 'import json, sys; print(json.load(open(sys.argv[1]))["run_id"])' $W/run.json)autonom report bundle --run $RUN --out $W/bundle > /dev/null# Export the report of a failed runautonom agent export --bundle $W/bundle --out $W/agent.jsonpython3 -m json.tool $W/agent.json# Inspect the failing stepautonom agent inspect --bundle $W/bundle --step 6 | tee $W/by-index.jsonautonom agent inspect --bundle $W/bundle --step src_4d88cd3b8ce7.1 > $W/by-id.jsoncmp $W/by-index.json $W/by-id.json && echo "index 6 and src_4d88cd3b8ce7.1 give the same answer"# Check that nothing changed and nothing leakedautonom report verify $W/bundlegrep -rl 271828 $W/bundle $W/agent.json || echo "the secret is in neither the bundle nor the report"# Know that screen evidence keeps what the screen showedsed 's/text="" resource-id="com.example.app:id\/pin"/text="271828" resource-id="com.example.app:id\/pin"/' $W/pin.xml > $W/shown.xmlecho "{\"ui_dump\": \"$W/shown.xml\"}" > $AUTONOM_FAKE_STATEautonom flow run $W/pin.yaml --secret APP_PIN > $W/run-shown.jsonSHOWN=$(python3 -c 'import json, sys; print(json.load(open(sys.argv[1]))["run_id"])' $W/run-shown.json)autonom report bundle --run $SHOWN --out $W/shown > /dev/nullautonom agent export --bundle $W/shown --out $W/shown-agent.json > /dev/nullgrep -rl 271828 $W/shown $W/shown-agent.json | sed "s|$W/||" | sortautonom report verify $W/shown > /dev/null && echo "the shown bundle still verifies"echo "{\"ui_dump\": \"$W/pin.xml\"}" > $AUTONOM_FAKE_STATE# Read the refusalsautonom agent inspect --bundle $W/bundle --step 7autonom session stop > /dev/nullautonom agent export --out $W/no-session.jsonautonom agent export --session $SESSION --run $RUN --out $W/after-stop.jsoncp -R $W/bundle $W/edited && echo '{}' >> $W/edited/steps/src_4d88cd3b8ce7.1.jsonautonom agent inspect --bundle $W/edited --step 6

Good to know

  • Branch on codes, not on messages. failure.error_code and each step's error.code are stable codes; the messages are for people.

  • Start from first_causal_failure. The summary names the first step that failed or broke, by its stable id; later failures often follow from it. Pass that id to agent inspect.

  • Store step ids, not indexes. An index counts runtime steps, so a conditional or a retry can shift it. A step id stays with the step.

  • Read a bundle when the run matters later. A bundle is verified on every read and keeps its own copy of the flow, so its report stays trustworthy after the session is gone.

  • Missing evidence is marked, not invented. A step's delta says for the screen, logs and requests whether evidence is available, partial or unavailable, instead of showing an empty list as if nothing happened.

  • The snapshot travels with the run. capabilities is the capability snapshot the run was checked against, and setup lists what the run applied and verified.

Reference

Both commands take the same sources.

CommandWhat it does
autonom agent export --out FILEWrites the autonom.agent-report/v1 file of the current session's latest run.
autonom agent inspect --step STEPPrints one step by runtime index or stable step id, with the attempt and that step's attachments.
--bundle DIRReads a report bundle, after verifying every file against its digest.
--run IDReads one run of the session instead of its latest.
--session IDReads from that session, running or stopped, instead of the current one.

The fields of autonom.agent-report/v1.

FieldWhat it holds
schemaautonom.agent-report/v1.
summaryThe run and attempt ids, status, proof_verdict, step counts by status, the attachment count, first_causal_failure, and the case_id and history_id that stay the same across runs of one flow.
failureThe run's first failure: error_code, error, failure_class, the command, the file, the line and the step index. null when the run passed.
setupWhat the run's setup applied, selected, used and verified, and the capabilities that were available.
capabilitiesThe autonom.provider/v1 snapshot the run was checked against.
replayHow the run can be rebuilt. A replay's own report adds the step it stopped at and cleanup_hooks_skipped.
stepsEvery step: step_id, index, name, command, status, error, delta and attachment_ids.
attachmentsEvery piece of evidence: id, type, name, MIME type, path relative to the session's directory and, in a bundle, blob (the file's place in the bundle), size and sha256.

The refusals, each with exit code 2.

Error codeWhen
flow_replay_step_not_reached--step names no step of the run.
report_integrity_failedThe bundle has no manifest, or a file in it no longer matches its digest.
no_active_sessionThere is no --bundle, no --session, and no current session.
usage_error--out is missing from agent export, or --step from agent inspect.

Next steps

Your agent can read a run as compact JSON and inspect any step.

Search documentation

Type to find a guide.

↑ ↓ to move · Enter to open · Esc to close