CI campaigns
Run your flows as one campaign on a CI runner. Autonom keeps every run as a verified report bundle, gates the runs on explicit rules, can publish the whole campaign, and ends with an exit code your pipeline can act on.
Try it autonom ci run flows --out ci-1
Needs a session on one target · flows
How it works
autonom ci run PATH --out DIR takes one flow file, or a directory of flows, and runs each flow on the current session's target. It works in phases, and writes the phase it has reached to ci-state.json in --out after each one:
capture: the flows run one after another, with full evidence. Each run is finalized as a report bundle underbundles/and packed as one shard underpacks/, andrunsrecords its flow, run id, status, bundle and pack. A flow that Autonom refuses to start is recorded undererrors, and the flows after it do not run.merge: the shards are merged intocampaign/campaign.json, which compares the shards it expected with the shards it received.finalize: every run is checked against the gate rules, and the verdict is written tocampaign/final.json.publish: only with--publish. The finalized campaign is copied to an empty directory, or sent to an HTTP endpoint.complete: the campaign is done, andci-state.jsonalso holds itsfinalverdict and itspublication.
A campaign expects one shard per flow, unless --expected-shards says how many to wait for. A campaign that receives fewer is incomplete and fails, even when every run that did arrive passed. In v0.36.1, missing_shards is the expected number minus the number of distinct attempts received, not of distinct shard ids. Merge drops a pack only when its attempt id was already seen, so a runner that reruns ci run with the same --shard-id adds a new attempt that counts as one more shard, and can hide a runner that never reported.
--env KEY=VALUE sets a variable that a flow reads as ${KEY}. --secret NAME reads a secret from the runner's environment by its name. Autonom records only the name, so the value is not written into the flow, the recorded command or the steps' arguments. Screen evidence keeps what the screen showed, as it does for any run.
Without --rules, the gate asks each run for status passed, a proof verdict of pass or not_applicable, no failed or broken step, and evidence for every failure. --rules FILE names a JSON object whose keys replace those defaults one by one; a key the file leaves out keeps its default.
The same work is split into commands for a campaign that runs on several machines: ci pack packs one bundle as a shard, ci merge merges the packs, ci finalize gates the merged campaign, and ci publish publishes a finalized one.
Of the ci commands, only ci pack verifies a bundle. ci merge reads shard.json and bundle/model/report.json from each pack and checks only the report's schema, not the bundle's digests, and ci finalize gates the reports merge copied. A pack changed between ci pack and ci merge, for example a report whose failed run was flipped to passed, is merged and gated as it is. When packs come from runners or channels you do not control, unzip each one and run autonom report verify on its bundle/ directory before the merge, or compare the sha256 that ci pack printed on the runner with the sha256 that campaign.json records for that shard before you finalize. Even a verified bundle only shows that it is whole, not who made it, as Report bundle explains.
ci run ends with one of three exit codes:
0, statuspassed: every expected shard arrived, every run passed its gate, and the publication, if you asked for one, succeeded.1, statusfailed: a run failed its gate, a shard is missing, or a flow could not start.2: the publication failed, whatever the status, andpublication.publication_statusisfailed. Exit code 2 with an error object on stderr instead means the campaign was refused before any flow ran.
The other commands keep to their own job. ci merge exits 1 while shards are missing. ci finalize exits 0 when the status is passed and 1 when it is not, even when its own --publish failed, so read publication.publication_status too. In v0.36.1 the top-level publication_status of ci finalize, and the one in final.json, always says not_requested, whether the publication failed, succeeded or was never asked for. ci publish exits 2 with ci_destination_failed when it cannot publish.
Use a new --out directory for every campaign. In v0.36.1, ci run adds to what the directory already holds, and the gate checks every run left in campaign/models/, including the runs of earlier campaigns. The same goes for ci merge --out: a directory that held an earlier merge keeps its runs and its final.json.
Examples
Every output here comes from Autonom's test emulator showing a PIN screen, so you can rerun it without a device. Two flows in flows/ pass: title.yaml looks for the title it gets from --env TITLE, and continue.yaml looks for the Continue button. sign-in.yaml types a PIN from the APP_PIN secret, taps Continue, then waits for a Welcome screen that never comes, so it fails. On a real device or Simulator the runs take as long as their flows, and the bundles hold real screenshots, so times, sizes and attachment counts differ. Reproduce these outputs, at the end of this section, has the exact commands.
Run a campaign
Goal: run every flow in a directory on the runner's device and get one verdict.
- Start a session on the runner's target.
autonom session start --serial emulator-5554 --app-id com.example.app - Run the flows as one campaign.
autonom ci run flows --out ci-1 --env TITLE="Enter your PIN" - Read what the campaign wrote, and the phase it reached.
ls ci-1 python3 -c 'import json; s = json.load(open("ci-1/ci-state.json")); print(json.dumps({k: s[k] for k in ("schema", "phase", "runs", "errors")}, indent=2))'
You should see status passed and exit code 0, then the directory and its state.
What to look at. execution counts the runs, and status is the gate's verdict on them. The flows ran in name order, each kept as a bundle and a pack, and the packs are named after the default --shard-id, shard. phase: complete says the campaign got to its end; a runner that stopped halfway leaves the phase it was in.
When a flow fails
Goal: see which rules a failed run broke.
- Run the failing flow, with its secret.
APP_PINis exported in the runner's environment.autonom ci run sign-in.yaml --out ci-2 --secret APP_PIN - Read the gate's failures from the campaign's verdict.
python3 -c 'import json; f = json.load(open("ci-2/campaign/final.json")); print(json.dumps(f["gates"][0]["failures"], indent=2))'
You should see status failed with exit code 1, then three broken rules.
What to look at. The run failed, its proof verdict is fail, and one step failed where none is allowed. require_evidence_for_failures is not in the list, because the failing step kept its screenshot and screen tree. The run's bundle is under ci-2/bundles/, ready for agent export or replay.
Keep secrets out of the campaign
Goal: check that the PIN the flow typed was not written into the campaign.
- Search every file of the campaign for the PIN's value.
grep -rl 271828 ci-2 || echo "the PIN is in no file under ci-2"
You should see that no file holds it.
What to look at. --secret APP_PIN passed the value from the environment, and the campaign recorded only the name. This test screen never shows what was typed. A real app that displays the PIN puts it into the screen trees and screenshots of its bundles, so use a test credential, and do not publish such a campaign as a public artifact.
Split a campaign across runners
Goal: run a campaign on several machines and judge it as one.
- On each runner, run its share with its own shard id.
Each runner writes its pack as
packs/<shard-id>-1.zip.autonom ci run flows/title.yaml --out runner-1 --shard-id one --env TITLE="Enter your PIN" autonom ci run flows/continue.yaml --out runner-2 --shard-id two - Merge the packs that have arrived, out of the three expected, and finalize.
autonom ci merge runner-1/packs/one-1.zip runner-2/packs/two-1.zip --out merged-1 --expected-shards 3 autonom ci finalize merged-1 - Pack the third runner's bundle.
That runner ran its flow with
flow runand kept it withreport bundle.autonom ci pack three --out three.zip --shard-id three - Merge every pack into a new directory, and finalize again.
The first pack is passed twice, as a retried upload would deliver it.
autonom ci merge runner-1/packs/one-1.zip runner-2/packs/two-1.zip three.zip runner-1/packs/one-1.zip --out merged-2 --expected-shards 3 autonom ci finalize merged-2
You should see incomplete and exit code 1 while a shard is missing, then complete and passed with exit code 0.
What to look at. Both runs that arrived first passed their gates, yet the campaign failed: missing_shards: 1 holds it back until the third shard comes in. The pack passed twice counts once, because merge keeps one copy of each attempt, so a retried upload does not count twice. A rerun of ci run on the same runner is a new attempt, though, and counts as one more shard. history groups the runs by flow across shards, which is how a flow that passed on one runner and failed on another shows as flaky.
Gate on your own rules
Goal: loosen one rule, and see what happens to a key Autonom does not know.
- Write a rules file.
max_failed_stepsallows one failed step.allowed_statusis a misspelling ofallowed_statuses.echo '{"max_failed_steps": 1, "allowed_status": ["passed", "failed"]}' > rules.json - Finalize the failed campaign again with it.
autonom ci finalize ci-2/campaign --rules rules.json
You should see status failed and exit code 1, now with two broken rules instead of three.
What to look at. max_failed_steps: 1 replaced its default, so the one failed step no longer breaks a rule; the other defaults still apply. Autonom v0.36.1 ignores a key it does not know: allowed_status is copied into rules as if it applied, and checks nothing, so allowed_statuses still asks for passed. Compare your keys with the rules in the reference below before you rely on a file.
Publish the result
Goal: keep the campaign where the pipeline can find it, and tell a failed upload from a failed test.
- Run a campaign and publish it to an empty directory.
autonom ci run flows --out ci-3 --env TITLE="Enter your PIN" --publish published - Run the next campaign, publishing to the same, now full, directory.
autonom ci run flows --out ci-4 --env TITLE="Enter your PIN" --publish published - Do the same with ci finalize, then publish to an endpoint that is down.
autonom ci finalize merged-2 --publish published autonom ci publish merged-2 http://127.0.0.1:9/upload
You should see published with exit code 0, then a failed publication with exit code 2 from ci run, but 0 from ci finalize, and an error from ci publish.
What to look at. A failed publication never changes the verdict: the second campaign still says passed. ci run exits 2 so the pipeline notices the missing upload, while ci finalize exits by the verdict alone, so a script around it has to read publication.publication_status. The top-level publication_status above still says not_requested although the upload failed, so do not test that one. A directory destination must be empty or absent. An HTTP destination receives the finalized campaign as one ZIP file by PUT, with an Idempotency-Key header holding its SHA-256 digest. Autonom sends no authorization header, so an endpoint that needs one can only take it in the URL, and the URL is not hidden: a successful publication prints it in full as destination, ci run also stores it in ci-state.json, and the error of ci publish repeats it. Use a short-lived URL that can write only this one object, and treat the CI log and ci-state.json as holding it. On Linux the refused connection reads [Errno 111].
Use a fresh --out for every campaign
Goal: see why a reused directory gives a wrong verdict.
- Run a passing flow into ci-2, the directory of the failed campaign.
autonom ci run flows/title.yaml --out ci-2 --env TITLE="Enter your PIN" - Count the runs the gate checked.
ls ci-2/campaign/models | wc -l
You should see status failed with exit code 1 for a campaign whose only run passed, and two runs.
What to look at. execution counts only this campaign's run, which passed. The gate also checked the failed run of the earlier campaign, still in campaign/models/, and so the verdict is failed. This is how Autonom v0.36.1 behaves. Give every campaign its own directory, for example one named after the pipeline's run number.
Read the refusals
Goal: tell why a command did nothing, from the error code alone.
Each refusal below is one error object on stderr, with exit code 2, and no flow runs.
What to look at. flow_secret_undefined names the variable to export on the runner. flow_command_invalid quotes the --env value that has no =. ci_destination_failed from ci publish means nothing was published; run ci finalize first. no_active_session asks for a session on the runner's target.
Reproduce these outputs
From the root of an Autonom checkout at tag v0.36.1 (commit ca2a280), in bash or zsh. The autonom function runs that checkout against its test emulator, and the script writes the PIN screen it shows and the four flows. Campaign ids, run ids, attempt ids, digests and times differ on every run, and ids are shortened to … on the page. On macOS the paths start with /private/tmp.
Good to know
One directory per campaign. Name
--outafter the pipeline's run, and merge into a new directory each time. A reused directory adds the runs it already holds to the verdict.Branch on the exit code, then on the fields. 0 means passed, 1 means a test or a shard is the problem, and 2 means the upload or the command line is.
ci finalizeis the exception: read itspublication.publication_status, not the top-levelpublication_status, which staysnot_requested.Keep the bundles. Every run of a campaign is a verified report bundle under
bundles/, with its own copy of the flow, so a failure can be replayed on another machine after the runner is gone.Shard by flow, not by step. A shard is one run of one flow. Give each runner its own
--shard-id, and tell the merge how many to expect with--expected-shards, so a runner that died shows asmissing_shardsinstead of a smaller green campaign. The count is of distinct attempts, so to retry a failed upload send the same pack again, which merge ignores, and after a rerun check that theshard_idvalues incampaign.jsonare the ones you expected.Check packs before the merge.
ci mergedoes not verify bundles, so a pack edited on its way to the merge host is merged as it is. Runautonom report verifyon each unzipped pack'sbundle/directory, or compare each pack'ssha256fromci packwithcampaign.json, before you trust the verdict.Secrets come from the runner's environment. Set them as the pipeline's secret variables and pass their names with
--secret. A screen that shows a typed value still puts it into the evidence.Unknown rule keys are ignored. v0.36.1 does not reject a misspelled key in
--rules; check the gate result'srulesthe first time you use a file.
Reference
The commands of a campaign.
| Command | What it does |
|---|---|
autonom ci run PATH --out DIR | Runs one flow, or every flow in a directory, on the current session's target, then merges, gates and, with --publish, publishes. |
--shard-id ID | Names this runner's packs ID-1.zip, ID-2.zip and so on. The default is shard. |
--expected-shards N | The number of shards the campaign waits for. The default is one per flow. |
--campaign-id ID | Sets the campaign id instead of a fresh one. |
--env KEY=VALUE | Sets a variable the flows read as ${KEY}. Repeat it for more. |
--secret NAME | Reads a secret from the environment by name. Only the name is recorded. |
--rules FILE | A JSON object of gate rules that replace the defaults key by key. |
--publish DEST | Publishes the finalized campaign to an empty directory or an HTTP endpoint. |
autonom ci pack BUNDLE --out FILE | Verifies a report bundle and packs it as one shard, a ZIP file. Takes --shard-id; the default is shard-1. |
autonom ci merge PACK… --out DIR | Merges packs into a campaign, one copy per attempt, and counts missing_shards by distinct attempts. Checks each report's schema, not the bundle's digests. Takes --expected-shards; the default is the number of distinct attempts received. |
autonom ci finalize DIR | Gates a merged campaign and writes final.json. Takes --rules and --publish. |
autonom ci publish DIR DEST | Publishes a finalized campaign to an empty directory or by HTTP PUT. |
What ci run writes into --out.
| Path | What it holds |
|---|---|
ci-state.json | autonom.ci-state/v1: the phase, the runs with their flow, run id, status, bundle and pack, and the errors. At the end, final and publication too. |
campaign-spec.json | autonom.campaign/v1: the campaign id and the shards it expects. |
bundles/<run-id>/ | The report bundle of each run. |
packs/<shard-id>-<n>.zip | The shard pack of each run: its bundle and a shard.json descriptor. |
campaign/campaign.json | autonom.campaign-result/v1: expected, received and missing shards, the shards, the history by flow and the summary. |
campaign/models/ | The report of every run the gate checks. |
campaign/final.json | autonom.campaign-final/v1: the verdict, the execution status, the missing shards and one gate result per run. |
The gate rules, with their defaults.
| Rule | Default | A run fails it when |
|---|---|---|
allowed_statuses | ["passed"] | Its status is not in the list. |
allowed_proof_verdicts | ["pass", "not_applicable"] | Its proof verdict is not in the list. |
max_failed_steps | 0 | More steps failed than this. |
max_broken_steps | 0 | More steps broke than this. |
require_evidence_for_failures | true | A failed or broken step kept no evidence. |
The exit codes.
| Command | 0 | 1 | 2 |
|---|---|---|---|
ci run | Status passed, and the publication, if requested, succeeded. | Status failed. | The publication failed, or the command was refused. |
ci merge | Every expected shard arrived. | Shards are missing. | Refused. |
ci finalize | Status passed, whatever the publication did. | Status failed. | Refused. |
ci pack, ci publish | Done. | Not used. | Refused, for example with ci_destination_failed. |
The error codes, each with exit code 2.
| Error code | When |
|---|---|
ci_destination_failed | The destination is not empty, is not a directory, or did not accept the upload, or the campaign is not finalized. From ci run and ci finalize it comes inside publication, in a successful answer. |
flow_secret_undefined | A --secret variable is not set in the environment. |
flow_command_invalid | An --env value has no =. |
no_active_session | ci run has no session to run on. |
report_integrity_failed | ci pack was given a bundle that fails verification. |
usage_error | --out is missing. |
Next steps
You can run a campaign, gate it and publish it from a pipeline.