Gates in a run

How the chapter runner's agents prove a test-run gate. What a gate is (its steps, and how a step is decided) is on Plans and nodes; this page is how a run works one.

Test-run gates

Gates are launchable too. The agent is given the gate's title as the plan states it, makes the change, runs the suites it names, and commits. The gate is done when a run under its tag ({plan}/{node}, already set in the agent's environment) finishes green and Stoat observes it, so the suite has to report to Stoat: the supervisor, or the suite's own Bobcat publisher. A suite that reports nowhere leaves the gate waiting however green it is.

Every process run-plan starts (the agent, --agent-setup, --verify-command) gets Stoat's run receiver added to its BOBCAT_MONITOR_URL: whatever it inherited, or bobcat ui's 5525 when nothing, plus STOAT_RUN_RECEIVER_URL (default http://localhost:5527). A Bobcat publisher sends to every URL in the list, so the run reaches the gate and bobcat ui alike, and nobody has to start the UI for a gate to resolve. A worktree that pins Bobcat 0.60.4 or earlier, which reads one URL, gets the receiver alone.

A suite that reports to no Bobcat monitor goes through stoat gate exec instead, and the prompt says so:

stoat gate exec -- dotnet test --project src/Persistence/PostgresqlTests

It runs the command, tells Stoat it started and how it exited (with the commit, and whether the tree was dirty), and exits with the command's own code. It reads the gate from BOBCAT_RUN_TAG, which the runner sets, or takes <plan>/<node> as its first argument; --step <id> reports one step of several, and the gate is done when every reported step passed. A step's declared cwd, env and timeout (Gate steps) ride the same line as --cwd, --env NAME=VALUE and --timeout, and the prompt writes them there; a run killed at its timeout exits 124. The tag is Stoat's to stamp: a step that sets its own BOBCAT_RUN_TAG, as --env BOBCAT_RUN_TAG=… or as BOBCAT_RUN_TAG=… command the way a shell writes it (a Nuke target that sets one, say), still runs, under Stoat's tag, and exec says so in one line: stoat gate exec: step core set BOBCAT_RUN_TAG=nuke/inner; replaced with my-plan/providers/core, the tag Stoat stamps. Stoat never fails the command: an unreachable Stoat, or one that refuses the evidence, is said on standard error and the exit code stands. Which evidence decides when a step has both a Bobcat run and an exec run is on Plans: How a step is decided. It records its own process id, so if the agent running it is killed mid-run, Stoat sees that process gone and the run reads failed straight away, which leaves the gate ready to run again instead of "running" for hours.

It says what it recorded, on standard error:

stoat: recorded step http for my-plan/providers (run 7f3a…)
stoat: step http for my-plan/providers failed (exit 1) after 7s

or NOT recorded after the reason. The finish carries how long the command ran and the last 1 KB of its output, both streams, since MSBuild writes its errors to standard output and a build that died before any test ran leaves standard error empty. The card, the step chip and the Activity show it as exited 1 after 7s: …error NETSDK1005…: a description of the failure, never evidence.

A pass under another command than the step's declared run: never passes the step; see Checked when it is written, and when it runs.

Running every step: stoat gate run

A gate that declares several steps can be run whole:

stoat gate run <plan>/<node> [--parallel N] [--step <id>]... [--url <stoat>]

It reads the gate's declared steps from Stoat and runs each one exactly as stoat gate exec --step <id> runs it, with the step's run, cwd, env and timeout, so every step's start and finish is recorded on the gate as it happens and the card's step chips move while the rest are still running. Nothing waits for the whole gate to finish. Steps start in the order the plan declares them, at most --parallel at once; the default is 1, one after another. Every selected step runs whatever an earlier one did, since the gate needs each step's own evidence. --step (repeatable) runs only the steps named. Like gate exec, it reads the gate from BOBCAT_RUN_TAG when none is named, and Stoat from --url, then STOAT_URL, then http://localhost:5526.

A step's cwd is relative to the root of the checkout gate run is started in. A gate that names its repo: runs only from a checkout of that repository; from any other it is refused with the words run-plan skips it with (in JasperFx/marten, not this checkout (JasperFx/stoat) — run the loop from that repository), exit code 2. A step with no run: is said and skipped: there is nothing to run, so run it by hand with gate exec --step.

With steps side by side, every line of their output starts with [step], so two suites never split each other's lines. At the end it prints a line per step, and exits with the worst step's exit code (0 only when every step that ran passed):

stoat gate run wolverine-providers/providers-green --parallel 4

stoat: running 5 steps of wolverine-providers/providers-green, up to 4 at once: core, http, postgres, sqlserver, rabbitmq
[core] stoat: recorded step core for wolverine-providers/providers-green (run 7f3a09c1)
[http] stoat: recorded step http for wolverine-providers/providers-green (run 2be1d4f0)
...
stoat: gate run wolverine-providers/providers-green
  core       passed           2m 41s
  http       passed           1m 12s
  postgres   failed (exit 1)  4m 3s
  sqlserver  passed           3m 55s
  rabbitmq   passed           6m 20s

For a suite that cannot be run either way, an agent can assert the result with report_gate_result. That decides the gate only where a person pressed allow asserted on its card, and the card then marks it asserted, never as observed.

A gate takes no go-ahead, for the same reason a slice takes none: what proves it is fixed by the plan. It is judged on its committed branch exactly as a slice is (see "Merge when ready" is checked), and --verify-command re-runs its suite there. A failed gate stays ready, and is retried up to --max-attempts.

A gate that re-arms

reverify: on-branch-change on a test-run gate (YAML, add_plan_node, the editor's checkbox) makes a done gate stale, and so ready again while its dependents wait, once the integration branch's tree differs from the tree it passed on. The resident runner decides it, since it owns the branch, and reports it on its status. Comparing trees is what keeps a gate's own landing from re-arming it: with nothing else landed, the branch fast-forwards to the very commit it passed on. On a re-arm the runner resets the node's attempts once and removes its worktree when it is clean and landed, so the next attempt is cut from the branch as it is now. A gate without reverify never changes.

A person can ask for the same by hand: ⟲ re-verify on a done gate makes every pass from before the request stop counting.

BOBCAT_RUN_TAG — how a gate finds the right run

Every agent runs the whole suite from a worktree holding only its own slice's work. Without a tag, each reports on every other slice's identities from a tree that does not contain their fixes — and the last agent to finish decides the plan.

So the runner stamps BOBCAT_RUN_TAG=<plan>/<nodeId> on the process it spawns. Bobcat's publisher reads it with no configuration, and the gate reads it back.

WARNING

The prompt tells the agent not to change it. Agents left alone have rewritten it as a bare node id, a truncated id, or an id with a timestamp appended, and a run under a changed tag is invisible to its own gate: the work is done, and the node still runs to give-up.

Matching accepts the canonical form and the bare node id, and nothing looser. A tag matched to the wrong node would mark a slice done off another slice's run, and node ids often share long prefixes. An unrecognised tag is ignored; losing evidence is the safer error.

See also

• Gate steps — declaring steps, and how a step is decided
• Runs and evidence — the runs a gate reads