Workflow

Overview

Steward runs evaluations from a workspace directory that includes the eval-set definition, steward-specific options, a journal of the entire run, and human readable documents on run status. Evaluations are launched as background processes and monitored using a scheduled job, with coding agents and operators proactively notified when they are needed for a decision.

We’ll walk through the various phases of the workflow below, then cover options you can use to customize Steward’s behavior and other advanced topics like pausing runs and modifying runs in flight.

Workspace

To create a workspace for an evaluation run, use the steward init command:

Terminal

mkdir -p ~/runs/swe-evals
cd ~/runs/swe-evals
steward init

The workspace is initialized with the following files:

Workspace

swe-evals/
  _steward.yaml       # steward specific options 
  AGENTS.md           # agent instructions for managing a run
  CLAUDE.md           # agent instructions for managing a run
  evalset.py          # eval-set definition (or config.py / hawk.yaml)
  journal.jsonl       # append-only journal of workspace actions
  status.md           # the operator's page: where the run stands, what waits on them

Your eval set definition can be any script which ends with a call to eval_set() or a Flow or Hawk configuration. For example:

evalset.py

from inspect_ai import eval_set
from inspect_harbor import cais_swebenchpro, datacurve_deep_swe

eval_set(
    tasks=[cais_swebenchpro(),  datacurve_deep_swe()],
    model=["openai/gpt-5.6-sol", "anthropic/claude-opus-5"],
)

Your task before launching is to provide the eval set definition and adjust any settings you need in _steward.yaml. The defaults are generally suitable, but be sure to review Concurrency against the specifics of your evals and infrastructure.

Execution

Once you have defined your eval set, use a coding agent to launch the evaluation (you should typically run from a tmux detached terminal so the agent is persistent):

Coding Agent

───────────────────────────────────────────── swe-evals ─
❯ please run this evaluation.
─────────────────────────────────────────────────────────

The agent reads the Steward runbook and follows it. It starts with a short smoke run, skipping any task a configured log store already satisfies, to confirm that configuration, infrastructure and model access all work correctly. A failed smoke stops the launch until it is fixed or waived by the user.

After the smoke the agent launches the tasks and works the decisions the run raises as the tend loop runs. Keeping the agent attached is recommended but not required. If you detach it, configure notifications so you know when to reconnect to it.

Tend Loop

As the run proceeds, the steward tend command runs on a timer set up by steward launch. By default this timer runs every 10 minutes. The tend command performs the following actions automatically:

• Spawns or restarts tasks that should be running and are not.
• Automatically tunes concurrency based on available sandbox capacity and observed model throughput.
• Notes anomalies and other conditions (model or infra errors, stuck samples, scoring integrity issues, etc.) and either handles them automatically or escalates them for agent or human consideration.
• Rewrites status.md and appends to journal.jsonl.

Note that you or an agent can call steward tend at any time to get the current status of the run.

Agent Decisions

After tend runs, agents call steward collect, which reports the open decisions along with everything that has happened since it last looked. The agent handles the following on its own:

• Diagnoses errors and groups them into classes, and proposes what should be done about them (re-run, exclude, zero, score, accept, or dismiss) for you to rule on.
• Oversees the concurrency ramp, holding the climb when something looks wrong and tuning a worker down while containing an incident.
• Raises anything that needs you, and notifies you through the configured channel when it cannot proceed without an answer.
• Writes up the overall run investigation in analysis.md once the run is complete.

Human Decisions

There are some things within a run that require your judgement or the establishment of a policy that can be applied to all similar judgements. Both steward tend and the agent can raise things for human input, asking you to:

• Rule on the error classes the agent proposes, choosing to re-run, exclude, zero, or re-score the samples, or to accept or dismiss the class outright.
• Answer tool call approvals that have escalated to human review.
• Decide what to do with a task that has stalled, or a sample stuck past the point the agent may act on it.
• Sign off on the final results after reviewing anomalies and transcript analysis.

Notifications

Steward uses Apprise for sending notifications to human operators. You can configure Apprise to send Slack messages, emails, AWS SES or SNS messages, or many other types of message. Configure notifications by setting the STEWARD_NOTIFICATION environment variable or setting the notification option in _steward.yaml. For example:

.env

STEWARD_NOTIFICATION=slack://{OAuthToken}/{ChannelID}

It is strongly recommended that you configure notifications. If you don’t, you should just reconnect to your agent periodically and ask how things are going:

Coding Agent

───────────────────────────────────────────── swe-evals ─
❯ how is the eval going?
─────────────────────────────────────────────────────────

Steward notifies you when something needs a decision, when a run finishes, and when it cannot tend at all. In between, it also posts an hourly heartbeat so you know the latest status at a glance.

Signoff

Once all tasks are completed (with errors/anomalies decided on) and their transcripts scanned for scoring integrity issues, the run is eligible for signoff. If you have a coding agent managing the run then it will prompt you for signoff at the appropriate time. If not, you can manually run signoff with:

Terminal

steward signoff --by "norah"

Signoff verifies that the run is eligible for signoff, ensures a clean final log directory, finalizes the journal and status, and disarms the tend loop timer. If you have a Log Store configured, the agent will also ask whether to publish your logs to it — add --publish if you want that.

Eval Logs

Log Directory

Logs are written to logs/ inside the workspace. Two things can change this:

• If your eval set passes a log_dir to eval_set(), Steward uses that.

• The log_root option or STEWARD_LOG_ROOT environment variable, for a machine that keeps logs somewhere else (a bigger disk, or an S3 bucket). Steward makes one directory per workspace underneath it.

If you want your logs to be durable then we recommend setting STEWARD_LOG_ROOT to an S3 bucket or other permanent storage and letting Steward create subdirectories within it for each workspace.

Log Store

A log store lets several workspaces share results, so a task somebody has already run doesn’t run again. Point at one with the log_store option or the STEWARD_LOG_STORE environment variable:

.env

STEWARD_LOG_STORE=s3://our-bucket/eval-logs

The store can be a directory of .eval files, or a Flow store (which requires the flow extra).

A result is only reused when everything about the task matches: the task and its arguments, the model, the solver, the plan, the generate config, and the limits.

Reading from a store happens automatically once you configure one, at launch and in launch --smoke, which rehearses only the tasks the store does not satisfy. Publishing your own results into it does not, that only happens if you ask for it at signoff:

Terminal

steward signoff --by "norah" --publish

Options

Steward’s options go in _steward.yaml at the root of the workspace, or in the environment as STEWARD_<OPTION> in an .env file within the workspace or a parent directory.

Logs

Storage location for logs. These are typically specified at the machine level (.env). Use the _steward.yaml option to override for a given project.

_steward.yaml	Environment	Default	Description
log_root	STEWARD_LOG_ROOT	none	Root for this machine’s logs.
log_store	STEWARD_LOG_STORE	none	Logs to reuse rather than run: a directory of .eval files or a Flow store.
sync	STEWARD_SYNC	log_dir	Where the workspace’s files are mirrored on each tend.

See Log Directory and Log Store for additional details.

Supervision

Frequency of run tending and how to communicate notifications.

_steward.yaml	Environment	Default	Description
tend_interval	STEWARD_TEND_INTERVAL	10m	How often a scheduled tend runs, with a unit.
notification	STEWARD_NOTIFICATION	none	Apprise URL(s) or an Apprise config file.
heartbeat	STEWARD_HEARTBEAT	1h	How often a liveness heartbeat posts while a run is progressing, with a unit; false for none.

See Tend Loop and Notifications for additional details.

Concurrency

_steward.yaml	Environment	Default	Description
samples_ramp	STEWARD_SAMPLES_RAMP	[50, 200]	Range to discover sample concurrency in; false for no automatic ramp.
max_workers	STEWARD_MAX_WORKERS	one per task	Pack the run into this many worker processes.

See Automatic Ramp and Worker Processes for additional details.

Errors

_steward.yaml	Environment	Default	Description
stall_after	STEWARD_STALL_AFTER	2	Fruitless respawns before a task is given up on and reported.
stuck_after	STEWARD_STUCK_AFTER	5h	Report a sample with no activity for this long.

See Retry and Stuck Samples for additional details.

Standing Rules

Standing decision rules. These are typically written by the agent during a run as a result of operator decisions.

_steward.yaml	Environment	Default	Description
preauthorized	STEWARD_PREAUTHORIZED	none	Anomaly-class patterns mapped to rerun, exclude, zero or score.
stuck_cancel	STEWARD_STUCK_CANCEL	none	Stuck tool calls the agent may cancel unasked: true for any, or a list of tool names.
policies	STEWARD_POLICIES	none	Rules an agent applies, as prose or a list.

See Codify and Stuck Samples for additional details.

Scanning

_steward.yaml	Environment	Default	Description
scan_model	STEWARD_SCAN_MODEL	eval model	The model scanners use. false ignores a SCOUT_SCAN_MODEL in the environment and scans with the eval model.
scanners	STEWARD_SCANNERS	none	Scanners run in addition to the definition’s.

See Scan Model and Adding Scanners for additional details.

Advanced

Agent Oversight

An agent does not need to stay attached: steward tend runs on its timer either way. An agent adds judgment, diagnosing errors, proposing what to do about them, and escalating what needs you. There are three ways to work:

• Attached and reactive: the agent watches for new activity and responds immediately. Best for a run you are watching closely.
• Attached and periodic: the agent wakes on its own schedule, can run steward tend to take a turn early, and stands down once the run is finished and nothing open is its own.
• Disconnected: nobody is attached, and the next session picks up whatever accumulated. This is the common case.

An agent attaching to a run it did not start begins here:

Terminal

steward runbook             # the mechanics, which ship with the package
steward collect --since 0   # everything that has happened so far

along with _steward.yaml for the project’s standing rules. Reading consumes nothing: an open decision leaves the queue only when somebody acts on it.

While nothing is attached the run keeps converging, but anomalies accumulate without proposals and items sit unraised. If no agent has collected in the last two tends, Steward routes the agent’s items to you as well.

Inflight Changes

There is no amend command. To change an eval set that is already running, edit the definition and launch again. Steward compares the new definition against the one it committed and runs only what is new or changed; a task whose identifier survives the edit already has its log and is not re-run.

Launch reports the delta before committing it:

Row	Meaning
add	New work, not in the previous definition.
extend	Same task, more work than before (raised epochs, a grown dataset).
removed	Gone from the definition; its logs move to logs-archive/.
superseded	Displaced by a task of the same name and model under a different configuration; the old logs move to logs-archive/.
restore	Asked for again, and an archived log answers it; moved back instead of re-run.

A purely additive delta is committed without comment. Anything that would move results out of the log directory prints what it would move and refuses; --accept-archive commits it. The gate exists because a one-character change to a task argument produces a new identifier and looks the same as a deliberate removal, while costing a re-run of everything.

An edit you have not launched is reported on every turn as drift.

Pausing Runs

steward pause --reason "..." stops new work from being scheduled. Workers already running finish normally, and every turn reports the run as paused and spawns nothing. steward resume starts scheduling again, and the next tend converges from whatever it finds by then.

steward ramp hold --reason "..." freezes the concurrency climb without pausing the run, and steward ramp resume re-arms it. A hold stops growth only: the defensive cut on provider pushback stays active.

Timers

steward launch installs the timer that runs steward tend, using a launchd agent, a systemd user timer, or a marked block in your crontab. Arming is idempotent, so re-arming at a new interval leaves exactly one. Manage it directly with steward timer arm, disarm, and status, or launch without one using --no-timer.

Arming refuses if a scheduled tend would not inherit this shell’s credentials, since the alternative is a fleet that fails to authenticate all night. Put them in .env, which the workers already read, or pass --no-env-check.

If no tend runs for two intervals, the run reports itself unsupervised.