Inspect Steward

Automated execution and supervision of long-running evaluations.

Overview

Welcome to Steward, a system for automated execution and supervision of long-running evaluations. Steward enables you to launch a large run and walk away while a coding agent monitors it. Here’s how it works:

Define your eval set and policies that will govern the run (Steward works with eval_set() as well as with execution frameworks like Inspect Flow and Inspect Hawk)
Launch the eval set for unattended execution. Let standing policies and agents handle tuning and errors whenever possible. Receive notifications whenever something needs your attention and make whatever decisions are required.
At the end of the run, make final judgements about how to score anomolies and then signoff the completed run.

Quick Tour

Setup

First, install Steward from PyPI with:

Terminal

pip install inspect_steward

To use Steward, create a directory, switch to it, then call steward init:

Terminal

mkdir -p ~/runs/swe-evals
cd ~/runs/swe-evals
steward init

This will scaffold up a Steward workspace with a placeholder evalset.py, a _steward.yaml for your standing rules, and an AGENTS.md file that teaches the agent how to use Steward to run an evaluation. First, define your eval set:

evalset.py

from inspect_ai import eval_set
from inspect_harbor import cais_swebenchpro, datacurve_deep_swe

eval_set(
    tasks=[cais_swebenchpro(),  datacurve_deep_swe()],
    model=["openai/gpt-5.6-sol", "anthropic/claude-opus-5"],
)

Steward can manage any evalution defined with a script that ends in a call to eval_set() or alternatively a config.py from Flow or a hawk.yaml from Hawk.

Log Directory

One other thing you should typically define up front is a root directory for eval logs. If you define the STEWARD_LOG_ROOT environment variable then a log_dir matching the workspace directory name will be automatically created and used for your eval set. For example:

.env

STEWARD_LOG_ROOT=s3://inspect-logs

Run the Eval

Next, launch your coding agent from the workspace and tell it to run the evaluation (you should typically run from a tmux detached terminal so the agent is persistent):

Coding Agent

───────────────────────────────────────────── swe-evals ─
❯ please run this evaluation.
─────────────────────────────────────────────────────────

The agent will launch all of the tasks, setup a background monitoring process, and respond to alerts that require its intervention. From here you can detach and walk away.

Notifications

To be notified when things require your attention, we strongly recommend you also configure a notification channel. Put it in _steward.yaml, or — to keep the token out of your repository — in your .env file:

.env

STEWARD_NOTIFICATION=slack://{OAuthToken}/{ChannelID}

See Notifications for more details on configuring notifications for various targets including Slack, Email, and others.

The run depends on neither you nor the agent staying connected. When you come back, ask how it’s going:

Coding Agent

───────────────────────────────────────────── swe-evals ─
❯ how is the eval going?
─────────────────────────────────────────────────────────

The agent reports where the run stands: per-task progress, what it resolved on its own while you were gone, and anything that is waiting on a decision from you.

Make Decisions

Runs rarely go perfectly, and the questions that survive the agent’s own judgement come back to you. For example, the agent might prompt. you with this when you check in:

Coding Agent

89 samples in cais_swebenchpro failed with the same
sandbox timeout, all inside a 30-minute window last 
night. Automatic retry has already spent its attempts.

re-run them, score them as failed, or exclude them 
from scoring?

───────────────────────────────────────────── swe-evals ─
❯ re-run them, the disk issue is fixed
─────────────────────────────────────────────────────────

Learning More

Workflow is the comprehensive guide to a Steward run: the workspace directory, launching and tending runs, agent and human decisions, and how a run ends.
Concurrency describes the various options for tuning concurrency, including automatic ramping behavior and caps on resource utilization.
Error Handling covers standing policies for error retries and resolution, including dealing with errors that are never resolved.
Scanners explains how transcripts are scanned for scoring integrity problems as the run proceeds, and what a finding asks of you.
Agent Runbook provides the full operating instructions that coding agents use to supervise and make decisions during runs.

Integrations with execution environments are covered in Inspect Flow and Inspect Hawk.