Task Format

What is a Harbor Task?

Harbor is a framework for building, evaluating, and optimizing agents and models in containerized environments. A Harbor task is a self-contained evaluation unit that includes an instruction, execution environment, scoring criteria, and optionally a reference solution.

For comprehensive details about Harbor tasks, see the Harbor documentation.

Harbor Task File Structure

A typical Harbor task directory contains the following components:

my_task/
├── instruction.md             # Task instructions/prompt shown to the agent
├── task.toml                  # Metadata, timeouts, resource specs, env vars
├── environment/               # Environment setup - Dockerfile or docker-compose.yaml
│   └── Dockerfile             # Docker environment spec (varies by sandbox provider)
├── solution/                  # (Optional) Reference solution for sanity checking
│   ├── solve.sh / solve.bat   # Executable solution script used by Oracle solver
│   └── ...                    # Supporting solution files and dependencies
└── tests/                     # Verification and scoring
    ├── test.sh / test.bat     # Test script executed by verifier
    └── ...                    # Outputs reward.txt or reward.json to /logs/verifier/

.bat script variants are used when the task targets Windows (controlled by [environment].os); Linux tasks use .sh.

Harbor to Inspect Mapping

Inspect Harbor bridges Harbor tasks to the Inspect AI evaluation framework using the following mappings:

Harbor Concept	Inspect Concept	Description
Harbor Task	`Sample`	A single evaluation instance with instructions and environment
Harbor Dataset	`Task`	A collection of related evaluation instances
instruction.md	`Sample.input`	The prompt/instructions given to the agent
environment/	`SandboxEnvironmentSpec`	Docker/environment configuration for isolated execution
tests/test.sh / test.bat	`Scorer` (`harbor_scorer`)	Test script executed by the scorer to produce reward/metrics
solution/solve.sh / solve.bat	`Solver` (`oracle`)	Reference solution script executed by the Oracle solver for sanity checking
task.toml[task]	`Sample.metadata`	Task name, description, authors, keywords
task.toml[metadata]	`Sample.metadata`	Arbitrary custom fields (difficulty, category, tags)
task.toml[verifier]	Scorer timeout/env vars	Timeout, env vars, and user for verifier execution
task.toml[agent]	Agent solver env vars/user	Environment variables and user for agent execution. Agent `timeout_sec` is ignored.
task.toml[solution]	Oracle solver env vars	Environment variables to set when running the solution script
task.toml[environment]	`SandboxEnvironmentSpec.config`	Docker image, target OS (`linux`/`windows`), resource specs (CPU, memory, storage, GPU, internet), MCP servers, healthchecks. Overwrites resource limits in `environment/docker-compose.yaml`.

Unsupported `task.toml` Features

Loading a task that declares any of the below either raises or warns:

Field	Behavior
`[[steps]]` + `multi_step_reward_strategy` (multi-step tasks)	raise
`[environment].os = "windows"`	raise
`[environment].mcp_servers`	warn (not exposed to agent)
`[environment].skills_dir`	warn (not copied to agent skills dir)

[environment].build_timeout_sec, storage_mb, workdir, and config.source / config.artifacts are ignored.

Healthchecks

[environment].healthcheck maps onto the healthcheck of the default compose service: command becomes a CMD-SHELL test and the *_sec fields become compose durations. Inspect brings the environment up with docker compose up --wait, so the agent only starts once the check passes, as it does under Harbor. Two differences are worth knowing about:

If a task ships a docker-compose.yaml that already declares a healthcheck on the default service, that one is kept and the task.toml healthcheck is not enforced (a compose service carries only one healthcheck, whereas Harbor gates readiness on both). Loading such a task warns.
Inspect sizes its compose up --wait timeout as retries * (interval + timeout), which does not include start_period_sec. A task whose grace period is longer than that product can therefore be cut short at startup, even while Docker still considers the container starting — with Harbor’s defaults, a start_period_sec above 105s. Harbor itself runs up --wait untimed and only then polls the healthcheck, so it has no such ceiling.

LLM Judges in Verification

Some Harbor tasks use LLM judges for verification (e.g. evaluating open-ended responses or code quality). These tasks specify the model in their task.toml:

[verifier.env]
MODEL_NAME = "claude-haiku-4-5"
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"

The verifier script (tests/test.sh or test.bat) uses these environment variables to call the LLM. Make sure to set the appropriate API key (e.g. ANTHROPIC_API_KEY) when running tasks with LLM judges.