Inspect Harbor

Harbor is a framework for building, evaluating, and optimizing AI agents in containerized environments. Inspect Harbor provides an interface to run Harbor tasks using Inspect AI.

Installation

Install from PyPI:

pip install "inspect-harbor>=0.5"

Or with uv:

uv add "inspect-harbor>=0.5"

The >=0.5 floor is important: version 0.5 introduced the organization-qualified Harbor registry used by these docs. Without the floor, a dependency resolver may silently select an older release whose task names do not match the examples below.

OpenAI 3 compatibility

Warning

Harbor depends on LiteLLM, which currently declares openai<3, while Inspect’s OpenAI provider requires openai>=3.1. The two declared dependency ranges cannot be resolved together. This is tracked upstream in LiteLLM #37907.

If your uv project uses OpenAI models, add a temporary override for LiteLLM’s upper bound to the project’s existing pyproject.toml:

[tool.uv]
override-dependencies = ["openai>=3.1.0"]

Then add both direct dependencies:

uv add "inspect-harbor>=0.5" "openai>=3.1.0"

This override is limited to the Inspect Harbor execution path, where Inspect—not LiteLLM—runs the model. Do not use the resulting environment for Harbor’s own LiteLLM-backed model execution; LiteLLM does not yet declare OpenAI 3 support. If you do not need OpenAI models, no override is necessary. pip does not offer an equivalent persistent override; use a uv-managed environment for this temporary OpenAI configuration.

Prerequisites

Before running Harbor tasks, ensure you have:

Python 3.12 or higher – required by inspect_harbor.
Docker installed and running – required for execution when using Docker sandbox (default).
Model API keys – set appropriate environment variables (e.g. OPENAI_API_KEY, ANTHROPIC_API_KEY).

Quick Start

The fastest way to get started is to run a dataset from the Harbor registry.

CLI:

# Run Aider's Polyglot coding benchmark
inspect eval inspect_harbor/aider_polyglot --model openai/gpt-5-mini

# Run Terminal-Bench 2.0
inspect eval inspect_harbor/terminal_bench_2 --model openai/gpt-5

Python API:

from inspect_ai import eval
from inspect_harbor import aider_polyglot, terminal_bench_2

# Run Aider's Polyglot coding benchmark
eval(aider_polyglot(), model="openai/gpt-5-mini")

# Run Terminal-Bench 2.0
eval(terminal_bench_2(), model="openai/gpt-5")

What this does

Loads the dataset from the Harbor registry.
Downloads and caches all tasks in the dataset.
Solves the tasks with the default ReAct agent scaffold.
Executes in a Docker sandbox environment.
Stores results in ./logs.

See the Registry for the full list of available datasets, and the Using Harbor guides for more detail on datasets, task parameters, agents, and advanced features.