openai/swe-lancer-diamond-manager

Coding

Professional

A benchmark of freelance software engineering tasks from Upwork, valued at $1 million USD total in real-world payouts. Manager variant: picking between technical implementation proposals.

← Back to Registry

Run this task

CLI:

inspect eval inspect_harbor/openai_swe_lancer_diamond_manager --model openai/gpt-5

Python:

from inspect_ai import eval
from inspect_harbor import openai_swe_lancer_diamond_manager

eval(openai_swe_lancer_diamond_manager(), model="openai/gpt-5")

Dataset information

Harbor registry	openai/swe-lancer-diamond-manager
Inspect task	`openai_swe_lancer_diamond_manager`
Latest digest	sha256:f73d9dfa9e19f923dd4461cd065f10fb4f19102459a48356b4f751501477eac2
Samples	265
Paper	arxiv
Source	https://github.com/openai/SWELancer-Benchmark

See Task Parameters for the parameter set shared across all Harbor tasks.