orca-bench/orca-bench-verified

Coding

Professional

ORCA-bench Verified: root-cause analysis of oncall incidents — agents investigate telemetry from an instrumented microservices system to diagnose each incident; verified split of 40 tasks across 40 incidents (10 easy, 15 medium, 15 hard).

← Back to Registry

Run this task

CLI:

inspect eval inspect_harbor/orca_bench_verified --model openai/gpt-5

Python:

from inspect_ai import eval
from inspect_harbor import orca_bench_verified

eval(orca_bench_verified(), model="openai/gpt-5")

Dataset information

Harbor registry	orca-bench/orca-bench-verified
Inspect task	`orca_bench_verified`
Latest digest	sha256:263a25d01329e4409bc03bfcd9e6d9771d189f6060a4b2db831b5e89783f7162
Samples	40
Paper	arxiv
Source	https://github.com/orca-bench/ORCA-bench

See Task Parameters for the parameter set shared across all Harbor tasks.