frontier-harness-eval/frontier-harness-eval
FrontierHarness Eval v1.1.0: 17 Terminal Bench 4 tasks and 45 Terminal-Bench 2.1 tasks. TB4 quarter-size selection derived from proposal Appendix C; TB2.1 selection from Appendix E. Original tasks and verifiers pinned by SHA256. Report benchmark subsets separately; these are not official full-benchmark scores. TB4 favors shorter historical runtimes. TB2.1 preserves all 16 categories, with 29 medium, 14 hard and 2 easy tasks; source commit 7131e4375048a0e408a8fb404b5f499d726b695b. Selection and provenance: This selection reduces TB4 from 33 to 17 tasks using proportional domain quotas and the fastest known historical runtimes within each domain. Tasks with unknown historical runtimes are excluded. This version replaces protein-assembly with dna-insert, retaining the scientific-computing category and 1800-second timeout. It also replaces crack-7z-hash with vulnerable-secret, retaining medium security coverage and reducing the agent timeout from 1800 to 900 seconds.
Run this task
CLI:
inspect eval inspect_harbor/frontier_harness_eval --model openai/gpt-5Python:
from inspect_ai import eval
from inspect_harbor import frontier_harness_eval
eval(frontier_harness_eval(), model="openai/gpt-5")Dataset information
| Harbor registry | frontier-harness-eval/frontier-harness-eval |
| Inspect task | frontier_harness_eval |
| Latest digest | sha256:96c9b81553f50d8fb0f843b2aaeb11e4fb4989767c28202fc7607a3a56abd594 |
| Samples | 62 |
| Source | https://github.com/frontier-harness-eval/eval |
See Task Parameters for the parameter set shared across all Harbor tasks.