bigcode/humanevalfix

Coding

HumanEvalFix (OctoPack): buggy functions across Python, JavaScript, Java, Go, C++, and Rust that models must repair given the failing unit tests.

← Back to Registry

Run this task

CLI:

inspect eval inspect_harbor/bigcode_humanevalfix --model openai/gpt-5

Python:

from inspect_ai import eval
from inspect_harbor import bigcode_humanevalfix

eval(bigcode_humanevalfix(), model="openai/gpt-5")

Dataset information

Harbor registry bigcode/humanevalfix
Inspect task bigcode_humanevalfix
Latest digest sha256:f3188245370b2ea142c6442591299c955202cfdf822371b4c6e9e5baf7dafe32
Samples 164
Paper arxiv
Source https://github.com/bigcode-project/octopack

See Task Parameters for the parameter set shared across all Harbor tasks.