nl2repobench/nl2repobench

Coding

Agent generates a complete, installable Python library from a single natural-language requirements document, starting from an empty workspace.

← Back to Registry

Run this task

CLI:

inspect eval inspect_harbor/nl2repobench --model openai/gpt-5

Python:

from inspect_ai import eval
from inspect_harbor import nl2repobench

eval(nl2repobench(), model="openai/gpt-5")

Dataset information

Harbor registry nl2repobench/nl2repobench
Inspect task nl2repobench
Latest digest sha256:b0d58e327ee30a6e6584bd4843a53db52a3442e230630369da095ec564542712
Samples 104
Paper arxiv
Source https://github.com/multimodal-art-projection/NL2RepoBench

See Task Parameters for the parameter set shared across all Harbor tasks.