hwe-bench/hwe-bench

HWE-bench evaluates LLM agents on engineering-simulation tasks: the agent drives a real solver and is scored on whether the number it reports lands inside a defensible tolerance band, by a deterministic verifier rather than an LLM judge. This CAE edition covers combustion, battery, CFD and 2.5D packaging.

← Back to Registry

Run this task

CLI:

inspect eval inspect_harbor/hwe_bench --model openai/gpt-5

Python:

from inspect_ai import eval
from inspect_harbor import hwe_bench

eval(hwe_bench(), model="openai/gpt-5")

Dataset information

Harbor registry	hwe-bench/hwe-bench
Inspect task	hwe_bench
Latest digest	sha256:c65139ef5b04e1951e0896b42c10ea24e8aa2a7fd28ab76c12b3944b9bea9abc
Samples	77
Source	https://github.com/svd-ai-lab/hwe-bench

See Task Parameters for the parameter set shared across all Harbor tasks.