inspect_swarm.scorer
Member scores
member_scores
Score each member’s result, for an answer task or an artifact task with member artifacts: the best member and the mean member.
The scorer reads each member’s output from the swarm’s result record and grades it with scorer, as if that member had run alone. Its value is a dict, {"best_member": ..., "mean_member": ...}, over the members with a value. Its metadata holds each member’s outcome and value, and the record’s final and kind.
It holds the task’s callables, so Inspect cannot log or rebuild it as a scorer argument. Wrap it in a registered scorer factory of the task’s own, with no arguments, which re-scoring rebuilds by name:
@scorer(metrics=MEMBER_METRICS)
def math_members() -> Scorer:
return member_scores(match(), math_result())List it after the task’s own scorer, which stays the headline score.
For an answer task, scorer must be a function of the answer: the output, the sample’s input, metadata and target. It may use the sandbox to evaluate the answer, but never as a source of the answer, or it would credit the team’s work to a member. It grades each member on a copy of the state, with the member’s output as the output and the sample’s input followed by the member’s message as the messages. An output with only a completion and no message adds no message, as when the swarm returns it as its answer. Members are graded in roster order, each inside a span named after it:
- A member that never started has no value (
not_started), and is not graded. - A member whose grade is
None, NaN (Score.unscored(), or a valuevalue_to_floatconverts to NaN) or not a single value (a dict, list or tuple), is unavailable: no value, with a reason (the grade’s ownreasonfor NaN, elseunscored). - A member with no output, or one whose completion is only whitespace, gets
empty_valuewithout being graded. - Every other member gets its grade, converted with
value_to_float, and the grade’s ownreason, such asinvalid_response_format.
For an artifact task whose contract has member_artifacts, scorer grades the environment, usually as the task’s own scorer does. Each member that started is graded inside member_artifacts.materialize(name), which puts that member’s artifact where the scorer looks and restores the team’s on exit. Every such member is graded, whatever its completion: a member can leave a passing artifact and submit no text. A member with no output is graded with an empty one. An error from materialize or the scorer fails the sample.
A sample with no swarm, a plain agent, is scored as one member named solver, whose output is the sample’s. For an artifact task it is graded on the environment as it left it, without materialize. A task whose result is an artifact with no member_artifacts, a swarm without the contract, or one with another kind, gives no member values; then both keys are NaN, which Inspect counts as unscored, with the reason artifact_without_member_artifacts, kind_not_declared or kind_mismatch. So does a sample with no member value, with the reason no_member_valued.
The scorer returns None while state.completed is False: the solve can call the task’s scorers (react(attempts=...) and other attempt loops, live control scoring), and the record does not exist yet.
def member_scores(
scorer: Scorer,
result: ResultSpec,
*,
value_to_float: ValueToFloat = value_to_float(), # noqa: B008
empty_value: float = 0.0,
) -> ScorerscorerScorer-
The scorer that grades each member, usually the task’s own.
resultResultSpec-
The task’s result contract, the one it passes to
swarm(result=...). value_to_floatValueToFloat-
Converts a grade’s value to a float.
empty_valuefloat-
The value of an empty output, for an answer task.
MEMBER_METRICS
Metrics for a scorer that returns member_scores(): the mean and standard error of best_member and mean_member. Pass them to the task’s own scorer factory, @scorer(metrics=MEMBER_METRICS).
MEMBER_METRICS: dict[str, list[Metric]] = {
"best_member": [mean(), stderr()],
"mean_member": [mean(), stderr()],
}