Skip to main content

inspect_swarm.scorer

Per-member scores for tasks scored on their answer or on each member’s artifact.

Member scores

member_scores

Score each member’s result, for an answer task or an artifact task with member artifacts: the best member and the mean member.

The scorer reads each member’s output from the swarm’s result record and grades it with scorer, as if that member had run alone. Its value is a dict, {"best_member": ..., "mean_member": ...}, over the members with a value. Its metadata holds each member’s outcome and value, and the record’s final and kind.

It holds the task’s callables, so Inspect cannot log or rebuild it as a scorer argument. Wrap it in a registered scorer factory of the task’s own, with no arguments, which re-scoring rebuilds by name:

@scorer(metrics=MEMBER_METRICS)
def math_members() -> Scorer:
    return member_scores(match(), math_result())

List it after the task’s own scorer, which stays the headline score.

For an answer task, scorer must be a function of the answer: the output, the sample’s input, metadata and target. It may use the sandbox to evaluate the answer, but never as a source of the answer, or it would credit the team’s work to a member. It grades each member on a copy of the state, with the member’s output as the output and the sample’s input followed by the member’s message as the messages. An output with only a completion and no message adds no message, as when the swarm returns it as its answer. Members are graded in roster order, each inside a span named after it:

  • A member that never started has no value (not_started), and is not graded.
  • A member whose grade is None, NaN (Score.unscored(), or a value value_to_float converts to NaN) or not a single value (a dict, list or tuple), is unavailable: no value, with a reason (the grade’s own reason for NaN, else unscored).
  • A member with no output, or one whose completion is only whitespace, gets empty_value without being graded.
  • Every other member gets its grade, converted with value_to_float, and the grade’s own reason, such as invalid_response_format.

For an artifact task whose contract has member_artifacts, scorer grades the environment, usually as the task’s own scorer does. Each member that started is graded inside member_artifacts.materialize(name), which puts that member’s artifact where the scorer looks and restores the team’s on exit. Every such member is graded, whatever its completion: a member can leave a passing artifact and submit no text. A member with no output is graded with an empty one. An error from materialize or the scorer fails the sample.

A sample with no swarm, a plain agent, is scored as one member named solver, whose output is the sample’s. For an artifact task it is graded on the environment as it left it, without materialize. A task whose result is an artifact with no member_artifacts, a swarm without the contract, or one with another kind, gives no member values; then both keys are NaN, which Inspect counts as unscored, with the reason artifact_without_member_artifacts, kind_not_declared or kind_mismatch. So does a sample with no member value, with the reason no_member_valued.

The scorer returns None while state.completed is False: the solve can call the task’s scorers (react(attempts=...) and other attempt loops, live control scoring), and the record does not exist yet.

def member_scores(
    scorer: Scorer,
    result: ResultSpec,
    *,
    value_to_float: ValueToFloat = value_to_float(),  # noqa: B008
    empty_value: float = 0.0,
) -> Scorer
scorer Scorer

The scorer that grades each member, usually the task’s own.

result ResultSpec

The task’s result contract, the one it passes to swarm(result=...).

value_to_float ValueToFloat

Converts a grade’s value to a float.

empty_value float

The value of an empty output, for an answer task.

MEMBER_METRICS

Metrics for a scorer that returns member_scores(): the mean and standard error of best_member and mean_member. Pass them to the task’s own scorer factory, @scorer(metrics=MEMBER_METRICS).

MEMBER_METRICS: dict[str, list[Metric]] = {
    "best_member": [mean(), stderr()],
    "mean_member": [mean(), stderr()],
}