Validation

Overview

When developing scanners, it’s often desirable to create a feedback loop based on human labeling of transcripts that indicate expected scanner results. You can do this by creating a validation set and applying it during your scan:

The validation set is the set of labeled transcripts that are compared against scan results. Validation sets are typically associated with the domain of a particular scanner type (e.g. “evaluation awareness”, “refusal”, etc.) so you will likely develop many of them for use with different scanners.

Apply a validation set by passing it to scan(). For example:

from inspect_scout import scan, transcripts_from

from my_scanners import eval_awareness

scan(
    scanners=[eval_awareness()],
    transcripts=transcripts_from("./logs"),
    validation="eval-awareness.csv"
)

Or from the command line:

scout scan eval_awareness.py -V eval-awarness.csv

Validation sets are stored in CSV, YAML, JSON, or JSONL text files, however you don’t need to edit them in their raw format. The next section describes recommended workflows for editing and managing validation sets.

Validation Sets

The easiest way to work with validation data is to use Scout View, which provides inline editing of validation cases as well as various tools for editing and refining validation sets.

Transcript View

When viewing any transcript, you can activate the validation case editor by clicking the button at the top right of the content view:

A validation case maps a transcript to an expected target result. In the example above we indicate that this transcript does have evidence of evaluation awareness which should be detected by scanners.

Results View

Sometimes its more convenient to apply validation labels in the context of scan results. There is also a validation editor available in every result view:

It’s often very natural to create cases this way as reviewing scan results often leads to judgments about whether the scanner is working as intended.

Validation Pane

The Validation pane provides a top level view of all validation sets as well as various tools for managing them:

Use the validation pane to review and edit validation cases, manage splits, or copy and move validation cases between validation sets.

Validation Files

While you don’t often need to edit validation files directly, you can do so if necessary since they are ordinary CSV for YAML files. For example, here’s a validation set in CSV format:

eval-awareness.csv

id,target
Fg3KBpgFr6RSsEWmHBUqeo,true
VFkCH7gXWpJYUYonvfHxrG,false
SiEXpECj7U9nNAvM3H7JqB,true

If you are editing validation files directly you will need a way to discover trancript IDs. Use the Copy button in the transcript view to copy the UUID of the transcript you are viewing:

See the File Formats section below for complete details on validation set files.

Scanning

Adding Validation

You’ll typically create a distinct validation set (with potentially multiple splits) for each scanner, and then pass the validation sets to scan() as a dict mapping scanner to set:

scanning.py

from inspect_scout import scan, transcripts_from

scan(
    scanners=[ctf_environment(), eval_awareness()],
    transcripts=transcripts_from("./logs"),
    validation={
        "ctf_environment": "ctf-environment.csv",
        "eval_awareness": "eval-awareness.csv"
    }
)

If you have only only a single scanner you can pass the validation set without the mapping:

scanning.py

scan(
    scanners=[ctf_environment()],
    transcripts=transcripts_from("./logs"),
    validation="ctf-environment.csv"
)

You can also specify validation sets on the command line. If the above scans were defined in a @scanjob you could add a validation set from the CLI using the -V option as follows:

# single validation set
scout scan scanning.py -V ctf-environment.csv

# multiple validation sets
scout scan scanning.py \
    -V ctf_environment:ctf-environment.csv \
    -V eval_awareness:eval-awareness.csv

Validation Results

Validation results are reported in the scan status/summary UI:

The validation metric reported in the task summary is the balanced accurary, which is good overall metric especially for unbalanced datasets (which is often the case for validation sets). Other metrics (precision, recall, and f1) are available in Scout View.

Scout View

Scout View will also show validation results alongside scanner values (sorting validated scans to the top for easy review):

Validation results are reported using four standard classification metrics:

Metric	Description
Accuracy	Balanced accuracy: average of recall and specificity. Accounts for class imbalance better than raw accuracy.
Precision	When the scanner flags something, how often is it correct? High precision means few false alarms.
Recall	Of all items that should be flagged, how many did the scanner find? High recall means few missed cases.
F1	Harmonic mean of precision and recall. Useful when you need to balance both concerns.

In practice, there’s often a tradeoff between precision and recall. A conservative scanner may have high precision but miss cases (low recall), while an aggressive scanner catches more cases (high recall) but with more false positives (lower precision). The right balance depends on your use case. Here are some resources that cover this in more depth:

• Precision and Recall (Wikipedia) — Comprehensive overview of precision, recall, F1, and related metrics.
• Classification Metrics (Google ML Crash Course) — Interactive tutorial on precision, recall, and the tradeoffs between them.

Validation Splits

Validation cases can be organized into named groups called “splits” (e.g., “dev”, “test”, etc.). This enables you to iterate your scanner prompts on a subset of cases (e.g. “dev”) then do final testing on a holdout set (e.g. “test”). Maintaining this separation is often critical to ensure that your scanner prompts aren’t overfit to your validation set.

Defining Splits

Use the validation pane to assign splits to validation cases:

Select multiple transcripts to assign several splits at once:

You can also directly assign splits from within the validation case editor in the transcripts and scan results views.

Splits are then written to validation set CSVs using a split column:

validation.csv

id,target,split
Fg3KBpgFr6RSsEWmHBUqeo,true,dev
VFkCH7gXWpJYUYonvfHxrG,false,dev
SiEXpECj7U9nNAvM3H7JqB,true,test

Applying Splits

Pass the split option to validation_set() to filter by split when creating your validation set:

from inspect_scout import validation_set

# Use only dev cases
validation = validation_set("validation.csv", split="dev")

# Use multiple splits
validation = validation_set("validation.csv", split=["dev", "test"])

When filtering by split, only cases with matching split values are included. Cases without a split value are excluded when a split filter is applied.

Predicates

By default, validation compares scanner results to targets using equality (eq). You can specify different comparison predicates either per-case or as a default for all cases.

Per-Case Predicates

When specifying a non-boolean target for a validation case, you can also customize the predicate used for comparison:

Within a validation set CSV, add a predicate column to specify comparison logic for individual cases:

validation.csv

id,target,predicate
SiEXpECj7U9nNAvM3H7JqB,true,eq
VFkCH7gXWpJYUYonvfHxrG,hello,contains
Fg3KBpgFr6RSsEWmHBUqeo,5,gte

In this example:

• First case passes if scanner result equals true
• Second case passes if scanner result contains “hello”
• Third case passes if scanner result is ≥ 5

Available Predicates

Predicate	Description
eq	Equal (default)
ne	Not equal
gt	Greater than
gte	Greater than or equal
lt	Less than
lte	Less than or equal
contains	String contains substring
startswith	String starts with prefix
endswith	String ends with suffix
icontains	Case-insensitive contains
iequals	Case-insensitive equals

Default Predicate

You can also set a default predicate for all cases using the predicate parameter of validation_set():

validation_set(cases="validation.csv", predicate="gte")

When both are specified, per-case predicates take precedence over the default. This allows you to set a common predicate while overriding it for specific cases:

validation.csv

id,target,predicate
Fg3KBpgFr6RSsEWmHBUqeo,5,
VFkCH7gXWpJYUYonvfHxrG,3,lt
SiEXpECj7U9nNAvM3H7JqB,10,

# Default is "gte", but second case uses "lt"
validation_set(cases="validation.csv", predicate="gte")

Advanced

Non-Transcript IDs

In the above examples, we provided a validation set of transcript_id => boolean. Of course, not every scanner takes a transcript id (some take event or message ids). All of these other variations are supported (including lists of events or messages yielded by a custom Loader).

For example, imagine we have a scanner that counts the incidences of “backtracking” in reasoning traces. In this case our scanner yields a number rather than a boolean. So our validation set would be message_id => number:

backtracking.csv

id,target
Fg3KBpgFr6RSsEWmHBUqeo,2
VFkCH7gXWpJYUYonvfHxrG,0
SiEXpECj7U9nNAvM3H7JqB,3

In the case of a custom loader (.e.g. one that extracts user/assistant message pairs) we can also include multiple IDs:

validation.csv

id,target
"Fg3KBpgFr6RSsEWmHBUqeo,VFkCH7gXWpJYUYonvfHxrG",true

When specifying multiple IDs in CSV format, you can use either:

• Comma-separated format: "id1,id2" (quote the entire value)
• JSON array format: "[\"id1\",\"id2\"]"

Both formats are automatically parsed into a list of IDs.

Complex Targets

Some scanners yield complex values (e.g. lists or dictionaries). For purposes of validation it’s often much better to deal with simple values, however if you want to validate against lists and dicts this is fully supported (although you can’t define validation cases with non-scaner targets in the Scout View UI).

One middle ground is that if your scanner is yielding a dict (e.g for Structured Answers) you can still mark one of the dict fields as the “value” using alias="value". For example:

class Refusal(BaseModel):
    refusal_exists: bool = Field(
        alias="value",
        description="Whether the assistant refused or declined to complete the user's requested task.",
    )

    type: str = Field(
        alias="label",
        description=(
            "The type of refusal: `NO_REFUSAL`, `PARTIAL_REFUSAL`, `INDIRECT_REFUSAL` or `CRITICAL_REFUSAL`"
        )
    )

This enables you to combine complex results with more straightforward validation. See the File Formats section below for details on specifying dict values as validation targets.

File Formats

You can specify a ValidationSet either in code, as a CSV, YAML, JSON, or JSONL file.

CSV

Here are the various ways you can structure a validation CSV for different scenarios:

Format	Header Row	Example
Single target	id,target	id,target abc123,true
Dict targets	id,target_*,...	id,target_foo,target_bar abc123,true,42
Label validation	id,label_*,...	id,label_deception,label_jailbreak abc123,true,false
With splits	id,target,split	id,target,split abc123,true,dev

YAML

Here is what a YAML file would look like for a single target:

validation.yaml

- id: Fg3KBpgFr6RSsEWmHBUqeo
  target: true

- id: VFkCH7gXWpJYUYonvfHxrG
  target: false

Here is a YAML file for multiple targets:

validation.yaml

- id: Fg3KBpgFr6RSsEWmHBUqeo
  target:
     deception: true
     backtracks: 2

- id: VFkCH7gXWpJYUYonvfHxrG
  target:
     deception: false
     backtracks: 0

Here is a YAML file for label-based validation (resultsets):

validation.yaml

- id: Fg3KBpgFr6RSsEWmHBUqeo
  labels:
     deception: true
     jailbreak: false
     misconfig: false

- id: VFkCH7gXWpJYUYonvfHxrG
  labels:
     deception: false
     jailbreak: true
     misconfig: false

Here is a YAML file with splits using the flat format (split field on each case):

validation.yaml

- id: Fg3KBpgFr6RSsEWmHBUqeo
  target: true
  split: dev

- id: VFkCH7gXWpJYUYonvfHxrG
  target: false
  split: test

You can also use a nested format that groups cases by split:

validation.yaml

- split: dev
  cases:
    - id: Fg3KBpgFr6RSsEWmHBUqeo
      target: true
    - id: VFkCH7gXWpJYUYonvfHxrG
      target: false

- split: test
  cases:
    - id: SiEXpECj7U9nNAvM3H7JqB
      target: true

JSON

JSON files use the same structure as YAML:

validation.json

[
  {"id": "Fg3KBpgFr6RSsEWmHBUqeo", "target": true},
  {"id": "VFkCH7gXWpJYUYonvfHxrG", "target": false}
]

JSONL (JSON Lines) format has one case per line:

validation.jsonl

{"id": "Fg3KBpgFr6RSsEWmHBUqeo", "target": true}
{"id": "VFkCH7gXWpJYUYonvfHxrG", "target": false}