Scanner API

Scanner

Scan transcript content.

class Scanner(Protocol[T]):
    def __call__(self, input: T, /) -> Awaitable[Result | list[Result]]

input T: Input to scan.

ScannerInput

Union of all valid scanner input types.

Source

ScannerInput = Union[
    Transcript,
    ChatMessage,
    Sequence[ChatMessage],
    Event,
    Sequence[Event],
]

Result

Scan result.

Source

class Result(BaseModel)

Attributes

uuid str | None: Unique identifer for scan result.
value JsonValue: Scan value.
answer str | None: Answer extracted from model output (optional)
explanation str | None: Explanation of result (optional).
metadata dict[str, Any] | None: Additional metadata related to the result (optional)
references list[Reference]: References to relevant messages or events.
label str | None: Label for result to indicate its origin.
type str | None: Type to designate contents of ‘value’ (used in value_type field in result data frames).

Reference

Reference to scanned content.

Source

class Reference(BaseModel)

Attributes

type Literal['message', 'event']

Reference type.

cite str | None

Cite text used when the entity was referenced (optional).

For example, a model may have pointed to a message using something like [M22], which is the cite.

id str

Reference id (message or event id)

Error

Scan error (runtime error which occurred during scan).

Source

class Error(BaseModel)

Attributes

transcript_id str: Target transcript id.
scanner str: Scanner name.
error str: Error message.
traceback str: Error traceback.

Loader

Load transcript data.

Source

class Loader(Protocol[TLoaderResult]):
    def __call__(
        self,
        transcript: Transcript,
    ) -> AsyncIterator[TLoaderResult]

transcript Transcript: Transcript to yield from.

LLM Scanner

llm_scanner

Create a scanner that uses an LLM to scan transcripts.

This scanner presents a conversation transcript to an LLM along with a custom prompt and answer specification, enabling automated analysis of conversations for specific patterns, behaviors, or outcomes.

Source

@scanner(messages="all")
def llm_scanner(
    *,
    question: str | Callable[[Transcript], Awaitable[str]],
    answer: Literal["boolean", "numeric", "string"]
    | list[str]
    | AnswerMultiLabel
    | AnswerStructured,
    template: str | None = None,
    template_variables: dict[str, Any]
    | Callable[[Transcript], dict[str, Any]]
    | None = None,
    preprocessor: MessagesPreprocessor | None = None,
    model: str | Model | None = None,
    name: str | None = None,
) -> Scanner[Transcript]

question str | Callable[[Transcript], Awaitable[str]]: Question for the scanner to answer. Can be a static string (e.g., “Did the assistant refuse the request?”) or a function that takes a Transcript and returns an string for dynamic questions based on transcript content.
answer Literal['boolean', 'numeric', 'string'] | list[str] | AnswerMultiLabel | AnswerStructured: Specification of the answer format. Pass “boolean”, “numeric”, or “string” for a simple answer; pass list[str] for a set of labels; or pass MultiLabels for multi-classification.
template str | None: Overall template for scanner prompt. The scanner template should include the following variables: - {{ question }} (question for the model to answer) - {{ messages }} (transcript message history as string) - {{ answer_prompt }} (prompt for a specific type of answer). - {{ answer_format }} (instructions on how to format the answer) In addition, scanner templates can bind to any data with the Transcript being processes (e.g. {{ transcript.score }})
template_variables dict[str, Any] | Callable[[Transcript], dict[str, Any]] | None: Additional variables to make available in the template. Optionally takes a function which receives the current Transcript which can return variables.
preprocessor MessagesPreprocessor | None: Transform conversation messages before analysis. Controls exclusion of system messages, reasoning tokens, and tool calls. Defaults to removing system messages.
model str | Model | None: Optional model specification. Can be a model name string or Model instance. If None, uses the default model
name str | None: Scanner name. Use this to assign a name when passing llm_scanner() directly to scan() rather than delegating to it from another scanner.

AnswerMultiLabel

Label descriptions for LLM scanner multi-classification.

Source

class AnswerMultiLabel(NamedTuple)

Attributes

labels list[str]

List of label descriptions.

Label values (e.g. A, B, C) will be provided automatically.

Utils

messages_as_str

Concatenate list of chat messages into a string.

Source

async def messages_as_str(
    messages: list[ChatMessage],
    *,
    preprocessor: MessagesPreprocessor | None = None,
    include_ids: Literal[True] | None = None,
) -> str | tuple[str, Callable[[str], list[Reference]]]

messages list[ChatMessage]: List of chat messages.
preprocessor MessagesPreprocessor | None: Content filter for messages.
include_ids Literal[True] | None: If True, prepend ordinal references (e.g., [M1], [M2]) to each message and return a function to extract references from text. If None (default), return plain formatted string.

MessagesPreprocessor

ChatMessage preprocessing transformations.

Provide a transform function for fully custom transformations. Use the higher-level options (e.g. exclude_system) to perform varioius common content removal transformations.

The default MessagesPreprocessor will exclude system messages and do no other transformations.

Source

class MessagesPreprocessor(NamedTuple)

Attributes

transform Callable[[list[ChatMessage]], Awaitable[list[ChatMessage]]] | None: Transform the list of messages.
exclude_system bool: Exclude system messages (defaults to True)
exclude_reasoning bool: Exclude reasoning content (defaults to False).
exclude_tool_usage bool: Exclude tool usage (defaults to False)

Types

MessageType

Message types.

Source

MessageType = Literal["system", "user", "assistant", "tool"]

EventType

Event types.

Source

EventType = Literal[
    "model",
    "tool",
    "approval",
    "sandbox",
    "info",
    "logger",
    "error",
    "span_begin",
    "span_end",
]

Registration

scanner

Decorator for registering scanners.

Source

def scanner(
    factory: ScannerFactory[P, T] | None = None,
    *,
    loader: Loader[TScan] | None = None,
    messages: list[MessageType] | Literal["all"] | None = None,
    events: list[EventType] | Literal["all"] | None = None,
    name: str | None = None,
    metrics: Sequence[Metric | Mapping[str, Sequence[Metric]]]
    | Mapping[str, Sequence[Metric]]
    | None = None,
) -> (
    ScannerFactory[P, T]
    | Callable[[ScannerFactory[P, T]], ScannerFactory[P, T]]
    | Callable[[ScannerFactory[P, TScan]], ScannerFactory[P, TScan]]
    | Callable[[ScannerFactory[P, TM]], ScannerFactory[P, ScannerInput]]
    | Callable[[ScannerFactory[P, TE]], ScannerFactory[P, ScannerInput]]
)

factory ScannerFactory[P, T] | None: Decorated scanner function.
loader Loader[TScan] | None: Custom data loader for scanner.
messages list[MessageType] | Literal['all'] | None: Message types to scan.
events list[EventType] | Literal['all'] | None: Event types to scan.
name str | None: Scanner name (defaults to function name).
metrics Sequence[Metric | Mapping[str, Sequence[Metric]]] | Mapping[str, Sequence[Metric]] | None: One or more metrics to calculate over the values (only used if scanner is converted to a scorer via as_scorer()).

loader

Decorator for registering loaders.

Source

def loader(
    *,
    name: str | None = None,
    messages: list[MessageType] | Literal["all"] | None = None,
    events: list[EventType] | Literal["all"] | None = None,
    content: TranscriptContent | None = None,
) -> Callable[[LoaderFactory[P, TLoaderResult]], LoaderFactory[P, TLoaderResult]]

name str | None: Loader name (defaults to function name).
messages list[MessageType] | Literal['all'] | None: Message types to load from.
events list[EventType] | Literal['all'] | None: Event types to load from.
content TranscriptContent | None: Transcript content filter.

as_scorer

Convert a Scanner to an Inspect Scorer.

Source

def as_scorer(
    scanner: Scanner[Transcript],
    metrics: Sequence[Metric | Mapping[str, Sequence[Metric]]]
    | Mapping[str, Sequence[Metric]]
    | None = None,
) -> Scorer

scanner Scanner[Transcript]: Scanner to convert (must take a Transcript).
metrics Sequence[Metric | Mapping[str, Sequence[Metric]]] | Mapping[str, Sequence[Metric]] | None: Metrics for scorer. Defaults to metrics specified on the @scanner decorator (or [accuracy(), stderr()] if none were specified).