Results

Results

scan_list

List completed and pending scans.

Source

def scan_list(scans_location: str) -> list[Status]

scans_location str

Location of scans to list.

scan_status

Status of scan.

Source

def scan_status(scan_location: str) -> Status

scan_location str

Location to get status for (e.g. directory or s3 bucket)

Status

Status of scan job.

Source

@dataclass
class Status

Attributes

complete bool

Is the job complete (all transcripts scanned).

spec ScanSpec

Scan spec (transcripts, scanners, options).

location str

Location of scan directory.

summary Summary

Summary of scan (results, errors, tokens, etc.)

errors list[Error]

Errors during last scan attempt.

Summary

Summary of scan results.

Source

class Summary(BaseModel)

Attributes

scanners dict[str, ScannerSummary]

Summary for each scanner.

scan_results_df

Scan results as Pandas data frames.

Source

def scan_results_df(
    scan_location: str,
    *,
    scanner: str | None = None,
    rows: Literal["results", "transcripts"] = "results",
) -> ScanResultsDF

scan_location str

Location of scan (e.g. directory or s3 bucket).

scanner str | None

Scanner name (defaults to all scanners).

rows Literal['results', 'transcripts']

Row granularity. Specify “results” to yield a row for each scanner result (potentially multiple per transcript); Specify “transcript” to yield a row for each transcript (in which case multiple results will be packed into the value field as a JSON list of Result).

ScanResultsDF

Scan results as pandas data frames.

Source

@dataclass
class ScanResultsDF(Status)

Attributes

scanners dict[str, pd.DataFrame]

Dict of scanner name to pandas data frame.

Validation

validation_set

Create a validation set by reading cases from a file or data frame.

Source

def validation_set(
    cases: str | Path | pd.DataFrame,
    predicate: ValidationPredicate | None = "eq",
) -> ValidationSet

cases str | Path | pd.DataFrame

Path to a CSV, YAML, JSON, or JSONL file with validation cases, or data frame with validation cases.

predicate ValidationPredicate | None

Predicate for comparing scanner results to validation targets (defaults to equality comparison). For single-value targets, compares value to target directly. For dict targets, string/single-value predicates are applied to each key, while multi-value predicates receive the full dicts.

ValidationSet

Validation set for a scanner.

Source

class ValidationSet(BaseModel)

Attributes

cases list[ValidationCase]

Cases to compare scanner values against.

predicate ValidationPredicate | None

Predicate for comparing scanner results to validation targets.

For single-value targets, the predicate compares value to target directly. For dict targets, string/single-value predicates are applied to each key, while multi-value predicates receive the full dicts.

ValidationCase

Validation case for comparing to scanner results.

A ValidationCase specifies the ground truth for a scan of particular id (e.g. transcript id, message id, etc.

Use target for single-value or dict validation. Use labels for validating resultsets with label-specific expectations.

Source

class ValidationCase(BaseModel)

Attributes

id str | list[str]

Target id (e.g. transcript_id, message, id, etc.)

target JsonValue | None

Target value that the scanner is expected to output.

For single-value results, this is the expected value. For dict-valued results, this is a dict of expected values.

labels dict[str, JsonValue] | None

Label-specific target values for resultset validation.

Maps result labels to their expected values. Used when validating scanners that return multiple labeled results per transcript.

Methods

model_post_init

Validate that exactly one of target or labels is set.

Source

def model_post_init(self, __context: Any) -> None

__context Any

ValidationPredicate

Predicate used to compare scanner result with target value.

Source

ValidationPredicate: TypeAlias = (
    Literal[
        "gt",
        "gte",
        "lt",
        "lte",
        "eq",
        "ne",
        "contains",
        "startswith",
        "endswith",
        "icontains",
        "iequals",
    ]
    | PredicateFn
)