Inspect Viz

Data visualization for Inspect AI large language model evalutions.

Welcome

Welcome to Inspect Viz, a data visualization library for Inspect AI. Inspect Viz provides flexible tools for high quality interactive visualizations from Inspect evaluations.

Inspect Viz is built on top of the Mosaic web-based visualization framework. You can include these visualization in any Jupyter Notebook or web publishing system that supports Jupyter widgets (e.g. Quarto).

Note

While inspect_viz will soon include plots, tables, and inputs pre-built to work with data from Inspect logs, it currently provides only lower-level plotting and interactivity primitives. If you are interested in helping to build out these functions, we’d very much welcome your contributions to the inspect_viz.sandbox module.

Getting Started

First, install the inspect_ai and inspect_viz package from GitHub as follows:

pip install git+https://github.com/UKGovernmentBEIS/inspect_ai
pip install git+https://github.com/meridianlabs-ai/inspect_viz

Then, read on below for a tutorial on the basics of Inspect Viz. Once you are up to speed on the basics, read the Concepts article for a more in depth explanation of the library’s components and how they fit together.

Tip

Inspect Viz is built on top of the Moasic data visualization system which is in turn built on Observable Plot.

Below we’ll describe the the Inspect Viz Python API, which typically maps quite closely to the Observable Plot JavaScript API. Once you start creating your own plots and are using Google or an LLM to help with development, asking how to do things in Observable Plot will typically yield actionable advice.

Plots

Plots in inspect_viz are composed of one or more marks, which can do either higher level plotting (e.g. dot(), bar_x(), bar_y(), area(), heatmap(), etc.) or lower level drawing on the plot canvas (e.g. text(), image(), arrow(), etc.)

Dot Plot

Here is an example of a simple dot plot using the Palmer Penquins dataset:

from inspect_viz import Data
from inspect_viz.plot import plot
from inspect_viz.mark import dot

penguins = Data.from_file("penguins.parquet")

plot( 
   dot(penguins, x="body_mass", y="flipper_length")
)

1: Read the dataset from a parquest file. You can can also use Data.from_dataframe() to read data from any Pandas, Polars, or PyArrow data frame.
2: Plot using a dot() mark. The plot() function takes one or more marks or interactors.

Facets

The penguins data set includes various species of penguin, so lets add some elements to our plot that enable us to see the difference between species. We’ll also add grid lines to make the plot a bit easier to read.

plot(
    dot(penguins, x="body_mass", y="flipper_length",  
        stroke="species", symbol="species"),
    legend="symbol",
    grid=True
)

Here we mapped the “species” column to the stroke and symbol facets of the plot (causing each species to have its own color and symbol). We also add a legend to the plot as a key to this mapping.

Bar Plot

Here is a simple horizontal bar plot that counts the number of each species:

from inspect_viz.mark import bar_x
from inspect_viz.transform import count

plot(
    bar_x(penguins, x=count(), y="species", fill="species"),
    y_label=None,
    height=200,
    margin_left=60
)

The x axis for this plot is not mapped to a column, but rather to a count() transform ( transforms enable you to perform computations on columns for ploatting). The fill option gives each species it’s own color. We also specify that we don’t want a y_label (as the species names serve that purpose) and a smaller than normal height.

Tables

You can also display data in a tabular layout using the table() function:

from inspect_viz.table import table

table(penguins)

You can sort and filter tables by column, use a scrolling or paginated display, and customize several other aspects of table appearance and behavior.

Filters

Use inputs to enable filtering datasets and dynamically updating plots. For example, here we add a select() input that filters on the species column:

from inspect_viz.input import select
from inspect_viz.layout import vconcat

vconcat(
   select(penguins, label="Species", column="species"),
   plot(
      dot(penguins, x="body_mass", y="flipper_length",  
          stroke="species", symbol="species"),
      legend="symbol",
      color_domain="fixed"
   )
)

We’ve introduced a few new things here:

The vconcat() function from the layout module lets us stack inputs on top of our plot.
The select() function from the input module binds a select box to the species column.
The color_domain="fixed" argument to plot() indicates that we want to preserve species colors even when the plot is filtered.

Marks

So far the plots we’ve created include only a single mark, however many of the more interesting plots you’ll create will include multiple marks.

For example, here we explore the relationships between the height, weight, and sex of olympic athletes using dot() and regression_y() marks:

from inspect_viz.mark import regression_y

athletes = Data.from_file("athletes.parquet")

plot(
    dot(athletes, x="weight", y="height", fill="sex", opacity=0.1),
    regression_y(athletes, x="weight", y="height", stroke="sex"),
    legend="color"
)

Note that we set the opacity of the dot mark to 0.1 to help mitigate oversaturation that results from large numbers of data points being stacked on top of eachother.

Marks can also be used to draw lines, arrows, text, or images on a plot.

Params

As illustrated above, inputs can be used to filter dataset selections. Inputs can also be used to set Param values that make various aspects of plots dynamic. For example, here is a density plot of flight delays which uses a slider() input to vary the amount of smooth ing by setting the kernel bandwidth:

from inspect_viz import Param
from inspect_viz.input import slider
from inspect_viz.mark import density_y

flights = Data.from_file("flights.parquet")

bandwidth = Param(0.1)

vconcat(
   slider(
      label="Bandwidth (σ)", target=bandwidth,
      min=0.1, max=100, step=0.1
   ),
   plot(
      density_y(
         flights, x="delay", fill="steelblue", bandwidth=bandwidth
      ),
      x_domain="fixed",
      y_axis=None,
      height=250,
   )
)

1: Create a bandwidth parameter with a default value of 0.1.
2: Bind the slider() to the bandwidth parameter.
3: Apply the bandwidth to the plot (plot automatically redraws when the bandwidth changes).

Selections

Above in Filtering we began exploring dataset selections. Inputs are one way to set selections, but you can also set selections through direct interaction with plots.

For example, below we stack two plots vertically, the dot() plot from above along with a bar_x() plot that counts the sex column. We then add an interval_x() interactor that enables us to filter the dataset using selections on the dot plot.

There are a number of new things introduced here, click on the numbers near the right margin for additional explanation.

from inspect_viz import Selection
from inspect_viz.interactor import Brush, interval_x

range = Selection.intersect()

vconcat(
   plot(
      dot(athletes, x="weight", y="height", fill="sex", opacity=0.1),
      regression_y(athletes, x="weight", y="height", stroke="sex"),
      interval_x(
         target=range,
         brush=Brush(fill="none", stroke="#888")
      ),
      legend="color"
   ),
   plot(
      bar_x(
         athletes, filter_by=range,
         x=count(), y="sex", fill="sex"
      ), 
      y_label=None,
      height=150,
      x_domain="fixed"
   )
)

1: A Selection is a means of filtering datasets based on interactions. Here we use an “intersect” selection for application of a simple filter from dot plot to bar plot.
2: The range selection is set via the interval_x() interactor (which enables using the mouse to select an x-range).
3: The Brush defines the color of the interactor (in this case #888, a medium-gray).
4: The range selection is consumed using the filter_by parameter.
5: We set the x_domain for the bar plot to “fixed” so that the scale doesn’t change as the dataset is filtered.

Try using the mouse to brush over regions on the dot plot—the bar plot will update accordingly.

Crossfilter

In many cases you’ll want to have an input or interactor that both consumes and produces the same selection (i.e. filtered based on interactions with other inputs or interactors, but also able to provide its own filtering).

Inputs

This example demonstrates crossfiltering across inputs. We plot shot types taken during the 2023 WNBA season, providing a select() input that filters by team, and another select() input that filters by player (which in turn is also filtered by the currently selected team). Click on the numbers at right for additional explanation of the code.

from inspect_viz.layout import hconcat

shots = Data.from_file("wnba-shots-2023.parquet")

filter = Selection.crossfilter()

vconcat(
   hconcat(
      select(
         shots, label="Team", column="team_name", 
         target=filter
      ),
      select(
         shots, label="Athlete", column="athlete_name", 
         filter_by=filter, target=filter
      )
   ),
   plot(
      bar_x(
         shots, filter_by=filter,
         x=count(), y="category", fill="category"
      ),
      y_label=None,
      color_domain="fixed",
      y_domain=["Jump", "Layup", "Hook"],
      height=200,
      margin_left=60
   )
)

1: Create a crossfilter selection, which enables inputs to both consume and produce the same selection (conditioning their available choices on other inputs).
2: The team select box targets the filter selection (filtering both the choices in the athelte select box and what is displayed in the plot).
3: The athlete select box is both filtered by and targets the filter selection, enabling it to both confine itself to the selected team as well as filter what is displayed in the plot.
4: As different teams and players are selected, the y-axis may take on differnet values and ordering. These options ensure that the y-axis remains stable across selections.

Interactors

This example demonstrates crossfiltering across plot interactors. We plot histograms showing arrival delay and departure time for flights. When you select a range in one plot, the other plot updates to show only the data within that selection—and vice versa. This bidirectional filtering is achieved using Selection.crossfilter(), which ensures each plot’s selection affects all other plots except itself. Click on the numbers at right for additional explanation of the code.

from inspect_viz.mark import rect_y
from inspect_viz.transform import count, bin

flights = Data.from_file("flights.parquet")

brush = Selection.crossfilter()

def flights_plot(x, label):
   return plot(
      rect_y(
         flights, filter_by=brush, 
         x=bin(x), y=count(), fill="steelblue"
      ),
      interval_x(target=brush),
      height=200,
      x_label=label,
      x_domain="fixed",
      y_tick_format="s"
   )

vconcat(
   flights_plot("delay", "Arrival Delay (min)"),
   flights_plot("time", "Departure Time (hour)")
)

1: Create a crossfilter selection, which ensures each plot’s selection affects all other plots except itself.
2: Our two plots are identical save for the x value and the x_label so factor out into a function.
3: The interval_x() interactor enables horizontal selection (targeting the crossfiltering brush).
4: Use a "fixed" domain so that the x-axis remains stable even when being filtered.

Learning More

Use these resources to learn more about using Inspect Viz:

Concepts describes the core components of the library and how they fit together.
Reference provides details on the available marks, interactors, transforms, and inputs.
Examples demonstrates more advanced plotting and interactivity features.
Sandbox includes examples from the inspect_viz.sandbox module as well as information on contributing your own functions to the sandbox.