Model Observability with Assays

How to use Wallaroo Assays to monitor the environment and know when to retrain the model based on changes to data.

Features:

Observe

Models:

Linear-Regression

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

Model Drift Observability with Assays

The Model Insights feature lets you monitor how the environment that your model operates within may be changing in ways that affect it’s predictions so that you can intervene (retrain) in an efficient and timely manner. Changes in the inputs, data drift, can occur due to errors in the data processing pipeline or due to changes in the environment such as user preference or behavior.

This notebook focuses on interactive exploration over historical data. After you are comfortable with how your data has behaved historically, you can schedule this same analysis (called an assay) to automatically run periodically, looking for indications of data drift or concept drift.

In this notebook, we will be running a drift assay on an ONNX model pre-trained to predict house prices.

Goal

Model insights monitors the output of the spam classifier model over a designated time window and compares it to an expected baseline distribution. We measure the difference between the window distribution and the baseline distribution; large differences indicate that the behavior of the model (or its inputs) has changed from what we expect. This possibly indicates a change that should be accounted for, possibly by retraining the models.

Resources

This tutorial provides the following:

Models:
- models/rf_model.onnx: The champion model that has been used in this environment for some time.
- Various inputs:
  - smallinputs.df.json: A set of house inputs that tends to generate low house price values.
  - biginputs.df.json: A set of house inputs that tends to generate high house price values.

Prerequisites

A deployed Wallaroo instance
The following Python libraries installed:
- wallaroo: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.
- pandas: Pandas, mainly used for Pandas DataFrame

Steps

Deploying a sample ML model used to determine house prices based on a set of input parameters.
Build an assay baseline from a set of baseline start and end dates, and an assay baseline from a numpy array.
Preview the assay and show different assay configurations.
Upload the assay.
View assay results.
Pause and resume the assay.

Import Libraries

The first step will be to import our libraries, and set variables used through this tutorial.

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

from IPython.display import display

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)

import datetime
import time

workspace_name = f'assay-demonstration-tutorial-2'
main_pipeline_name = f'assay-demonstration-tutorial'
model_name_control = f'house-price-estimator'
model_file_name_control = './models/rf_model.onnx'

# Set the name of the assay
assay_name=f"house price assay demo"

# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()

Create Workspace

We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace.

Workspace, pipeline, and model names should be unique to each user, so we’ll add in a randomly generated suffix so multiple people can run this tutorial in a Wallaroo instance without effecting each other.

workspace = wl.get_workspace(name=workspace_name, create_if_not_exist=True)

wl.set_current_workspace(workspace)

{'name': 'assay-demonstration-tutorial-2', 'id': 11, 'archived': False, 'created_by': '3eeab554-fc2e-41ac-8b4e-0d778e8a471b', 'created_at': '2024-10-09T19:50:23.231562+00:00', 'models': [{'name': 'house-price-estimator', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2024, 10, 16, 22, 0, 48, 467677, tzinfo=tzutc()), 'created_at': datetime.datetime(2024, 10, 9, 19, 50, 24, 679975, tzinfo=tzutc())}], 'pipelines': [{'name': 'assay-demonstration-tutorial', 'create_time': datetime.datetime(2024, 10, 9, 19, 50, 25, 224073, tzinfo=tzutc()), 'definition': '[]'}]}

Upload The Champion Model

For our example, we will upload the champion model that has been trained to derive house prices from a variety of inputs. The model file is rf_model.onnx, and is uploaded with the name house-price-estimator.

housing_model_control = (wl.upload_model(model_name_control, 
                                        model_file_name_control, 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

Build the Pipeline

This pipeline is made to be an example of an existing situation where a model is deployed and being used for inferences in a production environment. We’ll call it assay-demonstration-tutorial, set housing_model_control as a pipeline step, then run a few sample inferences.

This pipeline will be a simple one - just a single pipeline step.

# mainpipeline = wl.build_pipeline(main_pipeline_name)
# # clear the steps if used before
# mainpipeline.clear()

# mainpipeline.add_model_step(housing_model_control)

mainpipeline = wl.get_pipeline(main_pipeline_name)

#minimum deployment config
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

mainpipeline.deploy(deployment_config = deploy_config)

ok

name	assay-demonstration-tutorial
created	2024-10-09 19:50:25.224073+00:00
last_updated	2024-10-18 15:50:20.759830+00:00
deployed	True
workspace_id	11
workspace_name	assay-demonstration-tutorial-2
arch	x86
accel	none
tags
versions	ba6b344c-83a0-4308-bc62-f198f8af8a5d, 0740ad84-a770-4cf9-b1ce-1f45dbe9c1c4, a2494de7-4d3a-428f-a241-9ddff1ae6772, 6ff08bf3-9be8-484c-b35f-cfde42854ad7, 54380aa2-8722-4b80-b511-b494c6f42c4b, b7a878c6-4f62-4e57-843c-23604c38704b
steps	house-price-estimator
published	False

Testing

We’ll use two inferences as a quick sample test - one that has a house that should be determined around $700k, the other with a house determined to be around $1.5 million.

normal_input = pd.DataFrame.from_records({"tensor": [[4.0, 2.5, 2900.0, 5505.0, 2.0, 0.0, 0.0, 3.0, 8.0, 2900.0, 0.0, 47.6063, -122.02, 2970.0, 5251.0, 12.0, 0.0, 0.0]]})
result = mainpipeline.infer(normal_input)
display(result)

	time	in.tensor	out.variable	anomaly.count
0	2024-10-18 15:50:21.222	[4.0, 2.5, 2900.0, 5505.0, 2.0, 0.0, 0.0, 3.0, 8.0, 2900.0, 0.0, 47.6063, -122.02, 2970.0, 5251.0, 12.0, 0.0, 0.0]	[718013.7]	0

large_house_input = pd.DataFrame.from_records({'tensor': [[4.0, 3.0, 3710.0, 20000.0, 2.0, 0.0, 2.0, 5.0, 10.0, 2760.0, 950.0, 47.6696, -122.261, 3970.0, 20000.0, 79.0, 0.0, 0.0]]})
large_house_result = mainpipeline.infer(large_house_input)
display(large_house_result)

	time	in.tensor	out.variable	anomaly.count
0	2024-10-18 15:50:21.270	[4.0, 3.0, 3710.0, 20000.0, 2.0, 0.0, 2.0, 5.0, 10.0, 2760.0, 950.0, 47.6696, -122.261, 3970.0, 20000.0, 79.0, 0.0, 0.0]	[1514079.4]	0

Generate Sample Data

Before creating the assays, we must generate data for the assays to build from.

For this example, we will:

Perform sample inferences based on lower priced houses and use that as our baseline.
Generate inferences from specific set of high priced houses create inference outputs that will be outside the baseline. This is used in later steps to demonstrate baseline comparison against assay analyses.

Inference Results History Generation

To start the demonstration, we’ll create a set of inferences based on houses with small estimated prices.

We will save the beginning and end periods of inferences to the variables assay_baseline_start and assay_baseline_end. These will be used later to create assay baselines from a range of dates.

small_houses_inputs = pd.read_json('./data/smallinputs.df.json')
baseline_size = 500

# Where the baseline data will start
assay_baseline_start = datetime.datetime.now()

# These inputs will be random samples of small priced houses.  Around 30,000 is a good number
small_houses = small_houses_inputs.sample(baseline_size, replace=True).reset_index(drop=True)

# Wait 60 seconds to set this data apart from the rest
time.sleep(60)
small_results = mainpipeline.infer(small_houses)

# Set the baseline end

assay_baseline_end = datetime.datetime.now()

Generate Numpy Baseline Values

From our inference outputs, we will create an array of numpy values. These are used for assays baselines created from numpy values.

# get the numpy values

# set the results to a non-array value
small_results_baseline_df = small_results.copy()
small_results_baseline_df['variable']=small_results['out.variable'].map(lambda x: x[0])
small_results_baseline_df

# set the numpy array
small_results_baseline = small_results_baseline_df['variable'].to_numpy()

Assay Test Data

The following will generate inference data for us to test against the assay baseline. For this, we will add in house data that generate higher house prices than the baseline data we used earlier.

This process should take 6 minutes to generate the historical data we’ll later use in our assays. We store the DateTime assay_window_start to determine where to start out assay analyses.

# Get a spread of house values

# # Set the start for our assay window period.
assay_window_start = datetime.datetime.now()

for x in range(10):
    time.sleep(65)
    inference_size = 1000

    # And a spread of small house values

    small_houses_inputs = pd.read_json('./data/smallinputs.df.json', orient="records")
    small_houses = small_houses_inputs.sample(inference_size, replace=True).reset_index(drop=True)

    mainpipeline.infer(small_houses)

time.sleep(65)

# Get a spread of large house values

for x in range(10):
    time.sleep(65)
    inference_size = 1000

    # And a spread of large house values

    big_houses_inputs = pd.read_json('./data/biginputs.df.json', orient="records")
    big_houses = big_houses_inputs.sample(inference_size, replace=True).reset_index(drop=True)

    mainpipeline.infer(big_houses)

time.sleep(65)

Model Insights via the Wallaroo Dashboard SDK

Assays generated through the Wallaroo SDK can be previewed, configured, and uploaded to the Wallaroo Ops instance. The following is a condensed version of this process. For full details see the Wallaroo SDK Essentials Guide: Assays Management guide.

Model drift detection with assays using the Wallaroo SDK follows this general process.

Define the Baseline: From either historical inference data for a specific model in a pipeline, or from a pre-determine array of data, a baseline is formed.
Assay Preview: Once the baseline is formed, we preview the assay and configure the different options until we have the the best method of detecting environment or model drift.
Create Assay: With the previews and configuration complete, we upload the assay. The assay will perform an analysis on a regular scheduled based on the configuration.
Get Assay Results: Retrieve the analyses and use them to detect model drift and possible sources.
Pause/Resume Assay: Pause or restart an assay as needed.

Define the Baseline

Assay baselines are defined with the wallaroo.client.build_assay method. Through this process we define the baseline from either a range of dates or pre-generated values.

wallaroo.client.build_assay take the following parameters:

Parameter	Type	Description
assay_name	String (Required) - required	The name of the assay. Assay names must be unique across the Wallaroo instance.
pipeline	wallaroo.pipeline.Pipeline (Required)	The pipeline the assay is monitoring.
model_name	String (Optional) / `None`	The name of the model to monitor. This field should only be used to track the inputs/outputs for a specific model step in a pipeline. If no `model_name` is to be included, then the parameters must be passed a named parameters not positional ones.
iopath	String (Required)	The input/output data for the model being tracked in the format `input/output field index`. Only one value is tracked for any assay. For example, to track the output of the model’s field `house_value` at index `0`, the `iopath` is `'output house_value 0`.
baseline_start	datetime.datetime (Optional)	The start time for the inferences to use as the baseline. Must be included with `baseline_end`. Cannot be included with `baseline_data`.
baseline_end	datetime.datetime (Optional)	The end time of the baseline window. the baseline. Windows start immediately after the baseline window and are run at regular intervals continuously until the assay is deactivated or deleted. Must be included with `baseline_start`. Cannot be included with `baseline_data`..
baseline_data	numpy.array (Optional)	The baseline data in numpy array format. Cannot be included with either `baseline_start` or `baseline_data`.

Note that model_name is an optional parameters when parameters are named. For example:

assay_builder_from_dates = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

or:

assay_builder_from_dates = wl.build_assay("assays from date baseline", 
                                          mainpipeline, 
                                          None, ## since we are using positional parameters, `None` must be included for the model parameter
                                          "output variable 0",
                                          assay_baseline_start, 
                                          assay_baseline_end)

Baselines are created in one of two mutually exclusive methods:

Date Range: The baseline_start and baseline_end retrieves the inference requests and results for the pipeline from the start and end period. This data is summarized and used to create the baseline. For our examples, we’re using the variables assay_baseline_start and assay_baseline_end to represent a range of dates, with assay_baseline_start being set before assay_baseline_end.
Numpy Values: The baseline_data sets the baseline from a provided numpy array. This allows assay baselines to be created without first performing inferences in Wallaroo.

Define the Baseline Example

This example shows two methods of defining the baseline for an assay:

"assays from date baseline": This assay uses historical inference requests to define the baseline. This assay is saved to the variable assay_builder_from_dates.
"assays from numpy": This assay uses a pre-generated numpy array to define the baseline. This assay is saved to the variable assay_builder_from_numpy.

In both cases, the following parameters are used:

Parameter	Value
assay_name	`"assays from date baseline"` and `"assays from numpy"`
pipeline	`mainpipeline`: A pipeline with a ML model that predicts house prices. The output field for this model is `variable`.
iopath	These assays monitor the model’s output field variable at index 0 for the pipeline. From this, the `iopath` setting is `"output variable 0"`.

The difference between the two assays’ parameters determines how the baseline is generated.

"assays from date baseline": Uses the baseline_start and baseline_end to set the time period of inference requests and results to gather data from.
"assays from numpy": Uses a pre-generated numpy array as for the baseline data.

First we generate an assay baseline from a range of historical inferences performed through the specified pipeline deployment.

# Build the assay, based on the start and end of our baseline time, 
# and tracking the output variable index 0

display(assay_baseline_start)
display(assay_baseline_end)

assay_baseline_from_dates = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# create the baseline from the dates
assay_baseline_run_from_dates = assay_baseline_from_dates.build().interactive_baseline_run()

datetime.datetime(2024, 10, 18, 15, 50, 21, 296181)
datetime.datetime(2024, 10, 18, 15, 51, 21, 378013)

# storing the workspace id for later requests
workspace_id = workspace.id()
workspace_id

In this code sample, we generate the assay baseline from a preset numpy array called small_results_baseline.

# build the baseline from the numpy array
display(small_results_baseline[0:5])

# assay builder by baseline
assay_baseline_from_numpy = wl.build_assay(assay_name="assays from numpy", 
                               pipeline=mainpipeline, 
                               iopath="output variable 0", 
                               baseline_data = small_results_baseline)

# create the baseline from the numpy array
assay_baseline_run_from_numpy = assay_baseline_from_numpy.build().interactive_baseline_run()

array([ 450867.7 , 1140733.8 ,  974485.3 ,  873848.44,  249227.83])

Baseline Chart

The baseline chart is displayed with wallaroo.assay.AssayAnalysis.chart(), which returns a chart with:

baseline mean: The mean value of the baseline values.
baseline median: The median value of the baseline values.
bin_mode: The binning mode. See Binning Mode
aggregation: The aggregation type. See Aggregation Options
metric: The assay’s metric type. See Score Metric
weighted: Whether the binning mode is weighted. See Binning Mode

The first chart is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_run_from_dates.chart()

baseline mean = 539895.63075
baseline median = 450283.78125
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False

This chart is from an assay baseline generated from a set of numpy values.

assay_baseline_run_from_numpy.chart()

baseline mean = 539895.6332000002
baseline median = 450283.79
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False

Baseline DataFrame

The method wallaroo.assay_config.AssayBuilder.baseline_dataframe returns a DataFrame of the assay baseline generated from the provided parameters. This includes:

metadata: The inference metadata with the model information, inference time, and other related factors.
in data: Each input field assigned with the label in.{input field name}.
out data: Each output field assigned with the label out.{output field name}

Note that for assays generated from numpy values, there is only the out data based on the supplied baseline data.

In the following example, the baseline DataFrame is retrieved.

This baseline DataFrame is from an assay baseline generated from a set of inferences across a range of dates.

display(assay_baseline_from_dates.baseline_dataframe())

	time	metadata	input_tensor_0	input_tensor_1	input_tensor_2	input_tensor_3	input_tensor_4	input_tensor_5	input_tensor_6	input_tensor_7	input_tensor_8	input_tensor_9	input_tensor_10	input_tensor_11	input_tensor_12	input_tensor_13	input_tensor_14	input_tensor_15	input_tensor_16	input_tensor_17	output_variable_0
0	1729266681362	{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': 'ba6b344c-83a0-4308-bc62-f198f8af8a5d', 'elapsed': [2131469, 3616390], 'dropped': [], 'partition': 'engine-67d6ff94cb-mtn8m'}	3.0	1.00	1030.0	3000.0	1.0	0.0	0.0	3.0	7.0	830.0	200.0	47.681301	-122.317001	1830.0	3000.0	90.0	0.0	0.0	4.508677e+05
1	1729266681362	{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': 'ba6b344c-83a0-4308-bc62-f198f8af8a5d', 'elapsed': [2131469, 3616390], 'dropped': [], 'partition': 'engine-67d6ff94cb-mtn8m'}	4.0	4.25	3500.0	8750.0	1.0	0.0	4.0	5.0	9.0	2140.0	1360.0	47.722198	-122.366997	3110.0	8750.0	63.0	0.0	0.0	1.140734e+06
2	1729266681362	{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': 'ba6b344c-83a0-4308-bc62-f198f8af8a5d', 'elapsed': [2131469, 3616390], 'dropped': [], 'partition': 'engine-67d6ff94cb-mtn8m'}	5.0	3.00	3320.0	5354.0	2.0	0.0	0.0	3.0	9.0	3320.0	0.0	47.654202	-122.331001	2330.0	4040.0	11.0	0.0	0.0	9.744853e+05
3	1729266681362	{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': 'ba6b344c-83a0-4308-bc62-f198f8af8a5d', 'elapsed': [2131469, 3616390], 'dropped': [], 'partition': 'engine-67d6ff94cb-mtn8m'}	4.0	3.50	3660.0	4903.0	2.0	0.0	0.0	3.0	9.0	2760.0	900.0	47.718399	-122.155998	3630.0	4992.0	0.0	0.0	0.0	8.738484e+05
4	1729266681362	{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': 'ba6b344c-83a0-4308-bc62-f198f8af8a5d', 'elapsed': [2131469, 3616390], 'dropped': [], 'partition': 'engine-67d6ff94cb-mtn8m'}	2.0	2.00	1390.0	1302.0	2.0	0.0	0.0	3.0	7.0	1390.0	0.0	47.308899	-122.330002	1390.0	1302.0	28.0	0.0	0.0	2.492278e+05
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
495	1729266681362	{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': 'ba6b344c-83a0-4308-bc62-f198f8af8a5d', 'elapsed': [2131469, 3616390], 'dropped': [], 'partition': 'engine-67d6ff94cb-mtn8m'}	1.0	1.00	1090.0	32010.0	1.0	0.0	0.0	4.0	6.0	1090.0	0.0	47.692799	-121.870003	1870.0	25346.0	56.0	0.0	0.0	4.444080e+05
496	1729266681362	{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': 'ba6b344c-83a0-4308-bc62-f198f8af8a5d', 'elapsed': [2131469, 3616390], 'dropped': [], 'partition': 'engine-67d6ff94cb-mtn8m'}	3.0	1.75	1630.0	15600.0	1.0	0.0	0.0	3.0	7.0	1630.0	0.0	47.680000	-122.165001	1830.0	10850.0	56.0	0.0	0.0	5.596311e+05
497	1729266681362	{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': 'ba6b344c-83a0-4308-bc62-f198f8af8a5d', 'elapsed': [2131469, 3616390], 'dropped': [], 'partition': 'engine-67d6ff94cb-mtn8m'}	2.0	1.00	770.0	5040.0	1.0	0.0	0.0	3.0	5.0	770.0	0.0	47.596401	-122.299004	1330.0	2580.0	85.0	0.0	0.0	4.486278e+05
498	1729266681362	{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': 'ba6b344c-83a0-4308-bc62-f198f8af8a5d', 'elapsed': [2131469, 3616390], 'dropped': [], 'partition': 'engine-67d6ff94cb-mtn8m'}	5.0	2.50	2600.0	3839.0	2.0	0.0	0.0	3.0	7.0	2600.0	0.0	47.432400	-122.144997	2180.0	4800.0	9.0	0.0	0.0	4.006765e+05
499	1729266681362	{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': 'ba6b344c-83a0-4308-bc62-f198f8af8a5d', 'elapsed': [2131469, 3616390], 'dropped': [], 'partition': 'engine-67d6ff94cb-mtn8m'}	3.0	1.75	1520.0	12282.0	1.0	0.0	0.0	3.0	8.0	1220.0	300.0	47.751598	-122.299004	1860.0	13500.0	36.0	0.0	0.0	4.249666e+05

500 rows × 21 columns

This baseline DataFrame is from an assay baseline generated from an array of numpy values.

display(assay_baseline_from_numpy.baseline_dataframe())

	output_variable_0
0	450867.70
1	1140733.80
2	974485.30
3	873848.44
4	249227.83
...	...
495	444408.00
496	559631.06
497	448627.80
498	400676.47
499	424966.60

500 rows × 1 columns

Baseline Stats

The method wallaroo.assay.AssayAnalysis.baseline_stats() returns a pandas.core.frame.DataFrame of the baseline stats.

The baseline stats for each assay are displayed in the examples below.

This baseline states DataFrame is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_run_from_dates.baseline_stats()

	Baseline
count	500
min	236238.671875
max	1489624.25
mean	539895.63075
median	450283.78125
std	252708.493978
start	2024-10-18T15:50:21.296181+00:00
end	2024-10-18T15:51:21.377181+00:00

This baseline states DataFrame is from an assay baseline generated from an array of numpy values.

assay_baseline_run_from_numpy.baseline_stats()

	Baseline
count	500
min	236238.67
max	1489624.3
mean	539895.6332
median	450283.79
std	252708.498359
start	None
end	None

Baseline Bins

The method wallaroo.assay.AssayAnalysis.baseline_bins a simple dataframe to with the edge/bin data for a baseline.

These baseline bins DataFrame is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_run_from_dates.baseline_bins()

	b_edges	b_edge_names	b_aggregated_values	b_aggregation
0	2.362387e+05	left_outlier	0.000	Aggregation.DENSITY
1	3.383579e+05	q_20	0.202	Aggregation.DENSITY
2	4.371780e+05	q_40	0.212	Aggregation.DENSITY
3	5.235762e+05	q_60	0.186	Aggregation.DENSITY
4	7.184457e+05	q_80	0.204	Aggregation.DENSITY
5	1.489624e+06	q_100	0.196	Aggregation.DENSITY
6	inf	right_outlier	0.000	Aggregation.DENSITY

These baseline bins DataFrame is from an assay baseline generated from an array of numpy values.

assay_baseline_run_from_numpy.baseline_bins()

	b_edges	b_edge_names	b_aggregated_values	b_aggregation
0	236238.67	left_outlier	0.000	Aggregation.DENSITY
1	338357.88	q_20	0.202	Aggregation.DENSITY
2	437177.97	q_40	0.212	Aggregation.DENSITY
3	523576.20	q_60	0.186	Aggregation.DENSITY
4	718445.70	q_80	0.204	Aggregation.DENSITY
5	1489624.30	q_100	0.196	Aggregation.DENSITY
6	inf	right_outlier	0.000	Aggregation.DENSITY

Baseline Histogram Chart

The method wallaroo.assay_config.AssayBuilder.baseline_histogram returns a histogram chart of the assay baseline generated from the provided parameters.

These chart is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_from_dates.baseline_histogram()

Baseline KDE Chart

The method wallaroo.assay_config.AssayBuilder.baseline_kde returns a Kernel Density Estimation (KDE) chart of the assay baseline generated from the provided parameters.

These chart is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_from_dates.baseline_kde()

Baseline ECDF Chart

The method wallaroo.assay_config.AssayBuilder.baseline_ecdf returns a Empirical Cumulative Distribution Function (CDF) chart of the assay baseline generated from the provided parameters.

These chart is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_from_dates.baseline_ecdf()

Assay Preview

Now that the baseline is defined, we look at different configuration options and view how the assay baseline and results changes. Once we determine what gives us the best method of determining model drift, we can create the assay.

Analysis List Chart Scores

Analysis List scores show the assay scores for each assay result interval in one chart. Values that are outside of the alert threshold are colored red, while scores within the alert threshold are green.

Assay chart scores are displayed with the method wallaroo.assay.AssayAnalysisList.chart_scores, with ability to display an optional title with the chart. This takes the following parameters:

Parameter	Type	Description
title	String (Optional)	The title to assign the chart.
nth_x_tick	Integer (Optional) (Default: 4)	The density of the x ticks.
start	DateTime (Optional)	The start time of the chart. If `start` is set, the parameter `end` must be set.
end	DateTime (Optional)	The end time of the chart. If `end` is set, the parameter `start` must be set.

The following example shows retrieving the assay results and displaying the chart scores. From our example, we have two windows - the first should be green, and the second is red showing that values were outside the alert threshold.

assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show the analyses chart
assay_results.chart_scores()

The following chart uses the nth_x_tick setting to space out the x-axis labels further.

# set different nth_x_tick
assay_results.chart_scores(nth_x_tick=10)

Analysis Chart

The method wallaroo.assay.AssayAnalysis.chart() displays a comparison between the baseline and an interval of inference data.

This is compared to the Chart Scores, which is a list of all of the inference data split into intervals, while the Analysis Chart shows the breakdown of one set of inference data against the baseline.

Score from the Analysis List Chart Scores and each element from the Analysis List DataFrame generates

The following fields are included.

Field	Type	Description
baseline mean	Float	The mean of the baseline values.
window mean	Float	The mean of the window values.
baseline median	Float	The median of the baseline values.
window median	Float	The median of the window values.
bin_mode	String	The binning mode used for the assay.
aggregation	String	The aggregation mode used for the assay.
metric	String	The metric mode used for the assay.
weighted	Bool	Whether the bins were manually weighted.
score	Float	The score from the assay window.
scores	List(Float)	The score from each assay window bin.
index	Integer/None	The window index. Interactive assay runs are `None`.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline sample", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display one of the analysis from the total results
assay_results[0].chart()

baseline mean = 539895.63075
window mean = 520529.742265625
baseline median = 450283.78125
window median = 450867.6875
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.027998899384779063
scores = [0.0, 0.00031071866653010913, 0.00030773024662236867, 0.002769365409558061, 0.015958199760662362, 0.008652885301406161, 0.0]
index = None

Analysis List DataFrame

wallaroo.assay.AssayAnalysisList.to_dataframe() returns a DataFrame showing the assay results for each window aka individual analysis. This DataFrame contains the following fields:

Field	Type	Description
assay_id	Integer/None	The assay id. Only provided from uploaded and executed assays.
name	String/None	The name of the assay. Only provided from uploaded and executed assays.
iopath	String/None	The iopath of the assay. Only provided from uploaded and executed assays.
score	Float	The assay score.
start	DateTime	The DateTime start of the assay window.
min	Float	The minimum value in the assay window.
max	Float	The maximum value in the assay window.
mean	Float	The mean value in the assay window.
median	Float	The median value in the assay window.
std	Float	The standard deviation value in the assay window.
warning_threshold	Float/None	The warning threshold of the assay window.
alert_threshold	Float/None	The alert threshold of the assay window.
status	String	The assay window status. Values are: `OK`: The score is within accepted thresholds. `Warning`: The score has triggered the `warning_threshold` if exists, but not the `alert_threshold`. `Alert`: The score has triggered the the `alert_threshold`.

For this example, the assay analysis list DataFrame is listed.

From this tutorial, we should have 2 windows of dta to look at, each one minute apart. The first window should show status: OK, with the second window with the very large house prices will show status: alert

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display the dataframe from the analyses
assay_results.to_dataframe()

	id	assay_id	assay_name	pipeline_id	workspace_id	workspace_name	score	start	min	max	mean	median	std	warning_threshold	alert_threshold	status
0	None	None	assays from date baseline	None	None	None	0.027999	2024-10-18T15:52:21.396951+00:00	2.362387e+05	1489624.250	5.205297e+05	4.508677e+05	226843.358512	None	0.25	Ok
1	None	None	assays from date baseline	None	None	None	0.016821	2024-10-18T15:53:21.396951+00:00	2.362387e+05	1489624.250	5.231229e+05	4.508677e+05	226845.586466	None	0.25	Ok
2	None	None	assays from date baseline	None	None	None	0.010160	2024-10-18T15:54:21.396951+00:00	2.362387e+05	1489624.250	5.183006e+05	4.486278e+05	227582.307451	None	0.25	Ok
3	None	None	assays from date baseline	None	None	None	0.024660	2024-10-18T15:55:21.396951+00:00	2.362387e+05	1489624.250	5.178753e+05	4.486278e+05	228060.575593	None	0.25	Ok
4	None	None	assays from date baseline	None	None	None	0.026467	2024-10-18T15:56:21.396951+00:00	2.362387e+05	1489624.250	5.120811e+05	4.486278e+05	218875.510587	None	0.25	Ok
5	None	None	assays from date baseline	None	None	None	0.012931	2024-10-18T15:57:21.396951+00:00	2.362387e+05	1489624.250	5.246340e+05	4.500488e+05	233440.321856	None	0.25	Ok
6	None	None	assays from date baseline	None	None	None	0.023760	2024-10-18T15:58:21.396951+00:00	2.362387e+05	1489624.250	5.024677e+05	4.486278e+05	214154.238483	None	0.25	Ok
7	None	None	assays from date baseline	None	None	None	0.014865	2024-10-18T15:59:21.396951+00:00	2.362387e+05	1489624.250	5.240553e+05	4.508677e+05	243061.741658	None	0.25	Ok
8	None	None	assays from date baseline	None	None	None	0.047626	2024-10-18T16:00:21.396951+00:00	2.362387e+05	1489624.250	5.184981e+05	4.508677e+05	228599.592756	None	0.25	Ok
9	None	None	assays from date baseline	None	None	None	0.014559	2024-10-18T16:01:21.396951+00:00	2.362387e+05	1489624.250	5.219783e+05	4.508677e+05	234342.656919	None	0.25	Ok
10	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:04:21.396951+00:00	1.514079e+06	2016006.125	1.878114e+06	1.946438e+06	165762.752603	None	0.25	Alert
11	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:05:21.396951+00:00	1.514079e+06	2016006.125	1.869432e+06	1.946438e+06	169654.651904	None	0.25	Alert
12	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:06:21.396951+00:00	1.514079e+06	2016006.125	1.881715e+06	1.946438e+06	162160.610713	None	0.25	Alert
13	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:07:21.396951+00:00	1.514079e+06	2016006.125	1.883215e+06	1.946438e+06	160483.084939	None	0.25	Alert
14	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:08:21.396951+00:00	1.514079e+06	2016006.125	1.881695e+06	1.946438e+06	159266.187983	None	0.25	Alert
15	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:09:21.396951+00:00	1.514079e+06	2016006.125	1.884403e+06	1.946438e+06	156081.344812	None	0.25	Alert
16	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:10:21.396951+00:00	1.514079e+06	2016006.125	1.877560e+06	1.946438e+06	160930.722136	None	0.25	Alert
17	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:11:21.396951+00:00	1.514079e+06	2016006.125	1.879128e+06	1.946438e+06	162368.666688	None	0.25	Alert
18	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:12:21.396951+00:00	1.514079e+06	2016006.125	1.887236e+06	1.946438e+06	153937.005271	None	0.25	Alert
19	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:13:21.396951+00:00	1.514079e+06	2016006.125	1.880186e+06	1.946438e+06	161706.985040	None	0.25	Alert

Analysis List Full DataFrame

wallaroo.assay.AssayAnalysisList.to_full_dataframe() returns a DataFrame showing all values, including the inputs and outputs from the assay results for each window aka individual analysis. This DataFrame contains the following fields:

pipeline_id	warning_threshold	bin_index	created_at

Field	Type	Description
window_start	DateTime	The date and time when the window period began.
analyzed_at	DateTime	The date and time when the assay analysis was performed.
elapsed_millis	Integer	How long the analysis took to perform in milliseconds.
baseline_summary_count	Integer	The number of data elements from the baseline.
baseline_summary_min	Float	The minimum value from the baseline summary.
baseline_summary_max	Float	The maximum value from the baseline summary.
baseline_summary_mean	Float	The mean value of the baseline summary.
baseline_summary_median	Float	The median value of the baseline summary.
baseline_summary_std	Float	The standard deviation value of the baseline summary.
baseline_summary_edges_{0…n}	Float	The baseline summary edges for each baseline edge from 0 to number of edges.
summarizer_type	String	The type of summarizer used for the baseline. See `wallaroo.assay_config` for other summarizer types.
summarizer_bin_weights	List / None	If baseline bin weights were provided, the list of those weights. Otherwise, `None`.
summarizer_provided_edges	List / None	If baseline bin edges were provided, the list of those edges. Otherwise, `None`.
status	String	The assay window status. Values are: `OK`: The score is within accepted thresholds. `Warning`: The score has triggered the `warning_threshold` if exists, but not the `alert_threshold`. `Alert`: The score has triggered the the `alert_threshold`.
id	Integer/None	The id for the window aka analysis. Only provided from uploaded and executed assays.
assay_id	Integer/None	The assay id. Only provided from uploaded and executed assays.
pipeline_id	Integer/None	The pipeline id. Only provided from uploaded and executed assays.
warning_threshold	Float	The warning threshold set for the assay.
warning_threshold	Float	The warning threshold set for the assay.
bin_index	Integer/None	The bin index for the window aka analysis.
created_at	Datetime/None	The date and time the window aka analysis was generated. Only provided from uploaded and executed assays.

For this example, full DataFrame from an assay preview is generated.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display the full dataframe from the analyses
assay_results.to_full_dataframe()

	window_start	analyzed_at	elapsed_millis	baseline_summary_count	baseline_summary_min	baseline_summary_max	baseline_summary_mean	baseline_summary_median	baseline_summary_std	baseline_summary_edges_0	baseline_summary_edges_1	baseline_summary_edges_2	baseline_summary_edges_3	baseline_summary_edges_4	baseline_summary_edges_5	baseline_summary_edges_6	baseline_summary_edge_names_0	baseline_summary_edge_names_1	baseline_summary_edge_names_2	baseline_summary_edge_names_3	baseline_summary_edge_names_4	baseline_summary_edge_names_5	baseline_summary_edge_names_6	baseline_summary_aggregated_values_1	baseline_summary_aggregated_values_2	baseline_summary_aggregated_values_3	baseline_summary_aggregated_values_4	baseline_summary_aggregated_values_5	baseline_summary_aggregation	baseline_summary_start	baseline_summary_end	window_summary_count	window_summary_min	window_summary_max	window_summary_mean	window_summary_median	window_summary_std	window_summary_edges_0	window_summary_edges_1	window_summary_edges_2	window_summary_edges_3	window_summary_edges_4	window_summary_edges_5	window_summary_edges_6	window_summary_edge_names_0	window_summary_edge_names_1	window_summary_edge_names_2	window_summary_edge_names_3	window_summary_edge_names_4	window_summary_edge_names_5	window_summary_edge_names_6	window_summary_aggregated_values_1	window_summary_aggregated_values_2	window_summary_aggregated_values_3	window_summary_aggregated_values_4	window_summary_aggregated_values_5	window_summary_aggregated_values_6	window_summary_aggregation	window_summary_start	window_summary_end	alert_threshold	score	scores_1	scores_2	scores_3	scores_4	scores_5	scores_6	summarizer_bin_mode	summarizer_aggregation	summarizer_metric	summarizer_num_bins	summarizer_type	summarizer_bin_weights	summarizer_provided_edges	status	id	assay_id	pipeline_id	workspace_id	workspace_name	warning_threshold	bin_index	created_at
0	2024-10-18T15:52:21.396951+00:00	2024-10-18T16:16:51.372362+00:00	118	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	2.362387e+05	1489624.250	5.205297e+05	4.508677e+05	226843.358512	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.210	0.204	0.164	0.265	0.157	0.0	Density	2024-10-18T15:52:21.396951+00:00	2024-10-18T15:53:21.396951+00:00	0.25	0.027999	0.000311	0.000308	0.002769	0.015958	0.008653	0.000000	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Ok	None	None	None	None	None	None	None	None
1	2024-10-18T15:53:21.396951+00:00	2024-10-18T16:16:51.372433+00:00	174	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	2.362387e+05	1489624.250	5.231229e+05	4.508677e+05	226845.586466	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.200	0.219	0.172	0.248	0.161	0.0	Density	2024-10-18T15:53:21.396951+00:00	2024-10-18T15:54:21.396951+00:00	0.25	0.016821	0.000020	0.000227	0.001096	0.008594	0.006885	0.000000	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Ok	None	None	None	None	None	None	None	None
2	2024-10-18T15:54:21.396951+00:00	2024-10-18T16:16:51.372476+00:00	186	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	2.362387e+05	1489624.250	5.183006e+05	4.486278e+05	227582.307451	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.218	0.220	0.162	0.227	0.173	0.0	Density	2024-10-18T15:54:21.396951+00:00	2024-10-18T15:55:21.396951+00:00	0.25	0.010160	0.001220	0.000296	0.003316	0.002457	0.002871	0.000000	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Ok	None	None	None	None	None	None	None	None
3	2024-10-18T15:55:21.396951+00:00	2024-10-18T16:16:51.372528+00:00	195	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	2.362387e+05	1489624.250	5.178753e+05	4.486278e+05	228060.575593	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.235	0.199	0.156	0.246	0.164	0.0	Density	2024-10-18T15:55:21.396951+00:00	2024-10-18T15:56:21.396951+00:00	0.25	0.024660	0.004993	0.000823	0.005277	0.007863	0.005704	0.000000	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Ok	None	None	None	None	None	None	None	None
4	2024-10-18T15:56:21.396951+00:00	2024-10-18T16:16:51.372569+00:00	182	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	2.362387e+05	1489624.250	5.120811e+05	4.486278e+05	218875.510587	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.219	0.206	0.180	0.250	0.145	0.0	Density	2024-10-18T15:56:21.396951+00:00	2024-10-18T15:57:21.396951+00:00	0.25	0.026467	0.001374	0.000172	0.000197	0.009354	0.015370	0.000000	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Ok	None	None	None	None	None	None	None	None
5	2024-10-18T15:57:21.396951+00:00	2024-10-18T16:16:51.372633+00:00	200	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	2.362387e+05	1489624.250	5.246340e+05	4.500488e+05	233440.321856	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.209	0.216	0.162	0.241	0.172	0.0	Density	2024-10-18T15:57:21.396951+00:00	2024-10-18T15:58:21.396951+00:00	0.25	0.012931	0.000238	0.000075	0.003316	0.006167	0.003135	0.000000	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Ok	None	None	None	None	None	None	None	None
6	2024-10-18T15:58:21.396951+00:00	2024-10-18T16:16:51.372689+00:00	223	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	2.362387e+05	1489624.250	5.024677e+05	4.486278e+05	214154.238483	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.228	0.238	0.161	0.223	0.150	0.0	Density	2024-10-18T15:58:21.396951+00:00	2024-10-18T15:59:21.396951+00:00	0.25	0.023760	0.003148	0.003008	0.003609	0.001692	0.012304	0.000000	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Ok	None	None	None	None	None	None	None	None
7	2024-10-18T15:59:21.396951+00:00	2024-10-18T16:16:51.372769+00:00	204	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	2.362387e+05	1489624.250	5.240553e+05	4.508677e+05	243061.741658	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.227	0.205	0.160	0.236	0.172	0.0	Density	2024-10-18T15:59:21.396951+00:00	2024-10-18T16:00:21.396951+00:00	0.25	0.014865	0.002917	0.000235	0.003915	0.004663	0.003135	0.000000	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Ok	None	None	None	None	None	None	None	None
8	2024-10-18T16:00:21.396951+00:00	2024-10-18T16:16:51.372832+00:00	209	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	2.362387e+05	1489624.250	5.184981e+05	4.508677e+05	228599.592756	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.228	0.207	0.140	0.273	0.152	0.0	Density	2024-10-18T16:00:21.396951+00:00	2024-10-18T16:01:21.396951+00:00	0.25	0.047626	0.003148	0.000119	0.013069	0.020103	0.011186	0.000000	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Ok	None	None	None	None	None	None	None	None
9	2024-10-18T16:01:21.396951+00:00	2024-10-18T16:16:51.372884+00:00	213	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	2.362387e+05	1489624.250	5.219783e+05	4.508677e+05	234342.656919	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.218	0.205	0.169	0.242	0.166	0.0	Density	2024-10-18T16:01:21.396951+00:00	2024-10-18T16:02:21.396951+00:00	0.25	0.014559	0.001220	0.000235	0.001629	0.006491	0.004984	0.000000	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Ok	None	None	None	None	None	None	None	None
10	2024-10-18T16:04:21.396951+00:00	2024-10-18T16:16:51.373043+00:00	217	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	1.514079e+06	2016006.125	1.878114e+06	1.946438e+06	165762.752603	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.000	0.000	0.000	0.000	0.000	1.0	Density	2024-10-18T16:04:21.396951+00:00	2024-10-18T16:05:21.396951+00:00	0.25	8.869452	0.728669	0.775660	0.654552	0.738028	0.700717	5.271826	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Alert	None	None	None	None	None	None	None	None
11	2024-10-18T16:05:21.396951+00:00	2024-10-18T16:16:51.373090+00:00	227	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	1.514079e+06	2016006.125	1.869432e+06	1.946438e+06	169654.651904	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.000	0.000	0.000	0.000	0.000	1.0	Density	2024-10-18T16:05:21.396951+00:00	2024-10-18T16:06:21.396951+00:00	0.25	8.869452	0.728669	0.775660	0.654552	0.738028	0.700717	5.271826	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Alert	None	None	None	None	None	None	None	None
12	2024-10-18T16:06:21.396951+00:00	2024-10-18T16:16:51.373140+00:00	237	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	1.514079e+06	2016006.125	1.881715e+06	1.946438e+06	162160.610713	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.000	0.000	0.000	0.000	0.000	1.0	Density	2024-10-18T16:06:21.396951+00:00	2024-10-18T16:07:21.396951+00:00	0.25	8.869452	0.728669	0.775660	0.654552	0.738028	0.700717	5.271826	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Alert	None	None	None	None	None	None	None	None
13	2024-10-18T16:07:21.396951+00:00	2024-10-18T16:16:51.373182+00:00	232	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	1.514079e+06	2016006.125	1.883215e+06	1.946438e+06	160483.084939	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.000	0.000	0.000	0.000	0.000	1.0	Density	2024-10-18T16:07:21.396951+00:00	2024-10-18T16:08:21.396951+00:00	0.25	8.869452	0.728669	0.775660	0.654552	0.738028	0.700717	5.271826	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Alert	None	None	None	None	None	None	None	None
14	2024-10-18T16:08:21.396951+00:00	2024-10-18T16:16:51.373237+00:00	242	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	1.514079e+06	2016006.125	1.881695e+06	1.946438e+06	159266.187983	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.000	0.000	0.000	0.000	0.000	1.0	Density	2024-10-18T16:08:21.396951+00:00	2024-10-18T16:09:21.396951+00:00	0.25	8.869452	0.728669	0.775660	0.654552	0.738028	0.700717	5.271826	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Alert	None	None	None	None	None	None	None	None
15	2024-10-18T16:09:21.396951+00:00	2024-10-18T16:16:51.373288+00:00	251	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	1.514079e+06	2016006.125	1.884403e+06	1.946438e+06	156081.344812	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.000	0.000	0.000	0.000	0.000	1.0	Density	2024-10-18T16:09:21.396951+00:00	2024-10-18T16:10:21.396951+00:00	0.25	8.869452	0.728669	0.775660	0.654552	0.738028	0.700717	5.271826	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Alert	None	None	None	None	None	None	None	None
16	2024-10-18T16:10:21.396951+00:00	2024-10-18T16:16:51.373330+00:00	163	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	1.514079e+06	2016006.125	1.877560e+06	1.946438e+06	160930.722136	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.000	0.000	0.000	0.000	0.000	1.0	Density	2024-10-18T16:10:21.396951+00:00	2024-10-18T16:11:21.396951+00:00	0.25	8.869452	0.728669	0.775660	0.654552	0.738028	0.700717	5.271826	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Alert	None	None	None	None	None	None	None	None
17	2024-10-18T16:11:21.396951+00:00	2024-10-18T16:16:51.373376+00:00	256	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	1.514079e+06	2016006.125	1.879128e+06	1.946438e+06	162368.666688	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.000	0.000	0.000	0.000	0.000	1.0	Density	2024-10-18T16:11:21.396951+00:00	2024-10-18T16:12:21.396951+00:00	0.25	8.869452	0.728669	0.775660	0.654552	0.738028	0.700717	5.271826	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Alert	None	None	None	None	None	None	None	None
18	2024-10-18T16:12:21.396951+00:00	2024-10-18T16:16:51.373421+00:00	261	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	1.514079e+06	2016006.125	1.887236e+06	1.946438e+06	153937.005271	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.000	0.000	0.000	0.000	0.000	1.0	Density	2024-10-18T16:12:21.396951+00:00	2024-10-18T16:13:21.396951+00:00	0.25	8.869452	0.728669	0.775660	0.654552	0.738028	0.700717	5.271826	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Alert	None	None	None	None	None	None	None	None
19	2024-10-18T16:13:21.396951+00:00	2024-10-18T16:16:51.373471+00:00	246	500	236238.671875	1489624.25	539895.63075	450283.78125	252708.493978	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	q_20	q_40	q_60	q_80	q_100	right_outlier	0.202	0.212	0.186	0.204	0.196	Density	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:51:21.377181+00:00	1000	1.514079e+06	2016006.125	1.880186e+06	1.946438e+06	161706.985040	236238.671875	338357.875	437177.96875	523576.1875	718445.6875	1489624.25	None	left_outlier	e_3.38e5	e_4.37e5	e_5.24e5	e_7.18e5	e_1.49e6	right_outlier	0.000	0.000	0.000	0.000	0.000	1.0	Density	2024-10-18T16:13:21.396951+00:00	2024-10-18T16:14:21.396951+00:00	0.25	8.869452	0.728669	0.775660	0.654552	0.738028	0.700717	5.271826	Quantile	Density	PSI	5	UnivariateContinuous	None	None	Alert	None	None	None	None	None	None	None	None

Analysis Compare Basic Stats

The method wallaroo.assay.AssayAnalysis.compare_basic_stats returns a DataFrame comparing one set of inference data against the baseline.

This is compared to the Analysis List DataFrame, which is a list of all of the inference data split into intervals, while the Analysis Compare Basic Stats shows the breakdown of one set of inference data against the baseline.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display one analysis against the baseline
assay_results[0].compare_basic_stats()

	Baseline	Window	diff	pct_diff
count	500.0	1000.0	500.000000	100.000000
min	236238.671875	236238.671875	0.000000	0.000000
max	1489624.25	1489624.25	0.000000	0.000000
mean	539895.63075	520529.742266	-19365.888484	-3.586969
median	450283.78125	450867.6875	583.906250	0.129675
std	252708.493978	226843.358512	-25865.135466	-10.235167
start	2024-10-18T15:50:21.296181+00:00	2024-10-18T15:52:21.396951+00:00	NaN	NaN
end	2024-10-18T15:51:21.377181+00:00	2024-10-18T15:53:21.396951+00:00	NaN	NaN

Configure Assays

Before creating the assay, configure the assay and continue to preview it until the best method for detecting drift is set. The following options are available.

Inference Interval and Inference Width

The inference interval aka window interval sets how often to run the assay analysis. This is set from the wallaroo.assay_config.AssayBuilder.window_builder.add_interval method to collect data expressed in time units: “hours=24”, “minutes=1”, etc.

For example, with an interval of 1 minute, the assay collects data every minute. Within an hour, 60 intervals of data is collected.

We can adjust the interval and see how the assays change based on how frequently they are run.

The width sets the time period from the wallaroo.assay_config.AssayBuilder.window_builder.add_width method to collect data expressed in time units: “hours=24”, “minutes=1”, etc.

For example, an interval of 1 minute and a width of 1 minute collects 1 minutes worth of data every minute. An interval of 1 minute with a width of 5 minutes collects 5 minute of inference data every minute.

By default, the interval and width is 24 hours.

For this example, we’ll adjust the width and interval from 1 minute to 5 minutes and see how the number of analyses and their score changes.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show the analyses chart
assay_results.chart_scores()

wl.get_current_workspace()

{'name': 'assay-demonstration-tutorial-2', 'id': 11, 'archived': False, 'created_by': '3eeab554-fc2e-41ac-8b4e-0d778e8a471b', 'created_at': '2024-10-09T19:50:23.231562+00:00', 'models': [{'name': 'house-price-estimator', 'versions': 3, 'owner_id': '""', 'last_update_time': datetime.datetime(2024, 10, 18, 15, 50, 20, 532959, tzinfo=tzutc()), 'created_at': datetime.datetime(2024, 10, 9, 19, 50, 24, 679975, tzinfo=tzutc())}], 'pipelines': [{'name': 'assay-demonstration-tutorial', 'create_time': datetime.datetime(2024, 10, 9, 19, 50, 25, 224073, tzinfo=tzutc()), 'definition': '[]'}]}

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to five minutes each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show the analyses chart
assay_results.chart_scores()

Add Run Until and Add Inference Start

For previewing assays, setting wallaroo.assay_config.AssayBuilder.add_run_until sets the end date and time for collecting inference data. When an assay is uploaded, this setting is no longer valid - assays run at the Inference Interval until the assay is paused.

Setting the wallaroo.assay_config.WindowBuilder.add_start sets the start date and time to collect inference data. When an assay is uploaded, this setting is included, and assay results will be displayed starting from that start date at the Inference Interval until the assay is paused. By default, add_start begins 24 hours after the assay is uploaded unless set in the assay configuration manually.

For the following example, the add_run_until setting is set to datetime.datetime.now() to collect all inference data from assay_window_start up until now, and the second example limits that example to only two minutes of data.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results, minus 2 minutes for the period to start gathering analyses
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start+datetime.timedelta(seconds=-120))

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show the analyses chart
assay_results.chart_scores()

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results plus 2 minutes 
assay_baseline_from_dates.add_run_until(assay_window_start+datetime.timedelta(seconds=120))

# Set the interval and window to one minute each
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show the analyses chart
assay_results.chart_scores()

Score Metric

The score is a distance between the baseline and the analysis window. The larger the score, the greater the difference between the baseline and the analysis window. The following methods are provided determining the score:

PSI (Default) - Population Stability Index (PSI).
MAXDIFF: Maximum difference between corresponding bins.
SUMDIFF: Mum of differences between corresponding bins.

The metric type used is updated with the wallaroo.assay_config.AssayBuilder.add_metric(metric: wallaroo.assay_config.Metric) method.

The following three charts use each of the metrics. Note how the scores change based on the score type used.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# set metric PSI mode
assay_baseline.summarizer_builder.add_metric(wallaroo.assay_config.Metric.PSI)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display one analysis from the results
assay_results[0].chart()

baseline mean = 539895.63075
window mean = 520529.742265625
baseline median = 450283.78125
window median = 450867.6875
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.027998899384779063
scores = [0.0, 0.00031071866653010913, 0.00030773024662236867, 0.002769365409558061, 0.015958199760662362, 0.008652885301406161, 0.0]
index = None

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# set metric MAXDIFF mode
assay_baseline.summarizer_builder.add_metric(wallaroo.assay_config.Metric.MAXDIFF)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display one analysis from the results
assay_results[0].chart()

baseline mean = 539895.63075
window mean = 520529.742265625
baseline median = 450283.78125
window median = 450867.6875
bin_mode = Quantile
aggregation = Density
metric = MaxDiff
weighted = False
score = 0.061000000000000026
scores = [0.0, 0.00799999999999998, 0.008000000000000007, 0.021999999999999992, 0.061000000000000026, 0.03900000000000001, 0.0]
index = 4

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# set metric SUMDIFF mode
assay_baseline.summarizer_builder.add_metric(wallaroo.assay_config.Metric.SUMDIFF)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display one analysis from the results
assay_results[0].chart()

baseline mean = 539895.63075
window mean = 520529.742265625
baseline median = 450283.78125
window median = 450867.6875
bin_mode = Quantile
aggregation = Density
metric = SumDiff
weighted = False
score = 0.069
scores = [0.0, 0.00799999999999998, 0.008000000000000007, 0.021999999999999992, 0.061000000000000026, 0.03900000000000001, 0.0]
index = None

Alert Threshold

Assay alert thresholds are modified with the wallaroo.assay_config.AssayBuilder.add_alert_threshold(alert_threshold: float) method. By default alert thresholds are 0.1.

The following example updates the alert threshold to 0.5.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# set the alert threshold
assay_baseline.add_alert_threshold(0.5)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show the analyses with the alert threshold
assay_results.to_dataframe()

	id	assay_id	assay_name	pipeline_id	workspace_id	workspace_name	score	start	min	max	mean	median	std	warning_threshold	alert_threshold	status
0	None	None	assays from date baseline	None	None	None	0.027999	2024-10-18T15:52:21.396951+00:00	2.362387e+05	1489624.250	5.205297e+05	4.508677e+05	226843.358512	None	0.5	Ok
1	None	None	assays from date baseline	None	None	None	0.016821	2024-10-18T15:53:21.396951+00:00	2.362387e+05	1489624.250	5.231229e+05	4.508677e+05	226845.586466	None	0.5	Ok
2	None	None	assays from date baseline	None	None	None	0.010160	2024-10-18T15:54:21.396951+00:00	2.362387e+05	1489624.250	5.183006e+05	4.486278e+05	227582.307451	None	0.5	Ok
3	None	None	assays from date baseline	None	None	None	0.024660	2024-10-18T15:55:21.396951+00:00	2.362387e+05	1489624.250	5.178753e+05	4.486278e+05	228060.575593	None	0.5	Ok
4	None	None	assays from date baseline	None	None	None	0.026467	2024-10-18T15:56:21.396951+00:00	2.362387e+05	1489624.250	5.120811e+05	4.486278e+05	218875.510587	None	0.5	Ok
5	None	None	assays from date baseline	None	None	None	0.012931	2024-10-18T15:57:21.396951+00:00	2.362387e+05	1489624.250	5.246340e+05	4.500488e+05	233440.321856	None	0.5	Ok
6	None	None	assays from date baseline	None	None	None	0.023760	2024-10-18T15:58:21.396951+00:00	2.362387e+05	1489624.250	5.024677e+05	4.486278e+05	214154.238483	None	0.5	Ok
7	None	None	assays from date baseline	None	None	None	0.014865	2024-10-18T15:59:21.396951+00:00	2.362387e+05	1489624.250	5.240553e+05	4.508677e+05	243061.741658	None	0.5	Ok
8	None	None	assays from date baseline	None	None	None	0.047626	2024-10-18T16:00:21.396951+00:00	2.362387e+05	1489624.250	5.184981e+05	4.508677e+05	228599.592756	None	0.5	Ok
9	None	None	assays from date baseline	None	None	None	0.014559	2024-10-18T16:01:21.396951+00:00	2.362387e+05	1489624.250	5.219783e+05	4.508677e+05	234342.656919	None	0.5	Ok
10	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:04:21.396951+00:00	1.514079e+06	2016006.125	1.878114e+06	1.946438e+06	165762.752603	None	0.5	Alert
11	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:05:21.396951+00:00	1.514079e+06	2016006.125	1.869432e+06	1.946438e+06	169654.651904	None	0.5	Alert
12	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:06:21.396951+00:00	1.514079e+06	2016006.125	1.881715e+06	1.946438e+06	162160.610713	None	0.5	Alert
13	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:07:21.396951+00:00	1.514079e+06	2016006.125	1.883215e+06	1.946438e+06	160483.084939	None	0.5	Alert
14	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:08:21.396951+00:00	1.514079e+06	2016006.125	1.881695e+06	1.946438e+06	159266.187983	None	0.5	Alert
15	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:09:21.396951+00:00	1.514079e+06	2016006.125	1.884403e+06	1.946438e+06	156081.344812	None	0.5	Alert
16	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:10:21.396951+00:00	1.514079e+06	2016006.125	1.877560e+06	1.946438e+06	160930.722136	None	0.5	Alert
17	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:11:21.396951+00:00	1.514079e+06	2016006.125	1.879128e+06	1.946438e+06	162368.666688	None	0.5	Alert
18	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:12:21.396951+00:00	1.514079e+06	2016006.125	1.887236e+06	1.946438e+06	153937.005271	None	0.5	Alert
19	None	None	assays from date baseline	None	None	None	8.869452	2024-10-18T16:13:21.396951+00:00	1.514079e+06	2016006.125	1.880186e+06	1.946438e+06	161706.985040	None	0.5	Alert

Number of Bins

Number of bins sets how the baseline data is partitioned. The total number of bins includes the set number plus the left_outlier and the right_outlier, so the total number of bins will be the total set + 2.

The number of bins is set with the wallaroo.assay_config.UnivariateContinousSummarizerBuilder.add_num_bins(num_bins: int) method.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# update number of bins here
assay_baseline.summarizer_builder.add_num_bins(10)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()

baseline mean = 539895.63075
window mean = 520529.742265625
baseline median = 450283.78125
window median = 450867.6875
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.05908866707776886
scores = [0.0, 0.005162673110806691, 0.0027300082724922136, 0.001619315394342909, 0.0042755780600660315, 0.0015054510098891697, 0.0012818719788154692, 0.006791725840796723, 0.009249964850087234, 0.0002488075477953185, 0.026223271012677105, 0.0]
index = None

Binning Mode

Binning Mode defines how the bins are separated. Binning modes are modified through the wallaroo.assay_config.UnivariateContinousSummarizerBuilder.add_bin_mode(bin_mode: bin_mode: wallaroo.assay_config.BinMode, edges: Optional[List[float]] = None).

Available bin_mode values from wallaroo.assay_config.Binmode are the following:

QUANTILE (Default): Based on percentages. If num_bins is 5 then quintiles so bins are created at the 20%, 40%, 60%, 80% and 100% points.
EQUAL: Evenly spaced bins where each bin is set with the formula min - max / num_bins
PROVIDED: The user provides the edge points for the bins.

If PROVIDED is supplied, then a List of float values must be provided for the edges parameter that matches the number of bins.

The following examples are used to show how each of the binning modes effects the bins.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# update binning mode here
assay_baseline.summarizer_builder.add_bin_mode(wallaroo.assay_config.BinMode.QUANTILE)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()

baseline mean = 539895.63075
window mean = 520529.742265625
baseline median = 450283.78125
window median = 450867.6875
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.027998899384779063
scores = [0.0, 0.00031071866653010913, 0.00030773024662236867, 0.002769365409558061, 0.015958199760662362, 0.008652885301406161, 0.0]
index = None

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# update binning mode here
assay_baseline.summarizer_builder.add_bin_mode(wallaroo.assay_config.BinMode.EQUAL)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()

baseline mean = 539895.63075
window mean = 520529.742265625
baseline median = 450283.78125
window median = 450867.6875
bin_mode = Equal
aggregation = Density
metric = PSI
weighted = False
score = 0.024337608548999315
scores = [0.0, 0.0004539311540674428, 0.0102623545559447, 0.0015340004581186216, 0.00863998698518399, 0.003447335395684558, 0.0]
index = None

The following example manually sets the bin values.

The values in this dataset run from 200000 to 1500000. We can specify the bins with the BinMode.PROVIDED and specifying a list of floats with the right hand / upper edge of each bin and optionally the lower edge of the smallest bin. If the lowest edge is not specified the threshold for left outliers is taken from the smallest value in the baseline dataset.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

edges = [200000.0, 400000.0, 600000.0, 800000.0, 1500000.0, 2000000.0]

# update binning mode here
assay_baseline.summarizer_builder.add_bin_mode(wallaroo.assay_config.BinMode.PROVIDED, edges)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()

baseline mean = 539895.63075
window mean = 520529.742265625
baseline median = 450283.78125
window median = 450867.6875
bin_mode = Provided
aggregation = Density
metric = PSI
weighted = False
score = 0.022475023501744996
scores = [0.0, 0.0012824905633907766, 0.0004230661205221289, 0.007031010052631133, 0.013738456765200958, 0.0, 0.0]
index = None

Aggregation Options

Assay aggregation options are modified with the wallaroo.assay_config.AssayBuilder.add_aggregation(aggregation: wallaroo.assay_config.Aggregation) method. The following options are provided:

Aggregation.DENSITY (Default): Count the number/percentage of values that fall in each bin.
Aggregation.CUMULATIVE: Empirical Cumulative Density Function style, which keeps a cumulative count of the values/percentages that fall in each bin.

The following example demonstrate the different results between the two.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

#Aggregation.DENSITY - the default
assay_baseline.summarizer_builder.add_aggregation(wallaroo.assay_config.Aggregation.DENSITY)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()

baseline mean = 539895.63075
window mean = 520529.742265625
baseline median = 450283.78125
window median = 450867.6875
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.027998899384779063
scores = [0.0, 0.00031071866653010913, 0.00030773024662236867, 0.002769365409558061, 0.015958199760662362, 0.008652885301406161, 0.0]
index = None

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

#Aggregation.CUMULATIVE
assay_baseline.summarizer_builder.add_aggregation(wallaroo.assay_config.Aggregation.CUMULATIVE)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()

baseline mean = 539895.63075
window mean = 520529.742265625
baseline median = 450283.78125
window median = 450867.6875
bin_mode = Quantile
aggregation = Cumulative
metric = PSI
weighted = False
score = 0.06899999999999992
scores = [0.0, 0.00799999999999998, 0.0, 0.02200000000000002, 0.03899999999999992, 0.0, 0.0]
index = None

Create Assay

With the assay previewed and configuration options determined, we officially create it by uploading it to the Wallaroo instance.

Once it is uploaded, the assay runs an analysis based on the window width, interval, and the other settings configured.

Assays are uploaded with the wallaroo.assay_config.upload() method. This uploads the assay into the Wallaroo database with the configurations applied and returns the assay id. Note that assay names must be unique across the Wallaroo instance; attempting to upload an assay with the same name as an existing one will return an error.

wallaroo.assay_config.upload() returns the assay id for the assay.

Typically we would just call wallaroo.assay_config.upload() after configuring the assay. For the example below, we will perform the complete configuration in one window to show all of the configuration steps at once before creating the assay.

# Build the assay, based on the start and end of our baseline time, 
# and tracking the output variable index 0
assay_baseline = wl.build_assay(assay_name="assays from date baseline tutorial samples", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# set the width, interval, and assay start date and time
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# add other options
assay_baseline.summarizer_builder.add_aggregation(wallaroo.assay_config.Aggregation.CUMULATIVE)
assay_baseline.summarizer_builder.add_metric(wallaroo.assay_config.Metric.MAXDIFF)
assay_baseline.add_alert_threshold(0.5)

assay_id = assay_baseline.upload()

# wait 65 seconds for the first analysis run performed
time.sleep(65)

The assay is now visible through the Wallaroo UI by selecting the workspace, then the pipeline, then Insights.

Get Assay Info

Assay information is retrieved with the wallaroo.client.get_assay_info() which takes the following parameters.

Parameter	Type	Description
assay_id	Integer (Required)	The numerical id of the assay.
workspace_id	(Int) (Optional)	The numerical identifier of the workspace to filter by.
workspace_name	(String) (Optional)	The name of the workspace to filter by.

This returns the following:

Parameter	Type	Description
id	Integer	The numerical id of the assay.
name	String	The name of the assay.
active	Boolean	`True`: The assay is active and generates analyses based on its configuration. `False`: The assay is disabled and will not generate new analyses.
pipeline_name	String	The name of the pipeline the assay references.
last_run	DateTime	The date and time the assay last ran.
next_run	DateTime	THe date and time the assay analysis will next run.
alert_threshold	Float	The alert threshold setting for the assay.
baseline	Dict	The baseline and settings as set from the assay configuration.
iopath	String	The `iopath` setting for the assay.
metric	String	The `metric` setting for the assay.
num_bins	Integer	The number of bins for the assay.
bin_weights	List/None	The bin weights used if any.
bin_mode	String	The binning mode used.
workspace_id	(Int)	The numerical identifier of the workspace the assay is associated with.
workspace_name	(String)	The name of the workspace the assay is associated with.

display(wl.get_assay_info(assay_id))

	id	name	active	status	pipeline_name	last_run	next_run	alert_threshold	workspace_id	workspace_name	baseline	iopath	metric	num_bins	bin_weights	bin_mode
0	5	assays from date baseline tutorial samples	True	{"run_at": "2024-10-18T16:36:35.344528612+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	assay-demonstration-tutorial	2024-10-18T16:36:35.344529+00:00	2024-10-18T16:36:21.396951+00:00	0.5	11	assay-demonstration-tutorial-2	Start:2024-10-18T15:50:21.296181+00:00, End:2024-10-18T15:51:21.377181+00:00	output variable 0	MaxDiff	5	None	Quantile

Get Assay Results

Once an assay is created the assay runs an analysis based on the window width, interval, and the other settings configured.

Assay results are retrieved with the wallaroo.client.get_assay_results method which takes the following parameters:

Parameter	Type	Description
assay_id	Integer (Required)	The numerical id of the assay.
start	Datetime.Datetime (Required)	The start date and time of historical data from the pipeline to start analyses from.
end	Datetime.Datetime (Required)	The end date and time of historical data from the pipeline to limit analyses to.
workspace_id	(Int) (Optional)	The numerical identifier of the workspace to filter by.
workspace_name	(String) (Optional)	The name of the workspace to filter by.

IMPORTANT NOTE: This process requires that additional historical data is generated from the time the assay is created to when the results are available. To add additional inference data, use the Assay Test Data section above.

assay_results = wl.get_assay_results(assay_id=assay_id,
                     start=(datetime.datetime.now()-datetime.timedelta(days=2)),
                     end=datetime.datetime.now())

# Preview the assay analyses
assay_results.chart_scores()

assay_results[0].chart()

baseline mean = 519725.40603168664
window mean = 1116046.53125
baseline median = 448627.8125
window median = 1116046.53125
bin_mode = Quantile
aggregation = Cumulative
metric = MaxDiff
weighted = False
score = 0.8043912
scores = [0.0, 0.20159680638722555, 0.42115768463073855, 0.6027944111776448, 0.8043912175648703, 0.0, 0.0]
index = 4

List Assays

A list of assays is retrieved with the wallaroo.client.list_assays() method and takes the following parameters:

Parameter	Type	Description
workspace_id	(Int) (Optional)	The numerical identifier of the workspace to filter by.
workspace_name	(String) (Optional)	The name of the workspace to filter by.

This returns a list of assays as filtered in reverse chronological order.

Parameter	Type	Description
Assay Id	Integer	The numerical id of the assay.
Assay Name	String	The name of the assay.
Active	Boolean	`True`: The assay is active and generates analyses based on its configuration. `False`: The assay is disabled and will not generate new analyses.
Status	Dict	The status of the assay including the
Warning Threshold	Float/None	The warning threshold if set.
Alert Threshold	Float	The alert threshold for the assay.
workspace_id	(Int)	The numerical identifier of the workspace the assay is associated with.
workspace_name	(String)	The name of the workspace the assay is associated with.

The errors for this method include:

If the parameter workspace_id is not an integer.
If the parameter workspace_name is not a String.

wl.list_assays()

Assay ID	Assay Name	Active	Status	Warning Threshold	Alert Threshold	Pipeline ID	Pipeline Name	Workspace ID	Workspace Name
5	assays from date baseline tutorial samples	True	{"run_at": "2024-10-18T16:36:35.344528612+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	None	0.5	7	assay-demonstration-tutorial	11	assay-demonstration-tutorial-2
3	assays from date baseline tutorial sample	True	{"run_at": "2024-10-17T15:21:19.691968096+00:00", "num_ok": 10, "num_warnings": 0, "num_alerts": 10}	None	0.5	16	assay-demonstration	14	assay-demonstration-tutorial
1	assays from date baseline tutorial 2	True	{"run_at": "2024-10-18T16:36:11.506948857+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	None	0.5	7	assay-demonstration-tutorial	11	assay-demonstration-tutorial-2

Set Assay Active Status

Assays active status is either:

True: The assay generates analyses based on the assay configuration.
False: The assay will not generate new analyses.

Assays are set to active or not active with the wallaroo.client.set_assay_active which takes the following parameters.

Parameter	Type	Description
assay_id	Integer	The numerical id of the assay.
active	Boolean	`True`: The assay status is set to `Active`. `False`: The assay status is `Not Active`.

First we will show the current active status.

In the following, set the assay status to False, then set the assay active status back to True.

display(wl.get_assay_info(assay_id))

	id	name	active	status	pipeline_name	last_run	next_run	alert_threshold	workspace_id	workspace_name	baseline	iopath	metric	num_bins	bin_weights	bin_mode
0	5	assays from date baseline tutorial samples	True	{"run_at": "2024-10-18T16:36:35.344528612+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	assay-demonstration-tutorial	2024-10-18T16:36:35.344529+00:00	2024-10-18T16:36:21.396951+00:00	0.5	11	assay-demonstration-tutorial-2	Start:2024-10-18T15:50:21.296181+00:00, End:2024-10-18T15:51:21.377181+00:00	output variable 0	MaxDiff	5	None	Quantile

Now we set the active status to False, and show the assay list to verify it is no longer active.

wl.set_assay_active(assay_id, False)
display(wl.get_assay_info(assay_id))

	id	name	active	status	pipeline_name	last_run	next_run	alert_threshold	workspace_id	workspace_name	baseline	iopath	metric	num_bins	bin_weights	bin_mode
0	5	assays from date baseline tutorial samples	False	{"run_at": "2024-10-18T16:36:35.344528612+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	assay-demonstration-tutorial	2024-10-18T16:36:35.344529+00:00	2024-10-18T16:36:21.396951+00:00	0.5	11	assay-demonstration-tutorial-2	Start:2024-10-18T15:50:21.296181+00:00, End:2024-10-18T15:51:21.377181+00:00	output variable 0	MaxDiff	5	None	Quantile

We resume the assay by setting it’s active status to True.

wl.set_assay_active(assay_id, True)
display(wl.get_assay_info(assay_id))

	id	name	active	status	pipeline_name	last_run	next_run	alert_threshold	workspace_id	workspace_name	baseline	iopath	metric	num_bins	bin_weights	bin_mode
0	5	assays from date baseline tutorial samples	True	{"run_at": "2024-10-18T16:36:35.344528612+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	assay-demonstration-tutorial	2024-10-18T16:36:35.344529+00:00	2024-10-18T16:36:21.396951+00:00	0.5	11	assay-demonstration-tutorial-2	Start:2024-10-18T15:50:21.296181+00:00, End:2024-10-18T15:51:21.377181+00:00	output variable 0	MaxDiff	5	None	Quantile

Undeploy Main Pipeline

With the examples and tutorial complete, we will undeploy the main pipeline and return the resources back to the Wallaroo instance.

mainpipeline.undeploy()

Waiting for undeployment - this will take up to 45s .................................... ok

name	assay-demonstration-tutorial
created	2024-10-09 19:50:25.224073+00:00
last_updated	2024-10-18 15:50:20.759830+00:00
deployed	False
workspace_id	11
workspace_name	assay-demonstration-tutorial-2
arch	x86
accel	none
tags
versions	ba6b344c-83a0-4308-bc62-f198f8af8a5d, 0740ad84-a770-4cf9-b1ce-1f45dbe9c1c4, a2494de7-4d3a-428f-a241-9ddff1ae6772, 6ff08bf3-9be8-484c-b35f-cfde42854ad7, 54380aa2-8722-4b80-b511-b494c6f42c4b, b7a878c6-4f62-4e57-843c-23604c38704b
steps	house-price-estimator
published	False

View Assay Filters

The following examples demonstrate various ways of listing assays and retrieving assay details, with filtering options.

List Assays with Filters

List all assays for workspaces the user is a member of.

wl.list_assays()

Assay ID	Assay Name	Active	Status	Warning Threshold	Alert Threshold	Pipeline ID	Pipeline Name	Workspace ID	Workspace Name
5	assays from date baseline tutorial samples	True	{"run_at": "2024-10-18T16:36:35.344528612+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	None	0.5	7	assay-demonstration-tutorial	11	assay-demonstration-tutorial-2
3	assays from date baseline tutorial sample	True	{"run_at": "2024-10-17T15:21:19.691968096+00:00", "num_ok": 10, "num_warnings": 0, "num_alerts": 10}	None	0.5	16	assay-demonstration	14	assay-demonstration-tutorial
1	assays from date baseline tutorial 2	True	{"run_at": "2024-10-18T16:36:11.506948857+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	None	0.5	7	assay-demonstration-tutorial	11	assay-demonstration-tutorial-2

List all assays filtered by workspace id.

wl.list_assays(workspace_id=workspace_id)

Assay ID	Assay Name	Active	Status	Warning Threshold	Alert Threshold	Pipeline ID	Pipeline Name	Workspace ID	Workspace Name
5	assays from date baseline tutorial samples	True	{"run_at": "2024-10-18T16:36:35.344528612+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	None	0.5	7	assay-demonstration-tutorial	11	assay-demonstration-tutorial-2
1	assays from date baseline tutorial 2	True	{"run_at": "2024-10-18T16:36:11.506948857+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	None	0.5	7	assay-demonstration-tutorial	11	assay-demonstration-tutorial-2

List assays filtered by workspace name.

wl.list_assays(workspace_name=workspace_name)

Assay ID	Assay Name	Active	Status	Warning Threshold	Alert Threshold	Pipeline ID	Pipeline Name	Workspace ID	Workspace Name
5	assays from date baseline tutorial samples	True	{"run_at": "2024-10-18T16:36:35.344528612+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	None	0.5	7	assay-demonstration-tutorial	11	assay-demonstration-tutorial-2
1	assays from date baseline tutorial 2	True	{"run_at": "2024-10-18T16:36:11.506948857+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	None	0.5	7	assay-demonstration-tutorial	11	assay-demonstration-tutorial-2

Get Assay Results by Filter

Get assay results.

assay_results = wl.get_assay_results(assay_id=assay_id,
                     start=assay_window_start,
                     end=datetime.datetime.now())

assay_results.chart_scores()

Get assay results filtered by workspace id.

assay_results = wl.get_assay_results(assay_id=assay_id,
                                     workspace_id=workspace_id,
                     start=assay_window_start,
                     end=datetime.datetime.now())

assay_results.chart_scores()

Get assay results filtered by workspace name.

assay_results = wl.get_assay_results(assay_id=assay_id,
                                     workspace_name=workspace_name,
                     start=assay_window_start,
                     end=datetime.datetime.now())

assay_results.chart_scores()

Get Assay Info with Filter

Get assay info.

wl.get_assay_info(assay_id=assay_id)

	id	name	active	status	pipeline_name	last_run	next_run	alert_threshold	workspace_id	workspace_name	baseline	iopath	metric	num_bins	bin_weights	bin_mode
0	5	assays from date baseline tutorial samples	True	{"run_at": "2024-10-18T16:36:35.344528612+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	assay-demonstration-tutorial	2024-10-18T16:36:35.344529+00:00	2024-10-18T16:36:21.396951+00:00	0.5	11	assay-demonstration-tutorial-2	Start:2024-10-18T15:50:21.296181+00:00, End:2024-10-18T15:51:21.377181+00:00	output variable 0	MaxDiff	5	None	Quantile

Get assay info filtered by workspace id.

wl.get_assay_info(assay_id=assay_id, 
                 workspace_id=workspace_id)

	id	name	active	status	pipeline_name	last_run	next_run	alert_threshold	workspace_id	workspace_name	baseline	iopath	metric	num_bins	bin_weights	bin_mode
0	5	assays from date baseline tutorial samples	True	{"run_at": "2024-10-18T16:36:35.344528612+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	assay-demonstration-tutorial	2024-10-18T16:36:35.344529+00:00	2024-10-18T16:36:21.396951+00:00	0.5	11	assay-demonstration-tutorial-2	Start:2024-10-18T15:50:21.296181+00:00, End:2024-10-18T15:51:21.377181+00:00	output variable 0	MaxDiff	5	None	Quantile

Get assay info filtered by workspace name.

wl.get_assay_info(assay_id=assay_id, 
                 workspace_name=workspace_name)

	id	name	active	status	pipeline_name	last_run	next_run	alert_threshold	workspace_id	workspace_name	baseline	iopath	metric	num_bins	bin_weights	bin_mode
0	5	assays from date baseline tutorial samples	True	{"run_at": "2024-10-18T16:36:35.344528612+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}	assay-demonstration-tutorial	2024-10-18T16:36:35.344529+00:00	2024-10-18T16:36:21.396951+00:00	0.5	11	assay-demonstration-tutorial-2	Start:2024-10-18T15:50:21.296181+00:00, End:2024-10-18T15:51:21.377181+00:00	output variable 0	MaxDiff	5	None	Quantile