Wallaroo SDK Essentials Guide: Assays Management

How to create and manage Wallaroo Assays through the Wallaroo SDK

Model Drift Detection with Assays

Wallaroo provides the ability to monitor the input and outputs of models to detect model drift by using model insights. Changes in the inputs or model predictions that fall outside of expected norms, known as data drift, can occur due to errors in the data processing pipeline or due to changes in the environment such as user preference or behavior.

Monitoring tasks called assays provides model insights by comparing a model’s predictions or the input data coming into the model against an established baseline. Changes in the distribution of this data can be an indication of model drift, or of a change in the environment that the model trained for. This can provide tips on whether a model needs to be retrained or the environment data analyzed for accuracy or other needs.

Model Insights provides interactive assays to explore data from a pipeline and learn how the data is behaving. With this information and the knowledge of your particular business use case you can then choose appropriate settings to launch an assay to run on a given frequency as more data is collected.

Assay Details

Assays contain the following attributes:

AttributeDefaultDescription
Name The name of the assay. Assay names must be unique.
Baseline Data Data that is known to be “typical” (typically distributed) and can be used to determine whether the distribution of new data has changed.
ScheduleEvery 24 hours at 1 AMConfigure the start time and frequency of when the new analysis will run. New assays are configured to run a new analysis for every 24 hours starting at the end of the baseline period. This period can be configured through the SDK.
Group ResultsDailyHow the results are grouped: Daily (Default), Every Minute, Weekly, or Monthly.
MetricPSIPopulation Stability Index (PSI) is an entropy-based measure of the difference between distributions. Maximum Difference of Bins measures the maximum difference between the baseline and current distributions (as estimated using the bins). Sum of the difference of bins sums up the difference of occurrences in each bin between the baseline and current distributions.
Threshold0.1Threshold for deciding the difference between distributions is similar(small) or different(large), as evaluated by the metric. The default of 0.1 is generally a good threshold when using PSI as the metric.
Number of Bins5Number of bins used to partition the baseline data. By default, the binning scheme is percentile (quantile) based. The binning scheme can be configured (see Bin Mode, below). Note that the total number of bins will include the set number plus the left_outlier and the right_outlier, so the total number of bins will be the total set + 2.
Bin ModeQuantileSpecify the Binning Scheme. Available options are: Quantile binning defines the bins using percentile ranges (each bin holds the same percentage of the baseline data). Equal binning defines the bins using equally spaced data value ranges, like a histogram. Custom allows users to set the range of values for each bin, with the Left Outlier always starting at Min (below the minimum values detected from the baseline) and the Right Outlier always ending at Max (above the maximum values detected from the baseline).
Bin WeightEqually WeightedThe weight applied to each bin. The bin weights can be either set to Equally Weighted (the default) where each bin is weighted equally, or Custom where the bin weights can be adjusted depending on which are considered more important for detecting model drift.

Model Insights via the Wallaroo Dashboard SDK

Assays generated through the Wallaroo SDK can be previewed, configured, and uploaded to the Wallaroo Ops instance. The following is a condensed version of this process. For full details see the Wallaroo SDK Essentials Guide: Assays Management guide.

Model drift detection with assays using the Wallaroo SDK follows this general process.

  • Define the Baseline: From either historical inference data for a specific model in a pipeline, or from a pre-determine array of data, a baseline is formed.
  • Assay Preview: Once the baseline is formed, we preview the assay and configure the different options until we have the the best method of detecting environment or model drift.
  • Create Assay: With the previews and configuration complete, we upload the assay. The assay will perform an analysis on a regular scheduled based on the configuration.
  • Get Assay Results: Retrieve the analyses and use them to detect model drift and possible sources.
  • Pause/Resume Assay: Pause or restart an assay as needed.

Define the Baseline

Assay baselines are defined with the wallaroo.client.build_assay method. Through this process we define the baseline from either a range of dates or pre-generated values.

wallaroo.client.build_assay take the following parameters:

ParameterTypeDescription
assay_nameString (Required) - requiredThe name of the assay. Assay names must be unique across the Wallaroo instance.
pipelinewallaroo.pipeline.Pipeline (Required)The pipeline the assay is monitoring.
model_nameString (Optional) / NoneThe name of the model to monitor. This field should only be used to track the inputs/outputs for a specific model step in a pipeline. If no model_name is to be included, then the parameters must be passed a named parameters not positional ones.
iopathString (Required)The input/output data for the model being tracked in the format input/output field index. Only one value is tracked for any assay. For example, to track the output of the model’s field house_value at index 0, the iopath is 'output house_value 0.
baseline_startdatetime.datetime (Optional)The start time for the inferences to use as the baseline. Must be included with baseline_end. Cannot be included with baseline_data.
baseline_enddatetime.datetime (Optional)The end time of the baseline window. the baseline. Windows start immediately after the baseline window and are run at regular intervals continuously until the assay is deactivated or deleted. Must be included with baseline_start. Cannot be included with baseline_data..
baseline_datanumpy.array (Optional)The baseline data in numpy array format. Cannot be included with either baseline_start or baseline_data.

Note that model_name is an optional parameters when parameters are named. For example:

assay_builder_from_dates = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

or:

assay_builder_from_dates = wl.build_assay("assays from date baseline", 
                                          mainpipeline, 
                                          None, ## since we are using positional parameters, `None` must be included for the model parameter
                                          "output variable 0",
                                          assay_baseline_start, 
                                          assay_baseline_end)

Baselines are created in one of two mutually exclusive methods:

  • Date Range: The baseline_start and baseline_end retrieves the inference requests and results for the pipeline from the start and end period. This data is summarized and used to create the baseline. For our examples, we’re using the variables assay_baseline_start and assay_baseline_end to represent a range of dates, with assay_baseline_start being set before assay_baseline_end.
  • Numpy Values: The baseline_data sets the baseline from a provided numpy array. This allows assay baselines to be created without first performing inferences in Wallaroo.

Define the Baseline Example

This example shows two methods of defining the baseline for an assay:

  • "assays from date baseline": This assay uses historical inference requests to define the baseline. This assay is saved to the variable assay_builder_from_dates.
  • "assays from numpy": This assay uses a pre-generated numpy array to define the baseline. This assay is saved to the variable assay_builder_from_numpy.

In both cases, the following parameters are used:

ParameterValue
assay_name"assays from date baseline" and "assays from numpy"
pipelinemainpipeline: A pipeline with a ML model that predicts house prices. The output field for this model is variable.
iopathThese assays monitor the model’s output field variable at index 0 for the pipeline. From this, the iopath setting is "output variable 0".

The difference between the two assays’ parameters determines how the baseline is generated.

  • "assays from date baseline": Uses the baseline_start and baseline_end to set the time period of inference requests and results to gather data from.
  • "assays from numpy": Uses a pre-generated numpy array as for the baseline data.

First we generate an assay baseline from a range of historical inferences performed through the specified pipeline deployment.

# Build the assay, based on the start and end of our baseline time, 
# and tracking the output variable index 0

display(assay_baseline_start)
display(assay_baseline_end)

assay_baseline_from_dates = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# create the baseline from the dates
assay_baseline_run_from_dates = assay_baseline_from_dates.build().interactive_baseline_run()
datetime.datetime(2024, 7, 16, 17, 12, 27, 123262)

datetime.datetime(2024, 7, 16, 17, 13, 27, 222295)

In this code sample, we generate the assay baseline from a preset numpy array called small_results_baseline.

# build the baseline from the numpy array
display(small_results_baseline[0:5])

# assay builder by baseline
assay_baseline_from_numpy = wl.build_assay(assay_name="assays from numpy", 
                               pipeline=mainpipeline, 
                               iopath="output variable 0", 
                               baseline_data = small_results_baseline)

# create the baseline from the numpy array
assay_baseline_run_from_numpy = assay_baseline_from_numpy.build().interactive_baseline_run()
array([ 318011.4 ,  383833.88, 1295531.8 ,  437177.97,  513264.66])

Baseline Chart

The baseline chart is displayed with wallaroo.assay.AssayAnalysis.chart(), which returns a chart with:

  • baseline mean: The mean value of the baseline values.
  • baseline median: The median value of the baseline values.
  • bin_mode: The binning mode. See Binning Mode
  • aggregation: The aggregation type. See Aggregation Options
  • metric: The assay’s metric type. See Score Metric
  • weighted: Whether the binning mode is weighted. See Binning Mode

The first chart is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_run_from_dates.chart()
baseline mean = 513781.205375
baseline median = 448627.8125
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False

This chart is from an assay baseline generated from a set of numpy values.

assay_baseline_run_from_numpy.chart()
baseline mean = 513781.2065599997
baseline median = 448627.8
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False

Baseline DataFrame

The method wallaroo.assay_config.AssayBuilder.baseline_dataframe returns a DataFrame of the assay baseline generated from the provided parameters. This includes:

  • metadata: The inference metadata with the model information, inference time, and other related factors.
  • in data: Each input field assigned with the label in.{input field name}.
  • out data: Each output field assigned with the label out.{output field name}

Note that for assays generated from numpy values, there is only the out data based on the supplied baseline data.

In the following example, the baseline DataFrame is retrieved.

This baseline DataFrame is from an assay baseline generated from a set of inferences across a range of dates.

display(assay_baseline_from_dates.baseline_dataframe())
timemetadatainput_tensor_0input_tensor_1input_tensor_2input_tensor_3input_tensor_4input_tensor_5input_tensor_6input_tensor_7...input_tensor_9input_tensor_10input_tensor_11input_tensor_12input_tensor_13input_tensor_14input_tensor_15input_tensor_16input_tensor_17output_variable_0
01721150007209{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '3307faae-8a2f-4079-866b-f474d99f55aa', 'elapsed': [2159729, 3663809], 'dropped': [], 'partition': 'engine-5cfd766455-k42dw'}3.02.001590.087120.01.00.03.03.0...1590.00.047.224098-122.0719992780.0183161.016.00.00.03.180114e+05
11721150007209{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '3307faae-8a2f-4079-866b-f474d99f55aa', 'elapsed': [2159729, 3663809], 'dropped': [], 'partition': 'engine-5cfd766455-k42dw'}4.02.503200.06691.02.00.00.03.0...3200.00.047.367001-122.0309982610.06510.013.00.00.03.838339e+05
21721150007209{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '3307faae-8a2f-4079-866b-f474d99f55aa', 'elapsed': [2159729, 3663809], 'dropped': [], 'partition': 'engine-5cfd766455-k42dw'}4.03.255180.019850.02.00.03.03.0...3540.01640.047.562000-122.1620033160.09750.09.00.00.01.295532e+06
31721150007209{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '3307faae-8a2f-4079-866b-f474d99f55aa', 'elapsed': [2159729, 3663809], 'dropped': [], 'partition': 'engine-5cfd766455-k42dw'}3.02.001790.06800.01.00.00.04.0...1240.0550.047.728001-122.3389971470.06800.051.00.00.04.371780e+05
41721150007209{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '3307faae-8a2f-4079-866b-f474d99f55aa', 'elapsed': [2159729, 3663809], 'dropped': [], 'partition': 'engine-5cfd766455-k42dw'}3.01.752220.011646.01.00.00.03.0...1270.0950.047.776199-122.2699971490.010003.064.00.00.05.132647e+05
..................................................................
4951721150007209{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '3307faae-8a2f-4079-866b-f474d99f55aa', 'elapsed': [2159729, 3663809], 'dropped': [], 'partition': 'engine-5cfd766455-k42dw'}3.02.502160.08809.01.00.00.03.0...1540.0620.047.699402-122.348999930.05420.00.00.00.07.261818e+05
4961721150007209{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '3307faae-8a2f-4079-866b-f474d99f55aa', 'elapsed': [2159729, 3663809], 'dropped': [], 'partition': 'engine-5cfd766455-k42dw'}2.02.501785.0779.02.00.00.03.0...1595.0190.047.595901-122.1979981780.0794.039.00.00.05.596311e+05
4971721150007209{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '3307faae-8a2f-4079-866b-f474d99f55aa', 'elapsed': [2159729, 3663809], 'dropped': [], 'partition': 'engine-5cfd766455-k42dw'}4.02.501750.06397.02.00.00.03.0...1750.00.047.308201-122.3580021940.06502.027.00.00.02.912398e+05
4981721150007209{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '3307faae-8a2f-4079-866b-f474d99f55aa', 'elapsed': [2159729, 3663809], 'dropped': [], 'partition': 'engine-5cfd766455-k42dw'}4.02.252130.08499.01.00.00.04.0...1600.0530.047.365700-122.2099991890.011368.039.00.00.03.342577e+05
4991721150007209{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '3307faae-8a2f-4079-866b-f474d99f55aa', 'elapsed': [2159729, 3663809], 'dropped': [], 'partition': 'engine-5cfd766455-k42dw'}2.02.251660.02128.02.00.00.04.0...1660.00.047.752800-122.2519991640.02128.041.00.00.04.371780e+05

500 rows × 21 columns

This baseline DataFrame is from an assay baseline generated from an array of numpy values.

display(assay_baseline_from_numpy.baseline_dataframe())
output_variable_0
0318011.40
1383833.88
21295531.80
3437177.97
4513264.66
......
495726181.75
496559631.06
497291239.75
498334257.70
499437177.97

500 rows × 1 columns

Baseline Stats

The method wallaroo.assay.AssayAnalysis.baseline_stats() returns a pandas.core.frame.DataFrame of the baseline stats.

The baseline stats for each assay are displayed in the examples below.

This baseline states DataFrame is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_run_from_dates.baseline_stats()
Baseline
count500
min236238.671875
max1295531.75
mean513781.205375
median448627.8125
std233066.759431
start2024-07-16T17:12:27.123262+00:00
end2024-07-16T17:13:27.222262+00:00

This baseline states DataFrame is from an assay baseline generated from an array of numpy values.

assay_baseline_run_from_numpy.baseline_stats()
Baseline
count500
min236238.67
max1295531.8
mean513781.20656
median448627.8
std233066.761363
startNone
endNone

Baseline Bins

The method wallaroo.assay.AssayAnalysis.baseline_bins a simple dataframe to with the edge/bin data for a baseline.

These baseline bins DataFrame is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_run_from_dates.baseline_bins()
b_edgesb_edge_namesb_aggregated_valuesb_aggregation
02.362387e+05left_outlier0.0Aggregation.DENSITY
13.109929e+05q_200.2Aggregation.DENSITY
24.319922e+05q_400.2Aggregation.DENSITY
34.759715e+05q_600.2Aggregation.DENSITY
47.019407e+05q_800.2Aggregation.DENSITY
51.295532e+06q_1000.2Aggregation.DENSITY
6infright_outlier0.0Aggregation.DENSITY

These baseline bins DataFrame is from an assay baseline generated from an array of numpy values.

assay_baseline_run_from_numpy.baseline_bins()
b_edgesb_edge_namesb_aggregated_valuesb_aggregation
0236238.67left_outlier0.0Aggregation.DENSITY
1310992.94q_200.2Aggregation.DENSITY
2431992.22q_400.2Aggregation.DENSITY
3475971.50q_600.2Aggregation.DENSITY
4701940.70q_800.2Aggregation.DENSITY
51295531.80q_1000.2Aggregation.DENSITY
6infright_outlier0.0Aggregation.DENSITY

Baseline Histogram Chart

The method wallaroo.assay_config.AssayBuilder.baseline_histogram returns a histogram chart of the assay baseline generated from the provided parameters.

These chart is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_from_dates.baseline_histogram()

Baseline KDE Chart

The method wallaroo.assay_config.AssayBuilder.baseline_kde returns a Kernel Density Estimation (KDE) chart of the assay baseline generated from the provided parameters.

These chart is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_from_dates.baseline_kde()

Baseline ECDF Chart

The method wallaroo.assay_config.AssayBuilder.baseline_ecdf returns a Empirical Cumulative Distribution Function (CDF) chart of the assay baseline generated from the provided parameters.

These chart is from an assay baseline generated from a set of inferences across a range of dates.

assay_baseline_from_dates.baseline_ecdf()

Assay Preview

Now that the baseline is defined, we look at different configuration options and view how the assay baseline and results changes. Once we determine what gives us the best method of determining model drift, we can create the assay.

Analysis List Chart Scores

Analysis List scores show the assay scores for each assay result interval in one chart. Values that are outside of the alert threshold are colored red, while scores within the alert threshold are green.

Assay chart scores are displayed with the method wallaroo.assay.AssayAnalysisList.chart_scores(title: Optional[str] = None), with ability to display an optional title with the chart.

The following example shows retrieving the assay results and displaying the chart scores. From our example, we have two windows - the first should be green, and the second is red showing that values were outside the alert threshold.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# Preview the assay analyses
assay_results.chart_scores()

Analysis Chart

The method wallaroo.assay.AssayAnalysis.chart() displays a comparison between the baseline and an interval of inference data.

This is compared to the Chart Scores, which is a list of all of the inference data split into intervals, while the Analysis Chart shows the breakdown of one set of inference data against the baseline.

Score from the Analysis List Chart Scores and each element from the Analysis List DataFrame generates

The following fields are included.

FieldTypeDescription
baseline meanFloatThe mean of the baseline values.
window meanFloatThe mean of the window values.
baseline medianFloatThe median of the baseline values.
window medianFloatThe median of the window values.
bin_modeStringThe binning mode used for the assay.
aggregationStringThe aggregation mode used for the assay.
metricStringThe metric mode used for the assay.
weightedBoolWhether the bins were manually weighted.
scoreFloatThe score from the assay window.
scoresList(Float)The score from each assay window bin.
indexInteger/NoneThe window index. Interactive assay runs are None.
# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display one of the analysis from the total results
assay_results[0].chart()
baseline mean = 513781.205375
window mean = 512779.469375
baseline median = 448627.8125
window median = 448627.8125
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.008604680420075936
scores = [0.0, 0.001015989699687698, 0.0009472210786334042, 0.0033382848156130714, 0.002503651312977887, 0.0001265890399214496, 0.0006729444732424258]
index = None

Analysis List DataFrame

wallaroo.assay.AssayAnalysisList.to_dataframe() returns a DataFrame showing the assay results for each window aka individual analysis. This DataFrame contains the following fields:

FieldTypeDescription
assay_idInteger/NoneThe assay id. Only provided from uploaded and executed assays.
nameString/NoneThe name of the assay. Only provided from uploaded and executed assays.
iopathString/NoneThe iopath of the assay. Only provided from uploaded and executed assays.
scoreFloatThe assay score.
startDateTimeThe DateTime start of the assay window.
minFloatThe minimum value in the assay window.
maxFloatThe maximum value in the assay window.
meanFloatThe mean value in the assay window.
medianFloatThe median value in the assay window.
stdFloatThe standard deviation value in the assay window.
warning_thresholdFloat/NoneThe warning threshold of the assay window.
alert_thresholdFloat/NoneThe alert threshold of the assay window.
statusStringThe assay window status. Values are:
  • OK: The score is within accepted thresholds.
  • Warning: The score has triggered the warning_threshold if exists, but not the alert_threshold.
  • Alert: The score has triggered the the alert_threshold.

For this example, the assay analysis list DataFrame is listed.

From this tutorial, we should have 2 windows of dta to look at, each one minute apart. The first window should show status: OK, with the second window with the very large house prices will show status: alert

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display the dataframe from the analyses
assay_results.to_dataframe()
idassay_idassay_nameiopathpipeline_idpipeline_nameworkspace_idworkspace_namescorestartminmaxmeanmedianstdwarning_thresholdalert_thresholdstatus
0NoneNoneassays from date baselineNoneNoneNone0.0086052024-07-16T17:14:27.239563+00:002.362387e+051489624.2505.127795e+054.486278e+05226255.370366None0.25Ok
1NoneNoneassays from date baselineNoneNoneNone8.8684832024-07-16T17:16:27.239563+00:001.514079e+062016006.1251.881163e+061.946438e+06162405.590663None0.25Alert

Analysis List Full DataFrame

wallaroo.assay.AssayAnalysisList.to_full_dataframe() returns a DataFrame showing all values, including the inputs and outputs from the assay results for each window aka individual analysis. This DataFrame contains the following fields:

pipeline_id	warning_threshold	bin_index	created_at
FieldTypeDescription
window_startDateTimeThe date and time when the window period began.
analyzed_atDateTimeThe date and time when the assay analysis was performed.
elapsed_millisIntegerHow long the analysis took to perform in milliseconds.
baseline_summary_countIntegerThe number of data elements from the baseline.
baseline_summary_minFloatThe minimum value from the baseline summary.
baseline_summary_maxFloatThe maximum value from the baseline summary.
baseline_summary_meanFloatThe mean value of the baseline summary.
baseline_summary_medianFloatThe median value of the baseline summary.
baseline_summary_stdFloatThe standard deviation value of the baseline summary.
baseline_summary_edges_{0…n}FloatThe baseline summary edges for each baseline edge from 0 to number of edges.
summarizer_typeStringThe type of summarizer used for the baseline. See wallaroo.assay_config for other summarizer types.
summarizer_bin_weightsList / NoneIf baseline bin weights were provided, the list of those weights. Otherwise, None.
summarizer_provided_edgesList / NoneIf baseline bin edges were provided, the list of those edges. Otherwise, None.
statusStringThe assay window status. Values are:
  • OK: The score is within accepted thresholds.
  • Warning: The score has triggered the warning_threshold if exists, but not the alert_threshold.
  • Alert: The score has triggered the the alert_threshold.
idInteger/NoneThe id for the window aka analysis. Only provided from uploaded and executed assays.
assay_idInteger/NoneThe assay id. Only provided from uploaded and executed assays.
pipeline_idInteger/NoneThe pipeline id. Only provided from uploaded and executed assays.
warning_thresholdFloatThe warning threshold set for the assay.
warning_thresholdFloatThe warning threshold set for the assay.
bin_indexInteger/NoneThe bin index for the window aka analysis.
created_atDatetime/NoneThe date and time the window aka analysis was generated. Only provided from uploaded and executed assays.

For this example, full DataFrame from an assay preview is generated.

From this tutorial, we should have 2 windows of dta to look at, each one minute apart. The first window should show status: OK, with the second window with the very large house prices will show status: alert

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display the full dataframe from the analyses
assay_results.to_full_dataframe()
window_startanalyzed_atelapsed_millisbaseline_summary_countbaseline_summary_minbaseline_summary_maxbaseline_summary_meanbaseline_summary_medianbaseline_summary_stdbaseline_summary_edges_0...summarizer_provided_edgesstatusidassay_idpipeline_idworkspace_idworkspace_namewarning_thresholdbin_indexcreated_at
02024-07-16T17:14:27.239563+00:002024-07-16T17:17:49.440491+00:0017500236238.6718751295531.75513781.205375448627.8125233066.759431236238.671875...NoneOkNoneNoneNoneNoneNoneNoneNoneNone
12024-07-16T17:16:27.239563+00:002024-07-16T17:17:49.440649+00:009500236238.6718751295531.75513781.205375448627.8125233066.759431236238.671875...NoneAlertNoneNoneNoneNoneNoneNoneNoneNone

2 rows × 88 columns

Analysis Compare Basic Stats

The method wallaroo.assay.AssayAnalysis.compare_basic_stats returns a DataFrame comparing one set of inference data against the baseline.

This is compared to the Analysis List DataFrame, which is a list of all of the inference data split into intervals, while the Analysis Compare Basic Stats shows the breakdown of one set of inference data against the baseline.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display one analysis against the baseline
assay_results[0].compare_basic_stats()
BaselineWindowdiffpct_diff
count500.01000.0500.000000100.000000
min236238.671875236238.6718750.0000000.000000
max1295531.751489624.25194092.50000014.981686
mean513781.205375512779.469375-1001.736000-0.194973
median448627.8125448627.81250.0000000.000000
std233066.759431226255.370366-6811.389065-2.922506
start2024-07-16T17:12:27.123262+00:002024-07-16T17:14:27.239563+00:00NaNNaN
end2024-07-16T17:13:27.222262+00:002024-07-16T17:15:27.239563+00:00NaNNaN

Configure Assays

Before creating the assay, configure the assay and continue to preview it until the best method for detecting drift is set. The following options are available.

Inference Interval and Inference Width

The inference interval aka window interval sets how often to run the assay analysis. This is set from the wallaroo.assay_config.AssayBuilder.window_builder.add_interval method to collect data expressed in time units: “hours=24”, “minutes=1”, etc.

For example, with an interval of 1 minute, the assay collects data every minute. Within an hour, 60 intervals of data is collected.

We can adjust the interval and see how the assays change based on how frequently they are run.

The width sets the time period from the wallaroo.assay_config.AssayBuilder.window_builder.add_width method to collect data expressed in time units: “hours=24”, “minutes=1”, etc.

For example, an interval of 1 minute and a width of 1 minute collects 1 minutes worth of data every minute. An interval of 1 minute with a width of 5 minutes collects 5 minute of inference data every minute.

By default, the interval and width is 24 hours.

For this example, we’ll adjust the width and interval from 1 minute to 5 minutes and see how the number of analyses and their score changes.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show the analyses chart
assay_results.chart_scores()
# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to five minutes each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=5).add_interval(minutes=5).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show the analyses chart
assay_results.chart_scores()
Add Run Until and Add Inference Start

For previewing assays, setting wallaroo.assay_config.AssayBuilder.add_run_until sets the end date and time for collecting inference data. When an assay is uploaded, this setting is no longer valid - assays run at the Inference Interval until the assay is paused.

Setting the wallaroo.assay_config.WindowBuilder.add_start sets the start date and time to collect inference data. When an assay is uploaded, this setting is included, and assay results will be displayed starting from that start date at the Inference Interval until the assay is paused. By default, add_start begins 24 hours after the assay is uploaded unless set in the assay configuration manually.

For the following example, the add_run_until setting is set to datetime.datetime.now() to collect all inference data from assay_window_start up until now, and the second example limits that example to only two minutes of data.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results, minus 2 minutes for the period to start gathering analyses
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start+datetime.timedelta(seconds=-120))

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show the analyses chart
assay_results.chart_scores()
# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results plus 2 minutes 
assay_baseline.add_run_until(assay_window_start+datetime.timedelta(seconds=120))

# Set the interval and window to one minute each
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show the analyses chart
assay_results.chart_scores()
Score Metric

The score is a distance between the baseline and the analysis window. The larger the score, the greater the difference between the baseline and the analysis window. The following methods are provided determining the score:

  • PSI (Default) - Population Stability Index (PSI).
  • MAXDIFF: Maximum difference between corresponding bins.
  • SUMDIFF: Mum of differences between corresponding bins.

The metric type used is updated with the wallaroo.assay_config.AssayBuilder.add_metric(metric: wallaroo.assay_config.Metric) method.

The following three charts use each of the metrics. Note how the scores change based on the score type used.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# set metric PSI mode
assay_baseline.summarizer_builder.add_metric(wallaroo.assay_config.Metric.PSI)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display one analysis from the results
assay_results[0].chart()
baseline mean = 513781.205375
window mean = 512779.469375
baseline median = 448627.8125
window median = 448627.8125
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.008604680420075936
scores = [0.0, 0.001015989699687698, 0.0009472210786334042, 0.0033382848156130714, 0.002503651312977887, 0.0001265890399214496, 0.0006729444732424258]
index = None
# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# set metric MAXDIFF mode
assay_baseline.summarizer_builder.add_metric(wallaroo.assay_config.Metric.MAXDIFF)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display one analysis from the results
assay_results[0].chart()
baseline mean = 513781.205375
window mean = 512779.469375
baseline median = 448627.8125
window median = 448627.8125
bin_mode = Quantile
aggregation = Density
metric = MaxDiff
weighted = False
score = 0.025000000000000026
scores = [0.0, 0.014000000000000012, 0.013999999999999985, 0.025000000000000026, 0.022999999999999993, 0.0050000000000000044, 0.007]
index = 3
# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# set metric SUMDIFF mode
assay_baseline.summarizer_builder.add_metric(wallaroo.assay_config.Metric.SUMDIFF)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# display one analysis from the results
assay_results[0].chart()
baseline mean = 513781.205375
window mean = 512779.469375
baseline median = 448627.8125
window median = 448627.8125
bin_mode = Quantile
aggregation = Density
metric = SumDiff
weighted = False
score = 0.04400000000000001
scores = [0.0, 0.014000000000000012, 0.013999999999999985, 0.025000000000000026, 0.022999999999999993, 0.0050000000000000044, 0.007]
index = None
Alert Threshold

Assay alert thresholds are modified with the wallaroo.assay_config.AssayBuilder.add_alert_threshold(alert_threshold: float) method. By default alert thresholds are 0.1.

The following example updates the alert threshold to 0.5.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# set the alert threshold
assay_baseline.add_alert_threshold(0.5)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show the analyses with the alert threshold
assay_results.to_dataframe()
idassay_idassay_nameiopathpipeline_idpipeline_nameworkspace_idworkspace_namescorestartminmaxmeanmedianstdwarning_thresholdalert_thresholdstatus
0NoneNoneassays from date baselineNoneNoneNone0.0086052024-07-16T17:14:27.239563+00:002.362387e+051489624.2505.127795e+054.486278e+05226255.370366None0.5Ok
1NoneNoneassays from date baselineNoneNoneNone8.8684832024-07-16T17:16:27.239563+00:001.514079e+062016006.1251.881163e+061.946438e+06162405.590663None0.5Alert
Number of Bins

Number of bins sets how the baseline data is partitioned. The total number of bins includes the set number plus the left_outlier and the right_outlier, so the total number of bins will be the total set + 2.

The number of bins is set with the wallaroo.assay_config.UnivariateContinousSummarizerBuilder.add_num_bins(num_bins: int) method.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# update number of bins here
assay_baseline.summarizer_builder.add_num_bins(10)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()
baseline mean = 513781.205375
window mean = 512779.469375
baseline median = 448627.8125
window median = 448627.8125
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.049245130581621346
scores = [0.0, 0.0071920518112945295, 0.0011479601685666688, 0.00024635524503391294, 0.003431094801012864, 0.0005425807707100881, 0.0041791943627767685, 0.016044296997867993, 0.00495016900394247, 0.00364643113587909, 0.0071920518112945295, 0.0006729444732424258]
index = None
Binning Mode

Binning Mode defines how the bins are separated. Binning modes are modified through the wallaroo.assay_config.UnivariateContinousSummarizerBuilder.add_bin_mode(bin_mode: bin_mode: wallaroo.assay_config.BinMode, edges: Optional[List[float]] = None).

Available bin_mode values from wallaroo.assay_config.Binmode are the following:

  • QUANTILE (Default): Based on percentages. If num_bins is 5 then quintiles so bins are created at the 20%, 40%, 60%, 80% and 100% points.
  • EQUAL: Evenly spaced bins where each bin is set with the formula min - max / num_bins
  • PROVIDED: The user provides the edge points for the bins.

If PROVIDED is supplied, then a List of float values must be provided for the edges parameter that matches the number of bins.

The following examples are used to show how each of the binning modes effects the bins.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# update binning mode here
assay_baseline.summarizer_builder.add_bin_mode(wallaroo.assay_config.BinMode.QUANTILE)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()
baseline mean = 513781.205375
window mean = 512779.469375
baseline median = 448627.8125
window median = 448627.8125
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.008604680420075936
scores = [0.0, 0.001015989699687698, 0.0009472210786334042, 0.0033382848156130714, 0.002503651312977887, 0.0001265890399214496, 0.0006729444732424258]
index = None
# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# update binning mode here
assay_baseline.summarizer_builder.add_bin_mode(wallaroo.assay_config.BinMode.EQUAL)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()
baseline mean = 513781.205375
window mean = 512779.469375
baseline median = 448627.8125
window median = 448627.8125
bin_mode = Equal
aggregation = Density
metric = PSI
weighted = False
score = 0.014403886718133243
scores = [0.0, 2.0986366569212073e-06, 0.000233593237538099, 0.001922311458522506, 0.009091699485742576, 0.002481239426430717, 0.0006729444732424258]
index = None

The following example manually sets the bin values.

The values in this dataset run from 200000 to 1500000. We can specify the bins with the BinMode.PROVIDED and specifying a list of floats with the right hand / upper edge of each bin and optionally the lower edge of the smallest bin. If the lowest edge is not specified the threshold for left outliers is taken from the smallest value in the baseline dataset.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

edges = [200000.0, 400000.0, 600000.0, 800000.0, 1500000.0, 2000000.0]

# update binning mode here
assay_baseline.summarizer_builder.add_bin_mode(wallaroo.assay_config.BinMode.PROVIDED, edges)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()
baseline mean = 513781.205375
window mean = 512779.469375
baseline median = 448627.8125
window median = 448627.8125
bin_mode = Provided
aggregation = Density
metric = PSI
weighted = False
score = 0.005462658417539512
scores = [0.0, 0.0002882043853549172, 0.0017843411661165069, 5.438215377767015e-05, 0.003335730712290418, 0.0, 0.0]
index = None
Aggregation Options

Assay aggregation options are modified with the wallaroo.assay_config.AssayBuilder.add_aggregation(aggregation: wallaroo.assay_config.Aggregation) method. The following options are provided:

  • Aggregation.DENSITY (Default): Count the number/percentage of values that fall in each bin.
  • Aggregation.CUMULATIVE: Empirical Cumulative Density Function style, which keeps a cumulative count of the values/percentages that fall in each bin.

The following example demonstrate the different results between the two.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

#Aggregation.DENSITY - the default
assay_baseline.summarizer_builder.add_aggregation(wallaroo.assay_config.Aggregation.DENSITY)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()
baseline mean = 513781.205375
window mean = 512779.469375
baseline median = 448627.8125
window median = 448627.8125
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.008604680420075936
scores = [0.0, 0.001015989699687698, 0.0009472210786334042, 0.0033382848156130714, 0.002503651312977887, 0.0001265890399214496, 0.0006729444732424258]
index = None
# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

#Aggregation.CUMULATIVE
assay_baseline.summarizer_builder.add_aggregation(wallaroo.assay_config.Aggregation.CUMULATIVE)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# show one analysis with the updated bins
assay_results[0].chart()
baseline mean = 513781.205375
window mean = 512779.469375
baseline median = 448627.8125
window median = 448627.8125
bin_mode = Quantile
aggregation = Cumulative
metric = PSI
weighted = False
score = 0.04800000000000004
scores = [0.0, 0.014000000000000012, 0.0, 0.025000000000000026, 0.0020000000000000018, 0.007000000000000006, 0.0]
index = None

Create Assay

With the assay previewed and configuration options determined, we officially create it by uploading it to the Wallaroo instance.

Once it is uploaded, the assay runs an analysis based on the window width, interval, and the other settings configured.

Assays are uploaded with the wallaroo.assay_config.upload() method. This uploads the assay into the Wallaroo database with the configurations applied and returns the assay id. Note that assay names must be unique across the Wallaroo instance; attempting to upload an assay with the same name as an existing one will return an error.

wallaroo.assay_config.upload() returns the assay id for the assay.

Typically we would just call wallaroo.assay_config.upload() after configuring the assay. For the example below, we will perform the complete configuration in one window to show all of the configuration steps at once before creating the assay.

# Build the assay, based on the start and end of our baseline time, 
# and tracking the output variable index 0
assay_baseline = wl.build_assay(assay_name="assays from date baseline tutorial 2", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# set the width, interval, and assay start date and time
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# add other options
assay_baseline.summarizer_builder.add_aggregation(wallaroo.assay_config.Aggregation.CUMULATIVE)
assay_baseline.summarizer_builder.add_metric(wallaroo.assay_config.Metric.MAXDIFF)
assay_baseline.add_alert_threshold(0.5)

assay_id = assay_baseline.upload()

# wait 65 seconds for the first analysis run performed
time.sleep(65)

The assay is now visible through the Wallaroo UI by selecting the workspace, then the pipeline, then Insights.

Get Assay Info

Assay information is retrieved with the wallaroo.client.get_assay_info() which takes the following parameters.

ParameterTypeDescription
assay_idInteger (Required)The numerical id of the assay.
workspace_id(Int) (Optional)The numerical identifier of the workspace to filter by.
workspace_name(String) (Optional)The name of the workspace to filter by.

Assay information is returned for assays in workspaces the user is a member of. Admin users have unrestricted access to all workspaces. For more details, see Wallaroo Enterprise User Management.

This returns the following:

ParameterTypeDescription
idIntegerThe numerical id of the assay.
nameStringThe name of the assay.
activeBooleanTrue: The assay is active and generates analyses based on its configuration. False: The assay is disabled and will not generate new analyses.
pipeline_nameStringThe name of the pipeline the assay references.
last_runDateTimeThe date and time the assay last ran.
next_runDateTimeTHe date and time the assay analysis will next run.
alert_thresholdFloatThe alert threshold setting for the assay.
baselineDictThe baseline and settings as set from the assay configuration.
iopathStringThe iopath setting for the assay.
metricStringThe metric setting for the assay.
num_binsIntegerThe number of bins for the assay.
bin_weightsList/NoneThe bin weights used if any.
bin_modeStringThe binning mode used.
workspace_id(Int)The numerical identifier of the workspace the assay is associated with.
workspace_name(String)The name of the workspace the assay is associated with.
display(wl.get_assay_info(assay_id))
idnameactivestatuspipeline_namelast_runnext_runalert_thresholdworkspace_idworkspace_namebaselineiopathmetricnum_binsbin_weightsbin_mode
02assays from date baseline tutorial 2True{"run_at": "2024-07-16T18:42:03.041484407+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}assay-demonstration-tutorial2024-07-16T18:42:03.041484+00:002024-07-16T18:41:27.239563+00:000.529assay-demonstration-tutorial-2Start:2024-07-16T17:12:27.123262+00:00, End:2024-07-16T17:13:27.222262+00:00output variable 0MaxDiff5NoneQuantile

Get Assay Results

Once an assay is created the assay runs an analysis based on the window width, interval, and the other settings configured.

Assay results are retrieved with the wallaroo.client.get_assay_results method which takes the following parameters:

ParameterTypeDescription
assay_idInteger (Required)The numerical id of the assay.
startDatetime.Datetime (Required)The start date and time of historical data from the pipeline to start analyses from.
endDatetime.Datetime (Required)The end date and time of historical data from the pipeline to limit analyses to.
workspace_id(Int) (Optional)The numerical identifier of the workspace to filter by.
workspace_name(String) (Optional)The name of the workspace to filter by.

Assay results only show assays for workspaces the user is a member of. Admin users have unrestricted access to all workspaces. For more details, see Wallaroo Enterprise User Management.

  • IMPORTANT NOTE: This process requires that additional historical data is generated from the time the assay is created to when the results are available. To add additional inference data, use the Assay Test Data section above.
assay_results = wl.get_assay_results(assay_id=assay_id,
                     start=assay_window_start,
                     end=datetime.datetime.now())

assay_results.chart_scores()
assay_results[0].chart()
baseline mean = 513781.205375
window mean = 512779.469375
baseline median = 448627.8125
window median = 448627.8125
bin_mode = Quantile
aggregation = Cumulative
metric = MaxDiff
weighted = False
score = 0.025
scores = [0.0, 0.014000000000000012, 0.0, 0.025000000000000026, 0.0020000000000000018, 0.007000000000000006, 0.0]
index = 3

List Assays

A list of assays is retrieved with the wallaroo.client.list_assays() method and takes the following parameters:

ParameterTypeDescription
workspace_id(Int) (Optional)The numerical identifier of the workspace to filter by.
workspace_name(String) (Optional)The name of the workspace to filter by.

Assay information is returned for assays in workspaces the user is a member of. Admin users have unrestricted access to all workspaces. For more details, see Wallaroo Enterprise User Management.

This returns a list of assays as filtered in reverse chronological order.

FieldTypeDescription
Assay IdIntegerThe numerical id of the assay.
Assay NameStringThe name of the assay.
ActiveBooleanTrue: The assay is active and generates analyses based on its configuration. False: The assay is disabled and will not generate new analyses.
StatusDictThe status of the assay including the
Warning ThresholdFloat/NoneThe warning threshold if set.
Alert ThresholdFloatThe alert threshold for the assay.
Pipeline IdIntegerThe numerical id of the pipeline the assay references.
Pipeline NameStringThe name of the pipeline the assay references.
workspace_id(Int)The numerical identifier of the workspace the assay is associated with.
workspace_name(String)The name of the workspace the assay is associated with.

The errors for this method include:

  • If the parameter workspace_id is not an integer.
  • If the parameter workspace_name is not a String.
wl.list_assays()
Assay IDAssay NameActiveStatusWarning ThresholdAlert ThresholdPipeline IDPipeline NameWorkspace IDWorkspace Name
2assays from date baseline tutorial 2True{"run_at": "2024-07-16T18:42:03.041484407+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}None0.528assay-demonstration-tutorial29assay-demonstration-tutorial-2
1assays from date baseline tutorialTrue{"run_at": "2024-07-16T17:08:02.913332768+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}None0.524assay-demonstration-tutorial27assay-demonstration-tutorial

Set Assay Active Status

Assays active status is either:

  • True: The assay generates analyses based on the assay configuration.
  • False: The assay will not generate new analyses.

Assays are set to active or not active with the wallaroo.client.set_assay_active which takes the following parameters.

ParameterTypeDescription
assay_idIntegerThe numerical id of the assay.
activeBooleanTrue: The assay status is set to Active. False: The assay status is Not Active.

First we will show the current active status.

In the following, set the assay status to False, then set the assay active status back to True.

display(wl.get_assay_info(assay_id))
idnameactivestatuspipeline_namelast_runnext_runalert_thresholdworkspace_idworkspace_namebaselineiopathmetricnum_binsbin_weightsbin_mode
02assays from date baseline tutorial 2True{"run_at": "2024-07-16T18:42:03.041484407+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}assay-demonstration-tutorial2024-07-16T18:42:03.041484+00:002024-07-16T18:41:27.239563+00:000.529assay-demonstration-tutorial-2Start:2024-07-16T17:12:27.123262+00:00, End:2024-07-16T17:13:27.222262+00:00output variable 0MaxDiff5NoneQuantile

Now we set the active status to False, and show the assay list to verify it is no longer active.

wl.set_assay_active(assay_id, False)
display(wl.get_assay_info(assay_id))
idnameactivestatuspipeline_namelast_runnext_runalert_thresholdworkspace_idworkspace_namebaselineiopathmetricnum_binsbin_weightsbin_mode
02assays from date baseline tutorial 2False{"run_at": "2024-07-16T18:42:03.041484407+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}assay-demonstration-tutorial2024-07-16T18:42:03.041484+00:002024-07-16T18:41:27.239563+00:000.529assay-demonstration-tutorial-2Start:2024-07-16T17:12:27.123262+00:00, End:2024-07-16T17:13:27.222262+00:00output variable 0MaxDiff5NoneQuantile

We resume the assay by setting it’s active status to True.

wl.set_assay_active(assay_id, True)
display(wl.get_assay_info(assay_id))
idnameactivestatuspipeline_namelast_runnext_runalert_thresholdworkspace_idworkspace_namebaselineiopathmetricnum_binsbin_weightsbin_mode
02assays from date baseline tutorial 2True{"run_at": "2024-07-16T18:42:03.041484407+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}assay-demonstration-tutorial2024-07-16T18:42:03.041484+00:002024-07-16T18:41:27.239563+00:000.529assay-demonstration-tutorial-2Start:2024-07-16T17:12:27.123262+00:00, End:2024-07-16T17:13:27.222262+00:00output variable 0MaxDiff5NoneQuantile

View Assay Filters

The following examples demonstrate various ways of listing assays and retrieving assay details, with filtering options.

List Assays Filters

List all assays for workspaces the user is a member of.

wl.list_assays()
Assay IDAssay NameActiveStatusWarning ThresholdAlert ThresholdPipeline IDPipeline NameWorkspace IDWorkspace Name
2assays from date baseline tutorial 2True{"run_at": "2024-07-16T18:29:03.024210200+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}None0.528assay-demonstration-tutorial29assay-demonstration-tutorial-2
1assays from date baseline tutorialTrue{"run_at": "2024-07-16T17:08:02.913332768+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}None0.524assay-demonstration-tutorial27assay-demonstration-tutorial

List all assays filtered by workspace id.

wl.list_assays(workspace_id=27)
Assay IDAssay NameActiveStatusWarning ThresholdAlert ThresholdPipeline IDPipeline NameWorkspace IDWorkspace Name
1assays from date baseline tutorialTrue{"run_at": "2024-07-16T17:08:02.913332768+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}None0.524assay-demonstration-tutorial27assay-demonstration-tutorial

List assays filtered by workspace name.

wl.list_assays(workspace_name="assay-demonstration-tutorial-2")
Assay IDAssay NameActiveStatusWarning ThresholdAlert ThresholdPipeline IDPipeline NameWorkspace IDWorkspace Name
2assays from date baseline tutorial 2True{"run_at": "2024-07-16T18:30:03.029976427+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}None0.528assay-demonstration-tutorial29assay-demonstration-tutorial-2

Get Assay Results

Get assay results.

assay_results = wl.get_assay_results(assay_id=assay_id,
                     start=assay_window_start,
                     end=datetime.datetime.now())

assay_results.chart_scores()
Get assay results filtered by workspace id.
assay_results = wl.get_assay_results(assay_id=assay_id,
                                     workspace_id=29,
                     start=assay_window_start,
                     end=datetime.datetime.now())

assay_results.chart_scores()
Get assay results filtered by workspace name.
assay_results = wl.get_assay_results(assay_id=assay_id,
                                     workspace_name="assay-demonstration-tutorial-2",
                     start=assay_window_start,
                     end=datetime.datetime.now())

assay_results.chart_scores()

Get Assay Info

Get assay info.

wl.get_assay_info(assay_id=assay_id)
idnameactivestatuspipeline_namelast_runnext_runalert_thresholdworkspace_idworkspace_namebaselineiopathmetricnum_binsbin_weightsbin_mode
02assays from date baseline tutorial 2True{"run_at": "2024-07-16T18:35:03.031351761+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}assay-demonstration-tutorial2024-07-16T18:35:03.031352+00:002024-07-16T18:34:27.239563+00:000.529assay-demonstration-tutorial-2Start:2024-07-16T17:12:27.123262+00:00, End:2024-07-16T17:13:27.222262+00:00output variable 0MaxDiff5NoneQuantile

Get assay info filtered by workspace id.

wl.get_assay_info(assay_id=assay_id, 
                 workspace_id=29)
idnameactivestatuspipeline_namelast_runnext_runalert_thresholdworkspace_idworkspace_namebaselineiopathmetricnum_binsbin_weightsbin_mode
02assays from date baseline tutorial 2True{"run_at": "2024-07-16T18:36:03.030744191+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}assay-demonstration-tutorial2024-07-16T18:36:03.030744+00:002024-07-16T18:35:27.239563+00:000.529assay-demonstration-tutorial-2Start:2024-07-16T17:12:27.123262+00:00, End:2024-07-16T17:13:27.222262+00:00output variable 0MaxDiff5NoneQuantile

Get assay info filtered by workspace name.

wl.get_assay_info(assay_id=assay_id, 
                 workspace_name="assay-demonstration-tutorial-2")
idnameactivestatuspipeline_namelast_runnext_runalert_thresholdworkspace_idworkspace_namebaselineiopathmetricnum_binsbin_weightsbin_mode
02assays from date baseline tutorial 2True{"run_at": "2024-07-16T18:36:03.030744191+00:00", "num_ok": 0, "num_warnings": 0, "num_alerts": 0}assay-demonstration-tutorial2024-07-16T18:36:03.030744+00:002024-07-16T18:35:27.239563+00:000.529assay-demonstration-tutorial-2Start:2024-07-16T17:12:27.123262+00:00, End:2024-07-16T17:13:27.222262+00:00output variable 0MaxDiff5NoneQuantile