Model Insights and Interactive Analysis Introduction
Wallaroo provides the ability to perform interactive analysis so organizations can explore the data from a pipeline and learn how the data is behaving. With this information and the knowledge of your particular business use case you can then choose appropriate thresholds for persistent automatic assays as desired.
-
IMPORTANT NOTE
Model insights operates over time and is difficult to demo in a notebook without pre-canned data. We assume you have an active pipeline that has been running and making predictions over time and show you the code you may use to analyze your pipeline.
Monitoring tasks called assays monitors a model’s predictions or the data coming into the model against an established baseline. Changes in the distribution of this data can be an indication of model drift, or of a change in the environment that the model trained for. This can provide tips on whether a model needs to be retrained or the environment data analyzed for accuracy or other needs.
Assay Details
Assays contain the following attributes:
Attribute | Default | Description |
---|---|---|
Name | Â | The name of the assay. Assay names must be unique. |
Baseline Data | Â | Data that is known to be “typical” (typically distributed) and can be used to determine whether the distribution of new data has changed. |
Schedule | Every 24 hours at 1 AM | Configure the start time and frequency of when the new analysis will run. New assays are configured to run a new analysis for every 24 hours starting at the end of the baseline period. This period can be configured through the SDK. |
Group Results | Daily | How the results are grouped: Daily (Default), Every Minute, Weekly, or Monthly. |
Metric | PSI | Population Stability Index (PSI) is an entropy-based measure of the difference between distributions. Maximum Difference of Bins measures the maximum difference between the baseline and current distributions (as estimated using the bins). Sum of the difference of bins sums up the difference of occurrences in each bin between the baseline and current distributions. |
Threshold | 0.1 | Threshold for deciding the difference between distributions is similar(small) or different(large), as evaluated by the metric. The default of 0.1 is generally a good threshold when using PSI as the metric. |
Number of Bins | 5 | Number of bins used to partition the baseline data. By default, the binning scheme is percentile (quantile) based. The binning scheme can be configured (see Bin Mode, below). Note that the total number of bins will include the set number plus the left_outlier and the right_outlier , so the total number of bins will be the total set + 2. |
Bin Mode | Quantile | Specify the Binning Scheme. Available options are: Quantile binning defines the bins using percentile ranges (each bin holds the same percentage of the baseline data). Equal binning defines the bins using equally spaced data value ranges, like a histogram. Custom allows users to set the range of values for each bin, with the Left Outlier always starting at Min (below the minimum values detected from the baseline) and the Right Outlier always ending at Max (above the maximum values detected from the baseline). |
Bin Weight | Equally Weighted | The weight applied to each bin. The bin weights can be either set to Equally Weighted (the default) where each bin is weighted equally, or Custom where the bin weights can be adjusted depending on which are considered more important for detecting model drift. |
Manage Assays via the Wallaroo SDK
List Assays
Assays are listed through the Wallaroo Client list_assays
method.
wl.list_assays()
name | active | status | warning_threshold | alert_threshold | pipeline_name |
---|---|---|---|---|---|
api_assay | True | created | 0.0 | 0.1 | housepricepipe |
Create Assay
Assays are created with the wallaroo.client.build_assay
method.
Before creating an assay, at least one inference must be completed with the Wallaroo pipeline for the specific model the assay targets in the iopath
settings.
Create Assay Parameters
wallaroo.client.build_assay
accepts the following parameters.
window start date is the first inference data - you need inferences first. at least one.
Parameter | Type | Description |
---|---|---|
assay_name | String (Required) - required | The name of the assay. Assay names must be unique across the Wallaroo instance. |
pipeline | wallaroo.pipeline.Pipeline (Required) | The pipeline the assay is monitoring. |
model_name | String (Required) | The name of the model to monitor. |
iopath | String (Required) | The input/output data for the model being tracked in the format `input |
baseline_start | datetime.datetime (Optional) | The start time for the inferences to use as the baseline. |
baseline_end | datetime.datetime (Optional) | The end time of the baseline window. the baseline. Windows start immediately after the baseline window and are run at regular intervals continously until the assay is deactivated or deleted. |
baseline_data | numpy.array (Optional) | The baseline data in numpy array format. |
For assay baseline data one of the following must be set:
baseline_start
andbaseline_end
: These two parameters must be used together, and can not be used withbaseline_data
.baseline_data
: This option can not be used withbaseline_start
orbaseline_end
.
Create Assay Returns
Returns the wallaroo.assay_config.AssayBuilder object.
Note that object is local to the SDK session, and is not saved in the Wallaroo system until the method wallaroo.Client.upload_assay(config: wallaroo.assay_config)
is called.
Create Assay Example
The following demonstrates creating an assay for the model house_price_estimator
from pre-set baseline data stored in the file baseline_data.csv
.
# load the baseline data
baseline_data = pd.read_csv("baseline_data.csv")
# convert the output_dense_2_0 field to numpy
baseline_np_array = baseline_data["output_dense_2_0"].to_numpy()
assay_builder = client.build_assay("sample_assay",
pipeline,
"house_price_estimator",
iopath="output dense_2 0",
baseline_data=baseline_np_array)
Add Location
Assays are filtered by locations of each pipeline. By default, the Wallaroo Ops pipeline is automatically added to the assay locations. The method wallaroo.assay_config.AssayBuilder.add_locations(locations: List[string]) replaces filter to the pipeline partitions submitted.
Add Location Parameters
wallaroo.assay_config.AssayBuilder.add_location_filter(locations: List[string])
takes the following parameters.
Parameter | Type | Description |
---|---|---|
location |
List(string) | The list of pipeline partitions. For example: ["engine-5f57f86bfc-mjs8k", "engine-65b69cdd99-jmdm6"] The list of pipeline partitions is retrieved through the wallaroo.pipeline.list_edges method. The location list for pipelines is set when edges are added to a Pipeline publish. For more details, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication. |
Add Location Examples
The following example replaces the locations filter for an assay with edge pipeline deployments. For information on publishing pipelines and deploying them on edge devices, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.
assay_name = f"EdgeOnly"
from datetime import datetime
# day1="2023-10-18 01:28:45.524"
day1=datetime.fromisoformat('2023-10-27T00:13:50+00:00')
day2=datetime.fromisoformat('2023-10-27T14:00:00+00:00')
# default assay will have the Wallaroo Ops pipeline partition
assay_builder = wl.build_assay(assay_name, p1, "m1", "output dense_1 0", day1, day2)
# replace the assay_builder locations with edge devices
assay_builder.add_location_filter(["engine-5f57f86bfc-mjs8k", "engine-65b69cdd99-jmdm6"])
built = assay_builder.build()
print(built.to_json())
{ “name”: “EdgeOnly”, “pipeline_id”: 44, “pipeline_name”: “p1”, “active”: true, “status”: “created”, “baseline”: { “static”: { “count”: 500, “min”: 0.001497417688369751, “max”: 0.001497417688369751, “mean”: 0.001497417688369751, “median”: 0.001497417688369751, “std”: 0.0, “edges”: [ 0.001497417688369751, 0.001497417688369751, 0.001497417688369751, 0.001497417688369751, 0.001497417688369751, 0.001497417688369751, null ], “edge_names”: [ “left_outlier”, “q_20”, “q_40”, “q_60”, “q_80”, “q_100”, “right_outlier” ], “aggregated_values”: [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0 ], “aggregation”: “Density”, “start_at”: “2023-10-27T00:13:50+00:00”, “end_at”: “2023-10-27T14:00:00+00:00” } }, “window”: { “pipeline_name”: “p1”, “model_name”: “m1”, “width”: “24 hours”, “start”: null, “interval”: null, “path”: “output dense_1 0”, “workspace_id”: 47 }, “summarizer”: { “type”: “UnivariateContinuous”, “bin_mode”: “Quantile”, “aggregation”: “Density”, “metric”: “PSI”, “num_bins”: 5, “bin_weights”: null, “bin_width”: null, “provided_edges”: null, “add_outlier_edges”: true }, “warning_threshold”: null, “alert_threshold”: 0.25, “run_until”: null, “workspace_id”: 47 }
Interactive Baseline Runs
Once the assay is generated, an interactive baseline run gathers information from the assay baseline. This does not save the assay in the Wallaroo database; the data collected is stored only in the SDK session until the assay is uploaded.
The interactive baseline run is performed through the wallaroo.assay_config.AssayBuilder.build().interactive_baseline_run()
method that collects inference data and creates the baseline based on the parameters supplied in the build assay
process.
Interactive Baseline Run Returns
wallaroo.assay_config.AssayBuilder.build().interactive_baseline_run()
returns the wallaroo.assay object.
Interactive Baseline Run Example
The following demonstrates running an interactive baseline on a created assay.
assay_builder = client.build_assay("sample_assay",
pipeline,
"house_price_estimator",
iopath="output dense_2 0",
baseline_data=baseline_np_array)
baseline_run = assay_builder.build().interactive_baseline_run()
Baseline Statistics
The following methods display information about the baseline configurations.
Baseline Chart
The method wallaroo.assay.AssayAnalysis.chart()
displays a chart with the baseline statistics displayed on the chart and other data. This is generated after running the interactive baseline run.
assay_builder = client.build_assay("sample_assay",
pipeline,
"house_price_estimator",
iopath="output dense_2 0",
baseline_data=baseline_np_array)
baseline_run = assay_builder.build().interactive_baseline_run()
baseline_run.chart()
baseline mean = 12.940910643273655
baseline median = 12.884286880493164
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False

Baseline Stats
The method wallaroo.assay.AssayAnalysis.baseline_stats()
returns a pandas.core.frame.DataFrame
of the baseline stats.
The following example shows the baseline from the created assay.
assay_builder = client.build_assay("sample_assay",
pipeline,
"house_price_estimator",
iopath="output dense_2 0",
baseline_data=baseline_np_array)
baseline_run = assay_builder.build().interactive_baseline_run()
baseline_run.baseline_stats()
 | Baseline |
---|---|
count | 500 |
min | 236238.65625 |
max | 1489624.5 |
mean | 517129.749094 |
median | 448627.71875 |
std | 233679.013558 |
start | 2023-07-19T19:58:51.399138Z |
end | 2023-07-19T19:59:22.406972Z |
Baseline Bins
The method wallaroo.assay.AssayAnalysis.baseline_stats()
returns a pandas.core.frame.DataFrame
of the baseline binning schemes.
The following example shows the baseline bins and binning schemes from the created assay.
assay_builder = client.build_assay("sample_assay",
pipeline,
"house_price_estimator",
iopath="output dense_2 0",
baseline_data=baseline_np_array)
baseline_run = assay_builder.build().interactive_baseline_run()
baseline_run.baseline_bins()
b_edges | b_edge_names | b_aggregated_values | b_aggregation | |
---|---|---|---|---|
0 | 12.00 | left_outlier | 0.00 | Density |
1 | 12.55 | q_20 | 0.20 | Density |
2 | 12.81 | q_40 | 0.20 | Density |
3 | 12.98 | q_60 | 0.20 | Density |
4 | 13.33 | q_80 | 0.20 | Density |
5 | 14.97 | q_100 | 0.20 | Density |
6 | inf | right_outlier | 0.00 | Density |
Interactive Assay Runs
By default the assay builder creates an assay that runs a new analysis for every 24 hours starting at the end of the baseline period. We can perform an interactive run to collect the data currently available and evaluate the details.
This requires two methods be used:
wallaroo.assay_config.add_run_until(run_until: datetime.datetime)
: Sets how long the assay should run to limit the number of analyses. This is used to narrow down the number of records. Typically the datetime is set todatetime.now()
to run the assay until the current time.wallaroo.assay_config.interactive_run()
: Runs this assay interactively and returns awallaroo.assay.AssayAnalysisList
object. The assay is not saved to the Wallaroo database nor are analysis records saved to a Plateau topic. Useful for exploring pipeline inference data and experimenting with thresholds.
The following example demonstrates performing an interactive assay.
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_config = assay_builder.add_run_until(last_day).build()
assay_results = assay_config.interactive_run()
Assay Statistics
Different assay details and statistics are pulled from interactive assay run objects via the following methods.
Assay Chart Scores
Assay chart scores show the assay scores for each assay result interval. Values that are outside of the alert threshold are colored red, while scores within the alert threshold are green.
Assay chart scores are displayed with the method wallaroo.assay.AssayAnalysisList.chart_scores(title: Optional[str] = None)
, with ability to display an optional title with the chart.
The following example shows retrieving the assay results and displaying the chart scores.
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_config = assay_builder.add_run_until(last_day).build()
assay_results = assay_config.interactive_run()
assay_results.chart_scores()

Assay Results to DataFrame
Assay result values are displayed with the following methods:
wallaroo.assay.AssayAnalysisList.to_dataframe
: Returns a DataFrame with a summary of assay results.wallaroo.assay.AssayAnalysisList.to_full_dataframe
: Returns a DataFrame with all values including inputs and outputs from the assay results.
The following example demonstrates displaying a summary of assay values as a DataFrame, and displaying a subset of those values.
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_config = assay_builder.add_run_until(last_day).build()
assay_results = assay_config.interactive_run()
assay_df = assay_results.to_dataframe()
assay_df.loc[:, ~assay_df.columns.isin(['assay_id', 'iopath', 'name', 'warning_threshold'])]
score | start | min | max | mean | median | std | alert_threshold | status | |
---|---|---|---|---|---|---|---|---|---|
0 | 0.00 | 2023-01-02T00:00:00+00:00 | 12.05 | 14.71 | 12.97 | 12.90 | 0.48 | 0.25 | Ok |
1 | 0.09 | 2023-01-03T00:00:00+00:00 | 12.04 | 14.65 | 12.96 | 12.93 | 0.41 | 0.25 | Ok |
2 | 0.04 | 2023-01-04T00:00:00+00:00 | 11.87 | 14.02 | 12.98 | 12.95 | 0.46 | 0.25 | Ok |
3 | 0.06 | 2023-01-05T00:00:00+00:00 | 11.92 | 14.46 | 12.93 | 12.87 | 0.46 | 0.25 | Ok |
4 | 0.02 | 2023-01-06T00:00:00+00:00 | 12.02 | 14.15 | 12.95 | 12.90 | 0.43 | 0.25 | Ok |
5 | 0.03 | 2023-01-07T00:00:00+00:00 | 12.18 | 14.58 | 12.96 | 12.93 | 0.44 | 0.25 | Ok |
6 | 0.02 | 2023-01-08T00:00:00+00:00 | 12.01 | 14.60 | 12.92 | 12.90 | 0.46 | 0.25 | Ok |
7 | 0.04 | 2023-01-09T00:00:00+00:00 | 12.01 | 14.40 | 13.00 | 12.97 | 0.45 | 0.25 | Ok |
8 | 0.06 | 2023-01-10T00:00:00+00:00 | 11.99 | 14.79 | 12.94 | 12.91 | 0.46 | 0.25 | Ok |
9 | 0.02 | 2023-01-11T00:00:00+00:00 | 11.90 | 14.66 | 12.91 | 12.88 | 0.45 | 0.25 | Ok |
10 | 0.02 | 2023-01-12T00:00:00+00:00 | 11.96 | 14.82 | 12.94 | 12.90 | 0.46 | 0.25 | Ok |
11 | 0.03 | 2023-01-13T00:00:00+00:00 | 12.07 | 14.61 | 12.96 | 12.93 | 0.47 | 0.25 | Ok |
12 | 0.15 | 2023-01-14T00:00:00+00:00 | 12.00 | 14.20 | 13.06 | 13.03 | 0.43 | 0.25 | Ok |
13 | 2.92 | 2023-01-15T00:00:00+00:00 | 12.74 | 15.62 | 14.00 | 14.01 | 0.57 | 0.25 | Alert |
14 | 7.89 | 2023-01-16T00:00:00+00:00 | 14.64 | 17.19 | 15.91 | 15.87 | 0.63 | 0.25 | Alert |
15 | 8.87 | 2023-01-17T00:00:00+00:00 | 16.60 | 19.23 | 17.94 | 17.94 | 0.63 | 0.25 | Alert |
16 | 8.87 | 2023-01-18T00:00:00+00:00 | 18.67 | 21.29 | 20.01 | 20.04 | 0.64 | 0.25 | Alert |
17 | 8.87 | 2023-01-19T00:00:00+00:00 | 20.72 | 23.57 | 22.17 | 22.18 | 0.65 | 0.25 | Alert |
18 | 8.87 | 2023-01-20T00:00:00+00:00 | 23.04 | 25.72 | 24.32 | 24.33 | 0.66 | 0.25 | Alert |
19 | 8.87 | 2023-01-21T00:00:00+00:00 | 25.06 | 27.67 | 26.48 | 26.49 | 0.63 | 0.25 | Alert |
20 | 8.87 | 2023-01-22T00:00:00+00:00 | 27.21 | 29.89 | 28.63 | 28.58 | 0.65 | 0.25 | Alert |
21 | 8.87 | 2023-01-23T00:00:00+00:00 | 29.36 | 32.18 | 30.82 | 30.80 | 0.67 | 0.25 | Alert |
22 | 8.87 | 2023-01-24T00:00:00+00:00 | 31.56 | 34.35 | 32.98 | 32.98 | 0.65 | 0.25 | Alert |
23 | 8.87 | 2023-01-25T00:00:00+00:00 | 33.68 | 36.44 | 35.14 | 35.14 | 0.66 | 0.25 | Alert |
24 | 8.87 | 2023-01-26T00:00:00+00:00 | 35.93 | 38.51 | 37.31 | 37.33 | 0.65 | 0.25 | Alert |
25 | 3.69 | 2023-01-27T00:00:00+00:00 | 12.06 | 39.91 | 29.29 | 38.65 | 12.66 | 0.25 | Alert |
26 | 0.05 | 2023-01-28T00:00:00+00:00 | 11.87 | 13.88 | 12.92 | 12.90 | 0.38 | 0.25 | Ok |
27 | 0.10 | 2023-01-29T00:00:00+00:00 | 12.02 | 14.36 | 12.98 | 12.96 | 0.38 | 0.25 | Ok |
28 | 0.11 | 2023-01-30T00:00:00+00:00 | 11.99 | 14.44 | 12.89 | 12.88 | 0.37 | 0.25 | Ok |
29 | 0.01 | 2023-01-31T00:00:00+00:00 | 12.00 | 14.64 | 12.92 | 12.89 | 0.40 | 0.25 | Ok |
Assay Results Chart
An element of the wallaroo.assay.AssayAnalysisList
is a wallaroo.assay.AssayAnalysis
object. A chart from this object is pulled from the wallaroo.assay.AssayAnalysis.chart(show_scores=True)
.
The following demonstrates pulling a different assay results and display their charts.
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_config = assay_builder.add_run_until(last_day).build()
assay_results = assay_config.interactive_run()
assay_results[0].chart()
baseline mean = 12.940910643273655
window mean = 12.969964654406132
baseline median = 12.884286880493164
window median = 12.899214744567873
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.0029273068646199748
scores = [0.0, 0.000514261205558409, 0.0002139202456922972, 0.0012617897456473992, 0.0002139202456922972, 0.0007234154220295724, 0.0]
index = None

Other days, however are significantly different.
assay_results[12].chart()
baseline mean = 12.940910643273655
window mean = 13.06380216891949
baseline median = 12.884286880493164
window median = 13.027600288391112
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.15060511096978788
scores = [4.6637149189075455e-05, 0.05969428191167242, 0.00806617426854112, 0.008316273402678306, 0.07090885609902021, 0.003572888138686759, 0.0]
index = None

assay_results[13].chart()
baseline mean = 12.940910643273655
window mean = 14.004728427908038
baseline median = 12.884286880493164
window median = 14.009637832641602
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 2.9220486095961196
scores = [0.0, 0.7090936334784107, 0.7130482300184766, 0.33500731896676245, 0.12171058214520876, 0.9038825518183468, 0.1393062931689142]
index = None

Compare Bins
The method wallaroo.assay.AssayAnalysis.compare_bins()
returns a simple dataframe to compare the bin/edge information of baseline and window.
For example:
assay_bins = ar.compare_bins()
display(assay_bins.loc[:, assay_bins.columns!='w_aggregation'])
b_edges | b_edge_names | b_aggregated_values | b_aggregation | w_edges | w_edge_names | w_aggregated_values | diff_in_pcts | |
---|---|---|---|---|---|---|---|---|
0 | 12.00 | left_outlier | 0.00 | Density | 12.00 | left_outlier | 0.00 | 0.00 |
1 | 12.55 | q_20 | 0.20 | Density | 12.55 | e_1.26e1 | 0.19 | -0.01 |
2 | 12.81 | q_40 | 0.20 | Density | 12.81 | e_1.28e1 | 0.21 | 0.01 |
3 | 12.98 | q_60 | 0.20 | Density | 12.98 | e_1.30e1 | 0.18 | -0.02 |
4 | 13.33 | q_80 | 0.20 | Density | 13.33 | e_1.33e1 | 0.21 | 0.01 |
5 | 14.97 | q_100 | 0.20 | Density | 14.97 | e_1.50e1 | 0.21 | 0.01 |
6 | NaN | right_outlier | 0.00 | Density | NaN | right_outlier | 0.00 | 0.00 |
Compare Basic Stats
The method wallaroo.assay.AssayAnalysis.compare_basic_stats
returns a DataFrame with basic stats for the baseline and window data.
assay_results = assay_builder.build().interactive_run()
ar = assay_results[0]
ar.compare_basic_stats()
Baseline | Window | diff | pct_diff | |
---|---|---|---|---|
count | 182.00 | 181.00 | -1.00 | -0.55 |
min | 12.00 | 12.05 | 0.04 | 0.36 |
max | 14.97 | 14.71 | -0.26 | -1.71 |
mean | 12.94 | 12.97 | 0.03 | 0.22 |
median | 12.88 | 12.90 | 0.01 | 0.12 |
std | 0.45 | 0.48 | 0.03 | 5.68 |
start | 2023-01-01T00:00:00+00:00 | 2023-01-02T00:00:00+00:00 | NaN | NaN |
end | 2023-01-02T00:00:00+00:00 | 2023-01-03T00:00:00+00:00 | NaN | NaN |
Assay Configurations
By default the assay builder creates an assay with the following defaults:
- Interval: 24 hours. Every 24 hours the assay will execute.
- Window: 24 hours. The inference data collected is from the previous 24 hours to the interval execution.
- Number of bins: 5
- Binning method: Quintiles
Assay configurations are modified through the following methods.
Number of Bins
The number of bins are set with the wallaroo.assay_config.UnivariateContinousSummarizerBuilder.add_num_bins(num_bins: int)
method. This may have an effect on the binning mode operations.
The following example sets the number of bins to 10 and displays the updated results of an assay result.
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_bin_mode(BinMode.QUANTILE).add_num_bins(10)
assay_results = assay_builder.build().interactive_run()
assay_results_df = assay_results[1].compare_bins()
display(assay_results_df.loc[:, ~assay_results_df.columns.isin(['b_aggregation', 'w_aggregation'])])
assay_results[1].chart()
b_edges | b_edge_names | b_aggregated_values | w_edges | w_edge_names | w_aggregated_values | diff_in_pcts | |
---|---|---|---|---|---|---|---|
0 | 12.00 | left_outlier | 0.00 | 12.00 | left_outlier | 0.00 | 0.00 |
1 | 12.41 | q_10 | 0.10 | 12.41 | e_1.24e1 | 0.09 | -0.00 |
2 | 12.55 | q_20 | 0.10 | 12.55 | e_1.26e1 | 0.04 | -0.05 |
3 | 12.72 | q_30 | 0.10 | 12.72 | e_1.27e1 | 0.14 | 0.03 |
4 | 12.81 | q_40 | 0.10 | 12.81 | e_1.28e1 | 0.05 | -0.05 |
5 | 12.88 | q_50 | 0.10 | 12.88 | e_1.29e1 | 0.12 | 0.02 |
6 | 12.98 | q_60 | 0.10 | 12.98 | e_1.30e1 | 0.09 | -0.01 |
7 | 13.15 | q_70 | 0.10 | 13.15 | e_1.32e1 | 0.18 | 0.08 |
8 | 13.33 | q_80 | 0.10 | 13.33 | e_1.33e1 | 0.14 | 0.03 |
9 | 13.47 | q_90 | 0.10 | 13.47 | e_1.35e1 | 0.07 | -0.03 |
10 | 14.97 | q_100 | 0.10 | 14.97 | e_1.50e1 | 0.08 | -0.02 |
11 | NaN | right_outlier | 0.00 | NaN | right_outlier | 0.00 | 0.00 |
baseline mean = 12.940910643273655
window mean = 12.956829186961135
baseline median = 12.884286880493164
window median = 12.929338455200195
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.16591076620684958
scores = [0.0, 0.0002571306027792045, 0.044058279699182114, 0.009441459631493015, 0.03381618572319047, 0.0027335446937028877, 0.0011792419836838435, 0.051023062424253904, 0.009441459631493015, 0.008662563542113508, 0.0052978382749576496, 0.0]
index = None

Bin Weights
Bin weights are set with the wallaroo.assay_config.UnivariateContinousSummarizerBuilder.add_bin_weights(weights: [List[float]])
.
The following demonstrates modifying the bin weights to keep lower values outside of the score.
weights = [0] * 6
weights.extend([1] * 6)
print("Using weights: ", weights)
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_bin_mode(BinMode.QUANTILE).add_num_bins(10).add_bin_weights(weights)
assay_results = assay_builder.build().interactive_run()
assay_results_df = assay_results[1].compare_bins()
display(assay_results_df.loc[:, ~assay_results_df.columns.isin(['b_aggregation', 'w_aggregation'])])
assay_results[1].chart()
Using weights: [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
b_edges | b_edge_names | b_aggregated_values | w_edges | w_edge_names | w_aggregated_values | diff_in_pcts | |
---|---|---|---|---|---|---|---|
0 | 12.00 | left_outlier | 0.00 | 12.00 | left_outlier | 0.00 | 0.00 |
1 | 12.41 | q_10 | 0.10 | 12.41 | e_1.24e1 | 0.09 | -0.00 |
2 | 12.55 | q_20 | 0.10 | 12.55 | e_1.26e1 | 0.04 | -0.05 |
3 | 12.72 | q_30 | 0.10 | 12.72 | e_1.27e1 | 0.14 | 0.03 |
4 | 12.81 | q_40 | 0.10 | 12.81 | e_1.28e1 | 0.05 | -0.05 |
5 | 12.88 | q_50 | 0.10 | 12.88 | e_1.29e1 | 0.12 | 0.02 |
6 | 12.98 | q_60 | 0.10 | 12.98 | e_1.30e1 | 0.09 | -0.01 |
7 | 13.15 | q_70 | 0.10 | 13.15 | e_1.32e1 | 0.18 | 0.08 |
8 | 13.33 | q_80 | 0.10 | 13.33 | e_1.33e1 | 0.14 | 0.03 |
9 | 13.47 | q_90 | 0.10 | 13.47 | e_1.35e1 | 0.07 | -0.03 |
10 | 14.97 | q_100 | 0.10 | 14.97 | e_1.50e1 | 0.08 | -0.02 |
11 | NaN | right_outlier | 0.00 | NaN | right_outlier | 0.00 | 0.00 |
baseline mean = 12.940910643273655
window mean = 12.956829186961135
baseline median = 12.884286880493164
window median = 12.929338455200195
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = True
score = 0.012600694309416988
scores = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00019654033061397393, 0.00850384373737565, 0.0015735766052488358, 0.0014437605903522511, 0.000882973045826275, 0.0]
index = None

Binning Modes
Binning mores are modified through the wallaroo.assay_config.UnivariateContinousSummarizerBuilder.add_bin_mode(bin_mode: bin_mode: wallaroo.assay_config.BinMode, edges: Optional[List[float]] = None)
.
bin_mode
values are the following:
EQUAL
: Evenly spaced bins where each bin is set with the formulamin - max / num_bins
QUANTILE
: Based on percentages. Ifnum_bins
is 5 then quintiles so bins are created at the 20%, 40%, 60%, 80% and 100% points.PROVIDED
: The user provides the edge points for the bins.
If PROVIDED
is supplies, then a List of float values must be provided for the edges
parameter that matches the number of bins.
The previous assay examples used quintiles so all of the bins had the same percentage/count of samples.
The following example uses BinMode.EQUAL
, then displays the baseline bins created and charts resulting from that binning mode.
equal_bin_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
equal_bin_builder.summarizer_builder.add_bin_mode(BinMode.EQUAL)
equal_baseline = equal_bin_builder.build().interactive_baseline_run()
equal_baseline.chart()
baseline mean = 12.940910643273655
baseline median = 12.884286880493164
bin_mode = Equal
aggregation = Density
metric = PSI
weighted = False

equal_baseline.baseline_bins()
b_edges | b_edge_names | b_aggregated_values | b_aggregation | |
---|---|---|---|---|
0 | 12.00 | left_outlier | 0.00 | Density |
1 | 12.60 | p_1.26e1 | 0.24 | Density |
2 | 13.19 | p_1.32e1 | 0.49 | Density |
3 | 13.78 | p_1.38e1 | 0.22 | Density |
4 | 14.38 | p_1.44e1 | 0.04 | Density |
5 | 14.97 | p_1.50e1 | 0.01 | Density |
6 | inf | right_outlier | 0.00 | Density |
assay_results.chart_scores()

The following example manually sets the bin values.
The values in this dataset run from ~11.6 to ~15.81. And lets say we had a business reason to use specific bin edges. We can specify them with the BinMode.PROVIDED
and specifying a list of floats with the right hand / upper edge of each bin and optionally the lower edge of the smallest bin. If the lowest edge is not specified the threshold for left outliers is taken from the smallest value in the baseline dataset.
edges = [11.0, 12.0, 13.0, 14.0, 15.0, 16.0]
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_bin_mode(BinMode.PROVIDED, edges)
assay_results = assay_builder.build().interactive_run()
assay_results_df = assay_results[0].compare_bins()
display(assay_results_df.loc[:, ~assay_results_df.columns.isin(['b_aggregation', 'w_aggregation'])])
assay_results[0].chart()
b_edges | b_edge_names | b_aggregated_values | w_edges | w_edge_names | w_aggregated_values | diff_in_pcts | |
---|---|---|---|---|---|---|---|
0 | 11.00 | left_outlier | 0.00 | 11.00 | left_outlier | 0.00 | 0.00 |
1 | 12.00 | e_1.20e1 | 0.00 | 12.00 | e_1.20e1 | 0.00 | 0.00 |
2 | 13.00 | e_1.30e1 | 0.62 | 13.00 | e_1.30e1 | 0.59 | -0.03 |
3 | 14.00 | e_1.40e1 | 0.36 | 14.00 | e_1.40e1 | 0.35 | -0.00 |
4 | 15.00 | e_1.50e1 | 0.02 | 15.00 | e_1.50e1 | 0.06 | 0.03 |
5 | 16.00 | e_1.60e1 | 0.00 | 16.00 | e_1.60e1 | 0.00 | 0.00 |
6 | NaN | right_outlier | 0.00 | NaN | right_outlier | 0.00 | 0.00 |
baseline mean = 12.940910643273655
window mean = 12.969964654406132
baseline median = 12.884286880493164
window median = 12.899214744567873
bin_mode = Provided
aggregation = Density
metric = PSI
weighted = False
score = 0.0321620386600679
scores = [0.0, 0.0, 0.0014576920813015586, 3.549754401142936e-05, 0.030668849034754912, 0.0, 0.0]
index = None

Add Alert Threshold
Assay alert thresholds are modified with the wallaroo.assay_config.AssayBuilder.add_alert_threshold(alert_threshold: float)
method. By default alert thresholds are 0.1
.
The following example updates the alert threshold to 0.5
:
alert_threshold = 0.5
import string
import random
assay_name = "example assay"
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_alert_threshold(alert_threshold)
Metrics
The score
is a distance or dis-similarity measure. The larger it is the less similar the two distributions are. The following methods are provided determining the score:
PSI
(Default) - Population Stability Index (PSI).MAXDIFF
: Maximum difference between corresponding bins.SUMDIFF
: Mum of differences between corresponding bins.
The metric type used is updated with the wallaroo.assay_config.AssayBuilder.add_metric(metric: wallaroo.assay_config.Metric)
method.
The following three charts use each of the metrics. Note how the scores change based on the score type used.
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_results = assay_builder.build().interactive_run()
assay_results[0].chart() # PSI Score
baseline mean = 12.940910643273655
window mean = 12.969964654406132
baseline median = 12.884286880493164
window median = 12.899214744567873
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.0029273068646199748
scores = [0.0, 0.000514261205558409, 0.0002139202456922972, 0.0012617897456473992, 0.0002139202456922972, 0.0007234154220295724, 0.0]
index = None

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_metric(Metric.SUMDIFF)
assay_results = assay_builder.build().interactive_run()
assay_results[0].chart()
baseline mean = 12.940910643273655
window mean = 12.969964654406132
baseline median = 12.884286880493164
window median = 12.899214744567873
bin_mode = Quantile
aggregation = Density
metric = SumDiff
weighted = False
score = 0.025438649748041997
scores = [0.0, 0.009956893934794486, 0.006648048084512165, 0.01548175581324751, 0.006648048084512165, 0.012142553579017668, 0.0]
index = None

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_metric(Metric.MAXDIFF)
assay_results = assay_builder.build().interactive_run()
assay_results[0].chart()
baseline mean = 12.940910643273655
window mean = 12.969964654406132
baseline median = 12.884286880493164
window median = 12.899214744567873
bin_mode = Quantile
aggregation = Density
metric = MaxDiff
weighted = False
score = 0.01548175581324751
scores = [0.0, 0.009956893934794486, 0.006648048084512165, 0.01548175581324751, 0.006648048084512165, 0.012142553579017668, 0.0]
index = 3

Aggregation Options
Assay aggregation options are modified with the wallaroo.assay_config.AssayBuilder.add_aggregation(aggregation: wallaroo.assay_config.Aggregation)
method. The following options are provided:
Aggregation.DENSITY
(Default): Count the number/percentage of values that fall in each bin.Aggregation.CUMULATIVE
: Empirical Cumulative Density Function style, which keeps a cumulative count of the values/percentages that fall in each bin.
The following example demonstrate the different results between the two.
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_aggregation(Aggregation.DENSITY)
assay_results = assay_builder.build().interactive_run()
assay_results[0].chart()
baseline mean = 12.940910643273655
window mean = 12.969964654406132
baseline median = 12.884286880493164
window median = 12.899214744567873
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.0029273068646199748
scores = [0.0, 0.000514261205558409, 0.0002139202456922972, 0.0012617897456473992, 0.0002139202456922972, 0.0007234154220295724, 0.0]
index = None

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_aggregation(Aggregation.CUMULATIVE)
assay_results = assay_builder.build().interactive_run()
assay_results[0].chart()
baseline mean = 12.940910643273655
window mean = 12.969964654406132
baseline median = 12.884286880493164
window median = 12.899214744567873
bin_mode = Quantile
aggregation = Cumulative
metric = PSI
weighted = False
score = 0.04419889502762442
scores = [0.0, 0.009956893934794486, 0.0033088458502823492, 0.01879060166352986, 0.012142553579017725, 0.0, 0.0]
index = None

Scheduling Assays
By default assays are scheduled to run every 24 hours starting immediately after the baseline period ends. This is referred to as the window which has two settings:
width
(Default:hours=24
): The time period to measure from the current assay run. For example:hours=12
would accumulate the previous 12 hours of inferences from the assay run.interval
(Default:hours=24
): The time period between each asay run.
This is modified by the wallaroo.assay_config.WindowBuilder.add_width(**kwargs: int)
and wallaroo.assay_config.WindowBuilder.add_interval(**kwargs: int)
methods, where kwargs
are:
weeks
days
hours
minutes
To recap:
- The window width is the size of the window. The default is 24 hours.
- The interval is how often the analysis is run, how far the window is slid into the future based on the last run. The default is the window width.
- The window start is when the analysis should start. The default is the end of the baseline period if the baseline
baseline_start
andbaseline_end
are set. Otherwise, it is from the first inference performed.
The following example sets the assay analysis window to execute every 12 hours (the interval) on the previous 24 hours of data (the width)
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_builder = assay_builder.add_run_until(last_day)
assay_builder.window_builder().add_width(hours=24).add_interval(hours=12)
assay_config = assay_builder.build()
assay_results = assay_config.interactive_run()
print(f"Generated {len(assay_results)} analyses")
Generated 59 analyses
assay_results.chart_scores()

The following example adapts the width and interval to 1 week at a specified start date.
report_start = datetime.datetime.fromisoformat('2022-01-03T00:00:00+00:00')
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_builder = assay_builder.add_run_until(last_day)
assay_builder.window_builder().add_width(weeks=1).add_interval(weeks=1).add_start(report_start)
assay_config = assay_builder.build()
assay_results = assay_config.interactive_run()
print(f"Generated {len(assay_results)} analyses")
Generated 5 analyses
assay_results.chart_scores()

Upload Assay
Assays are uploaded with the wallaroo.assay_config.upload()
method. This uploads the assay into the Wallaroo database with the configurations applied. Note that assay names must be unique across the Wallaroo instance; attempting to upload an assay with the same name as an existing one will return an error.
This returns the assay id as an int.
For example:
# create a fresh assay builder with the correct parameters
assay_builder = ( wl.build_assay(assay_name, pipeline, model_name,
baseline_start, baseline_end)
.add_iopath("output variable 0") )
# this assay runs every 24 hours on a 24 hour window
assay_builder.window_builder().add_width(hours=24)
assay_builder.add_alert_threshold(0.1)
# now schedule the assay
assay_id = assay_builder.upload()
5