Wallaroo SDK Essentials Guide: Assays Management

How to create and manage Wallaroo Assays through the Wallaroo SDK

Model Insights and Interactive Analysis Introduction

Wallaroo provides the ability to perform interactive analysis so organizations can explore the data from a pipeline and learn how the data is behaving. With this information and the knowledge of your particular business use case you can then choose appropriate thresholds for persistent automatic assays as desired.

  • IMPORTANT NOTE

    Model insights operates over time and is difficult to demo in a notebook without pre-canned data. We assume you have an active pipeline that has been running and making predictions over time and show you the code you may use to analyze your pipeline.

Monitoring tasks called assays monitors a model’s predictions or the data coming into the model against an established baseline. Changes in the distribution of this data can be an indication of model drift, or of a change in the environment that the model trained for. This can provide tips on whether a model needs to be retrained or the environment data analyzed for accuracy or other needs.

Assay Details

Assays contain the following attributes:

Attribute Default Description
Name   The name of the assay. Assay names must be unique.
Baseline Data   Data that is known to be “typical” (typically distributed) and can be used to determine whether the distribution of new data has changed.
Schedule Every 24 hours at 1 AM Configure the start time and frequency of when the new analysis will run. New assays are configured to run a new analysis for every 24 hours starting at the end of the baseline period. This period can be configured through the SDK.
Group Results Daily How the results are grouped: Daily (Default), Every Minute, Weekly, or Monthly.
Metric PSI Population Stability Index (PSI) is an entropy-based measure of the difference between distributions. Maximum Difference of Bins measures the maximum difference between the baseline and current distributions (as estimated using the bins). Sum of the difference of bins sums up the difference of occurrences in each bin between the baseline and current distributions.
Threshold 0.1 Threshold for deciding the difference between distributions is similar(small) or different(large), as evaluated by the metric. The default of 0.1 is generally a good threshold when using PSI as the metric.
Number of Bins 5 Number of bins used to partition the baseline data. By default, the binning scheme is percentile (quantile) based. The binning scheme can be configured (see Bin Mode, below). Note that the total number of bins will include the set number plus the left_outlier and the right_outlier, so the total number of bins will be the total set + 2.
Bin Mode Quantile Specify the Binning Scheme. Available options are: Quantile binning defines the bins using percentile ranges (each bin holds the same percentage of the baseline data). Equal binning defines the bins using equally spaced data value ranges, like a histogram. Custom allows users to set the range of values for each bin, with the Left Outlier always starting at Min (below the minimum values detected from the baseline) and the Right Outlier always ending at Max (above the maximum values detected from the baseline).
Bin Weight Equally Weighted The weight applied to each bin. The bin weights can be either set to Equally Weighted (the default) where each bin is weighted equally, or Custom where the bin weights can be adjusted depending on which are considered more important for detecting model drift.

Manage Assays via the Wallaroo SDK

List Assays

Assays are listed through the Wallaroo Client list_assays method.

wl.list_assays()
nameactivestatuswarning_thresholdalert_thresholdpipeline_name
api_assayTruecreated0.00.1housepricepipe

Create Assay

Assays are created with the wallaroo.client.build_assay method.

Before creating an assay, at least one inference must be completed with the Wallaroo pipeline for the specific model the assay targets in the iopath settings.

Create Assay Parameters

wallaroo.client.build_assay accepts the following parameters.

window start date is the first inference data - you need inferences first. at least one.

Parameter Type Description
assay_name String (Required) - required The name of the assay. Assay names must be unique across the Wallaroo instance.
pipeline wallaroo.pipeline.Pipeline (Required) The pipeline the assay is monitoring.
model_name String (Required) The name of the model to monitor.
iopath String (Required) The input/output data for the model being tracked in the format `input
baseline_start datetime.datetime (Optional) The start time for the inferences to use as the baseline.
baseline_end datetime.datetime (Optional) The end time of the baseline window. the baseline. Windows start immediately after the baseline window and are run at regular intervals continously until the assay is deactivated or deleted.
baseline_data numpy.array (Optional) The baseline data in numpy array format.

For assay baseline data one of the following must be set:

  • baseline_start and baseline_end: These two parameters must be used together, and can not be used with baseline_data.
  • baseline_data: This option can not be used with baseline_start or baseline_end.

Create Assay Returns

Returns the wallaroo.assay_config.AssayBuilder object.

Note that object is local to the SDK session, and is not saved in the Wallaroo system until the method wallaroo.Client.upload_assay(config: wallaroo.assay_config) is called.

Create Assay Example

The following demonstrates creating an assay for the model house_price_estimator from pre-set baseline data stored in the file baseline_data.csv.

# load the baseline data
baseline_data = pd.read_csv("baseline_data.csv")
# convert the output_dense_2_0 field to numpy
baseline_np_array = baseline_data["output_dense_2_0"].to_numpy()

assay_builder = client.build_assay("sample_assay", 
                                  pipeline, 
                                  "house_price_estimator", 
                                  iopath="output dense_2 0", 
                                  baseline_data=baseline_np_array)

Add Location

Assays are filtered by locations of each pipeline. By default, the Wallaroo Ops pipeline is automatically added to the assay locations. The method wallaroo.assay_config.AssayBuilder.add_locations(locations: List[string]) replaces filter to the pipeline partitions submitted.

Add Location Parameters

wallaroo.assay_config.AssayBuilder.add_location_filter(locations: List[string]) takes the following parameters.

Parameter Type Description
location List(string) The list of pipeline partitions. For example: ["engine-5f57f86bfc-mjs8k", "engine-65b69cdd99-jmdm6"] The list of pipeline partitions is retrieved through the wallaroo.pipeline.list_edges method. The location list for pipelines is set when edges are added to a Pipeline publish. For more details, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.

Add Location Examples

The following example replaces the locations filter for an assay with edge pipeline deployments. For information on publishing pipelines and deploying them on edge devices, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.

assay_name = f"EdgeOnly"
from datetime import datetime
# day1="2023-10-18 01:28:45.524"
day1=datetime.fromisoformat('2023-10-27T00:13:50+00:00')
day2=datetime.fromisoformat('2023-10-27T14:00:00+00:00')

# default assay will have the Wallaroo Ops pipeline partition
assay_builder = wl.build_assay(assay_name, p1, "m1", "output dense_1 0", day1, day2)
# replace the assay_builder locations with edge devices
assay_builder.add_location_filter(["engine-5f57f86bfc-mjs8k", "engine-65b69cdd99-jmdm6"])
built = assay_builder.build()
print(built.to_json())

{ “name”: “EdgeOnly”, “pipeline_id”: 44, “pipeline_name”: “p1”, “active”: true, “status”: “created”, “baseline”: { “static”: { “count”: 500, “min”: 0.001497417688369751, “max”: 0.001497417688369751, “mean”: 0.001497417688369751, “median”: 0.001497417688369751, “std”: 0.0, “edges”: [ 0.001497417688369751, 0.001497417688369751, 0.001497417688369751, 0.001497417688369751, 0.001497417688369751, 0.001497417688369751, null ], “edge_names”: [ “left_outlier”, “q_20”, “q_40”, “q_60”, “q_80”, “q_100”, “right_outlier” ], “aggregated_values”: [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0 ], “aggregation”: “Density”, “start_at”: “2023-10-27T00:13:50+00:00”, “end_at”: “2023-10-27T14:00:00+00:00” } }, “window”: { “pipeline_name”: “p1”, “model_name”: “m1”, “width”: “24 hours”, “start”: null, “interval”: null, “path”: “output dense_1 0”, “workspace_id”: 47 }, “summarizer”: { “type”: “UnivariateContinuous”, “bin_mode”: “Quantile”, “aggregation”: “Density”, “metric”: “PSI”, “num_bins”: 5, “bin_weights”: null, “bin_width”: null, “provided_edges”: null, “add_outlier_edges”: true }, “warning_threshold”: null, “alert_threshold”: 0.25, “run_until”: null, “workspace_id”: 47 }

Interactive Baseline Runs

Once the assay is generated, an interactive baseline run gathers information from the assay baseline. This does not save the assay in the Wallaroo database; the data collected is stored only in the SDK session until the assay is uploaded.

The interactive baseline run is performed through the wallaroo.assay_config.AssayBuilder.build().interactive_baseline_run() method that collects inference data and creates the baseline based on the parameters supplied in the build assay process.

Interactive Baseline Run Returns

wallaroo.assay_config.AssayBuilder.build().interactive_baseline_run() returns the wallaroo.assay object.

Interactive Baseline Run Example

The following demonstrates running an interactive baseline on a created assay.

assay_builder = client.build_assay("sample_assay", 
                                  pipeline, 
                                  "house_price_estimator", 
                                  iopath="output dense_2 0", 
                                  baseline_data=baseline_np_array)
baseline_run = assay_builder.build().interactive_baseline_run()

Baseline Statistics

The following methods display information about the baseline configurations.

Baseline Chart

The method wallaroo.assay.AssayAnalysis.chart() displays a chart with the baseline statistics displayed on the chart and other data. This is generated after running the interactive baseline run.

assay_builder = client.build_assay("sample_assay", 
                                  pipeline, 
                                  "house_price_estimator", 
                                  iopath="output dense_2 0", 
                                  baseline_data=baseline_np_array)
baseline_run = assay_builder.build().interactive_baseline_run()
baseline_run.chart()

baseline mean = 12.940910643273655
baseline median = 12.884286880493164
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False

Baseline Stats

The method wallaroo.assay.AssayAnalysis.baseline_stats() returns a pandas.core.frame.DataFrame of the baseline stats.

The following example shows the baseline from the created assay.

assay_builder = client.build_assay("sample_assay", 
                                  pipeline, 
                                  "house_price_estimator", 
                                  iopath="output dense_2 0", 
                                  baseline_data=baseline_np_array)
baseline_run = assay_builder.build().interactive_baseline_run()
baseline_run.baseline_stats()
  Baseline
count 500
min 236238.65625
max 1489624.5
mean 517129.749094
median 448627.71875
std 233679.013558
start 2023-07-19T19:58:51.399138Z
end 2023-07-19T19:59:22.406972Z

Baseline Bins

The method wallaroo.assay.AssayAnalysis.baseline_stats() returns a pandas.core.frame.DataFrame of the baseline binning schemes.

The following example shows the baseline bins and binning schemes from the created assay.

assay_builder = client.build_assay("sample_assay", 
                                  pipeline, 
                                  "house_price_estimator", 
                                  iopath="output dense_2 0", 
                                  baseline_data=baseline_np_array)
baseline_run = assay_builder.build().interactive_baseline_run()

baseline_run.baseline_bins()
b_edges b_edge_names b_aggregated_values b_aggregation
0 12.00 left_outlier 0.00 Density
1 12.55 q_20 0.20 Density
2 12.81 q_40 0.20 Density
3 12.98 q_60 0.20 Density
4 13.33 q_80 0.20 Density
5 14.97 q_100 0.20 Density
6 inf right_outlier 0.00 Density

Interactive Assay Runs

By default the assay builder creates an assay that runs a new analysis for every 24 hours starting at the end of the baseline period. We can perform an interactive run to collect the data currently available and evaluate the details.

This requires two methods be used:

  • wallaroo.assay_config.add_run_until(run_until: datetime.datetime): Sets how long the assay should run to limit the number of analyses. This is used to narrow down the number of records. Typically the datetime is set to datetime.now() to run the assay until the current time.
  • wallaroo.assay_config.interactive_run(): Runs this assay interactively and returns a wallaroo.assay.AssayAnalysisList object. The assay is not saved to the Wallaroo database nor are analysis records saved to a Plateau topic. Useful for exploring pipeline inference data and experimenting with thresholds.

The following example demonstrates performing an interactive assay.

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_config = assay_builder.add_run_until(last_day).build()
assay_results = assay_config.interactive_run()

Assay Statistics

Different assay details and statistics are pulled from interactive assay run objects via the following methods.

Assay Chart Scores

Assay chart scores show the assay scores for each assay result interval. Values that are outside of the alert threshold are colored red, while scores within the alert threshold are green.

Assay chart scores are displayed with the method wallaroo.assay.AssayAnalysisList.chart_scores(title: Optional[str] = None), with ability to display an optional title with the chart.

The following example shows retrieving the assay results and displaying the chart scores.

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_config = assay_builder.add_run_until(last_day).build()
assay_results = assay_config.interactive_run()

assay_results.chart_scores()

Assay Results to DataFrame

Assay result values are displayed with the following methods:

The following example demonstrates displaying a summary of assay values as a DataFrame, and displaying a subset of those values.

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_config = assay_builder.add_run_until(last_day).build()
assay_results = assay_config.interactive_run()

assay_df = assay_results.to_dataframe()
assay_df.loc[:, ~assay_df.columns.isin(['assay_id', 'iopath', 'name', 'warning_threshold'])]
score start min max mean median std alert_threshold status
0 0.00 2023-01-02T00:00:00+00:00 12.05 14.71 12.97 12.90 0.48 0.25 Ok
1 0.09 2023-01-03T00:00:00+00:00 12.04 14.65 12.96 12.93 0.41 0.25 Ok
2 0.04 2023-01-04T00:00:00+00:00 11.87 14.02 12.98 12.95 0.46 0.25 Ok
3 0.06 2023-01-05T00:00:00+00:00 11.92 14.46 12.93 12.87 0.46 0.25 Ok
4 0.02 2023-01-06T00:00:00+00:00 12.02 14.15 12.95 12.90 0.43 0.25 Ok
5 0.03 2023-01-07T00:00:00+00:00 12.18 14.58 12.96 12.93 0.44 0.25 Ok
6 0.02 2023-01-08T00:00:00+00:00 12.01 14.60 12.92 12.90 0.46 0.25 Ok
7 0.04 2023-01-09T00:00:00+00:00 12.01 14.40 13.00 12.97 0.45 0.25 Ok
8 0.06 2023-01-10T00:00:00+00:00 11.99 14.79 12.94 12.91 0.46 0.25 Ok
9 0.02 2023-01-11T00:00:00+00:00 11.90 14.66 12.91 12.88 0.45 0.25 Ok
10 0.02 2023-01-12T00:00:00+00:00 11.96 14.82 12.94 12.90 0.46 0.25 Ok
11 0.03 2023-01-13T00:00:00+00:00 12.07 14.61 12.96 12.93 0.47 0.25 Ok
12 0.15 2023-01-14T00:00:00+00:00 12.00 14.20 13.06 13.03 0.43 0.25 Ok
13 2.92 2023-01-15T00:00:00+00:00 12.74 15.62 14.00 14.01 0.57 0.25 Alert
14 7.89 2023-01-16T00:00:00+00:00 14.64 17.19 15.91 15.87 0.63 0.25 Alert
15 8.87 2023-01-17T00:00:00+00:00 16.60 19.23 17.94 17.94 0.63 0.25 Alert
16 8.87 2023-01-18T00:00:00+00:00 18.67 21.29 20.01 20.04 0.64 0.25 Alert
17 8.87 2023-01-19T00:00:00+00:00 20.72 23.57 22.17 22.18 0.65 0.25 Alert
18 8.87 2023-01-20T00:00:00+00:00 23.04 25.72 24.32 24.33 0.66 0.25 Alert
19 8.87 2023-01-21T00:00:00+00:00 25.06 27.67 26.48 26.49 0.63 0.25 Alert
20 8.87 2023-01-22T00:00:00+00:00 27.21 29.89 28.63 28.58 0.65 0.25 Alert
21 8.87 2023-01-23T00:00:00+00:00 29.36 32.18 30.82 30.80 0.67 0.25 Alert
22 8.87 2023-01-24T00:00:00+00:00 31.56 34.35 32.98 32.98 0.65 0.25 Alert
23 8.87 2023-01-25T00:00:00+00:00 33.68 36.44 35.14 35.14 0.66 0.25 Alert
24 8.87 2023-01-26T00:00:00+00:00 35.93 38.51 37.31 37.33 0.65 0.25 Alert
25 3.69 2023-01-27T00:00:00+00:00 12.06 39.91 29.29 38.65 12.66 0.25 Alert
26 0.05 2023-01-28T00:00:00+00:00 11.87 13.88 12.92 12.90 0.38 0.25 Ok
27 0.10 2023-01-29T00:00:00+00:00 12.02 14.36 12.98 12.96 0.38 0.25 Ok
28 0.11 2023-01-30T00:00:00+00:00 11.99 14.44 12.89 12.88 0.37 0.25 Ok
29 0.01 2023-01-31T00:00:00+00:00 12.00 14.64 12.92 12.89 0.40 0.25 Ok

Assay Results Chart

An element of the wallaroo.assay.AssayAnalysisList is a wallaroo.assay.AssayAnalysis object. A chart from this object is pulled from the wallaroo.assay.AssayAnalysis.chart(show_scores=True).

The following demonstrates pulling a different assay results and display their charts.

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_config = assay_builder.add_run_until(last_day).build()
assay_results = assay_config.interactive_run()

assay_results[0].chart()

    baseline mean = 12.940910643273655
    window mean = 12.969964654406132
    baseline median = 12.884286880493164
    window median = 12.899214744567873
    bin_mode = Quantile
    aggregation = Density
    metric = PSI
    weighted = False
    score = 0.0029273068646199748
    scores = [0.0, 0.000514261205558409, 0.0002139202456922972, 0.0012617897456473992, 0.0002139202456922972, 0.0007234154220295724, 0.0]
    index = None

Other days, however are significantly different.

assay_results[12].chart()

    baseline mean = 12.940910643273655
    window mean = 13.06380216891949
    baseline median = 12.884286880493164
    window median = 13.027600288391112
    bin_mode = Quantile
    aggregation = Density
    metric = PSI
    weighted = False
    score = 0.15060511096978788
    scores = [4.6637149189075455e-05, 0.05969428191167242, 0.00806617426854112, 0.008316273402678306, 0.07090885609902021, 0.003572888138686759, 0.0]
    index = None

assay_results[13].chart()

    baseline mean = 12.940910643273655
    window mean = 14.004728427908038
    baseline median = 12.884286880493164
    window median = 14.009637832641602
    bin_mode = Quantile
    aggregation = Density
    metric = PSI
    weighted = False
    score = 2.9220486095961196
    scores = [0.0, 0.7090936334784107, 0.7130482300184766, 0.33500731896676245, 0.12171058214520876, 0.9038825518183468, 0.1393062931689142]
    index = None

Compare Bins

The method wallaroo.assay.AssayAnalysis.compare_bins() returns a simple dataframe to compare the bin/edge information of baseline and window.

For example:

assay_bins = ar.compare_bins()
display(assay_bins.loc[:, assay_bins.columns!='w_aggregation'])
b_edges b_edge_names b_aggregated_values b_aggregation w_edges w_edge_names w_aggregated_values diff_in_pcts
0 12.00 left_outlier 0.00 Density 12.00 left_outlier 0.00 0.00
1 12.55 q_20 0.20 Density 12.55 e_1.26e1 0.19 -0.01
2 12.81 q_40 0.20 Density 12.81 e_1.28e1 0.21 0.01
3 12.98 q_60 0.20 Density 12.98 e_1.30e1 0.18 -0.02
4 13.33 q_80 0.20 Density 13.33 e_1.33e1 0.21 0.01
5 14.97 q_100 0.20 Density 14.97 e_1.50e1 0.21 0.01
6 NaN right_outlier 0.00 Density NaN right_outlier 0.00 0.00

Compare Basic Stats

The method wallaroo.assay.AssayAnalysis.compare_basic_stats returns a DataFrame with basic stats for the baseline and window data.

assay_results = assay_builder.build().interactive_run()
ar = assay_results[0]

ar.compare_basic_stats()
Baseline Window diff pct_diff
count 182.00 181.00 -1.00 -0.55
min 12.00 12.05 0.04 0.36
max 14.97 14.71 -0.26 -1.71
mean 12.94 12.97 0.03 0.22
median 12.88 12.90 0.01 0.12
std 0.45 0.48 0.03 5.68
start 2023-01-01T00:00:00+00:00 2023-01-02T00:00:00+00:00 NaN NaN
end 2023-01-02T00:00:00+00:00 2023-01-03T00:00:00+00:00 NaN NaN

Assay Configurations

By default the assay builder creates an assay with the following defaults:

  • Interval: 24 hours. Every 24 hours the assay will execute.
  • Window: 24 hours. The inference data collected is from the previous 24 hours to the interval execution.
  • Number of bins: 5
  • Binning method: Quintiles

Assay configurations are modified through the following methods.

Number of Bins

The number of bins are set with the wallaroo.assay_config.UnivariateContinousSummarizerBuilder.add_num_bins(num_bins: int) method. This may have an effect on the binning mode operations.

The following example sets the number of bins to 10 and displays the updated results of an assay result.

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_bin_mode(BinMode.QUANTILE).add_num_bins(10)
assay_results = assay_builder.build().interactive_run()
assay_results_df = assay_results[1].compare_bins()
display(assay_results_df.loc[:, ~assay_results_df.columns.isin(['b_aggregation', 'w_aggregation'])])
assay_results[1].chart()
b_edges b_edge_names b_aggregated_values w_edges w_edge_names w_aggregated_values diff_in_pcts
0 12.00 left_outlier 0.00 12.00 left_outlier 0.00 0.00
1 12.41 q_10 0.10 12.41 e_1.24e1 0.09 -0.00
2 12.55 q_20 0.10 12.55 e_1.26e1 0.04 -0.05
3 12.72 q_30 0.10 12.72 e_1.27e1 0.14 0.03
4 12.81 q_40 0.10 12.81 e_1.28e1 0.05 -0.05
5 12.88 q_50 0.10 12.88 e_1.29e1 0.12 0.02
6 12.98 q_60 0.10 12.98 e_1.30e1 0.09 -0.01
7 13.15 q_70 0.10 13.15 e_1.32e1 0.18 0.08
8 13.33 q_80 0.10 13.33 e_1.33e1 0.14 0.03
9 13.47 q_90 0.10 13.47 e_1.35e1 0.07 -0.03
10 14.97 q_100 0.10 14.97 e_1.50e1 0.08 -0.02
11 NaN right_outlier 0.00 NaN right_outlier 0.00 0.00 
   baseline mean = 12.940910643273655
   window mean = 12.956829186961135
   baseline median = 12.884286880493164
   window median = 12.929338455200195
   bin_mode = Quantile
   aggregation = Density
   metric = PSI
   weighted = False
   score = 0.16591076620684958
   scores = [0.0, 0.0002571306027792045, 0.044058279699182114, 0.009441459631493015, 0.03381618572319047, 0.0027335446937028877, 0.0011792419836838435, 0.051023062424253904, 0.009441459631493015, 0.008662563542113508, 0.0052978382749576496, 0.0]
   index = None

Bin Weights

Bin weights are set with the wallaroo.assay_config.UnivariateContinousSummarizerBuilder.add_bin_weights(weights: [List[float]]).

The following demonstrates modifying the bin weights to keep lower values outside of the score.

weights = [0] * 6
weights.extend([1] * 6)
print("Using weights: ", weights)
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_bin_mode(BinMode.QUANTILE).add_num_bins(10).add_bin_weights(weights)
assay_results = assay_builder.build().interactive_run()
assay_results_df = assay_results[1].compare_bins()
display(assay_results_df.loc[:, ~assay_results_df.columns.isin(['b_aggregation', 'w_aggregation'])])
assay_results[1].chart()

    Using weights:  [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
b_edges b_edge_names b_aggregated_values w_edges w_edge_names w_aggregated_values diff_in_pcts
0 12.00 left_outlier 0.00 12.00 left_outlier 0.00 0.00
1 12.41 q_10 0.10 12.41 e_1.24e1 0.09 -0.00
2 12.55 q_20 0.10 12.55 e_1.26e1 0.04 -0.05
3 12.72 q_30 0.10 12.72 e_1.27e1 0.14 0.03
4 12.81 q_40 0.10 12.81 e_1.28e1 0.05 -0.05
5 12.88 q_50 0.10 12.88 e_1.29e1 0.12 0.02
6 12.98 q_60 0.10 12.98 e_1.30e1 0.09 -0.01
7 13.15 q_70 0.10 13.15 e_1.32e1 0.18 0.08
8 13.33 q_80 0.10 13.33 e_1.33e1 0.14 0.03
9 13.47 q_90 0.10 13.47 e_1.35e1 0.07 -0.03
10 14.97 q_100 0.10 14.97 e_1.50e1 0.08 -0.02
11 NaN right_outlier 0.00 NaN right_outlier 0.00 0.00 
    baseline mean = 12.940910643273655
    window mean = 12.956829186961135
    baseline median = 12.884286880493164
    window median = 12.929338455200195
    bin_mode = Quantile
    aggregation = Density
    metric = PSI
    weighted = True
    score = 0.012600694309416988
    scores = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00019654033061397393, 0.00850384373737565, 0.0015735766052488358, 0.0014437605903522511, 0.000882973045826275, 0.0]
    index = None

Binning Modes

Binning mores are modified through the wallaroo.assay_config.UnivariateContinousSummarizerBuilder.add_bin_mode(bin_mode: bin_mode: wallaroo.assay_config.BinMode, edges: Optional[List[float]] = None).

bin_mode values are the following:

  • EQUAL: Evenly spaced bins where each bin is set with the formula min - max / num_bins
  • QUANTILE: Based on percentages. If num_bins is 5 then quintiles so bins are created at the 20%, 40%, 60%, 80% and 100% points.
  • PROVIDED: The user provides the edge points for the bins.

If PROVIDED is supplies, then a List of float values must be provided for the edges parameter that matches the number of bins.

The previous assay examples used quintiles so all of the bins had the same percentage/count of samples.

The following example uses BinMode.EQUAL, then displays the baseline bins created and charts resulting from that binning mode.

equal_bin_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
equal_bin_builder.summarizer_builder.add_bin_mode(BinMode.EQUAL)
equal_baseline = equal_bin_builder.build().interactive_baseline_run()
equal_baseline.chart()


    baseline mean = 12.940910643273655
    baseline median = 12.884286880493164
    bin_mode = Equal
    aggregation = Density
    metric = PSI
    weighted = False

equal_baseline.baseline_bins()
b_edges b_edge_names b_aggregated_values b_aggregation
0 12.00 left_outlier 0.00 Density
1 12.60 p_1.26e1 0.24 Density
2 13.19 p_1.32e1 0.49 Density
3 13.78 p_1.38e1 0.22 Density
4 14.38 p_1.44e1 0.04 Density
5 14.97 p_1.50e1 0.01 Density
6 inf right_outlier 0.00 Density 
assay_results.chart_scores()

The following example manually sets the bin values.

The values in this dataset run from ~11.6 to ~15.81. And lets say we had a business reason to use specific bin edges. We can specify them with the BinMode.PROVIDED and specifying a list of floats with the right hand / upper edge of each bin and optionally the lower edge of the smallest bin. If the lowest edge is not specified the threshold for left outliers is taken from the smallest value in the baseline dataset.

edges = [11.0, 12.0, 13.0, 14.0, 15.0, 16.0]
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_bin_mode(BinMode.PROVIDED, edges)
assay_results = assay_builder.build().interactive_run()
assay_results_df = assay_results[0].compare_bins()
display(assay_results_df.loc[:, ~assay_results_df.columns.isin(['b_aggregation', 'w_aggregation'])])
assay_results[0].chart()
b_edges b_edge_names b_aggregated_values w_edges w_edge_names w_aggregated_values diff_in_pcts
0 11.00 left_outlier 0.00 11.00 left_outlier 0.00 0.00
1 12.00 e_1.20e1 0.00 12.00 e_1.20e1 0.00 0.00
2 13.00 e_1.30e1 0.62 13.00 e_1.30e1 0.59 -0.03
3 14.00 e_1.40e1 0.36 14.00 e_1.40e1 0.35 -0.00
4 15.00 e_1.50e1 0.02 15.00 e_1.50e1 0.06 0.03
5 16.00 e_1.60e1 0.00 16.00 e_1.60e1 0.00 0.00
6 NaN right_outlier 0.00 NaN right_outlier 0.00 0.00 
    baseline mean = 12.940910643273655
    window mean = 12.969964654406132
    baseline median = 12.884286880493164
    window median = 12.899214744567873
    bin_mode = Provided
    aggregation = Density
    metric = PSI
    weighted = False
    score = 0.0321620386600679
    scores = [0.0, 0.0, 0.0014576920813015586, 3.549754401142936e-05, 0.030668849034754912, 0.0, 0.0]
    index = None

Add Alert Threshold

Assay alert thresholds are modified with the wallaroo.assay_config.AssayBuilder.add_alert_threshold(alert_threshold: float) method. By default alert thresholds are 0.1.

The following example updates the alert threshold to 0.5:

alert_threshold = 0.5
import string
import random

assay_name = "example assay"
assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_alert_threshold(alert_threshold)

Metrics

The score is a distance or dis-similarity measure. The larger it is the less similar the two distributions are. The following methods are provided determining the score:

  • PSI (Default) - Population Stability Index (PSI).
  • MAXDIFF: Maximum difference between corresponding bins.
  • SUMDIFF: Mum of differences between corresponding bins.

The metric type used is updated with the wallaroo.assay_config.AssayBuilder.add_metric(metric: wallaroo.assay_config.Metric) method.

The following three charts use each of the metrics. Note how the scores change based on the score type used.

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_results = assay_builder.build().interactive_run()
assay_results[0].chart() # PSI Score

    baseline mean = 12.940910643273655
    window mean = 12.969964654406132
    baseline median = 12.884286880493164
    window median = 12.899214744567873
    bin_mode = Quantile
    aggregation = Density
    metric = PSI
    weighted = False
    score = 0.0029273068646199748
    scores = [0.0, 0.000514261205558409, 0.0002139202456922972, 0.0012617897456473992, 0.0002139202456922972, 0.0007234154220295724, 0.0]
    index = None

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_metric(Metric.SUMDIFF)
assay_results = assay_builder.build().interactive_run()
assay_results[0].chart()

    baseline mean = 12.940910643273655
    window mean = 12.969964654406132
    baseline median = 12.884286880493164
    window median = 12.899214744567873
    bin_mode = Quantile
    aggregation = Density
    metric = SumDiff
    weighted = False
    score = 0.025438649748041997
    scores = [0.0, 0.009956893934794486, 0.006648048084512165, 0.01548175581324751, 0.006648048084512165, 0.012142553579017668, 0.0]
    index = None

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_metric(Metric.MAXDIFF)
assay_results = assay_builder.build().interactive_run()
assay_results[0].chart()

    baseline mean = 12.940910643273655
    window mean = 12.969964654406132
    baseline median = 12.884286880493164
    window median = 12.899214744567873
    bin_mode = Quantile
    aggregation = Density
    metric = MaxDiff
    weighted = False
    score = 0.01548175581324751
    scores = [0.0, 0.009956893934794486, 0.006648048084512165, 0.01548175581324751, 0.006648048084512165, 0.012142553579017668, 0.0]
    index = 3

Aggregation Options

Assay aggregation options are modified with the wallaroo.assay_config.AssayBuilder.add_aggregation(aggregation: wallaroo.assay_config.Aggregation) method. The following options are provided:

  • Aggregation.DENSITY (Default): Count the number/percentage of values that fall in each bin.
  • Aggregation.CUMULATIVE: Empirical Cumulative Density Function style, which keeps a cumulative count of the values/percentages that fall in each bin.

The following example demonstrate the different results between the two.

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_aggregation(Aggregation.DENSITY)
assay_results = assay_builder.build().interactive_run()
assay_results[0].chart()

    baseline mean = 12.940910643273655
    window mean = 12.969964654406132
    baseline median = 12.884286880493164
    window median = 12.899214744567873
    bin_mode = Quantile
    aggregation = Density
    metric = PSI
    weighted = False
    score = 0.0029273068646199748
    scores = [0.0, 0.000514261205558409, 0.0002139202456922972, 0.0012617897456473992, 0.0002139202456922972, 0.0007234154220295724, 0.0]
    index = None

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end).add_run_until(last_day)
assay_builder.summarizer_builder.add_aggregation(Aggregation.CUMULATIVE)
assay_results = assay_builder.build().interactive_run()
assay_results[0].chart()

    baseline mean = 12.940910643273655
    window mean = 12.969964654406132
    baseline median = 12.884286880493164
    window median = 12.899214744567873
    bin_mode = Quantile
    aggregation = Cumulative
    metric = PSI
    weighted = False
    score = 0.04419889502762442
    scores = [0.0, 0.009956893934794486, 0.0033088458502823492, 0.01879060166352986, 0.012142553579017725, 0.0, 0.0]
    index = None

Scheduling Assays

By default assays are scheduled to run every 24 hours starting immediately after the baseline period ends. This is referred to as the window which has two settings:

  • width (Default: hours=24): The time period to measure from the current assay run. For example: hours=12 would accumulate the previous 12 hours of inferences from the assay run.
  • interval (Default: hours=24): The time period between each asay run.

This is modified by the wallaroo.assay_config.WindowBuilder.add_width(**kwargs: int) and wallaroo.assay_config.WindowBuilder.add_interval(**kwargs: int) methods, where kwargs are:

  • weeks
  • days
  • hours
  • minutes

To recap:

  • The window width is the size of the window. The default is 24 hours.
  • The interval is how often the analysis is run, how far the window is slid into the future based on the last run. The default is the window width.
  • The window start is when the analysis should start. The default is the end of the baseline period if the baseline baseline_start and baseline_end are set. Otherwise, it is from the first inference performed.

The following example sets the assay analysis window to execute every 12 hours (the interval) on the previous 24 hours of data (the width)

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_builder = assay_builder.add_run_until(last_day)

assay_builder.window_builder().add_width(hours=24).add_interval(hours=12)

assay_config = assay_builder.build()

assay_results = assay_config.interactive_run()
print(f"Generated {len(assay_results)} analyses")

    Generated 59 analyses

assay_results.chart_scores()

The following example adapts the width and interval to 1 week at a specified start date.

report_start = datetime.datetime.fromisoformat('2022-01-03T00:00:00+00:00')

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_builder = assay_builder.add_run_until(last_day)

assay_builder.window_builder().add_width(weeks=1).add_interval(weeks=1).add_start(report_start)

assay_config = assay_builder.build()

assay_results = assay_config.interactive_run()
print(f"Generated {len(assay_results)} analyses")
    Generated 5 analyses

assay_results.chart_scores()

Upload Assay

Assays are uploaded with the wallaroo.assay_config.upload() method. This uploads the assay into the Wallaroo database with the configurations applied. Note that assay names must be unique across the Wallaroo instance; attempting to upload an assay with the same name as an existing one will return an error.

This returns the assay id as an int.

For example:

# create a fresh assay builder with the correct parameters
assay_builder = ( wl.build_assay(assay_name, pipeline, model_name, 
                     baseline_start, baseline_end)
                    .add_iopath("output variable 0") )

# this assay runs every 24 hours on a 24 hour window
assay_builder.window_builder().add_width(hours=24)
assay_builder.add_alert_threshold(0.1)

# now schedule the assay
assay_id = assay_builder.upload()
  5