Wallaroo Assays Management

How to create and use assays to monitor model inputs and outputs.

Model Insights and Interactive Analysis Introduction

Wallaroo provides the ability to perform interactive analysis so organizations can explore the data from a pipeline and learn how the data is behaving. With this information and the knowledge of your particular business use case you can then choose appropriate thresholds for persistent automatic assays as desired.

Monitoring tasks called assays monitors a model’s predictions or the data coming into the model against an established baseline. Changes in the distribution of this data can be an indication of model drift, or of a change in the environment that the model trained for. This can provide tips on whether a model needs to be retrained or the environment data analyzed for accuracy or other needs.

Assay Details

Assays contain the following attributes:

Attribute Default Description
Name   The name of the assay. Assay names must be unique.
Baseline Data   Data that is known to be “typical” (typically distributed) and can be used to determine whether the distribution of new data has changed.
Schedule Every 24 hours at 1 AM New assays are configured to run a new analysis for every 24 hours starting at the end of the baseline period. This period can be configured through the SDK. See Schedule Assay for more information.
Number of Bins 5 Sets the number of bins that will be used to partition the baseline data for comparison against how future data falls into these bins. By default, the binning scheme is percentile (quantile) based. The binning scheme can be configured (see Bin Mode, below).
Bin Mode Quantile Set the binning scheme. Quantile binning defines the bins using percentile ranges (each bin holds the same percentage of the baseline data). Equal binning defines the bins using equally spaced data value ranges, like a histogram.
Metric PSI Population Stability Index (PSI) is an entropy-based measure of the difference between distributions. Maximum Difference of Bins measures the maximum difference between the baseline and current distributions (as estimated using the bins). Sum of the difference of bins sums up the difference of occurrences in each bin between the baseline and current distributions.
Threshold 0.1 The threshold for deciding whether the difference between distributions, as evaluated by the above metric, is large (the distributions are different) or small (the distributions are similar). The default of 0.1 is generally a good threshold when using PSI as the metric.

Manage Assays via the Wallaroo Dashboard

Assays can be created and used via the Wallaroo Dashboard.

Accessing Assays Through the Pipeline Dashboard

Assays created through the Wallaroo Dashboard are accessed through the Pipeline Dashboard through the following process.

  1. Log into the Wallaroo Dashboard.
  2. Select the workspace containing the pipeline with the models being monitored from the Change Current Workspace and Workspace Management drop down.
  3. Select View Pipelines.
  4. Select the pipeline containing the models being monitored.
  5. Select Insights.

The Wallaroo Assay Dashboard contains the following elements. For more details of each configuration type, see the Model Insights and Assays Introduction.

Assay Dashboard User Interface
  • (A) Time Period for Assay Data: Set the time period for data to be used in displaying the assay results.
  • (B) Create Assay: Create a new Assay.
  • (C) Pause/Start Assay: Pause or Start an assay.
  • (D) Show Assay Details: View assay details. See Assay Details View for more details.
  • (E) Assay Events: Select an individual assay event to see more details. See View Assay Alert Details for more information.

Assay Details View

Assay UI Details

The following details are visible by selecting the Assay View Details icon:

  • (A) Assay Name: The name of the assay displayed.
  • (B) Model: The model being monitored.
  • (C) Baseline: The time period used to generate the baseline.
  • (D) Last Run: The date and time the assay was last run.
  • (E) Next Run: The future date and time the assay will be run again. NOTE: If the assay is paused, then it will not run at the scheduled time. When unpaused, the date will be updated to the next date and time that the assay will be run.
  • (F) Aggregation Type: The aggregation type used with the assay.
  • (G) Threshold: The threshold value used for the assay.
  • (H) Metric: The metric type used for the assay.
  • (I) Number of Bins: The number of bins used for the assay.
  • (J) Bin Weight: The weight applied to each bin.
  • (K) Bin Mode: The type of bin node applied to each bin.

View Assay Alert Details

To view details on an assay alert:

  1. Select the data with available alert data.
  2. Mouse hover of a specific Assay Event Alert to view the data and time of the event and the alert value.
  3. Select the Assay Event Alert to view the Baseline and Window details of the alert including the left_outlier and right_outlier.

Hover over a bar chart graph to view additional details.

  1. Select the ⊗ symbol to exit the Assay Event Alert details and return to the Assay View.

Build an Assay Through the Pipeline Dashboard

To create a new assay through the Wallaroo Pipeline Dashboard:

  1. Log into the Wallaroo Dashboard.

  2. Select the workspace containing the pipeline with the models being monitored from the Change Current Workspace and Workspace Management drop down.

  3. Select View Pipelines.

  4. Select the pipeline containing the models being monitored.

  5. Select Insights.

  6. Select +Create Assay.

  7. On the Assay Name module, enter the following:

    Assay Name Module
    1. Assay Name: The name of the new assay.
    2. Select Model to monitor: Select the model that will be monitored by the assay.
    3. Input or Output: Select whether to evaluate:
      1. Monitor output data: Data output as part of the inference process.
      2. Monitor input data: Data fed to the pipeline as part of the inference process.
    4. Select Next to continue.
  8. On the Specify Baseline Module:

    Baseline Module
    1. (A) Select the data to use for the baseline. This can either be set with a preset recent time period (last 30 seconds, last 60 seconds, etc) or with a custom date range.

    Once selected, a preview graph of the baseline values will be displayed (B). Note that this may take a few seconds to generate.

    1. Select Next to continue.
  9. On the Settings Module:

    1. Set the date and time range to view values generated by the assay. This can either be set with a preset recent time period (last 30 seconds, last 60 seconds, etc) or with a custom date range.

      New assays are configured to run a new analysis for every 24 hours starting at the end of the baseline period. For information on how to adjust the scheduling period and other settings for the assay scheduling window, see the SDK section on how to Schedule Assay.

    2. Set the following Advanced Settings.

      Baseline Module
      1. (A) Preview Date Range: The date and times to for the preview chart.
      2. (B) Preview: A preview of the assay results will be displayed based on the settings below.
      3. (C) Scheduling: Set the Frequency (Daily, Every Minute, Hourly, Weekly, Default: Daily) and the Time (increments of one hour Default: 1:00 AM).
      4. (D)Number of Bins: From 5 to 14. Default: 5
      5. (E)Bin Mode:
        1. Equally Spaced
        2. Default: Quantile
      6. (F)Metric:
        1. Default: Population Stability Index
        2. Maximum Difference of Bins
        3. Sum of the Difference of Bins
      7. (G)Threshold
        1. Default: 0.1
      8. Review the preview chart to verify the settings are correct.
    3. Select Build to complete the process and build the new assay.

Once created, it may take a few minutes for the assay to complete compiling data. If needed, reload the Pipeline Dashboard to view changes.

Manage Assays via the Wallaroo SDK

Build Assay Via the Wallaroo SDK

Assays are built with the Wallaroo client.build_assay(assayName, pipeline, modelName, baselineStart, baselineEnd), and returns the wallaroo.assay_config.AssayBuilder. The method requires the following parameters:

Parameter Type Description
assayName String The human friendly name of the created assay.
pipeline Wallaroo.pipeline The pipeline the assay is assigned to.
modelName String The model to perform the assay on.
baselineStart DateTime When to start the baseline period.
baselineStart DateTime When to end the baseline period.

When called, this method will then pool the pipeline between the baseline start and end periods to establish what values are considered normal inputs for the specified model.

Assays by default will run a new a new analysis every 24 hours starting at the end of the baseline period, using a 24 hour observation window.

In this example, an assay will be created named example assay and stored into the variable assay_builder.

import datetime
baseline_start = datetime.datetime.fromisoformat('2022-01-01T00:00:00+00:00')
baseline_end = datetime.datetime.fromisoformat('2022-01-02T00:00:00+00:00')
last_day = datetime.datetime.fromisoformat('2022-02-01T00:00:00+00:00')

assay_name = "example assay"
assay_builder = client.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)

Schedule Assay

By default assays are scheduled to run every 24 hours starting immediately after the baseline period ends. This scheduled period is referred to as the assay window and has the following properties:

  • width: The period of data included in the analysis. By default this is 24 hours.
  • interval:
    • How often the analysis is run (every 5 minutes, every 24 hours, etc). By default this is the window width.
  • start: When the analysis should start. By default this is at the end of the baseline period.

These are adjusted through the assay window_builder method that includes the following methods:

  • add_width: Sets the width of the window.
  • add_interval: Sets how often the analysis is run.

In this example, the assay will be set to run an analysis every 12 hours on the previous 24 hours of data:

assay_builder = wl.build_assay(assay_name, pipeline, model_name, baseline_start, baseline_end)
assay_builder = assay_builder.add_run_until(last_day)

assay_builder.window_builder().add_width(hours=24).add_interval(hours=12)

assay_config = assay_builder.build()

assay_results = assay_config.interactive_run()
print(f"Generated {len(assay_results)} analyses")
Generated 59 analyses

Perform Interactive Baseline

Interactive baselines can be run against an assay to generate a list of the values that are established in the baseline. This is done through the AssayBuilder.interactive_baseline_run() method, which returns the following:

Parameter Type Description
count Integer The number of records evaluated.
min Float The minimum value found
max Float The maximum value found
mean Float The mean value derived from the values evaluated.
median Float The median value derived from the values evaluated.
std Float The standard deviation from the values evaluated.
start DateTime The start date for the records to evaluate.
end DateTime The end date for the records to evaluate.

In this example, an interactive baseline will be run against a new assay, and the results displayed:

``` python
baseline_run = assay_builder.build().interactive_baseline_run()
baseline_run.baseline_stats()

                    Baseline
count                   1813
min                    11.95
max                    15.08
mean                   12.95
median                 12.91
std                     0.46
start   2022-01-01T00:00:00Z
end     2022-01-02T00:00:00Z

Display Assay Graphs

Histogram, kernel density estimate (KDE), and Empirical Cumulative Distribution (ecdf) charts can be generated from an assay to provide a visual representation of the values evaluated and where they fit within the established baseline.

These methods are part of the AssayBuilder object and are as follows:

Method Description
baseline_histogram() Creates a histogram chart from the assay baseline.
baseline_kde() Creates a kernel density estimate (KDE) chart from the assay baseline.
baseline_ecdf() Creates an Empirical Cumulative Distribution (ecdf) from the assay baseline.

In this example, each of the three different charts will be generated from an assay:

assay_builder.baseline_histogram()

assay_builder.baseline_kde()

assay_builder.baseline_ecdf()