Drift Detection for Edge And Multicloud Model Deployments

How to detect drift in Wallaroo for Edge And Multicloud AI Model deployments.

Edge Model Drift Detection with Model Insights via the Wallaroo Dashboard

Model Drift Detection with Model Insights via the Wallaroo Dashboard allows organizations to create assays that monitor inference requests and results against an established baseline.

The following focuses on model drift detection for model deployment on edge locations. For full details on managing assays and model drift detection through the Wallaroo Dashboard, see Model Drift Detection with Model Insights.

Prerequisites

To monitor inference requests and results on edge locations, the following prerequisites must be met:

Access the Model Insights Page

Assays created through the Wallaroo Dashboard are accessed through the Pipeline Dashboard through the following process.

  1. Log into the Wallaroo Dashboard.
  2. Select the workspace containing the pipeline with the models being monitored from the Change Current Workspace and Workspace Management drop down.
  3. Select View Pipelines.
  4. Select the pipeline containing the models being monitored.
  5. Select Insights.

For more details of the Model Insights page, see Model Drift Detection with Model Insights.

Filter By Locations

Edge deployments are filtered with the Filter Edges (B) option.

Assay Dashboard Insights Page

By default, all locations are displayed. Filter Edges provides a list of available edge deployments are displayed. Selecting one or more filters from the list limits the available metrics and logs displayed to only the selected locations.


    
Filter Edges

Build a Edge Location Filtered Assay Through the Pipeline Dashboard

To create a new assay through the Wallaroo Pipeline Dashboard:

  1. Log into the Wallaroo Dashboard.
  2. Select the workspace containing the pipeline with the models being monitored from the Change Current Workspace and Workspace Management drop down.
  3. Select View Pipelines.
  4. Select the pipeline containing the models being monitored.
  5. Select Insights.
  6. Select Filter Edges and select the edge locations to include. By default, assays include all locations. By specifying edges in the filter, this restricts the inference inputs used for the assay baseline and assay analysis results to the specified locations.
  7. Select +Create Assay.
  8. On the Create Assay module, enter the following:
    1. On the Assay Name section, enter the following:

      Assay Name Module
    2. Assay Name (A): The name of the new assay.

    3. Monitor output data or Monitor input data (B): Select whether to monitor input or output data.

    4. Select an output/input to monitor (C): Select the input or output to monitor.

      1. Named Field: The name of the field to monitor.
      2. Index: The index of the monitored field.
    5. On the Specify Baseline section, select one of the following options:

      1. (D) Select the data to use for the baseline. This can either be set with a preset recent time period (last 30 seconds, last 60 seconds, etc) or with a custom date range.
      Assay Name Module
      1. (E) Upload an assay baseline file as either a CSV or TXT file. These assay baselines must be a list of numpy (aka float) values that are comma and newline separated, terminating at the last record with no additional commas or returns.

        Assay Name Module

        For example:

        684577.200,
        921561.500,
        705013.440,
        725875.900,
        684577.200,
        379398.300,
        266405.600,
        256630.310
        

      Once selected, a preview graph of the baseline values will be displayed (C). Note that this may take a few seconds to generate.

    6. Select Next to continue.

  9. On the Settings Module:
    1. Set the date and time range to view values generated by the assay. This can either be set with a preset recent time period (last 30 seconds, last 60 seconds, etc) or with a custom date range.

      New assays are configured to run a new analysis for every 24 hours starting at the end of the baseline period. For information on how to adjust the scheduling period and other settings for the assay scheduling window, see the SDK section on how to Schedule Assay.

    2. Set the following Advanced Settings.

      Baseline Module
      1. (A) Preview Date Range: The date and times to for the preview chart.
      2. (B) Preview: A preview of the assay results will be displayed based on the settings below.
      3. (C) Scheduling: Set the Frequency (Daily, Every Minute, Hourly, Weekly, Default: Daily) and the Time (increments of one hour Default: 1:00 AM).
      4. (D) Group Results: How the results are grouped: Daily (Default), Every Minute, Weekly, or Monthly.
      5. (E) Aggregation Type: Density or Cumulative.
      6. (F) Threshold:
        1. Default: 0.1
      7. (G) Metric:
        1. Default: Population Stability Index
        2. Maximum Difference of Bins
        3. Sum of the Difference of Bins
      8. (H) Number of Bins: From 5 to 14. Default: 5
      9. (F) Bin Mode:
        1. Equally Spaced
        2. Default: Quantile
      10. (I) Bin Weights: The bin weights:
        1. Equally Weighted (Default)
        2. Custom: Users can assign their own bin weights as required.
    3. Review the preview chart to verify the settings are correct.

    4. Select Build to complete the process and build the new assay.

Once created, it may take a few minutes for the assay to complete compiling data.

Model Insights via the Wallaroo Dashboard SDK

Assays generated through the Wallaroo SDK can be previewed, configured, and uploaded to the Wallaroo Ops instance. The following is a condensed version of this process. For full details see the Wallaroo SDK Essentials Guide: Assays Management guide.

Model drift detection with assays using the Wallaroo SDK follows this general process.

  • Define the Baseline: From either historical inference data for a specific model in a pipeline, or from a pre-determine array of data, a baseline is formed.
  • Assay Preview: Once the baseline is formed, we preview the assay and configure the different options until we have the the best method of detecting environment or model drift.
  • Create Assay: With the previews and configuration complete, we upload the assay. The assay will perform an analysis on a regular scheduled based on the configuration.
  • Get Assay Results: Retrieve the analyses and use them to detect model drift and possible sources.
  • Pause/Resume Assay: Pause or restart an assay as needed.

Define the Baseline

Assay baselines are defined with the wallaroo.client.build_assay method. Through this process we define the baseline from either a range of dates or pre-generated values.

wallaroo.client.build_assay take the following parameters:

ParameterTypeDescription
assay_nameString (Required) - requiredThe name of the assay. Assay names must be unique across the Wallaroo instance.
pipelinewallaroo.pipeline.Pipeline (Required)The pipeline the assay is monitoring.
model_nameString (Optional) / NoneThe name of the model to monitor. This field should only be used to track the inputs/outputs for a specific model step in a pipeline. If no model_name is to be included, then the parameters must be passed a named parameters not positional ones.
iopathString (Required)The input/output data for the model being tracked in the format input/output field index. Only one value is tracked for any assay. For example, to track the output of the model’s field house_value at index 0, the iopath is 'output house_value 0.
baseline_startdatetime.datetime (Optional)The start time for the inferences to use as the baseline. Must be included with baseline_end. Cannot be included with baseline_data.
baseline_enddatetime.datetime (Optional)The end time of the baseline window. the baseline. Windows start immediately after the baseline window and are run at regular intervals continuously until the assay is deactivated or deleted. Must be included with baseline_start. Cannot be included with baseline_data..
baseline_datanumpy.array (Optional)The baseline data in numpy array format. Cannot be included with either baseline_start or baseline_data.

Note that model_name is an optional parameters when parameters are named. For example:

assay_builder_from_dates = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

or:

assay_builder_from_dates = wl.build_assay("assays from date baseline", 
                                          mainpipeline, 
                                          None, ## since we are using positional parameters, `None` must be included for the model parameter
                                          "output variable 0",
                                          assay_baseline_start, 
                                          assay_baseline_end)

Baselines are created in one of two mutually exclusive methods:

  • Date Range: The baseline_start and baseline_end retrieves the inference requests and results for the pipeline from the start and end period. This data is summarized and used to create the baseline. For our examples, we’re using the variables assay_baseline_start and assay_baseline_end to represent a range of dates, with assay_baseline_start being set before assay_baseline_end.
  • Numpy Values: The baseline_data sets the baseline from a provided numpy array. This allows assay baselines to be created without first performing inferences in Wallaroo.

Define the Baseline by Location Example

This example shows the assay defined from the date ranges from the inferences performed earlier.

By default, all locations are included in the assay location filters. For our example we use wallaroo.assay_config.WindowBuilder.add_location_filter to specify location_01 for our baseline comparison results.

# edge locations

location_01 = "houseprice-edge-demonstration-01"
location_02 = "houseprice-edge-demonstration-02"
# Build the assay, based on the start and end of our baseline time, 
# and tracking the output variable index 0

display(assay_baseline_start)
display(assay_baseline_end)

assay_baseline_from_dates = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# set the location to the edge location
assay_baseline_from_dates.window_builder().add_location_filter([location_01])

# create the baseline from the dates
assay_baseline_run_from_dates = assay_baseline_from_dates.build().interactive_baseline_run()
datetime.datetime(2024, 4, 19, 12, 57, 12, 837401)

datetime.datetime(2024, 4, 19, 12, 58, 18, 555032)

Baseline DataFrame

The method wallaroo.assay_config.AssayBuilder.baseline_dataframe returns a DataFrame of the assay baseline generated from the provided parameters. This includes:

  • metadata: The inference metadata with the model information, inference time, and other related factors.
  • in data: Each input field assigned with the label in.{input field name}.
  • out data: Each output field assigned with the label out.{output field name}

Note that for assays generated from numpy values, there is only the out data based on the supplied baseline data.

In the following example, the baseline DataFrame is retrieved. Note that in this example, the partition is set to houseprice-edge-demonstration-01, the location specified when we set the location when creating the baseline.

display(assay_baseline_from_dates.baseline_dataframe())
timemetadatainput_tensor_0input_tensor_1input_tensor_2input_tensor_3input_tensor_4input_tensor_5input_tensor_6input_tensor_7...input_tensor_9input_tensor_10input_tensor_11input_tensor_12input_tensor_13input_tensor_14input_tensor_15input_tensor_16input_tensor_17output_variable_0
01713553098362{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '2c70d1a3-1430-421d-b00e-7d5d8f93413a', 'elapsed': [4734876, 1569242], 'dropped': [], 'partition': 'houseprice-edge-demonstration-01'}4.02.502350.06958.02.00.00.03.0...2350.00.047.332100-122.1719972480.06395.016.00.00.0461279.12500
11713553098362{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '2c70d1a3-1430-421d-b00e-7d5d8f93413a', 'elapsed': [4734876, 1569242], 'dropped': [], 'partition': 'houseprice-edge-demonstration-01'}3.02.501350.0941.03.00.00.03.0...1350.00.047.626499-122.3639981640.01369.08.00.00.0684577.18750
21713553098362{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '2c70d1a3-1430-421d-b00e-7d5d8f93413a', 'elapsed': [4734876, 1569242], 'dropped': [], 'partition': 'houseprice-edge-demonstration-01'}3.02.752600.012860.01.00.00.03.0...1350.01250.047.695000-121.9179992260.012954.049.00.00.0703282.62500
31713553098362{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '2c70d1a3-1430-421d-b00e-7d5d8f93413a', 'elapsed': [4734876, 1569242], 'dropped': [], 'partition': 'houseprice-edge-demonstration-01'}4.02.501980.05909.02.00.00.03.0...1980.00.047.391300-122.1849982550.05487.011.00.00.0324875.06250
41713553098362{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '2c70d1a3-1430-421d-b00e-7d5d8f93413a', 'elapsed': [4734876, 1569242], 'dropped': [], 'partition': 'houseprice-edge-demonstration-01'}3.02.501750.07208.02.00.00.03.0...1750.00.047.431499-122.1920012050.07524.020.00.00.0311909.59375
..................................................................
4951713553098362{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '2c70d1a3-1430-421d-b00e-7d5d8f93413a', 'elapsed': [4734876, 1569242], 'dropped': [], 'partition': 'houseprice-edge-demonstration-01'}5.02.002300.07897.02.50.00.04.0...2300.00.047.755600-122.3560032030.07902.059.00.00.0523576.25000
4961713553098362{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '2c70d1a3-1430-421d-b00e-7d5d8f93413a', 'elapsed': [4734876, 1569242], 'dropped': [], 'partition': 'houseprice-edge-demonstration-01'}2.01.00870.04600.01.00.00.04.0...870.00.047.527401-122.378998930.04600.072.00.00.0313906.53125
4971713553098362{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '2c70d1a3-1430-421d-b00e-7d5d8f93413a', 'elapsed': [4734876, 1569242], 'dropped': [], 'partition': 'houseprice-edge-demonstration-01'}3.01.752350.020820.01.00.00.04.0...1800.0550.047.609501-122.0589982040.010800.036.00.00.0700294.18750
4981713553098362{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '2c70d1a3-1430-421d-b00e-7d5d8f93413a', 'elapsed': [4734876, 1569242], 'dropped': [], 'partition': 'houseprice-edge-demonstration-01'}4.02.501820.03899.02.00.00.03.0...1820.00.047.735001-121.9850011820.03899.016.00.00.0437177.84375
4991713553098362{'last_model': '{"model_name":"house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '2c70d1a3-1430-421d-b00e-7d5d8f93413a', 'elapsed': [4734876, 1569242], 'dropped': [], 'partition': 'houseprice-edge-demonstration-01'}3.02.501660.02890.02.00.00.03.0...1660.00.047.543400-122.2929991540.02890.014.00.00.0544392.12500

500 rows × 21 columns

Baseline Stats

The method wallaroo.assay.AssayAnalysis.baseline_stats() returns a pandas.core.frame.DataFrame of the baseline stats.

The baseline stats are displayed in the sample below.

assay_baseline_run_from_dates.baseline_stats()
Baseline
count500
min236238.65625
max1489624.5
mean527769.114156
median450867.5625
std235592.519303
start2024-04-19T18:57:12.837401+00:00
end2024-04-19T18:58:18.554401+00:00

Baseline Bins

The method wallaroo.assay.AssayAnalysis.baseline_bins a simple dataframe to with the edge/bin data for a baseline.

assay_baseline_run_from_dates.baseline_bins()
b_edgesb_edge_namesb_aggregated_valuesb_aggregation
02.362387e+05left_outlier0.000Aggregation.DENSITY
13.338782e+05q_200.202Aggregation.DENSITY
24.371778e+05q_400.212Aggregation.DENSITY
35.384368e+05q_600.192Aggregation.DENSITY
47.050134e+05q_800.194Aggregation.DENSITY
51.489624e+06q_1000.200Aggregation.DENSITY
6infright_outlier0.000Aggregation.DENSITY

Baseline Histogram Chart

The method wallaroo.assay_config.AssayBuilder.baseline_histogram returns a histogram chart of the assay baseline generated from the provided parameters.

assay_baseline_from_dates.baseline_histogram()

Assay Preview

Now that the baseline is defined, we look at different configuration options and view how the assay baseline and results changes. Once we determine what gives us the best method of determining model drift, we can create the assay.

The following examples show different methods of previewing the assay, then how to configure the assay by collecting data from different locations.

Analysis List Chart Scores

Analysis List scores show the assay scores for each assay result interval in one chart. Values that are outside of the alert threshold are colored red, while scores within the alert threshold are green.

Assay chart scores are displayed with the method wallaroo.assay.AssayAnalysisList.chart_scores(title: Optional[str] = None), with ability to display an optional title with the chart.

The following example shows retrieving the assay results and displaying the chart scores for each window interval for location_01

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# set the location to the edge location
assay_baseline.window_builder().add_location_filter([location_01])

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# Preview the assay analyses
assay_results.chart_scores()
# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# set the location to the edge location
assay_baseline.window_builder().add_location_filter([location_01, location_02])

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# Preview the assay analyses
assay_results.chart_scores()

Analysis Chart

The method wallaroo.assay.AssayAnalysis.chart() displays a comparison between the baseline and an interval of inference data.

This is compared to the Chart Scores, which is a list of all of the inference data split into intervals, while the Analysis Chart shows the breakdown of one set of inference data against the baseline.

Score from the Analysis List Chart Scores and each element from the Analysis List DataFrame generates

The following fields are included.

FieldTypeDescription
baseline meanFloatThe mean of the baseline values.
window meanFloatThe mean of the window values.
baseline medianFloatThe median of the baseline values.
window medianFloatThe median of the window values.
bin_modeStringThe binning mode used for the assay.
aggregationStringThe aggregation mode used for the assay.
metricStringThe metric mode used for the assay.
weightedBoolWhether the bins were manually weighted.
scoreFloatThe score from the assay window.
scoresList(Float)The score from each assay window bin.
indexInteger/NoneThe window index. Interactive assay runs are None.
# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# set the location to the edge location
assay_baseline.window_builder().add_location_filter([location_01])

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# Preview the assay analyses
assay_results[0].chart()
baseline mean = 527769.11415625
window mean = 539489.484203125
baseline median = 450867.5625
window median = 451046.9375
bin_mode = Quantile
aggregation = Density
metric = PSI
weighted = False
score = 0.004311757135528007
scores = [0.0, 0.0001809182290241255, 0.0011007381890103568, 0.000612816677154065, 4.603670902398101e-05, 2.010067170700292e-05, 0.002351146659608475]
index = None

Analysis List DataFrame

wallaroo.assay.AssayAnalysisList.to_dataframe() returns a DataFrame showing the assay results for each window aka individual analysis. This DataFrame contains the following fields:

FieldTypeDescription
assay_idInteger/NoneThe assay id. Only provided from uploaded and executed assays.
nameString/NoneThe name of the assay. Only provided from uploaded and executed assays.
iopathString/NoneThe iopath of the assay. Only provided from uploaded and executed assays.
scoreFloatThe assay score.
startDateTimeThe DateTime start of the assay window.
minFloatThe minimum value in the assay window.
maxFloatThe maximum value in the assay window.
meanFloatThe mean value in the assay window.
medianFloatThe median value in the assay window.
stdFloatThe standard deviation value in the assay window.
warning_thresholdFloat/NoneThe warning threshold of the assay window.
alert_thresholdFloat/NoneThe alert threshold of the assay window.
statusStringThe assay window status. Values are:
  • OK: The score is within accepted thresholds.
  • Warning: The score has triggered the warning_threshold if exists, but not the alert_threshold.
  • Alert: The score has triggered the the alert_threshold.

For this example, the assay analysis list DataFrame is listed.

From this tutorial, we should have 2 windows of dta to look at, each one minute apart.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# set the location to the edge location
assay_baseline.window_builder().add_location_filter([location_01])

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# Preview the assay analyses
assay_results.to_dataframe()
idassay_idassay_nameiopathpipeline_idpipeline_namescorestartminmaxmeanmedianstdwarning_thresholdalert_thresholdstatus
0NoneNoneassays from date baselineNone0.0043122024-04-19T18:59:23.561523+00:00236238.656252016006.0539489.484203451046.9375264051.044244None0.25Ok

Configure Assays

Before creating the assay, configure the assay and continue to preview it until the best method for detecting drift is set.

Location Filter

This tutorial focuses on the assay configuration method wallaroo.assay_config.WindowBuilder.add_location_filter.

Location Filter Parameters

add_location_filter takes the following parameters.

ParameterTypeDescription
locationsList(String)The list of model deployment locations for the assay.
Location Filter Example

By default, the locations parameter includes all locations as part of the pipeline. This is seen in the default where no location filter is set, and of the inference data is shown.

For our examples, we will show different locations and how the assay changes. For the first example, we set the location to location_01 which was used to create the baseline, and included inferences that were likely to not trigger a model drift detection alert.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# set the location to the edge location
assay_baseline.window_builder().add_location_filter([location_01])

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# Preview the assay analyses
assay_results.chart_scores()
assay_results.to_dataframe()
idassay_idassay_nameiopathpipeline_idpipeline_namescorestartminmaxmeanmedianstdwarning_thresholdalert_thresholdstatus
0NoneNoneassays from date baselineNone0.0043122024-04-19T18:59:23.561523+00:00236238.656252016006.0539489.484203451046.9375264051.044244None0.25Ok

Now we will set the location to location_02 which had a set of inferences likely to trigger a model drift alert.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# set the location to the edge location
assay_baseline.window_builder().add_location_filter([location_02])

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# Preview the assay analyses
assay_results.chart_scores()
assay_results.to_dataframe()
idassay_idassay_nameiopathpipeline_idpipeline_namescorestartminmaxmeanmedianstdwarning_thresholdalert_thresholdstatus
0NoneNoneassays from date baselineNone0.0043122024-04-19T19:00:23.561523+00:002.362387e+052016006.05.394895e+054.510469e+05264051.044244None0.25Ok
1NoneNoneassays from date baselineNone8.8691152024-04-19T19:01:23.561523+00:001.514080e+062016006.01.888635e+061.946437e+06157648.930624None0.25Alert

The next example includes location_01 and location_02. Since they were performed in distinct times, the model insights scores for each location is seen in the chart.

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# set the location to the edge location
assay_baseline.window_builder().add_location_filter([location_01, location_02])

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

# build the assay configuration
assay_config = assay_baseline.build()

# perform an interactive run and collect inference data
assay_results = assay_config.interactive_run()

# Preview the assay analyses
assay_results.chart_scores()
assay_results.to_dataframe()
idassay_idassay_nameiopathpipeline_idpipeline_namescorestartminmaxmeanmedianstdwarning_thresholdalert_thresholdstatus
0NoneNoneassays from date baselineNone0.0043122024-04-19T18:59:23.561523+00:002.362387e+052016006.05.394895e+054.510469e+05264051.044244None0.25Ok
1NoneNoneassays from date baselineNone0.0043122024-04-19T19:00:23.561523+00:002.362387e+052016006.05.394895e+054.510469e+05264051.044244None0.25Ok
2NoneNoneassays from date baselineNone8.8691152024-04-19T19:01:23.561523+00:001.514080e+062016006.01.888635e+061.946437e+06157648.930624None0.25Alert

Create Assay

With the assay previewed and configuration options determined, we officially create it by uploading it to the Wallaroo instance.

Once it is uploaded, the assay runs an analysis based on the window width, interval, and the other settings configured.

Assays are uploaded with the wallaroo.assay_config.upload() method. This uploads the assay into the Wallaroo database with the configurations applied and returns the assay id. Note that assay names must be unique across the Wallaroo instance; attempting to upload an assay with the same name as an existing one will return an error.

wallaroo.assay_config.upload() returns the assay id for the assay.

Typically we would just call wallaroo.assay_config.upload() after configuring the assay. For the example below, we will perform the complete configuration in one window to show all of the configuration steps at once before creating the assay, and narrow the locations to location_01 and location_02. By default, all locations associated with a pipeline are included in the assay results unless the add_location_filter method is applied to specify location(s).

# Create the assay baseline
assay_baseline = wl.build_assay(assay_name="assays from date baseline", 
                                          pipeline=mainpipeline, 
                                          iopath="output variable 0",
                                          baseline_start=assay_baseline_start, 
                                          baseline_end=assay_baseline_end)

# Set the assay parameters

# set the location to the edge location
assay_baseline.window_builder().add_location_filter([location_01])

# The end date to gather inference results
assay_baseline.add_run_until(datetime.datetime.now())

# Set the interval and window to one minute each, set the start date for gathering inference results
assay_baseline.window_builder().add_width(minutes=1).add_interval(minutes=1).add_start(assay_window_start)

assay_id = assay_baseline.upload()

The assay is now visible through the Wallaroo UI by selecting the workspace, then the pipeline, then Insights. The following is an example of another assay in the Wallaroo Dashboard.

Get Assay Info

Assay information is retrieved with the wallaroo.client.get_assay_info() which takes the following parameters.

ParameterTypeDescription
assay_idInteger (Required)The numerical id of the assay.

This returns the following:

ParameterTypeDescription
idIntegerThe numerical id of the assay.
nameStringThe name of the assay.
activeBooleanTrue: The assay is active and generates analyses based on its configuration. False: The assay is disabled and will not generate new analyses.
pipeline_nameStringThe name of the pipeline the assay references.
last_runDateTimeThe date and time the assay last ran.
next_runDateTimeTHe date and time the assay analysis will next run.
alert_thresholdFloatThe alert threshold setting for the assay.
baselineDictThe baseline and settings as set from the assay configuration.
iopathStringThe iopath setting for the assay.
metricStringThe metric setting for the assay.
num_binsIntegerThe number of bins for the assay.
bin_weightsList/NoneThe bin weights used if any.
bin_modeStringThe binning mode used.
display(wl.get_assay_info(assay_id))
idnameactivestatuspipeline_namelast_runnext_runalert_thresholdbaselineiopathmetricnum_binsbin_weightsbin_mode
02assays from date baselineTruecreatedassay-demonstration-tutorialNone2024-04-19T19:18:12.465292+00:000.25Start:2024-04-19T18:57:12.837401+00:00, End:2024-04-19T18:58:18.554401+00:00output variable 0PSI5NoneQuantile

Troubleshooting

Timeout Issues

If latency between the model’s edge deployment and the Wallaroo Ops Instance is higher than 0.5, the connection may time out and the inference results are not transmitted back to the Wallaroo Ops Center. To check the latency between the deployed model and the Wallaroo Ops Center, use the following command from the edge device:

curl -o /dev/null -s -w "%{time_total}\n" {Wallaroo Domain}:8443

For example, if the Wallaroo Domain of the Wallaroo Ops Center is example.wallaroo.ai, the command is:

curl -o /dev/null -s -w "%{time_total}\n" example.wallaroo.ai:8443
0.089316

For docker run deployments, increase the timeout with the option -e MTLS_CONNECTION_TIMEOUT=3.0. Increase the MTLS_CONNECTION_TIMEOUT further as needed for extreme low latency scenarios.

For example:

docker run -p 8080:8080 \
-v ./data:/persist \ 
-e DEBUG=true \
-e MTLS_CONNECTION_TIMEOUT=3.0 \ 
-e OCI_REGISTRY=$OCI_REGISTRY \
-e EDGE_BUNDLE=abc123 \
-e CONFIG_CPUS=1 \
-e OCI_USERNAME=$OCI_USERNAME \
-e OCI_PASSWORD=$OCI_PASSWORD \
-e PIPELINE_URL=example.registry.io/wallaroolabs/doc-samples/pipelines/edge-observability-pipeline:3b49fba8-94d8-4fca-aecc-c75257fd16c6 \
example.registry.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-main-4079