Enable Wallaroo Assays

Enable Wallaroo Assays for model drift detection.

Table of Contents

Model Drift Detection with Assays

Wallaroo provides the ability to monitor the input and outputs of models to detect model drift by using model insights. Changes in the inputs or model predictions that fall outside of expected norms, known as data drift, can occur due to errors in the data processing pipeline or due to changes in the environment such as user preference or behavior.

Monitoring tasks called assays provides model insights by comparing a model’s predictions or the input data coming into the model against an established baseline. Changes in the distribution of this data can be an indication of model drift, or of a change in the environment that the model trained for. This can provide tips on whether a model needs to be retrained or the environment data analyzed for accuracy or other needs.

Model Insights provides interactive assays to explore data from a pipeline and learn how the data is behaving. With this information and the knowledge of your particular business use case you can then choose appropriate settings to launch an assay to run on a given frequency as more data is collected.

Assay versions

Wallaroo defaults to Assay V1 and introduced Assays V2 in Wallaroo 2024.2. Assays V2 provides additional performance enhancements and multi-threaded processing of assay runs for realtime drift detection scenarios and offline drift detection scenarios. Except where noted, the Wallaroo SDK and Wallaroo MLOps API calls and returns are the same.

Configuring Assay Version

The assay version is set through platform operations, either during installation or via update.

When changing from one assay version to the other, the following effects will follow:

  • The Wallaroo Dashboard Pipeline Insights page only displays assays created in the configured assay version.
    • For example, if the assay was created in Assays V1, then the Wallaroo instance is configured to Assays V2, only assays created under Assays V2 are displayed in the Wallaroo Dashboard.
  • The Wallaroo SDK Assay methods must be set to same configured Wallaroo Assay version.
  • The Wallaroo MLOps API endpoints only support Assays V1.

Organizations should inform ML engineers and other developers on what Wallaroo Assay version is used so they can use the correct API and SDK settings.

It is highly recommended to set the Wallaroo Assay version at installation time, or before any assays are created. When moving from one Wallaroo Assay version to another, all existing assays should be migrated by recreating them with the same original settings.

Enable Assays

Wallaroo assays are enabled depending on the installation method: kots, through the Kots Administrative Dashboard, or helm installs.

Enable Assays via Kots

To enable Wallaroo assays via kots after Wallaroo is installed:

  1. Launch the Kots Administrative Dashboard through a terminal linked to the Kubernetes cluster. Replace the namespace with the one used in your installation.

    kubectl kots admin-console --namespace wallaroo

  2. Launch the dashboard, by default at http://localhost:8800.

  3. From the admin dashboard, select Config -> Observability -> Enable Assays.

  4. Select from one of the following options:

    1. Disable: Disable assays.
    2. v1: Enable Wallaroo Assays V1.
    3. v2: Enable Wallaroo Assays V2.

  5. Scroll to the bottom and select Save config.

  6. Deploy the updated configuration and version.

Updating enabled assays takes around 5 minutes to complete deployment.

Enable Assays via Helm

To enable or change the assay version via helm:

  1. Update the local-values.yaml file with the new Kubernetes secret set to the custTlsSecretName Helm variable. For details on other helm based settings, see the Wallaroo Helm Reference Guides.

  2. Update the local-values.yaml and set the assay version to use to either true or false.

    1. v1: Enable Assays V1.
    2. v2: Enable Assays V2.

    The following is a sample example of enabling assays.

    wallarooDomain: "wallaroo.example.com" # change to match the domain name

custTlsSecretName: cust-cert-secret

apilb:
  serviceType: LoadBalancer
  external_inference_endpoints_enabled: true
  ingress_mode: internal # internal (Default), external,or none

dashboard:
  clientName: "Wallaroo Helm Example" # Insert the name displayed in the Wallaroo Dashboard

kubernetes_distribution: ""   # Required. One of: aks, eks, gke, oke, or kurl.

assays:  #enable Wallaroo assays.  Select **one** of the following: `v1` or `v2`, or leave commented out to disable assays.
  enabled: true
  v1: false
  v2: false

  3. Update the helm based installation with the helm upgrade command in the following format:

    helm upgrade $RELEASE $REGISTRYURL --version $VERSION --values $LOCALVALUES.yaml --timeout 10m

    Where:

    1. $RELEASE: The name of the Helm release. By default, wallaroo.
    2. $REGISTRYURL: The URl for the Wallaroo container registry service.
    3. $VERSION: The version of Wallaroo to install. For this example, 2024.3.0-5913.
    4. $LOCALVALUES: The .yaml file containing the local values overrides. For this example, local-values.yaml.

    For example, for the registration wallaroo the command is:

    helm upgrade wallaroo oci://registry.replicated.com/wallaroo/2024-3/wallaroo --version 2024.3.0-5913 --values local-values.yaml --timeout 10m