Add Validations Method

Wallaroo provides validations to detect anomalous data from inference inputs and outputs.

Validations are added to a Wallaroo pipeline with the wallaroo.pipeline.add_validations method.

Adding validations to a pipeline takes the format:

pipeline.add_validations(
    validation_name_01 = polars.Expr,
    validation_name_02 = polars.Expr
    ...{additional validations}
)

Polars expressions are in the polars library version 0.18.5 Expression Python library.

Add Validations Method Parameters

wallaroo.pipeline.add_validations takes the following parameters.

Expressions are typically in the following format.

Add Validations Method Successful Returns

N/A: Nothing is returned on a successful add_validations request.

Add Validations Method Warning Returns

If any validations violate a warning condition, a warning is returned. Warning conditions include the following.

Validation Examples

Common Data Selection Expressions

The following sample expressions demonstrate different methods of selecting which model input or output data to validate.

• polars.col(in|out.{column_name}).list.get(index): Returns the index of a specific field. For example, pl.col("out.dense_1") returns from the inference the output the field dense_1, and list.get(0) returns the first value in that list. Most output values from a Wallaroo inference result are a List of at least length 1, making this a common validation expression.
• polars.col(in.price_ranges).list.max(): Returns from the inference request the input field price_ranges the maximum value from a list of values.
• polars.col(out.price_ranges).mean() returns the mean for all values from the output field price_ranges.

For example, to the following validation fraud detects values for the output of an inference request for the field dense_1 that are greater than 0.9, indicating a transaction has a high likelihood of fraud:

import polars as pl

pipeline.add_validations(
    fraud = fraud=pl.col("out.dense_1").list.get(0) > 0.9
)

The following inference output shows the detected anomaly from an inference output:

Detecting Input Anomalies

The following validation tests the inputs from sales figures for a week’s worth of sales:

To validate that any sales figure does not go below 500 units, the validation is:

import polars as pl

pipeline.add_validations(
    minimum_sales=pl.col("in.sales_count").list.min() < 500
)

pipeline.deploy()

pipeline.infer_from_file(previous_week_sales)

For the input provided, the minimum_sales validation would return True, indicating an anomaly.

Detecting Output Anomalies

The following validation detects an anomaly from a output.

• fraud: Detects when an inference output for the field dense_1 at index 0 is greater than 0.9, indicating fraud.

# create the pipeline
sample_pipeline = wallaroo.client.build_pipeline("sample-pipeline")

# add a model step
sample_pipeline.add_model_step(ccfraud_model)

# add validations to the pipeline
sample_pipeline.add_validations(
    fraud=pl.col("out.dense_1").list.get(0) > 0.9
    )
sample_pipeline.deploy()

sample_pipeline.infer_from_file("dev_high_fraud.json")

Multiple Validations

The following demonstrates multiple validations added to a pipeline at once and their results from inference requests. Two validations that track the same output field and index are applied to a pipeline:

• fraud: Detects an anomaly when the inference output field dense_1 at index 0 value is greater than 0.9.
• too_low: Detects an anomaly when the inference output field dense_1 at the index 0 value is lower than 0.05.

sample_pipeline.add_validations(
    fraud=pl.col("out.dense_1").list.get(0) > 0.9,
    too_low=pl.col("out.dense_1").list.get(0) < 0.05
    )

Two separate inferences where the output of the first is over 0.9 and the second is under 0.05 would be the following.

sample_pipeline.infer_from_file("high_fraud_example.json")

sample_pipeline.infer_from_file("low_fraud_example.json")

The following example tracks two validations for a model that takes the previous week’s sales and projects the next week’s average sales with the field predicted_sales.

• minimum_sales=pl.col("in.sales_count").list.min() < 500: The input field sales_count with a range of values has any minimum value under 500.
• average_sales_too_low=pl.col("out.predicted_sales").list.get(0) < 500: The output field predicted_sales is less than 500.

The following inputs return the following values. Note how the anomaly.count value changes by the number of validations that detect an anomaly.

Input 1:

In this example, one day had sales under 500, which triggers the minimum_sales validation to return True. The predicted sales are above 500, causing the average_sales_too_low validation to return False.

Output 1:

Input 2:

In this example, multiple days have sales under 500, which triggers the minimum_sales validation to return True. The predicted average sales for the next week are above 500, causing the average_sales_too_low validation to return True.

Output 2:

Input 3:

In this example, no sales day figures are below 500, which triggers the minimum_sales validation to return False. The predicted sales for the next week is below 500, causing the average_sales_too_low validation to return True.

Output 3:

Compound Validations

The following combines multiple field checks into a single validation. For this, we will check for values of out.dense_1 that are between 0.05 and 0.9.

Each expression is separated by (). For example:

• Expression 1: pl.col("out.dense_1").list.get(0) < 0.9
• Expression 2: pl.col("out.dense_1").list.get(0) > 0.001
• Compound Expression: (pl.col("out.dense_1").list.get(0) < 0.9) & (pl.col("out.dense_1").list.get(0) > 0.001)

sample_pipeline = sample_pipeline.add_validations(
    in_between_2=(pl.col("out.dense_1").list.get(0) < 0.9) & (pl.col("out.dense_1").list.get(0) > 0.001)
)

results = sample_pipeline.infer_from_file("./data/cc_data_1k.df.json")

results.loc[results['anomaly.in_between_2'] == True]