This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

ONNX Multiple Input Output Example

The following example demonstrates some of the data and input requirements when working with ONNX models in Wallaroo. This example will:

• Upload an ONNX model trained to accept multiple inputs and return multiple outputs.
• Deploy the model, and show how to format the data for inference requests through a Wallaroo pipeline.

For more information on using ONNX models with Wallaroo, see Wallaroo SDK Essentials Guide: Model Uploads and Registrations: ONNX.

Steps

Import Libraries

The first step is to import the libraries used for our demonstration - primarily the Wallaroo SDK, which is used to connect to the Wallaroo Ops instance, upload models, etc.

• References
  • Wallaroo SDK Install Guides

import wallaroo
from wallaroo.deployment_config import DeploymentConfigBuilder
from wallaroo.framework import Framework
import pyarrow as pa
import numpy as np
import pandas as pd 

Connect to the Wallaroo Instance

The next step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()

Create Workspace

We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace. If this tutorial has been run before, the helper function get_workspace will either create or connect to an existing workspace.

Workspace names must be unique; verify that no other workspaces have the same name when running this tutorial. We then set the current workspace to our new workspace; all model uploads and other requests will use this

def get_workspace(workspace_name, wallaroo_client):
    workspace = None
    for ws in wallaroo_client.list_workspaces():
        if ws.name() == workspace_name:
            workspace= ws
    if(workspace == None):
        workspace = wallaroo_client.create_workspace(workspace_name)
    return workspace

workspace_name = 'onnx-tutorial'

workspace = get_workspace(workspace_name, wl)

wl.set_current_workspace(workspace)

{'name': 'onnx-tutorial', 'id': 9, 'archived': False, 'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4', 'created_at': '2023-11-22T16:24:47.786643+00:00', 'models': [], 'pipelines': []}

Upload Model

The ONNX model ./models/multi_io.onnx will be uploaded with the wallaroo.client.upload_model method. This requires:

• The designated model name.
• The path for the file.
• The framework aka what kind of model it is based on the wallaroo.framework.Framework options.

If we wanted to overwrite the name of the input fields, we could use the wallaroo.client.upload_model.configure(tensor_fields[field_names]) option. This ONNX model takes the inputs input_1 and input_2.

• References
  • Wallaroo SDK Essentials Guide: Model Uploads and Registrations: ONNX

model = wl.upload_model('onnx-multi-io-model', 
                        "./models/multi_io.onnx", 
                        framework=Framework.ONNX)
model

Create the Pipeline and Add Steps

A new pipeline ‘multi-io-example’ is created with the wallaroo.client.build_pipeline method that creates a new Wallaroo pipeline within our current workspace. We then add our onnx-multi-io-model as a pipeline step.

• References
  • Wallaroo SDK Essentials Guide: Pipeline Management

pipeline_name = 'multi-io-example'

pipeline = wl.build_pipeline(pipeline_name)

# in case this pipeline was run before
pipeline.clear()
pipeline.add_model_step(model)

Deploy Pipeline

With the model set, deploy the pipeline with a deployment configuration. This sets the number of resources that the pipeline will be allocated from the Wallaroo Ops cluster and makes it available for inference requests.

• References
  • Wallaroo SDK Essentials Guide: Pipeline Management
  • Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration

deployment_config = DeploymentConfigBuilder() \
    .cpus(0.25).memory('1Gi') \
    .build()

pipeline.deploy(deployment_config=deployment_config)
pipeline.status()

{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.143',
   'name': 'engine-857444867-nldj5',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'multi-io-example',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'onnx-multi-io-model',
      'version': '7adb9245-53c2-43b4-95df-2c907bb88161',
      'sha': 'bb3e51dfdaa6440359c2396033a84a4248656d0f81ba1f662751520b3f93de27',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.155',
   'name': 'engine-lb-584f54c899-h647p',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': []}

Sample Inference

For our inference request, we will create a dummy DataFrame with the following fields:

• input_1: A list of randomly generated numbers.
• input_2: A list of randomly generated numbers.

10 rows will be created.

Inference requests for Wallaroo for ONNX models must meet the following criteria:

• Equal rows constraint: The number of input rows and output rows must match.
• All inputs are tensors: The inputs are tensor arrays with the same shape.
• Data Type Consistency: Data types within each tensor are of the same type.

Note that each input meets these requirements:

• Each input is one row, and will correspond to a single output row.
• Each input is a tensor. Field values are a list contained within their field.
• Each input is the same data type - for example, a list of floats.

For more details, see Wallaroo SDK Essentials Guide: Model Uploads and Registrations: ONNX.

np.random.seed(1)
mock_inference_data = [np.random.rand(10, 10), np.random.rand(10, 5)]
mock_dataframe = pd.DataFrame(
    {
        "input_1": mock_inference_data[0].tolist(),
        "input_2": mock_inference_data[1].tolist(),
    }
)

display(mock_dataframe)

We now perform an inference with our sample inference request with the wallaroo.pipeline.infer method. The returning DataFrame displays the input variables as in.{variable_name}, and the output variables as out.{variable_name}. Each inference output row corresponds with an input row.

• References
  • Wallaroo SDK Essentials Guide: Inference Management

results = pipeline.infer(mock_dataframe)
results

Undeploy the Pipeline

With the tutorial complete, we will undeploy the pipeline and return the resources back to the cluster.

pipeline.undeploy()