Computer Vision Tutorial Notebook 1: Build and Deploy a Model

For this tutorial, let’s pretend that you work for a retail store. You have developed a model that scans images and predicts what items are on a shelf, shopping card, or otherwise.

In this set of exercises, you will build to recognize images and deploy it to Wallaroo.

Before we start, let’s load some libraries that we will need for this notebook (note that this may not be a complete list).

• IMPORTANT NOTE: This tutorial is geared towards a Wallaroo 2023.2.1 environment.

# preload needed libraries 

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

from IPython.display import display

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
from wallaroo.framework import Framework

import json

import datetime
import time

# used for unique connection names

import string
import random

import pyarrow as pa

import sys
 
# setting path - only needed when running this from the `with-code` folder.
# sys.path.append('../')

from CVDemoUtils import CVDemo
cvDemo = CVDemo()
cvDemo.COCO_CLASSES_PATH = "../models/coco_classes.pickle"

Exercise: Build a model

The following models are examples of computer vision models that have been tested in Wallaroo.

• RESNET50
• Mobilenet

Models similar to these can be deployed and deployed in Wallaroo. It is recommended to convert them to the ONNX ML Model framework before deploying to Wallaroo. Instructions for converting them are included with in the N0-environment-prep-model-conversion notebook.

NOTE

If you prefer to shortcut this step, you can use one of the pre-trained models. These are available either through the Wallaroo Tutorials repository at https://github.com/WallarooLabs/Wallaroo_Tutorials/releases by selecting the most recent version and downloading computer_vision.zip, or by using the following command with the gcloud application:

gcloud storage cp "gs://wallaroo-model-zoo/open-source/computer-vision/models/*" .

It is highly recommended that these pre-trained models are stored in the ./models folder.

## Blank space for training model, if needed

Getting Ready to deploy

Wallaroo natively supports models in the ONNX, Python based models, Tensorflow frameworks, and other frameworks via containerization. For this exercise, we assume that you have a model that can be converted to the ONNX framework. The first steps to deploying in Wallaroo, then, is to convert your model to ONNX, and to add some extra functions to your processing modules so Wallaroo can call them.

For computer vision models, additional instructions are available in the N0-environment-prep-model-conversion notebook.

Exercise: Convert your Model to ONNX

Take the model that you created in the previous exercises, and convert it to ONNX supported by Wallaroo. If you need help, see the Wallaroo ONNX conversion tips. The model can also be deployed to Wallaroo if is supported by Wallaroo. See the Wallaroo SDK Essentials Guide: Model Uploads and Registrations

At the end of this exercise, you should have your model as a standalone artifact, for example, a file called model.onnx.

NOTE

If you prefer to shortcut this exercise, one of the two ONNX models in the ./models directory.

# Blank space to load for converting model, if needed

Get ready to work with Wallaroo

Now that you have a model ready to go, you can log into Wallaroo and set up a workspace to organize your deployment artifacts. A Wallaroo workspace is place to organize the deployment artifacts for a project, and to collaborate with other team members. For more information, see the Wallaroo 101.

Logging into Wallaroo via the cluster’s integrated JupyterLab is quite straightfoward:

# Login through local Wallaroo instance 
wl = wallaroo.Client()

See the documentation if you are logging into Wallaroo some other way.

Once you are logged in, you can create a workspace and set it as your working environment. To make the first exercise easier, here is a convenience function to get or create a workspace:

# return the workspace called <name>, or create it if it does not exist.
# this function assumes your connection to wallaroo is called wl
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

Then logging in and creating a workspace looks something like this:

# Login through local Wallaroo instance 
wl = wallaroo.Client()

Setting up the workspace may resemble this. Verify that the workspace name is unique across the Wallaroo instance.

# workspace names need to be globally unique, so add a random suffix to insure this
# especially important if the "main" workspace name is potentially a common one

suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
workspace_name = "my-workspace"+suffix

workspace = get_workspace(workspace_name)

# set your current workspace to the workspace that you just created
wl.set_current_workspace(workspace)

# optionally, examine your current workspace
wl.get_current_workspace()

Exercise: Log in and create a workspace

Log into wallaroo, and create a workspace for this tutorial. Then set that new workspace to your current workspace.
Make sure you remember the name that you gave the workspace, as you will need it for later notebooks. Set that workspace to be your working environment.

Notes

• Workspace names must be globally unique, so don’t pick something too common. The “random suffix” trick in the code snippet is one way to try to generate a unique workspace name, if you suspect you are using a common name.

At the end of the exercise, you should be in a new workspace to do further work.

## Blank spot to log in

wl = wallaroo.Client()

## Blank spot to connect to the workspace

import string
import random

suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))

workspace_name = f'computer-vision-tutorial'

workspace = get_workspace(workspace_name)

wl.set_current_workspace(workspace)

{'name': 'computer-vision-tutorialjohn', 'id': 20, 'archived': False, 'created_by': '0a36fba2-ad42-441b-9a8c-bac8c68d13fa', 'created_at': '2023-08-04T19:16:04.283819+00:00', 'models': [], 'pipelines': []}

Deploy a Simple Single-Step Pipeline

Once your model is in the ONNX format, and you have a workspace to work in, you can easily upload your model to Wallaroo’s production platform with just a few lines of code. For example, if you have a model called model.onnx, and you wish to upload it to Wallaroo with the name mymodel, then upload the model as follows (once you are in the appropriate workspace):

my_model = wl.upload_model("mymodel", "model.onnx", framework=Framework.ONNX).configure('onnx', batch_config="single")

The addition of .configure('onnx', batch_config="single") is used for these computer vision models to instruct Wallaroo how to handle the inputs.

See Wallaroo SDK Essentials Guide: Model Uploads and Registrations: ONNX for full details.

The function upload_model() returns a handle to the uploaded model that you will continue to work with in the SDK.

Upload Python Models

For this tutorial, we also recommend the use of the Python step model ./models/post-process-drift-detection-arrow.py. This takes the output of the models and averages the confidence values into one value avg_conf. This value is necessary for performing observations for data validation and Wallaroo assays.

To upload a Python model to Wallaroo, the following is required:

• The name of the model: What to designate the model name once uploaded to Wallaroo.
• Path of the Python script: This specifies the file location. For example: ./models/my-python-script.py
• The input and output schemas: These inform Wallaroo how inference request data will be formatted, and the shape of the data going out from the model. The schema is in Apache Arrow schema format, aka pyarrow.lib.Schema.. These are required to be in the For example, if the inference inputs are a pandas DataFrame with the following shape:

With the following output:

[
“forecast”: [235, 135, 175],
“average_forecast”: [181.66]
]

In this case, the input schema is represented as:

import pyarrow as py

input_schema = pa.schema([
    pa.field('tensor', pa.list_(pa.float64()))
])

output_schema = pa.schema([
    pa.field('forecast', pa.list_(pa.int64())),
    pa.field('average_forecast', pa.list_(pa.float64()))
])

Python models are uploaded to Wallaroo with the upload_model method, with the additional configure method specifying the python framework with the inputs and outputs. For example:

sample_model = (wl.upload_model(control_model_name, 
                                 control_model_file, 
                                 framework=Framework.PYTHON)
                                 .configure("python", 
                                 input_schema=input_schema, 
                                 output_schema=output_schema)
                )

For more information about Python models, see Wallaroo SDK Essentials Guide: Model Uploads and Registrations: Python Models.

Once the model has been uploaded, you can create a pipeline that contains the model. The pipeline is the mechanism that manages deployments. A pipeline contains a series of steps - sequential sets of models which take in the data from the preceding step, process it through the model, then return a result. Some pipelines can have just one step, while others may have multiple models with multiple steps or arranged for A/B testing. Deployed pipelines allocate resources and can then process data either through local files or through a deployment URL.

So for your model to accept inferences, you must add it to a pipeline. You can create a single step pipeline called mypipeline as follows.

# create the pipeline
my_pipeline = wl.build_pipeline("mypipeline").add_model_step(my_model)

# deploy the pipeline
my_pipeline = my_pipeline.deploy()

Multiple steps can be deployed to the same pipeline. The data from each previous step is passed to the next. To use the post process Python step provided, we can apply both model and post process step as follows:

# create the pipeline
my_pipeline = wl.build_pipeline("mypipeline")
my_pipeline.add_model_step(my_model)
my_pipeline.add_model_step(postprocessing-step)

# deploy the pipeline
my_pipeline = my_pipeline.deploy()

This pipeline now has two steps. If using our example Python step, the difference is the model returns the fields boxes, classes, and confidences, each as an array of values. The python step takes that data and adds the additional field avg_conf, which is an average of the confidences fields.

Deploying the pipeline means that resources from the cluster are allocated to the pipeline, and it is ready to accept inferences. You can “turn off” the pipeline with the call pipeline.undeploy(), which returns the resources back to the cluster. This is an important step - leaving pipeline deployed when they’re no longer needed takes up resources that may be needed by other pipelines or services.

See Wallaroo SDK Essentials Guide: Pipeline Management for full details.

More Hints

• workspace = wl.get_current_workspace() gives you a handle to the current workspace
• then workspace.models() will return a list of the models in the workspace
• and workspace.pipelines() will return a list of the pipelines in the workspace

Exercise: Upload and deploy your model

Upload and deploy the ONNX model that you created in the previous exercise. For simplicity, do any needed pre-processing in the notebook.

At the end of the exercise, you should have a model and a deployed pipeline in your workspace.

## blank space to upload model, and create the pipeline

mobilenet_model_name = 'mobilenet'
mobilenet_model_path = "../models/mobilenet.pt.onnx"

mobilenet_model = wl.upload_model(mobilenet_model_name, 
                                  mobilenet_model_path, 
                                  framework=Framework.ONNX).configure('onnx', 
                                                                      batch_config="single"
                                                                      )

# upload python step

field_boxes = pa.field('boxes', pa.list_(pa.list_(pa.float64(), 4)))
field_classes = pa.field('classes', pa.list_(pa.int32()))
field_confidences =  pa.field('confidences', pa.list_(pa.float64()))

# field_boxes - will have a flattened array of the 4 coordinates representing the boxes.  128 entries
# field_classes - will have 32 entries
# field_confidences - will have 32 entries
input_schema = pa.schema([field_boxes, field_classes, field_confidences])

output_schema = pa.schema([
    field_boxes,
    field_classes,
    field_confidences,
    pa.field('avg_conf', pa.list_(pa.float64()))
])

module_post_process = wl.upload_model("cv-post-process-drift-detection", 
                                      "../models/post-process-drift-detection-arrow.py",
                                      framework=Framework.PYTHON ).configure('python', 
                                                                             input_schema=input_schema, 
                                                                             output_schema=output_schema
                                    )

# create the pipeline

pipeline_name = 'cv-retail-pipeline'

pipeline = wl.build_pipeline(pipeline_name)
pipeline.undeploy()
pipeline.clear()
pipeline.add_model_step(mobilenet_model)
pipeline.add_model_step(module_post_process)
pipeline.deploy()

Sending Data to your Pipeline

ONNX models generally expect their input as an array in a dictionary, keyed by input name. In Wallaroo, the default input name is “tensor”. So (outside of Wallaroo), an ONNX model that expected three numeric values as its input would expect input data similar to the below: (Note: The below examples are only notional, they aren’t intended to work with our example models.)

# one datum
singleton = {'tensor': [[1, 2, 3]] }

# two datums
two_inputs = {'tensor': [[1, 2, 3], [4, 5, 6]] }

In the Wallaroo SDK, you can send a pandas DataFrame representation of this dictionary (pandas record format) to the pipeline, via the pipeline.infer() method.

import pandas as pd

# one datum (notional example)
sdf = pd.DataFrame(singleton)
sdf
#       tensor
# 0  [1, 2, 3]

# send the datum to a pipeline for inference
# notional example - not houseprice model!
result = my_pipeline.infer(sdf)

# two datums
# Note that the value of 'tensor' must be a list, not a numpy array 
twodf = pd.DataFrame(two_inputs)
twodf
#      tensor
# 0  [1, 2, 3]
# 1  [4, 5, 6]

# send the data to a pipeline for inference
# notional example, not houseprice model!
result = my_pipeline.infer(twodf)

To send data to a pipeline via the inference URL (for example, via CURL), you need the JSON representation of these data frames.

#
# notional examples, not houseprice model!
#
sdf.to_json(orient='records')
# '[{"tensor":[1,2,3]}]'

twodf.to_json(orient='records')
# '[{"tensor":[1,2,3]},{"tensor":[4,5,6]}]'

For these tutorials, images are converted into their tensor value using the cvDemo.loadImageAndConvertToDataframe(image_path, width, height). For example, the following converts the file ./data/images/input/example/dairy_bottles.png into its tensor values.

width, height = 640, 480
dfImage, resizedImage = cvDemo.loadImageAndConvertToDataframe('data/images/input/example/dairy_bottles.png', 
                                                              width, 
                                                              height
                                                              )
dfImage

	tensor
0	[[[[0.9372549 0.9529412 0.9490196 0.9450980...

Exercise: Send data to your pipeline for inference.

Create some test data from the housing data and send it to the pipeline that you deployed in the previous exercise.

• If you used the pre-provided models, then you can use the images from the ./data/images/input/example folder with the cvDemo.loadImageAndConvertToDataframe to translate it into a pandas DataFrame with the relevant tensor data.

At the end of the exercise, you should have a set of inference results that you got through the Wallaroo pipeline.

##  blank space to create test data, and send some data to your model

image = '../data/images/input/example/dairy_bottles.png'

width, height = 640, 480
dfImage, resizedImage = cvDemo.loadImageAndConvertToDataframe(image, 
                                                              width, 
                                                              height
                                                              )

results = pipeline.infer(dfImage)
display(results)

Render the Image

If using our sample models, this inference has returned the following fields:

• out.boxes: An array of the bounding boxes for detected objects.
• out.classes: Identifies the type of object detected.
• out.confidences: Displays the confidence level the model has in its classification.
• out.avg_conf: An average of all of the confidence values.

The provides CVDemoUtils contains a handy method cvDemo.drawDetectedObjectsFromInference(results) that takes these values and combines then with the images to show the bounding boxes of detected objects. Not that inf-results requires the inference results derived earlier.

elapsed = 1.0

sample = {
    'model_name' : mobilenet_model_name ,
    'pipeline_name' : pipeline_name,
    'width': width,
    'height': height,
    'image' : resizedImage,
    'inf-results' : results,
    'confidence-target' : 0.50,
    'inference-time': 0,
    'onnx-time' : int(elapsed) / 1e+9,                
    'color':(255,0,0)
}

image = cvDemo.drawDetectedObjectsFromInference(sample)

elapsed = 1.0

sample = {
    'model_name' : mobilenet_model_name ,
    'pipeline_name' : pipeline_name,
    'width': width,
    'height': height,
    'image' : resizedImage,
    'inf-results' : results,
    'confidence-target' : 0.50,
    'inference-time': 0,
    'onnx-time' : int(elapsed) / 1e+9,                
    'color':(255,0,0)
}

image = cvDemo.drawDetectedObjectsFromInference(sample)

Undeploying Your Pipeline

You should always undeploy your pipelines when you are done with them, or don’t need them for a while. This releases the resources that the pipeline is using for other processes to use. You can always redeploy the pipeline when you need it again. As a reminder, here are the commands to deploy and undeploy a pipeline:

# when the pipeline is deployed, it's ready to receive data and infer
pipeline.deploy()

# "turn off" the pipeline and releaase its resources
pipeline.undeploy()

If you are continuing on to the next notebook now, you can leave the pipeline deployed to keep working; but if you are taking a break, then you should undeploy.

## blank space to undeploy the pipeline, if needed

pipeline.undeploy()

Congratulations!

You have now

• Successfully trained a model
• Converted your model and uploaded it to Wallaroo
• Created and deployed a simple single-step pipeline
• Successfully send data to your pipeline for inference

In the next notebook, you will look at two different ways to evaluate your model against the real world environment.