1 - Aloha Quick Tutorial

The Aloha Quick Start Guide demonstrates how to use Wallaroo to determine malicious web sites from their URL.

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

Aloha Demo

In this notebook we will walk through a simple pipeline deployment to inference on a model. For this example we will be using an open source model that uses an Aloha CNN LSTM model for classifying Domain names as being either legitimate or being used for nefarious purposes such as malware distribution.

Prerequisites

  • An installed Wallaroo instance.
  • The following Python libraries installed:
    • os
    • wallaroo: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.
    • pandas: Pandas, mainly used for Pandas DataFrame
    • pyarrow: PyArrow for Apache Arrow support

Tutorial Goals

For our example, we will perform the following:

  • Create a workspace for our work.
  • Upload the Aloha model.
  • Create a pipeline that can ingest our submitted data, submit it to the model, and export the results
  • Run a sample inference through our pipeline by loading a file
  • Run a sample inference through our pipeline’s URL and store the results in a file.

All sample data and models are available through the Wallaroo Quick Start Guide Samples repository.

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.e/wallaroo-sdk-essentials-client/).

import wallaroo
from wallaroo.object import EntityNotFoundError

# to display dataframe tables
from IPython.display import display
# used to display dataframe information without truncating
import pandas as pd
pd.set_option('display.max_colwidth', None)
import pyarrow as pa
# Login through local Wallaroo instance

wl = wallaroo.Client()

Create the Workspace

We will create a workspace to work in and call it the “alohaworkspace”, then set it as current workspace environment. We’ll also create our pipeline in advance as alohapipeline. The model name and the model file will be specified for use in later steps.

To allow this tutorial to be run multiple times or by multiple users in the same Wallaroo instance, a random 4 character prefix will be added to the workspace, pipeline, and model.

import string
import random

# make a random 4 character suffix to verify uniqueness in tutorials
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
workspace_name = f'alohaworkspace{suffix}'
pipeline_name = f'alohapipeline{suffix}'
model_name = f'alohamodel{suffix}'
model_file_name = './alohacnnlstm.zip'
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

def get_pipeline(name):
    try:
        pipeline = wl.pipelines_by_name(name)[0]
    except EntityNotFoundError:
        pipeline = wl.build_pipeline(name)
    return pipeline
wl.list_workspaces()[0:5]
[{'name': 'john.hummel@wallaroo.ai - Default Workspace', 'id': 1, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-24T16:55:38.589652+00:00', 'models': [{'name': 'hf-summarization-demoyns2', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 25, 15, 49, 36, 877913, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 25, 15, 49, 36, 877913, tzinfo=tzutc())}], 'pipelines': [{'name': 'hf-summarization-pipeline-edge', 'create_time': datetime.datetime(2023, 8, 25, 15, 52, 2, 329988, tzinfo=tzutc()), 'definition': '[]'}]},
 {'name': 'edge-publish-demojohn', 'id': 5, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-24T16:57:29.048825+00:00', 'models': [{'name': 'ccfraud', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 24, 16, 57, 29, 410708, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 24, 16, 57, 29, 410708, tzinfo=tzutc())}], 'pipelines': [{'name': 'edge-pipeline', 'create_time': datetime.datetime(2023, 8, 24, 16, 57, 29, 486951, tzinfo=tzutc()), 'definition': '[]'}]},
 {'name': 'workshop-workspace-sample', 'id': 6, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-24T21:01:46.66355+00:00', 'models': [{'name': 'house-price-prime', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 24, 21, 9, 13, 887924, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 24, 21, 8, 35, 654241, tzinfo=tzutc())}], 'pipelines': [{'name': 'houseprice-estimator', 'create_time': datetime.datetime(2023, 8, 24, 21, 16, 0, 405934, tzinfo=tzutc()), 'definition': '[]'}]},
 {'name': 'john-is-nice', 'id': 7, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-24T21:05:57.618824+00:00', 'models': [], 'pipelines': []},
 {'name': 'edge-hf-summarization', 'id': 8, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-25T18:43:02.41099+00:00', 'models': [{'name': 'hf-summarization', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 25, 18, 49, 26, 195772, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 25, 18, 49, 26, 195772, tzinfo=tzutc())}], 'pipelines': [{'name': 'edge-hf-summarization', 'create_time': datetime.datetime(2023, 8, 25, 18, 53, 19, 465175, tzinfo=tzutc()), 'definition': '[]'}]}]
workspace = get_workspace(workspace_name)

wl.set_current_workspace(workspace)

aloha_pipeline = get_pipeline(pipeline_name)
aloha_pipeline
namealohapipelineudjo
created2023-08-28 17:10:05.043499+00:00
last_updated2023-08-28 17:10:05.043499+00:00
deployed(none)
tags
versionse3f6ddff-9a25-48cc-9f94-fbc7bd67db29
steps
publishedFalse

We can verify the workspace is created the current default workspace with the get_current_workspace() command.

wl.get_current_workspace()
{'name': 'alohaworkspaceudjo', 'id': 13, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-28T17:10:04.351972+00:00', 'models': [], 'pipelines': [{'name': 'alohapipelineudjo', 'create_time': datetime.datetime(2023, 8, 28, 17, 10, 5, 43499, tzinfo=tzutc()), 'definition': '[]'}]}

Upload the Models

Now we will upload our models. Note that for this example we are applying the model from a .ZIP file. The Aloha model is a protobuf file that has been defined for evaluating web pages, and we will configure it to use data in the tensorflow format.

from wallaroo.framework import Framework

model = wl.upload_model(model_name, 
                        model_file_name,
                        framework=Framework.TENSORFLOW
                        )

Deploy a model

Now that we have a model that we want to use we will create a deployment for it.

We will tell the deployment we are using a tensorflow model and give the deployment name and the configuration we want for the deployment.

To do this, we’ll create our pipeline that can ingest the data, pass the data to our Aloha model, and give us a final output. We’ll call our pipeline aloha-test-demo, then deploy it so it’s ready to receive data. The deployment process usually takes about 45 seconds.

  • Note: If you receive an error that the pipeline could not be deployed because there are not enough resources, undeploy any other pipelines and deploy this one again. This command can quickly undeploy all pipelines to regain resources. We recommend not running this command in a production environment since it will cancel any running pipelines:
for p in wl.list_pipelines(): p.undeploy()
aloha_pipeline.add_model_step(model)
namealohapipelineudjo
created2023-08-28 17:10:05.043499+00:00
last_updated2023-08-28 17:10:05.043499+00:00
deployed(none)
tags
versionse3f6ddff-9a25-48cc-9f94-fbc7bd67db29
steps
publishedFalse
aloha_pipeline.deploy()
namealohapipelineudjo
created2023-08-28 17:10:05.043499+00:00
last_updated2023-08-28 17:10:10.176032+00:00
deployedTrue
tags
versions750fec3b-f178-4abd-9643-2208e0fd1fbd, e3f6ddff-9a25-48cc-9f94-fbc7bd67db29
stepsalohamodeludjo
publishedFalse

We can verify that the pipeline is running and list what models are associated with it.

aloha_pipeline.status()
{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.99',
   'name': 'engine-7f6ccf6879-hmw2p',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'alohapipelineudjo',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'alohamodeludjo',
      'version': 'b99dd290-870d-425e-8835-953decd402be',
      'sha': 'd71d9ffc61aaac58c2b1ed70a2db13d1416fb9d3f5b891e5e4e2e97180fe22f8',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.114',
   'name': 'engine-lb-584f54c899-4g6rh',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': []}

Interferences

Infer 1 row

Now that the pipeline is deployed and our Aloha model is in place, we’ll perform a smoke test to verify the pipeline is up and running properly. We’ll use the infer_from_file command to load a single encoded URL into the inference engine and print the results back out.

The result should tell us that the tokenized URL is legitimate (0) or fraud (1). This sample data should return close to 1 in out.main.

smoke_test = pd.DataFrame.from_records(
    [
    {
        "text_input":[
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            0,
            28,
            16,
            32,
            23,
            29,
            32,
            30,
            19,
            26,
            17
        ]
    }
]
)

result = aloha_pipeline.infer(smoke_test)
display(result.loc[:, ["time","out.main"]])
timeout.main
02023-08-28 17:10:27.482[0.997564]

Infer From File

This time, we’ll give it a bigger set of data to infer. ./data/data_1k.arrow is an Apache Arrow table with 1,000 records in it. Once submitted, we’ll turn the result into a DataFrame and display the first five results.

result = aloha_pipeline.infer_from_file('./data/data_1k.arrow')
display(result.to_pandas().loc[:, ["time","out.main"]])
timeout.main
02023-08-28 17:10:28.399[0.997564]
12023-08-28 17:10:28.399[0.9885122]
22023-08-28 17:10:28.399[0.9993358]
32023-08-28 17:10:28.399[0.99999857]
42023-08-28 17:10:28.399[0.9984837]
.........
9952023-08-28 17:10:28.399[0.9999754]
9962023-08-28 17:10:28.399[0.9999727]
9972023-08-28 17:10:28.399[0.66066873]
9982023-08-28 17:10:28.399[0.9998954]
9992023-08-28 17:10:28.399[0.99999803]

1000 rows × 2 columns

outputs =  result.to_pandas()
display(outputs.loc[:5, ["time","out.main"]])
timeout.main
02023-08-28 17:10:28.399[0.997564]
12023-08-28 17:10:28.399[0.9885122]
22023-08-28 17:10:28.399[0.9993358]
32023-08-28 17:10:28.399[0.99999857]
42023-08-28 17:10:28.399[0.9984837]
52023-08-28 17:10:28.399[1.0]

Batch Inference

Now that our smoke test is successful, let’s really give it some data. We have two inference files we can use:

  • data_1k.arrow: Contains 10,000 inferences
  • data_25k.arrow: Contains 25,000 inferences

When Apache Arrow tables are submitted to a Wallaroo Pipeline, the inference is processed natively as an Arrow table, and the results are returned as an Arrow table. This allows for faster data processing than with JSON files or DataFrame objects.

We’ll pipe the data_25k.arrow file through the aloha_pipeline deployment URL, and place the results in a file named response.arrow. We’ll also display the time this takes. Note that for larger batches of 50,000 inferences or more can be difficult to view in Jupyter Hub because of its size, so we’ll only display the first five rows.

  • IMPORTANT NOTE: The _deployment._url() method will return an internal URL when using Python commands from within the Wallaroo instance - for example, the Wallaroo JupyterHub service. When connecting via an external connection, _deployment._url() returns an external URL. External URL connections requires the authentication be included in the HTTP request, and that Model Endpoints Guide external endpoints are enabled in the Wallaroo configuration options.
inference_url = aloha_pipeline._deployment._url()
inference_url
'https://doc-test.api.wallarooexample.ai/v1/api/pipelines/infer/alohapipelineudjo-15/alohapipelineudjo'
connection =wl.mlops().__dict__
token = connection['token']
dataFile="./data/data_25k.arrow"
contentType="application/vnd.apache.arrow.file"
!curl -X POST {inference_url} -H "Authorization: Bearer {token}" -H "Content-Type:{contentType}" --data-binary @{dataFile} > curl_response.df
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 25.9M  100 21.1M  100 4874k  1740k   391k  0:00:12  0:00:12 --:--:-- 4647k
cc_data_from_file =  pd.read_json('./curl_response.df', orient="records")
display(cc_data_from_file.head(5).loc[:5, ["time","out"]])
timeout
01693242632007{'banjori': [0.0015195821], 'corebot': [0.9829147500000001], 'cryptolocker': [0.012099549000000001], 'dircrypt': [4.7591115e-05], 'gozi': [2.0289428e-05], 'kraken': [0.00031977256999999996], 'locky': [0.011029262000000001], 'main': [0.997564], 'matsnu': [0.010341609], 'pykspa': [0.008038961], 'qakbot': [0.016155055], 'ramdo': [0.00623623], 'ramnit': [0.0009985747000000001], 'simda': [1.7933434e-26], 'suppobox': [1.388995e-27]}
11693242632007{'banjori': [7.447196e-18], 'corebot': [6.7359245e-08], 'cryptolocker': [0.1708199], 'dircrypt': [1.3220122000000002e-09], 'gozi': [1.2758705999999999e-24], 'kraken': [0.22559543], 'locky': [0.34209849999999997], 'main': [0.99999994], 'matsnu': [0.3080186], 'pykspa': [0.1828217], 'qakbot': [3.8022549999999994e-11], 'ramdo': [0.2062254], 'ramnit': [0.15215826], 'simda': [1.1701982e-30], 'suppobox': [3.1514454e-38]}
21693242632007{'banjori': [2.8598648999999997e-21], 'corebot': [9.302004000000001e-08], 'cryptolocker': [0.04445298], 'dircrypt': [6.1637580000000004e-09], 'gozi': [8.3496755e-23], 'kraken': [0.48234479999999996], 'locky': [0.26332903], 'main': [1.0], 'matsnu': [0.29800338], 'pykspa': [0.22361776], 'qakbot': [1.5238920999999999e-06], 'ramdo': [0.32820392], 'ramnit': [0.029332489000000003], 'simda': [1.1995622e-31], 'suppobox': [0.0]}
31693242632007{'banjori': [2.1387213e-15], 'corebot': [3.8817485e-10], 'cryptolocker': [0.045599736], 'dircrypt': [1.9090386e-07], 'gozi': [1.3140123e-25], 'kraken': [0.59542626], 'locky': [0.17374137], 'main': [0.9999996999999999], 'matsnu': [0.23151578], 'pykspa': [0.17591679999999998], 'qakbot': [1.0876152e-09], 'ramdo': [0.21832279999999998], 'ramnit': [0.0128692705], 'simda': [6.1588803e-28], 'suppobox': [1.4386237e-35]}
41693242632007{'banjori': [9.453342500000001e-15], 'corebot': [7.091151e-10], 'cryptolocker': [0.049815163], 'dircrypt': [5.2914135e-09], 'gozi': [7.4132087e-19], 'kraken': [1.5504574999999998e-13], 'locky': [1.079181e-15], 'main': [0.9999988999999999], 'matsnu': [1.5003075e-15], 'pykspa': [0.33075705], 'qakbot': [2.6258850000000004e-07], 'ramdo': [0.5036279], 'ramnit': [0.020393765], 'simda': [0.0], 'suppobox': [2.3292326e-38]}

Undeploy Pipeline

When finished with our tests, we will undeploy the pipeline so we have the Kubernetes resources back for other tasks. Note that if the deployment variable is unchanged aloha_pipeline.deploy() will restart the inference engine in the same configuration as before.

aloha_pipeline.undeploy()
namealohapipelineudjo
created2023-08-28 17:10:05.043499+00:00
last_updated2023-08-28 17:10:10.176032+00:00
deployedFalse
tags
versions750fec3b-f178-4abd-9643-2208e0fd1fbd, e3f6ddff-9a25-48cc-9f94-fbc7bd67db29
stepsalohamodeludjo
publishedFalse

2 - Computer Vision Tutorials

How to use Wallaroo with computer vision models to detect objects in images.

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

Step 00: Introduction and Setup

This tutorial demonstrates how to use the Wallaroo to detect objects in images through the following models:

  • rnn mobilenet: A single stage object detector that performs fast inferences. Mobilenet is typically good at identifying objects at a distance.
  • resnet50: A dual stage object detector with slower inferencing but but is able to detect objects that are closer to each other.

This tutorial series will demonstrate the following:

  • How to deploy a Wallaroo pipeline with trained rnn mobilenet model and perform sample inferences to detect objects in pictures, then display those objects.
  • How to deploy a Wallaroo pipeline with a trained resnet50 model and perform sample inferences to detect objects in pictures, then display those objects.
  • Use the Wallaroo feature shadow deploy to have both models perform inferences, then select the inference result with the higher confidence and show the objects detected.

This tutorial assumes that users have installed the Wallaroo SDK or are running these tutorials from within their Wallaroo instance’s JupyterHub service.

This demonstration should be run within a Wallaroo JupyterHub instance for best results.

Prerequisites

The included OpenCV class is included in this demonstration as CVDemoUtils.py, and requires the following dependencies:

  • ffmpeg
  • libsm
  • libxext

Internal JupyterHub Service

To install these dependencies in the Wallaroo JupyterHub service, use the following commands from a terminal shell via the following procedure:

  1. Launch the JupyterHub Service within the Wallaroo install.

  2. Select File->New->Terminal.

  3. Enter the following:

    sudo apt-get update
    
    sudo apt-get install ffmpeg libsm6 libxext6  -y
    

External SDK Users

For users using the Wallaroo SDK to connect with a remote Wallaroo instance, the following commands will install the required dependancies:

For Linux users, this can be installed with:

sudo apt-get update
sudo apt-get install ffmpeg libsm6 libxext6  -y

MacOS users can prepare their environments using a package manager such as Brew with the following:

brew install ffmpeg libsm libxext

Libraries and Dependencies

  1. This repository may use large file sizes for the models. Use the Wallaroo Tutorials Releases to download a .zip file of the most recent computer vision tutorial that includes the models.
  2. Import the following Python libraries into your environment:
    1. torch
    2. wallaroo
    3. torchvision
    4. opencv-python
    5. onnx
    6. onnxruntime
    7. imutils
    8. pytz
    9. ipywidgets

These can be installed by running the command below in the Wallaroo JupyterHub service. Note the use of pip install torch --no-cache-dir for low memory environments.

!pip install torchvision
!pip install torch --no-cache-dir
!pip install opencv-python
!pip install onnx
!pip install onnxruntime
!pip install imutils
!pip install pytz
!pip install ipywidgets
Requirement already satisfied: torchvision in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (0.14.1)
Requirement already satisfied: typing-extensions in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torchvision) (4.5.0)
Requirement already satisfied: numpy in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torchvision) (1.22.3)
Requirement already satisfied: requests in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torchvision) (2.25.1)
Requirement already satisfied: torch in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torchvision) (1.13.1)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torchvision) (10.1.0)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from requests->torchvision) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from requests->torchvision) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from requests->torchvision) (1.26.18)
Requirement already satisfied: certifi>=2017.4.17 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from requests->torchvision) (2023.7.22)
Requirement already satisfied: torch in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (1.13.1)
Requirement already satisfied: typing-extensions in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torch) (4.5.0)
Requirement already satisfied: opencv-python in /Users/johnhansarick/.local/lib/python3.8/site-packages (4.8.1.78)
Requirement already satisfied: numpy>=1.21.0 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from opencv-python) (1.22.3)
Requirement already satisfied: onnx in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (1.15.0)
Requirement already satisfied: numpy in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnx) (1.22.3)
Requirement already satisfied: protobuf>=3.20.2 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnx) (4.24.4)
Collecting onnxruntime
  Using cached onnxruntime-1.16.1-cp38-cp38-macosx_11_0_arm64.whl.metadata (4.1 kB)
Collecting coloredlogs (from onnxruntime)
  Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
Requirement already satisfied: flatbuffers in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnxruntime) (1.12)
Requirement already satisfied: numpy>=1.21.6 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnxruntime) (1.22.3)
Requirement already satisfied: packaging in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnxruntime) (23.1)
Requirement already satisfied: protobuf in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnxruntime) (4.24.4)
Collecting sympy (from onnxruntime)
  Using cached sympy-1.12-py3-none-any.whl (5.7 MB)
Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime)
  Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
Collecting mpmath>=0.19 (from sympy->onnxruntime)
  Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)
Using cached onnxruntime-1.16.1-cp38-cp38-macosx_11_0_arm64.whl (6.1 MB)
Installing collected packages: mpmath, sympy, humanfriendly, coloredlogs, onnxruntime
Successfully installed coloredlogs-15.0.1 humanfriendly-10.0 mpmath-1.3.0 onnxruntime-1.16.1 sympy-1.12
Collecting imutils
  Using cached imutils-0.5.4-py3-none-any.whl
Installing collected packages: imutils
Successfully installed imutils-0.5.4
Requirement already satisfied: pytz in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (2023.3.post1)
Collecting ipywidgets
  Using cached ipywidgets-8.1.1-py3-none-any.whl.metadata (2.4 kB)
Collecting comm>=0.1.3 (from ipywidgets)
  Using cached comm-0.1.4-py3-none-any.whl.metadata (4.2 kB)
Requirement already satisfied: ipython>=6.1.0 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipywidgets) (7.24.1)
Requirement already satisfied: traitlets>=4.3.1 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipywidgets) (5.12.0)
Collecting widgetsnbextension~=4.0.9 (from ipywidgets)
  Using cached widgetsnbextension-4.0.9-py3-none-any.whl.metadata (1.6 kB)
Collecting jupyterlab-widgets~=3.0.9 (from ipywidgets)
  Using cached jupyterlab_widgets-3.0.9-py3-none-any.whl.metadata (4.1 kB)
Requirement already satisfied: setuptools>=18.5 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (68.0.0)
Requirement already satisfied: jedi>=0.16 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.18.1)
Requirement already satisfied: decorator in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1)
Requirement already satisfied: pickleshare in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.36)
Requirement already satisfied: pygments in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (2.15.1)
Requirement already satisfied: backcall in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.2.0)
Requirement already satisfied: matplotlib-inline in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.6)
Requirement already satisfied: pexpect>4.3 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (4.8.0)
Requirement already satisfied: appnope in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.2)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.3)
Requirement already satisfied: ptyprocess>=0.5 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0)
Requirement already satisfied: wcwidth in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=6.1.0->ipywidgets) (0.2.5)
Using cached ipywidgets-8.1.1-py3-none-any.whl (139 kB)
Using cached comm-0.1.4-py3-none-any.whl (6.6 kB)
Using cached jupyterlab_widgets-3.0.9-py3-none-any.whl (214 kB)
Using cached widgetsnbextension-4.0.9-py3-none-any.whl (2.3 MB)
Installing collected packages: widgetsnbextension, jupyterlab-widgets, comm, ipywidgets
  Attempting uninstall: comm
    Found existing installation: comm 0.1.2
    Uninstalling comm-0.1.2:
      Successfully uninstalled comm-0.1.2
Successfully installed comm-0.1.4 ipywidgets-8.1.1 jupyterlab-widgets-3.0.9 widgetsnbextension-4.0.9

The rest of the tutorials will rely on these libraries and applications, so finish their installation before running the tutorials in this series.

Models for Wallaroo Computer Vision Tutorials

In order for the wallaroo tutorial notebooks to run properly, the videos directory must contain these models in the models directory.

To download the Wallaroo Computer Vision models, use the following link:

https://storage.googleapis.com/wallaroo-public-data/cv-demo-models/cv-retail-models.zip

Unzip the contents into the directory models.

Directory contents

  • coco_classes.pickle - contain the 80 COCO classifications used by resnet50 and mobilenet object detectors.
  • frcnn-resent.pt - PyTorch resnet50 model
  • frcnn-resnet.pt.onnx - PyTorch resnet50 model converted to onnx
  • mobilenet.pt - PyTorch mobilenet model
  • mobilenet.pt.onnx - PyTorch mobilenet model converted to onnx

2.1 - Step 01: Detecting Objects Using mobilenet

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

Step 01: Detecting Objects Using mobilenet

The following tutorial demonstrates how to use a trained mobilenet model deployed in Wallaroo to detect objects. This process will use the following steps:

  1. Create a Wallaroo workspace and pipeline.
  2. Upload a trained mobilenet ML model and add it as a pipeline step.
  3. Deploy the pipeline.
  4. Perform an inference on a sample image.
  5. Draw the detected objects, their bounding boxes, their classifications, and the confidence of the classifications on the provided image.
  6. Review our results.

Steps

Import Libraries

The first step will be to import our libraries. Please check with Step 00: Introduction and Setup and verify that the necessary libraries and applications are added to your environment.

import torch
import pickle
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

import numpy as np
import json
import requests
import time
import pandas as pd
from CVDemoUtils import CVDemo

# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

# used for unique connection names

import string
import random
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
suffix=''

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local service

wl = wallaroo.Client()

Set Variables

The following variables and methods are used later to create or connect to an existing workspace, pipeline, and model.

workspace_name = f'mobilenetworkspacetest{suffix}'
pipeline_name = f'mobilenetpipeline{suffix}'
model_name = f'mobilenet{suffix}'
model_file_name = 'models/mobilenet.pt.onnx'
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

def get_pipeline(name):
    try:
        pipeline = wl.pipelines_by_name(name)[0]
    except EntityNotFoundError:
        pipeline = wl.build_pipeline(name)
    return pipeline

Create Workspace

The workspace will be created or connected to, and set as the default workspace for this session. Once that is done, then all models and pipelines will be set in that workspace.

workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
wl.get_current_workspace()
{'name': 'mobilenetworkspacetest', 'id': 8, 'archived': False, 'created_by': '1b26fc10-d0f9-4f92-a1a2-ab342b3d069e', 'created_at': '2023-10-30T18:15:15.756292+00:00', 'models': [], 'pipelines': []}

Create Pipeline and Upload Model

We will now create or connect to an existing pipeline as named in the variables above.

pipeline = get_pipeline(pipeline_name)
mobilenet_model = wl.upload_model(model_name, model_file_name, framework=Framework.ONNX).configure(batch_config="single", tensor_fields=["tensor"])

Deploy Pipeline

With the model uploaded, we can add it is as a step in the pipeline, then deploy it. Once deployed, resources from the Wallaroo instance will be reserved and the pipeline will be ready to use the model to perform inference requests.

pipeline.add_model_step(mobilenet_model)

pipeline.deploy()
namemobilenetpipeline
created2023-10-30 18:15:17.969988+00:00
last_updated2023-10-30 18:15:25.186553+00:00
deployedTrue
archNone
tags
versions7a1d84f9-13f7-405a-a4d0-5ec7de59f03e, b236b83b-1755-41d9-a2ae-804c040b9038
stepsmobilenet
publishedFalse

Prepare input image

Next we will load a sample image and resize it to the width and height required for the object detector. Once complete, it the image will be converted to a numpy ndim array and added to a dictionary.


# The size the image will be resized to
width = 640
height = 480

# Only objects that have a confidence > confidence_target will be displayed on the image
cvDemo = CVDemo()

imagePath = 'data/images/input/example/dairy_bottles.png'

# The image width and height needs to be set to what the model was trained for.  In this case 640x480.
tensor, resizedImage = cvDemo.loadImageAndResize(imagePath, width, height)

# get npArray from the tensorFloat
npArray = tensor.cpu().numpy()

#creates a dictionary with the wallaroo "tensor" key and the numpy ndim array representing image as the value.

dictData = {"tensor":[npArray]}
dataframedata = pd.DataFrame(dictData)

Run Inference

With that done, we can have the model detect the objects on the image by running an inference through the pipeline, and storing the results for the next step.

startTime = time.time()
# pass the dataframe in 
#infResults = pipeline.infer(dataframedata, dataset=["*", "metadata.elapsed"])
infResults = pipeline.infer_from_file('./data/dairy_bottles.df.json', dataset=["*", "metadata.elapsed"])

endTime = time.time()

Draw the Inference Results

With our inference results, we can take them and use the Wallaroo CVDemo class and draw them onto the original image. The bounding boxes and the confidence value will only be drawn on images where the model returned a 90% confidence rate in the object’s identity.

df = pd.DataFrame(columns=['classification','confidence','x','y','width','height'])
pd.options.mode.chained_assignment = None  # default='warn'
pd.options.display.float_format = '{:.2%}'.format

# Points to where all the inference results are
boxList = infResults.loc[0]["out.boxes"]

# # reshape this to an array of bounding box coordinates converted to ints
boxA = np.array(boxList)
boxes = boxA.reshape(-1, 4)
boxes = boxes.astype(int)

df[['x', 'y','width','height']] = pd.DataFrame(boxes)

classes = infResults.loc[0]["out.classes"]
confidences = infResults.loc[0]["out.confidences"]

infResults = {
    'model_name' : model_name,
    'pipeline_name' : pipeline_name,
    'width': width,
    'height': height,
    'image' : resizedImage,
    'boxes' : boxes,
    'classes' : classes,
    'confidences' : confidences,
    'confidence-target' : 0.90,
    'inference-time': (endTime-startTime),
    'onnx-time' : int(infResults.loc[0]["metadata.elapsed"][1]) / 1e+9,                
    'color':(255,0,0)
}

image = cvDemo.drawAndDisplayDetectedObjectsWithClassification(infResults)

Extract the Inference Information

To show what is going on in the background, we’ll extract the inference results create a dataframe with columns representing the classification, confidence, and bounding boxes of the objects identified.

idx = 0 
for idx in range(0,len(classes)):
    df['classification'][idx] = cvDemo.CLASSES[classes[idx]] # Classes contains the 80 different COCO classificaitons
    df['confidence'][idx] = confidences[idx]
df
classificationconfidencexywidthheight
0bottle98.65%021085479
1bottle90.12%72197151468
2bottle60.78%211184277420
3bottle59.22%143203216448
4refrigerator53.73%1341640480
5bottle45.13%106206159463
6bottle43.73%278132193
7bottle43.09%462104510224
8bottle40.85%310135294
9bottle39.19%528268636475
10bottle35.76%220025890
11bottle31.81%55296600233
12bottle26.45%349040498
13bottle23.06%450264619472
14bottle20.48%261193307408
15bottle17.46%509101544235
16bottle17.31%592100633239
17bottle16.00%475297551468
18bottle14.91%368163423362
19book13.66%120017581
20book13.32%72014385
21bottle12.22%271200305274
22book12.13%161021385
23bottle11.96%162021483
24bottle11.53%310190367397
25bottle9.62%396166441360
26cake8.65%439256640473
27bottle7.84%544375636472
28vase7.23%272230696
29bottle6.28%453303524463
30bottle5.28%60994635211

Undeploy the Pipeline

With the inference complete, we can undeploy the pipeline and return the resources back to the Wallaroo instance.

pipeline.undeploy()
namemobilenetpipeline
created2023-10-30 18:15:17.969988+00:00
last_updated2023-10-30 18:15:25.186553+00:00
deployedFalse
archNone
tags
versions7a1d84f9-13f7-405a-a4d0-5ec7de59f03e, b236b83b-1755-41d9-a2ae-804c040b9038
stepsmobilenet
publishedFalse

2.2 - Step 02: Detecting Objects Using resnet50

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

Step 02: Detecting Objects Using resnet50

The following tutorial demonstrates how to use a trained mobilenet model deployed in Wallaroo to detect objects. This process will use the following steps:

  1. Create a Wallaroo workspace and pipeline.
  2. Upload a trained resnet50 ML model and add it as a pipeline step.
  3. Deploy the pipeline.
  4. Perform an inference on a sample image.
  5. Draw the detected objects, their bounding boxes, their classifications, and the confidence of the classifications on the provided image.
  6. Review our results.

Steps

Import Libraries

The first step will be to import our libraries. Please check with Step 00: Introduction and Setup and verify that the necessary libraries and applications are added to your environment.

import torch
import pickle
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

import numpy as np
import json
import requests
import time
import pandas as pd
from CVDemoUtils import CVDemo

# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

# used for unique connection names

import string
import random
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local service

wl = wallaroo.Client()

wl = wallaroo.Client()

Set Variables

The following variables and methods are used later to create or connect to an existing workspace, pipeline, and model. This example has both the resnet model, and a post process script.

workspace_name = f'resnetworkspace{suffix}'
pipeline_name = f'resnetnetpipeline{suffix}'
model_name = f'resnet50{suffix}'
model_file_name = 'models/frcnn-resnet.pt.onnx'
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

def get_pipeline(name):
    try:
        pipeline = wl.pipelines_by_name(name)[0]
    except EntityNotFoundError:
        pipeline = wl.build_pipeline(name)
    return pipeline

Create Workspace

The workspace will be created or connected to, and set as the default workspace for this session. Once that is done, then all models and pipelines will be set in that workspace.

workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
wl.get_current_workspace()
{'name': 'resnetworkspacestck', 'id': 9, 'archived': False, 'created_by': '6236ad2a-7eb8-4bbc-a8c9-39ce92767bad', 'created_at': '2023-10-24T16:52:27.179962+00:00', 'models': [], 'pipelines': []}

Create Pipeline and Upload Model

We will now create or connect to an existing pipeline as named in the variables above.

pipeline = get_pipeline(pipeline_name)

resnet_model = wl.upload_model(model_name, model_file_name, framework=Framework.ONNX).configure(batch_config="single", tensor_fields=["tensor"])

Deploy Pipeline

With the model uploaded, we can add it is as a step in the pipeline, then deploy it. Once deployed, resources from the Wallaroo instance will be reserved and the pipeline will be ready to use the model to perform inference requests.

pipeline.add_model_step(resnet_model)
nameresnetnetpipelinestck
created2023-10-24 16:52:29.027087+00:00
last_updated2023-10-24 16:52:29.027087+00:00
deployed(none)
tags
versions6e3874c5-e1bb-4847-a60d-2c42b7bc0a7e
steps
publishedFalse
pipeline.deploy()
nameresnetnetpipelinestck
created2023-10-24 16:52:29.027087+00:00
last_updated2023-10-24 16:52:40.680056+00:00
deployedTrue
tags
versions52cb9d23-29cc-431f-9b08-d95d337b5bea, 6e3874c5-e1bb-4847-a60d-2c42b7bc0a7e
stepsresnet50stck
publishedFalse

Test the pipeline by running inference on a sample image

Prepare input image

Next we will load a sample image and resize it to the width and height required for the object detector.

We will convert the image to a numpy ndim array and add it do a dictionary

#The size the image will be resized to
width = 640
height = 480

cvDemo = CVDemo()

imagePath = 'data/images/input/example/dairy_bottles.png'

# The image width and height needs to be set to what the model was trained for.  In this case 640x480.
tensor, resizedImage = cvDemo.loadImageAndResize(imagePath, width, height)

# get npArray from the tensorFloat
npArray = tensor.cpu().numpy()

dictData = {"tensor":[npArray]}
dataframedata = pd.DataFrame(dictData)

Run Inference

With that done, we can have the model detect the objects on the image by running an inference through the pipeline, and storing the results for the next step.

IMPORTANT NOTE: If necessary, add timeout=60 to the infer method if more time is needed to upload the data file for the inference request.

startTime = time.time()
# pass the dataframe in 
# infResults = pipeline.infer(dataframedata, dataset=["*", "metadata.elapsed"])
infResults = pipeline.infer_from_file('./data/dairy_bottles.df.json', dataset=["*", "metadata.elapsed"])
endTime = time.time()

Draw the Inference Results

With our inference results, we can take them and use the Wallaroo CVDemo class and draw them onto the original image. The bounding boxes and the confidence value will only be drawn on images where the model returned a 50% confidence rate in the object’s identity.

df = pd.DataFrame(columns=['classification','confidence','x','y','width','height'])
pd.options.mode.chained_assignment = None  # default='warn'
pd.options.display.float_format = '{:.2%}'.format

# Points to where all the inference results are
boxList = infResults.loc[0]["out.boxes"]

# # reshape this to an array of bounding box coordinates converted to ints

boxA = np.array(boxList)
boxes = boxA.reshape(-1, 4)
boxes = boxes.astype(int)

df[['x', 'y','width','height']] = pd.DataFrame(boxes)

classes = infResults.loc[0]["out.classes"]
confidences = infResults.loc[0]["out.confidences"]

infResults = {
    'model_name' : model_name,
    'pipeline_name' : pipeline_name,
    'width': width,
    'height': height,
    'image' : resizedImage,
    'boxes' : boxes,
    'classes' : classes,
    'confidences' : confidences,
    'confidence-target' : 0.90,
    'inference-time': (endTime-startTime),
    'onnx-time' : int(infResults.loc[0]["metadata.elapsed"][1]) / 1e+9,                
    'color':(255,0,0)
}

image = cvDemo.drawAndDisplayDetectedObjectsWithClassification(infResults)

Extract the Inference Information

To show what is going on in the background, we’ll extract the inference results create a dataframe with columns representing the classification, confidence, and bounding boxes of the objects identified.

idx = 0 
for idx in range(0,len(classes)):
    cocoClasses = cvDemo.getCocoClasses()
    df['classification'][idx] = cocoClasses[classes[idx]] # Classes contains the 80 different COCO classificaitons
    df['confidence'][idx] = confidences[idx]
df
classificationconfidencexywidthheight
0bottle99.65%219376475
1bottle98.83%61098639232
2bottle97.00%54498581230
3bottle96.96%454113484210
4bottle96.48%502331551476
.....................
95bottle5.72%556287580322
96refrigerator5.66%80161638480
97bottle5.60%455334480349
98bottle5.46%613267635375
99bottle5.37%345239599

100 rows × 6 columns

Undeploy the Pipeline

With the inference complete, we can undeploy the pipeline and return the resources back to the Wallaroo instance.

pipeline.undeploy()
nameresnetnetpipelinestck
created2023-10-24 16:52:29.027087+00:00
last_updated2023-10-24 16:52:40.680056+00:00
deployedFalse
tags
versions52cb9d23-29cc-431f-9b08-d95d337b5bea, 6e3874c5-e1bb-4847-a60d-2c42b7bc0a7e
stepsresnet50stck
publishedFalse

2.3 - Step 03: mobilenet and resnet50 Shadow Deploy

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

Step 03: Detecting Objects Using Shadow Deploy

The following tutorial demonstrates how to use two trained models, one based on the resnet50, the other on mobilenet, deployed in Wallaroo to detect objects. This builds on the previous tutorials in this series, Step 01: Detecting Objects Using mobilenet" and “Step 02: Detecting Objects Using resnet50”.

For this tutorial, the Wallaroo feature Shadow Deploy will be used to submit inference requests to both models at once. The mobilnet object detector is the control and the faster-rcnn object detector is the challenger. The results between the two will be compared for their confidence, and that confidence will be used to draw bounding boxes around identified objects.

This process will use the following steps:

  1. Create a Wallaroo workspace and pipeline.
  2. Upload a trained resnet50 ML model and trained mobilenet model and add them as a shadow deployed step with the mobilenet as the control model.
  3. Deploy the pipeline.
  4. Perform an inference on a sample image.
  5. Based on the
  6. Draw the detected objects, their bounding boxes, their classifications, and the confidence of the classifications on the provided image.
  7. Review our results.

Steps

Import Libraries

The first step will be to import our libraries. Please check with Step 00: Introduction and Setup and verify that the necessary libraries and applications are added to your environment.

import torch
import pickle
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

import numpy as np
import json
import requests
import time
import pandas as pd
from CVDemoUtils import CVDemo

# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

# used for unique connection names

import string
import random
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local service

wl = wallaroo.Client()

wl = wallaroo.Client()

Set Variables

The following variables and methods are used later to create or connect to an existing workspace, pipeline, and model. This example has both the resnet model, and a post process script.

workspace_name = f'shadowimageworkspacetest{suffix}'
pipeline_name = f'shadowimagepipelinetest{suffix}'
control_model_name = f'mobilenet{suffix}'
control_model_file_name = 'models/mobilenet.pt.onnx'
challenger_model_name = f'resnet50{suffix}'
challenger_model_file_name = 'models/frcnn-resnet.pt.onnx'
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

def get_pipeline(name):
    try:
        pipeline = wl.pipelines_by_name(name)[0]
    except EntityNotFoundError:
        pipeline = wl.build_pipeline(name)
    return pipeline

Create Workspace

The workspace will be created or connected to, and set as the default workspace for this session. Once that is done, then all models and pipelines will be set in that workspace.

workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
wl.get_current_workspace()
{'name': 'shadowimageworkspacetestkpld', 'id': 10, 'archived': False, 'created_by': '6236ad2a-7eb8-4bbc-a8c9-39ce92767bad', 'created_at': '2023-10-24T17:05:03.040658+00:00', 'models': [], 'pipelines': []}

Create Pipeline and Upload Model

We will now create or connect to an existing pipeline as named in the variables above, then upload each of the models.

pipeline = get_pipeline(pipeline_name)
control =  wl.upload_model(control_model_name, control_model_file_name, framework=Framework.ONNX).configure(batch_config="single", tensor_fields=["tensor"])
challenger = wl.upload_model(challenger_model_name, challenger_model_file_name, framework=Framework.ONNX).configure(batch_config="single", tensor_fields=["tensor"])

Shadow Deploy Pipeline

For this step, rather than deploying each model into a separate step, both will be deployed into a single step as a Shadow Deploy step. This will take the inference input data and process it through both pipelines at the same time. The inference results for the control will be stored in it’s ['outputs'] array, while the results for the challenger are stored the ['shadow_data'] array.

pipeline.add_shadow_deploy(control, [challenger])
nameshadowimagepipelinetestkpld
created2023-10-24 17:05:04.926467+00:00
last_updated2023-10-24 17:05:04.926467+00:00
deployed(none)
tags
versions2f6bee8b-bf31-41cf-b7d2-d4912bfdcca8
steps
publishedFalse
pipeline.deploy()
nameshadowimagepipelinetestkpld
created2023-10-24 17:05:04.926467+00:00
last_updated2023-10-24 17:05:22.613731+00:00
deployedTrue
tags
versions433aa539-cb64-4733-85d3-68f6f769dd36, 2f6bee8b-bf31-41cf-b7d2-d4912bfdcca8
stepsmobilenetkpld
publishedFalse
pipeline.status()
{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.35',
   'name': 'engine-5888d5b5d6-2dvlb',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'shadowimagepipelinetestkpld',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'mobilenetkpld',
      'version': 'be97c0ef-afb1-4835-9f94-ec6534fa9c07',
      'sha': '9044c970ee061cc47e0c77e20b05e884be37f2a20aa9c0c3ce1993dbd486a830',
      'status': 'Running'},
     {'name': 'resnet50kpld',
      'version': 'f0967b5f-4b17-4dbd-b1d6-49b3339f041b',
      'sha': '43326e50af639105c81372346fb9ddf453fea0fe46648b2053c375360d9c1647',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.58',
   'name': 'engine-lb-584f54c899-hl4lx',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': []}

Prepare input image

Next we will load a sample image and resize it to the width and height required for the object detector.

We will convert the image to a numpy ndim array and add it do a dictionary


imagePath = 'data/images/input/example/store-front.png'

# The image width and height needs to be set to what the model was trained for.  In this case 640x480.
cvDemo = CVDemo()

# The size the image will be resized to meet the input requirements of the object detector
width = 640
height = 480
tensor, controlImage = cvDemo.loadImageAndResize(imagePath, width, height)
challengerImage = controlImage.copy()

# get npArray from the tensorFloat
npArray = tensor.cpu().numpy()

#creates a dictionary with the wallaroo "tensor" key and the numpy ndim array representing image as the value.
# dictData = {"tensor": npArray.tolist()}

dictData = {"tensor":[npArray]}
dataframedata = pd.DataFrame(dictData)

Run Inference using Shadow Deployment

Now lets have the model detect the objects on the image by running inference and extracting the results

startTime = time.time()
infResults = pipeline.infer_from_file('./data/dairy_bottles.df.json', dataset=["*", "metadata.elapsed"])
#infResults = pipeline.infer(dataframedata, dataset=["*", "metadata.elapsed"])
endTime = time.time()

Extract Control Inference Results

First we’ll extract the inference result data for the control model and map it onto the image.

df = pd.DataFrame(columns=['classification','confidence','x','y','width','height'])
pd.options.mode.chained_assignment = None  # default='warn'
pd.options.display.float_format = '{:.2%}'.format

# Points to where all the inference results are
# boxList = infResults[0]["out.output"]
boxList = infResults.loc[0]["out.boxes"]

# # reshape this to an array of bounding box coordinates converted to ints
boxA = np.array(boxList)
controlBoxes = boxA.reshape(-1, 4)
controlBoxes = controlBoxes.astype(int)

df[['x', 'y','width','height']] = pd.DataFrame(controlBoxes)

controlClasses = infResults.loc[0]["out.classes"]
controlConfidences = infResults.loc[0]["out.confidences"]

results = {
    'model_name' : control.name(),
    'pipeline_name' : pipeline.name(),
    'width': width,
    'height': height,
    'image' : controlImage,
    'boxes' : controlBoxes,
    'classes' : controlClasses,
    'confidences' : controlConfidences,
    'confidence-target' : 0.9,
    'color':CVDemo.RED, # color to draw bounding boxes and the text in the statistics
    'inference-time': (endTime-startTime),
    'onnx-time' : 0,                
}
cvDemo.drawAndDisplayDetectedObjectsWithClassification(results)

Display the Control Results

Here we will use the Wallaroo CVDemo helper class to draw the control model results on the image.

The full results will be displayed in a dataframe with columns representing the classification, confidence, and bounding boxes of the objects identified.

Once extracted from the results we will want to reshape the flattened array into an array with 4 elements (x,y,width,height).

idx = 0 
cocoClasses = cvDemo.getCocoClasses()
for idx in range(0,len(controlClasses)):
    df['classification'][idx] = cocoClasses[controlClasses[idx]] # Classes contains the 80 different COCO classificaitons
    df['confidence'][idx] = controlConfidences[idx]
df
classificationconfidencexywidthheight
0bottle98.65%021085479
1bottle90.12%72197151468
2bottle60.78%211184277420
3bottle59.22%143203216448
4refrigerator53.73%1341640480
5bottle45.13%106206159463
6bottle43.73%278132193
7bottle43.09%462104510224
8bottle40.85%310135294
9bottle39.19%528268636475
10bottle35.76%220025890
11bottle31.81%55296600233
12bottle26.45%349040498
13bottle23.06%450264619472
14bottle20.48%261193307408
15bottle17.46%509101544235
16bottle17.31%592100633239
17bottle16.00%475297551468
18bottle14.91%368163423362
19book13.66%120017581
20book13.32%72014385
21bottle12.22%271200305274
22book12.13%161021385
23bottle11.96%162021483
24bottle11.53%310190367397
25bottle9.62%396166441360
26cake8.65%439256640473
27bottle7.84%544375636472
28vase7.23%272230696
29bottle6.28%453303524463
30bottle5.28%60994635211

Display the Challenger Results

Here we will use the Wallaroo CVDemo helper class to draw the challenger model results on the input image.

challengerDf = pd.DataFrame(columns=['classification','confidence','x','y','width','height'])
pd.options.mode.chained_assignment = None  # default='warn'
pd.options.display.float_format = '{:.2%}'.format

# Points to where all the inference results are
boxList = infResults.loc[0][f"out_{challenger_model_name}.boxes"]

# outputs = results['outputs']
# boxes = outputs[0]

# # reshape this to an array of bounding box coordinates converted to ints
# boxList = boxes['Float']['data']
boxA = np.array(boxList)
challengerBoxes = boxA.reshape(-1, 4)
challengerBoxes = challengerBoxes.astype(int)

challengerDf[['x', 'y','width','height']] = pd.DataFrame(challengerBoxes)

challengerClasses = infResults.loc[0][f"out_{challenger_model_name}.classes"]
challengerConfidences = infResults.loc[0][f"out_{challenger_model_name}.confidences"]

results = {
    'model_name' : challenger.name(),
    'pipeline_name' : pipeline.name(),
    'width': width,
    'height': height,
    'image' : challengerImage,
    'boxes' : challengerBoxes,
    'classes' : challengerClasses,
    'confidences' : challengerConfidences,
    'confidence-target' : 0.9,
    'color':CVDemo.RED, # color to draw bounding boxes and the text in the statistics
    'inference-time': (endTime-startTime),
    'onnx-time' : 0,                
}
cvDemo.drawAndDisplayDetectedObjectsWithClassification(results)

Display Challenger Results

The inference results for the objects detected by the challenger model will be displayed including the confidence values. Once extracted from the results we will want to reshape the flattened array into an array with 4 elements (x,y,width,height).

idx = 0 
for idx in range(0,len(challengerClasses)):
    challengerDf['classification'][idx] = cvDemo.CLASSES[challengerClasses[idx]] # Classes contains the 80 different COCO classificaitons
    challengerDf['confidence'][idx] = challengerConfidences[idx]
challengerDf
classificationconfidencexywidthheight
0bottle99.65%219376475
1bottle98.83%61098639232
2bottle97.00%54498581230
3bottle96.96%454113484210
4bottle96.48%502331551476
.....................
95bottle5.72%556287580322
96refrigerator5.66%80161638480
97bottle5.60%455334480349
98bottle5.46%613267635375
99bottle5.37%345239599

100 rows × 6 columns

pipeline.undeploy()
nameshadowimagepipelinetestkpld
created2023-10-24 17:05:04.926467+00:00
last_updated2023-10-24 17:05:22.613731+00:00
deployedFalse
tags
versions433aa539-cb64-4733-85d3-68f6f769dd36, 2f6bee8b-bf31-41cf-b7d2-d4912bfdcca8
stepsmobilenetkpld
publishedFalse

Conclusion

Notice the difference in the control confidence and the challenger confidence. Clearly we can see in this example the challenger resnet50 model is performing better than the control mobilenet model. This is likely due to the fact that frcnn resnet50 model is a 2 stage object detector vs the frcnn mobilenet is a single stage detector.

This completes using Wallaroo’s shadow deployment feature to compare different computer vision models.

3 - Computer Vision: Yolov8n Demonstration

How to use Wallaroo with the Yolov8n computer vision model to detect objects in images.

The following tutorial is available on the Wallaroo Github Repository.

Computer Vision Yolov8n Deployment in Wallaroo

The Yolov8 computer vision model is used for fast recognition of objects in images. This tutorial demonstrates how to deploy a Yolov8n pre-trained model into a Wallaroo Ops server and perform inferences on it.

For this tutorial, the helper module CVDemoUtils and WallarooUtils are used to transform a sample image into a pandas DataFrame. This DataFrame is then submitted to the Yolov8n model deployed in Wallaroo.

This demonstration follows these steps:

  • Upload the Yolo8 model to Wallaroo
  • Add the Yolo8 model as a Wallaroo pipeline step
  • Deploy the Wallaroo pipeline and allocate cluster resources to the pipeline
  • Perform sample inferences
  • Undeploy and return the resources

References

  • Wallaroo Workspaces: Workspaces are environments were users upload models, create pipelines and other artifacts. The workspace should be considered the fundamental area where work is done. Workspaces are shared with other users to give them access to the same models, pipelines, etc.
  • Wallaroo Model Upload and Registration: ML Models are uploaded to Wallaroo through the SDK or the MLOps API to a workspace. ML models include default runtimes (ONNX, Python Step, and TensorFlow) that are run directly through the Wallaroo engine, and containerized runtimes (Hugging Face, PyTorch, etc) that are run through in a container through the Wallaroo engine.
  • Wallaroo Pipelines: Pipelines are used to deploy models for inferencing. Each model is a pipeline step in a pipelines, where the inputs of the previous step are fed into the next. Pipeline steps can be ML models, Python scripts, or Arbitrary Python (these contain necessary models and artifacts for running a model).

Steps

Load Libraries

The first step is loading the required libraries including the Wallaroo Python module.

# Import Wallaroo Python SDK
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework
from CVDemoUtils import CVDemo
from WallarooUtils import Util
cvDemo = CVDemo()
util = Util()

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

Connect to the Wallaroo Instance through the User Interface

The next step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

wl = wallaroo.Client()

Create a New Workspace

We’ll use the SDK below to create our workspace , assign as our current workspace, then display all of the workspaces we have at the moment. We’ll also set up variables for our models and pipelines down the road, so we have one spot to change names to whatever fits your organization’s standards best.

To allow this tutorial to be run by multiple users in the same Wallaroo instance, a random 4 character prefix will be added to the workspace, pipeline, and model. Feel free to set suffix='' if this is not required.

# import string
# import random

# # make a random 4 character suffix to verify uniqueness in tutorials
# suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))

suffix = ''

model_name = 'yolov8n'
model_filename = 'models/yolov8n.onnx'
pipeline_name = 'yolo8demonstration'
workspace_name = f'yolo8-demonstration{suffix}'
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

Upload the Model

When a model is uploaded to a Wallaroo cluster, it is optimized and packaged to make it ready to run as part of a pipeline. In many times, the Wallaroo Server can natively run a model without any Python overhead. In other cases, such as a Python script, a custom Python environment will be automatically generated. This is comparable to the process of “containerizing” a model by adding a small HTTP server and other wrapping around it.

Our pretrained model is in ONNX format, which is specified in the framework parameter. For this model, the tensor fields are set to images to match the input parameters, and the batch configuration is set to single - only one record will be submitted at a time.

# Upload Retrained Yolo8 Model 
yolov8_model = (wl.upload_model(model_name, 
                               model_filename, 
                               framework=Framework.ONNX)
                               .configure(tensor_fields=['images'],
                                          batch_config="single"
                                          )
                )

Pipeline Deployment Configuration

For our pipeline we set the deployment configuration to only use 1 cpu and 1 GiB of RAM.

deployment_config = wallaroo.DeploymentConfigBuilder() \
                    .replica_count(1) \
                    .cpus(1) \
                    .memory("1Gi") \
                    .build()

Build and Deploy the Pipeline

Now we build our pipeline and set our Yolo8 model as a pipeline step, then deploy the pipeline using the deployment configuration above.

pipeline = wl.build_pipeline(pipeline_name) \
            .add_model_step(yolov8_model)        
pipeline.deploy(deployment_config=deployment_config)
nameyolo8demonstration
created2023-10-19 15:33:26.144685+00:00
last_updated2023-10-19 15:33:27.154726+00:00
deployedTrue
tags
versions39b02242-de8b-46cd-849e-8a896226a84a, bed60f2a-ddd6-4a48-a9fd-debe6b1e1bca
stepsyolov8n
publishedFalse

Convert Image to DataFrame

The sample image dogbike.png was converted to a DataFrame using the cvDemo helper modules. The converted DataFrame is stored as ./data/dogbike.df.json to save time.

The code sample below demonstrates how to use this module to convert the sample image to a DataFrame.

# convert the image to a tensor

width, height = 640, 640
tensor1, resizedImage1 = cvDemo.loadImageAndResize('dogbike.png', width, height)
tensor1.flatten()

# add the tensor to a DataFrame and save the DataFrame in pandas record format
df = util.convert_data(tensor1,'images')
df.to_json("data.json", orient = 'records')
# convert the image to a tensor

width, height = 640, 640
tensor1, resizedImage1 = cvDemo.loadImageAndResize('./data/dogbike.png', width, height)

tensor1.flatten()

# add the tensor to a DataFrame and save the DataFrame in pandas record format
df = util.convert_data(tensor1,'images')
df.to_json("data.json", orient = 'records')

Inference Request

We submit the DataFrame to the pipeline using wallaroo.pipeline.infer_from_file, and store the results in the variable inf1.

inf1 = pipeline.infer_from_file('./data/dogbike.df.json')

Display Bounding Boxes

Using our helper method cvDemo we’ll identify the objects detected in the photo and their bounding boxes. Only objects with a confidence threshold of 50% or more are shown.

inf1.loc[:, ['out.output0']]
out.output0
0[17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]
confidence_thres = 0.50
iou_thres = 0.25

cvDemo.drawYolo8Boxes(inf1, resizedImage1, width, height, confidence_thres, iou_thres, draw=True)
  Score: 86.47% | Class: Dog | Bounding Box: [108, 250, 149, 356]
  Score: 81.13% | Class: Bicycle | Bounding Box: [97, 149, 375, 323]
  Score: 63.16% | Class: Car | Bounding Box: [390, 85, 186, 108]
array([[[ 34,  34,  34],
        [ 35,  35,  35],
        [ 33,  33,  33],
        ...,
        [ 33,  33,  33],
        [ 33,  33,  33],
        [ 35,  35,  35]],

       [[ 33,  33,  33],
        [ 34,  34,  34],
        [ 34,  34,  34],
        ...,
        [ 34,  34,  34],
        [ 33,  33,  33],
        [ 34,  34,  34]],

       [[ 53,  54,  48],
        [ 54,  55,  49],
        [ 54,  55,  49],
        ...,
        [153, 178, 111],
        [151, 183, 108],
        [159, 176,  99]],

       ...,

       [[159, 167, 178],
        [159, 165, 177],
        [158, 163, 175],
        ...,
        [126, 127, 121],
        [127, 125, 120],
        [128, 120, 117]],

       [[160, 168, 179],
        [156, 162, 174],
        [152, 157, 169],
        ...,
        [126, 127, 121],
        [129, 127, 122],
        [127, 118, 116]],

       [[155, 163, 174],
        [155, 162, 174],
        [152, 158, 170],
        ...,
        [127, 127, 121],
        [130, 126, 122],
        [128, 119, 116]]], dtype=uint8)

Inference Through Pipeline API

Another method of performing an inference using the pipeline’s deployment url.

Performing an inference through an API requires the following:

  • The authentication token to authorize the connection to the pipeline.
  • The pipeline’s inference URL.
  • Inference data to sent to the pipeline - in JSON, DataFrame records format, or Apache Arrow.

Full details are available through the Wallaroo API Connection Guide on how retrieve an authorization token and perform inferences through the pipeline’s API.

For this demonstration we’ll submit the pandas record, request a pandas record as the return, and set the authorization header. The results will be stored in the file curl_response.df.

deploy_url = pipeline._deployment._url()

headers = wl.auth.auth_header()

headers['Content-Type']='application/json; format=pandas-records'
headers['Accept']='application/json; format=pandas-records'
!curl -X POST {deploy_url} \
    -H "Authorization:{headers['Authorization']}" \
    -H "Content-Type:application/json; format=pandas-records" \
    -H "Accept:application/json; format=pandas-records" \
    --data @./data/dogbike.df.json > curl_response.df
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 38.0M  100 22.9M  100 15.0M  5624k  3701k  0:00:04  0:00:04 --:--:-- 9334k
pipeline.undeploy()
nameyolo8demonstration
created2023-10-11 14:37:32.252497+00:00
last_updated2023-10-11 14:59:03.213137+00:00
deployedFalse
tags
versions2a3933c4-52db-40c6-b80f-9031664fd08a, 95bbabf1-1f15-4e4b-9e67-f7730c2b2cbd, 6c672144-ed4f-4505-97eb-a5b1763af847, 7149e0bc-089b-4d57-9a0b-5d4f4a9a4097, 329e394b-5105-4dc3-b0ff-5411623fc139, 7acaea4e-6ae3-426b-9f97-5e3dcc39c48e, a8b2c009-e7b5-4b96-81b9-40447797a05f, 09952a45-2401-4ebd-8e85-c678365b64a7, d870a558-10ef-448e-b00d-068c10c7e82b, fa531e16-1706-43c4-98d9-e0dd6355fe6f, 4c0b535e-b39b-40f4-82a7-34965b2f7c2a, 3507964d-382f-4e1c-84c7-64c5e27f819c, 9971f8dd-a17b-4d6a-ab72-d786d4990fab, b92a035f-903c-4039-8303-8ceb979a53c2
stepsyolov8n
publishedFalse

4 - Computer Vision: Image Detection for Health Care

How to use Wallaroo with computer vision models to detect mitochondria from images of cells.

This tutorial can be found on the Wallaroo Tutorials Github Repository.

Image Detection for Health Care Computer Vision Tutorial Part 00: Prerequisites

The following tutorial demonstrates how to use Wallaroo to detect mitochondria from high resolution images. For this example we will be using a high resolution 1536x2048 image that is broken down into “patches” of 256x256 images that can be quickly analyzed.

Mitochondria are known as the “powerhouse” of the cell, and having a healthy amount of mitochondria indicates that a patient has enough energy to live a healthy life, or may have underlying issues that a doctor can check for.

Scanning high resolution images of patient cells can be used to count how many mitochondria a patient has, but the process is laborious. The following ML Model is trained to examine an image of cells, then detect which structures are mitochondria. This is used to speed up the process of testing patients and determining next steps.

Prerequisites

The included TiffImagesUtils class is included in this demonstration as CVDemoUtils.py, and requires the following dependencies:

  • ffmpeg
  • libsm
  • libxext

Internal JupyterHub Service

To install these dependencies in the Wallaroo JupyterHub service, use the following commands from a terminal shell via the following procedure:

  1. Launch the JupyterHub Service within the Wallaroo install.

  2. Select File->New->Terminal.

  3. Enter the following:

    sudo apt-get update
    
    sudo apt-get install ffmpeg libsm6 libxext6  -y
    

External SDK Users

For users using the Wallaroo SDK to connect with a remote Wallaroo instance, the following commands will install the required dependancies:

For Linux users, this can be installed with:

sudo apt-get update
sudo apt-get install ffmpeg libsm6 libxext6  -y

MacOS users can prepare their environments using a package manager such as Brew with the following:

brew install ffmpeg libsm libxext

### Libraries and Dependencies

1. This repository may use large file sizes for the models.  If necessary, install [Git Large File Storage (LFS)](https://git-lfs.com) or use the [Wallaroo Tutorials Releases](https://github.com/WallarooLabs/Wallaroo_Tutorials/releases) to download a .zip file of the most recent computer vision tutorial that includes the models.
1. Import the following Python libraries into your environment:
    1. [torch](https://pypi.org/project/torch/)
    1. [wallaroo](https://pypi.org/project/wallaroo/)
    1. [torchvision](https://pypi.org/project/torchvision/)
    1. [opencv-python](https://pypi.org/project/opencv-python/)
    1. [onnx](https://pypi.org/project/onnx/)
    1. [onnxruntime](https://pypi.org/project/onnxruntime/)
    1. [imutils](https://pypi.org/project/imutils/)
    1. [pytz](https://pypi.org/project/pytz/)
    1. [ipywidgets](https://pypi.org/project/ipywidgets/)

These can be installed by running the command below in the Wallaroo JupyterHub service.  Note the use of `pip install torch --no-cache-dir` for low memory environments.

```python
!pip install torchvision==0.15.2
!pip install torch==2.0.1 --no-cache-dir
!pip install opencv-python==4.7.0.72
!pip install onnx==1.12.0
!pip install onnxruntime==1.15.0
!pip install imutils==0.5.4
!pip install pytz
!pip install ipywidgets==8.0.6
!pip install patchify==0.2.3
!pip install tifffile==2023.4.12
!pip install piexif==1.1.3
Requirement already satisfied: torchvision==0.15.2 in /opt/conda/lib/python3.9/site-packages (0.15.2)
Requirement already satisfied: requests in /opt/conda/lib/python3.9/site-packages (from torchvision==0.15.2) (2.25.1)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/lib/python3.9/site-packages (from torchvision==0.15.2) (9.2.0)
Requirement already satisfied: torch==2.0.1 in /opt/conda/lib/python3.9/site-packages (from torchvision==0.15.2) (2.0.1)
Requirement already satisfied: numpy in /opt/conda/lib/python3.9/site-packages (from torchvision==0.15.2) (1.22.3)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.7.99)
Requirement already satisfied: filelock in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (3.12.0)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (3.1.2)
Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (10.9.0.58)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (8.5.0.96)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.10.3.66)
Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.7.4.91)
Requirement already satisfied: networkx in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (3.1)
Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (2.14.3)
Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (10.2.10.91)
Requirement already satisfied: triton==2.0.0 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (2.0.0)
Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.7.91)
Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (4.3.0)
Requirement already satisfied: sympy in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (1.12)
Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.4.0.1)
Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.7.101)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.7.99)
Requirement already satisfied: wheel in /opt/conda/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1->torchvision==0.15.2) (0.37.1)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1->torchvision==0.15.2) (62.3.2)
Requirement already satisfied: cmake in /opt/conda/lib/python3.9/site-packages (from triton==2.0.0->torch==2.0.1->torchvision==0.15.2) (3.26.4)
Requirement already satisfied: lit in /opt/conda/lib/python3.9/site-packages (from triton==2.0.0->torch==2.0.1->torchvision==0.15.2) (16.0.5.post0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.9/site-packages (from requests->torchvision==0.15.2) (1.26.9)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.9/site-packages (from requests->torchvision==0.15.2) (2022.5.18.1)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.9/site-packages (from requests->torchvision==0.15.2) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.9/site-packages (from requests->torchvision==0.15.2) (4.0.0)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.9/site-packages (from jinja2->torch==2.0.1->torchvision==0.15.2) (2.1.1)
Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.9/site-packages (from sympy->torch==2.0.1->torchvision==0.15.2) (1.3.0)
Requirement already satisfied: torch==2.0.1 in /opt/conda/lib/python3.9/site-packages (2.0.1)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (3.1.2)
Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.7.4.91)
Requirement already satisfied: networkx in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (3.1)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.7.99)
Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (10.2.10.91)
Requirement already satisfied: triton==2.0.0 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (2.0.0)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.7.99)
Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (10.9.0.58)
Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (2.14.3)
Requirement already satisfied: filelock in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (3.12.0)
Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.7.101)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.10.3.66)
Requirement already satisfied: sympy in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (1.12)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (8.5.0.96)
Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.4.0.1)
Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.7.91)
Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (4.3.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (62.3.2)
Requirement already satisfied: wheel in /opt/conda/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (0.37.1)
Requirement already satisfied: cmake in /opt/conda/lib/python3.9/site-packages (from triton==2.0.0->torch==2.0.1) (3.26.4)
Requirement already satisfied: lit in /opt/conda/lib/python3.9/site-packages (from triton==2.0.0->torch==2.0.1) (16.0.5.post0)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.9/site-packages (from jinja2->torch==2.0.1) (2.1.1)
Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.9/site-packages (from sympy->torch==2.0.1) (1.3.0)
Requirement already satisfied: opencv-python==4.7.0.72 in /opt/conda/lib/python3.9/site-packages (4.7.0.72)
Requirement already satisfied: numpy>=1.19.3 in /opt/conda/lib/python3.9/site-packages (from opencv-python==4.7.0.72) (1.22.3)
Requirement already satisfied: onnx==1.12.0 in /opt/conda/lib/python3.9/site-packages (1.12.0)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /opt/conda/lib/python3.9/site-packages (from onnx==1.12.0) (4.3.0)
Requirement already satisfied: protobuf<=3.20.1,>=3.12.2 in /opt/conda/lib/python3.9/site-packages (from onnx==1.12.0) (3.19.5)
Requirement already satisfied: numpy>=1.16.6 in /opt/conda/lib/python3.9/site-packages (from onnx==1.12.0) (1.22.3)
Requirement already satisfied: onnxruntime==1.15.0 in /opt/conda/lib/python3.9/site-packages (1.15.0)
Requirement already satisfied: packaging in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (21.3)
Requirement already satisfied: flatbuffers in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (1.12)
Requirement already satisfied: protobuf in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (3.19.5)
Requirement already satisfied: numpy>=1.21.6 in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (1.22.3)
Requirement already satisfied: coloredlogs in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (15.0.1)
Requirement already satisfied: sympy in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (1.12)
Requirement already satisfied: humanfriendly>=9.1 in /opt/conda/lib/python3.9/site-packages (from coloredlogs->onnxruntime==1.15.0) (10.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.9/site-packages (from packaging->onnxruntime==1.15.0) (3.0.9)
Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.9/site-packages (from sympy->onnxruntime==1.15.0) (1.3.0)
Requirement already satisfied: imutils==0.5.4 in /opt/conda/lib/python3.9/site-packages (0.5.4)
Requirement already satisfied: pytz in /opt/conda/lib/python3.9/site-packages (2022.1)
Requirement already satisfied: ipywidgets==8.0.6 in /opt/conda/lib/python3.9/site-packages (8.0.6)
Requirement already satisfied: jupyterlab-widgets~=3.0.7 in /opt/conda/lib/python3.9/site-packages (from ipywidgets==8.0.6) (3.0.7)
Requirement already satisfied: ipython>=6.1.0 in /opt/conda/lib/python3.9/site-packages (from ipywidgets==8.0.6) (7.24.1)
Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.9/site-packages (from ipywidgets==8.0.6) (6.13.0)
Requirement already satisfied: traitlets>=4.3.1 in /opt/conda/lib/python3.9/site-packages (from ipywidgets==8.0.6) (5.2.1.post0)
Requirement already satisfied: widgetsnbextension~=4.0.7 in /opt/conda/lib/python3.9/site-packages (from ipywidgets==8.0.6) (4.0.7)
Requirement already satisfied: psutil in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (5.9.1)
Requirement already satisfied: matplotlib-inline>=0.1 in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (0.1.3)
Requirement already satisfied: debugpy>=1.0 in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (1.6.0)
Requirement already satisfied: nest-asyncio in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (1.5.5)
Requirement already satisfied: packaging in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (21.3)
Requirement already satisfied: tornado>=6.1 in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (6.1)
Requirement already satisfied: jupyter-client>=6.1.12 in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (7.3.1)
Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (4.8.0)
Requirement already satisfied: backcall in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (0.2.0)
Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (0.18.1)
Requirement already satisfied: pickleshare in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (0.7.5)
Requirement already satisfied: decorator in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (5.1.1)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (3.0.29)
Requirement already satisfied: setuptools>=18.5 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (62.3.2)
Requirement already satisfied: pygments in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (2.12.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.9/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets==8.0.6) (0.8.3)
Requirement already satisfied: entrypoints in /opt/conda/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets==8.0.6) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets==8.0.6) (2.8.2)
Requirement already satisfied: jupyter-core>=4.9.2 in /opt/conda/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets==8.0.6) (4.10.0)
Requirement already satisfied: pyzmq>=22.3 in /opt/conda/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets==8.0.6) (23.0.0)
Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.9/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets==8.0.6) (0.7.0)
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=6.1.0->ipywidgets==8.0.6) (0.2.5)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.9/site-packages (from packaging->ipykernel>=4.5.1->ipywidgets==8.0.6) (3.0.9)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.9/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets==8.0.6) (1.16.0)
Requirement already satisfied: patchify==0.2.3 in /opt/conda/lib/python3.9/site-packages (0.2.3)
Requirement already satisfied: numpy<2,>=1 in /opt/conda/lib/python3.9/site-packages (from patchify==0.2.3) (1.22.3)
Requirement already satisfied: tifffile==2023.4.12 in /opt/conda/lib/python3.9/site-packages (2023.4.12)
Requirement already satisfied: numpy in /opt/conda/lib/python3.9/site-packages (from tifffile==2023.4.12) (1.22.3)
Requirement already satisfied: piexif==1.1.3 in /opt/conda/lib/python3.9/site-packages (1.1.3)

Wallaroo SDK

The Wallaroo SDK is provided by default with the Wallaroo instance’s JupyterHub service. To install the Wallaroo SDK manually, it is provided from the Python Package Index and is installed with pip. Verify that the same version of the Wallaroo SDK is the same version as the Wallaroo instance. For example for Wallaroo release 2023.2, the SDK install command is:

pip install wallaroo==2023.4.1

See the Wallaroo SDK Install Guides for full details.

4.1 - Computer Vision: Image Detection for Health Care

How to use Wallaroo with computer vision models to detect mitochondria from images of cells.

This tutorial can be found on the Wallaroo Tutorials Github Repository.

Image Detection for Health Care Computer Vision Tutorial Part 01: Mitochondria Detection

The following tutorial demonstrates how to use Wallaroo to detect mitochondria from high resolution images. For this example we will be using a high resolution 1536x2048 image that is broken down into “patches” of 256x256 images that can be quickly analyzed.

Mitochondria are known as the “powerhouse” of the cell, and having a healthy amount of mitochondria indicates that a patient has enough energy to live a healthy life, or may have underling issues that a doctor can check for.

Scanning high resolution images of patient cells can be used to count how many mitochondria a patient has, but the process is laborious. The following ML Model is trained to examine an image of cells, then detect which structures are mitochondria. This is used to speed up the process of testing patients and determining next steps.

Tutorial Goals

This tutorial will perform the following:

  1. Upload and deploy the mitochondria_epochs_15.onnx model to a Wallaroo pipeline.
  2. Randomly select from from a selection of 256x256 images that were originally part of a larger 1536x2048 image.
  3. Convert the images into a numpy array inserted into a pandas DataFrame.
  4. Submit the DataFrame to the Wallaroo pipeline and use the results to create a mask image of where the model detects mitochondria.
  5. Compare the original image against a map of “ground truth” and the model’s mask image.
  6. Undeploy the pipeline and return the resources back to the Wallaroo instance.

Prerequisites

Complete the steps from Mitochondria Detection Computer Vision Tutorial Part 00: Prerequisites.

Mitochondria Computer Vision Detection Steps

Import Libraries

The first step is to import the necessary libraries. Included with this tutorial are the following custom modules:

  • tiff_utils: Organizes the tiff images to perform random image selections and other tasks.

Note that tensorflow may return warnings depending on the environment.

import json
import IPython.display as display
import time
import matplotlib.pyplot as plt
from IPython.display import clear_output, display
from lib.TiffImageUtils import TiffUtils
import tifffile as tiff

import pandas as pd

import wallaroo
from wallaroo.object import EntityNotFoundError

import numpy as np
from matplotlib import pyplot as plt
import cv2
from keras.utils import normalize

tiff_utils = TiffUtils()

# used for unique connection names

import string
import random

suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))

# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')

Open a Connection to Wallaroo

The next step is connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). If logging in externally, update the wallarooPrefix and wallarooSuffix variables with the proper DNS information. For more information on Wallaroo DNS settings, see the Wallaroo DNS Integration Guide.

wl = wallaroo.Client()

Create Workspace and Pipeline

We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace.

Workspace, pipeline, and model names should be unique to each Wallaroo instance, so we’ll add in a randomly generated suffix so multiple people can run this tutorial in a Wallaroo instance without affecting each other.

workspace_name = f'biolabsworkspace{suffix}'
pipeline_name = f'biolabspipeline{suffix}'
model_name = f'biolabsmodel{suffix}'
model_file_name = 'models/mitochondria_epochs_15.onnx'
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

def get_pipeline(name):
    try:
        pipeline = wl.pipelines_by_name(name)[0]
    except EntityNotFoundError:
        pipeline = wl.build_pipeline(name)
    return pipeline
workspace = get_workspace(workspace_name)

wl.set_current_workspace(workspace)

pipeline = get_pipeline(pipeline_name)
pipeline
namebiolabspipelinebspy
created2023-07-14 15:28:32.639523+00:00
last_updated2023-07-14 15:28:32.639523+00:00
deployed(none)
tags
versionsc70dbdfe-e380-41b0-9da6-97bbfae90554
steps

Upload the Models

Now we will:

  1. Upload our model.
  2. Apply it as a step in our pipeline.
  3. Create a pipeline deployment with enough memory to perform the inferences.
  4. Deploy the pipeline.
deployment_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(1).memory("2Gi").build()

model = (wl.upload_model(model_name, 
                         model_file_name, 
                         framework=wallaroo.framework.Framework.ONNX)
                         .configure(tensor_fields=["tensor"])
        )

pipeline = wl.build_pipeline(pipeline_name) \
            .add_model_step(model) \
            .deploy(deployment_config = deployment_config)

Retrieve Image and Convert to Data

The next step is to process the image into a numpy array that the model is trained to detect from.

We start by retrieving all the patch images from a recorded time series tiff recorded on one of our microscopes.

sample_mitochondria_patches_path = "./patches/ms-01-atl-3-22-23_9-50"

patches = tiff_utils.get_all_patches(sample_mitochondria_patches_path)

Randomly we will retrieve a 256x256 patch image and use it to do our semantic segmentation prediction.

We’ll then convert it into a numpy array and insert into a DataFrame for a single inference.

The following helper function loadImageAndConvertTiff is used to convert the image into a numpy, then insert that into the DataFrame. This allows a later command to take the randomly grabbed image perform the process on other images.

def loadImageAndConvertTiff(imagePath, width, height):
    img = cv2.imread(imagePath, 0)
    imgNorm = np.expand_dims(normalize(np.array(img), axis=1),2)
    imgNorm=imgNorm[:,:,0][:,:,None]
    imgNorm=np.expand_dims(imgNorm, 0)
    
    resizedImage = None
    #creates a dictionary with the wallaroo "tensor" key and the numpy ndim array representing image as the value.
    dictData = {"tensor":[imgNorm]}
    dataframedata = pd.DataFrame(dictData)
    # display(dataframedata)
    return dataframedata, resizedImage
def run_semantic_segmentation_inference(pipeline, input_tiff_image, width, height, threshold):
    
    tensor, resizedImage = loadImageAndConvertTiff(input_tiff_image, width, height)
    # print(tensor)

    # #
    # # run inference on the 256x256 patch image get the predicted mitochandria mask
    # #
    output = pipeline.infer(tensor)
    # print(output)

    # # Obtain the flattened predicted mitochandria mask result
    list1d = output.loc[0]["out.conv2d_37"]
    np1d = np.array(list1d)
    
    # # unflatten it
    predicted_mask = np1d.reshape(1,width,height,1)
    
    # # perform the element-wise comaprison operation using the threshold provided
    predicted_mask = (predicted_mask[0,:,:,0] > threshold).astype(np.uint8)
    
    # return predicted_mask
    return predicted_mask

Infer and Display Results

We will now perform our inferences and display the results. This results in a predicted mask showing us where the mitochondria cells are located.

  1. The first image is the input image.
  2. The 2nd image is the ground truth. The mask was created by a human who identified the mitochondria cells in the input image
  3. The 3rd image is the predicted mask after running inference on the Wallaroo pipeline.

We’ll perform this 10 times to show how quickly the inferences can be submitted.

for x in range(10):     
    # get a sample 256x256 mitochondria image
    random_patch = tiff_utils.get_random_patch_sample(patches)

    # build the path to the image
    patch_image_path = sample_mitochondria_patches_path + "/images/" + random_patch['patch_image_file']

    # run inference in order to get the predicted 256x256 mask
    predicted_mask = run_semantic_segmentation_inference(pipeline, patch_image_path, 256,256, 0.2)

    # # plot the results
    test_image = random_patch['patch_image'][:,:,0]
    test_image_title = f"Testing Image - {random_patch['index']}"

    ground_truth_image = random_patch['patch_mask'][:,:,0]
    ground_truth_image_title = "Ground Truth Mask"

    predicted_mask_title = 'Predicted Mask'

    tiff_utils.plot_test_results(test_image, test_image_title, \
                            ground_truth_image, ground_truth_image_title, \
                            predicted_mask, predicted_mask_title)

Complete Tutorial

With the demonstration complete, the pipeline is undeployed and the resources returned back to the Wallaroo instance.

pipeline.undeploy()
namebiolabspipelinebspy
created2023-07-14 15:28:32.639523+00:00
last_updated2023-07-14 15:28:38.163950+00:00
deployedFalse
tags
versions0460ef47-3de3-43b2-8f62-16e76be8ce93, ef41bc7b-8213-4dd4-a1b9-54ac1253c652, c70dbdfe-e380-41b0-9da6-97bbfae90554
stepsbiolabsmodelbspy

4.2 - Computer Vision: Image Detection for Health Care

How to use Wallaroo with computer vision models to detect mitochondria from images of cells.

This tutorial can be found on the Wallaroo Tutorials Github Repository.

Image Detection for Health Care Computer Vision Tutorial Part 02: External Data Connection Stores

The first example in this tutorial series showed selecting from 10 random images of cells and detecting mitochondria from within them.

This tutorial will expand on that by using a Wallaroo connection to retrieve a high resolution 1536x2048 image, break it down into 256x256 “patches” that can be quickly analyzed.

Wallaroo connections are definitions set by MLOps engineers that are used by other Wallaroo users for connection information to a data source. For this example, the data source will be a GitHub URL, but they could be Google BigQuery databases, Kafka topics, or any number of user defined examples.

Tutorial Goals

This tutorial will perform the following:

  1. Upload and deploy the mitochondria_epochs_15.onnx model to a Wallaroo pipeline.
  2. Create a Wallaroo connection pointing to a location for a high resolution image of cells.
  3. Break down the image into 256x256 images based on the how the model was trained to detect mitochondria.
  4. Convert the images into a numpy array inserted into a pandas DataFrame.
  5. Submit the DataFrame to the Wallaroo pipeline and use the results to create a mask image of where the model detects mitochondria.
  6. Compare the original image against a map of “ground truth” and the model’s mask image.
  7. Undeploy the pipeline and return the resources back to the Wallaroo instance.

Prerequisites

Prerequisites

Complete the steps from Mitochondria Detection Computer Vision Tutorial Part 00: Prerequisites.

Mitochondria Computer Vision Detection Steps

Import Libraries

The first step is to import the necessary libraries. Included with this tutorial are the following custom modules:

  • tiff_utils: Organizes the tiff images to perform random image selections and other tasks.
import json
import IPython.display as display
import time
import matplotlib.pyplot as plt
from IPython.display import clear_output, display
from lib.TiffImageUtils import TiffUtils
import tifffile as tiff

import pandas as pd

import wallaroo
from wallaroo.object import EntityNotFoundError, RequiredAttributeMissing

import numpy as np
from matplotlib import pyplot as plt
import cv2
from keras.utils import normalize

tiff_utils = TiffUtils()

# used for unique connection names

import string
import random

suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))

# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')

Open a Connection to Wallaroo

The next step is connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). If logging in externally, update the wallarooPrefix and wallarooSuffix variables with the proper DNS information. For more information on Wallaroo DNS settings, see the Wallaroo DNS Integration Guide.

wl = wallaroo.Client()

Create Workspace and Pipeline

We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace.

Workspace, pipeline, and model names should be unique to each Wallaroo instance, so we’ll add in a randomly generated suffix so multiple people can run this tutorial in a Wallaroo instance without effecting each other.

workspace_name = f'biolabsconnectionworkspace{suffix}'
pipeline_name = f'biolabsconnectionpipeline{suffix}'
model_name = f'biolabsconnectionmodel{suffix}'
model_file_name = 'models/mitochondria_epochs_15.onnx'
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

def get_pipeline(name):
    try:
        pipeline = wl.pipelines_by_name(name)[0]
    except EntityNotFoundError:
        pipeline = wl.build_pipeline(name)
    return pipeline
workspace = get_workspace(workspace_name)

wl.set_current_workspace(workspace)

pipeline = get_pipeline(pipeline_name)
pipeline
namebiolabsconnectionpipelinepdrv
created2023-07-14 15:29:16.969523+00:00
last_updated2023-07-14 15:29:16.969523+00:00
deployed(none)
tags
versions79d3b6ca-823f-406d-bf23-88a23b49ff1b
steps

Upload the Models

Now we will:

  1. Upload our model.
  2. Apply it as a step in our pipeline.
  3. Create a pipeline deployment with enough memory to perform the inferences.
  4. Deploy the pipeline.
deployment_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(1).memory("2Gi").build()

model = (wl.upload_model(model_name, 
                         model_file_name, 
                         framework=wallaroo.framework.Framework.ONNX)
                         .configure(tensor_fields=["tensor"])
        )

pipeline = wl.build_pipeline(pipeline_name) \
            .add_model_step(model) \
            .deploy(deployment_config = deployment_config)

Create the Wallaroo Connection

The Wallaroo connection will point to the location of our high resolution images:

  • One the cell photos to be retrieved
  • The other “ground truth” masks that are mapped compared against the model’s predictions.

The images will be retrieved, then parsed into a series of 256x256 images.

image_connection_name = f'mitochondria_image_source{suffix}'
image_connection_type = "HTTP"
image_connection_argument = {
    'cell_images':'https://storage.googleapis.com/wallaroo-public-data/csa_demo/computer-vision/examples/medical/bio-labs/atl-lab/images/ms-01-atl-3-22-23_9-50.tiff',
    'ground_truth_masks': 'https://storage.googleapis.com/wallaroo-public-data/csa_demo/computer-vision/examples/medical/bio-labs/atl-lab/masks/ms-01-atl-3-22-23_9-50-masks.tiff'
    }

connection = wl.create_connection(image_connection_name, image_connection_type, image_connection_argument)

Retrieve Images

We’ll use our new connection to reach out, retrieve the images and store them locally for processing. The information is stored in the details() method for the connection, which is hidden by default when showing the connection, but the data can be retrieved when necessary.

inference_source_connection = wl.get_connection(name=image_connection_name)
display(inference_source_connection)
FieldValue
Namemitochondria_image_sourcepdrv
Connection TypeHTTP
Details*****
Created At2023-07-14T15:32:00.930954+00:00
Linked Workspaces[]
patches_dict = tiff_utils.build_patches("downloaded_patches", 
                                        (256,256), 
                                        256, 
                                        inference_source_connection.details()['cell_images'], 
                                        inference_source_connection.details()['ground_truth_masks'] )
created dir downloaded_patches/ms-01-atl-3-22-23_9-50
saving file downloaded_patches/ms-01-atl-3-22-23_9-50/ms-01-atl-3-22-23_9-50.tiff

Retrieve Image and Convert to Data

The next step is to process the image into a numpy array that the model is trained to detect from.

We start by retrieving all the patch images from a recorded time series tiff recorded on one of our microscopes.

sample_mitochondria_patches_path = "./downloaded_patches/ms-01-atl-3-22-23_9-50"

patches = tiff_utils.get_all_patches(sample_mitochondria_patches_path)

Randomly we will retrieve a 256x256 patch image and use it to do our semantic segmentation prediction.

We’ll then convert it into a numpy array and insert into a DataFrame for a single inference.

The following helper function loadImageAndConvertTiff is used to convert the image into a numpy, then insert that into the DataFrame. This allows a later command to take the randomly grabbed image perform the process on other images.

def loadImageAndConvertTiff(imagePath, width, height):
    img = cv2.imread(imagePath, 0)
    imgNorm = np.expand_dims(normalize(np.array(img), axis=1),2)
    imgNorm=imgNorm[:,:,0][:,:,None]
    imgNorm=np.expand_dims(imgNorm, 0)
    
    resizedImage = None
    #creates a dictionary with the wallaroo "tensor" key and the numpy ndim array representing image as the value.
    dictData = {"tensor":[imgNorm]}
    dataframedata = pd.DataFrame(dictData)
    # display(dataframedata)
    return dataframedata, resizedImage
def run_semantic_segmentation_inference(pipeline, input_tiff_image, width, height, threshold):
    
    tensor, resizedImage = loadImageAndConvertTiff(input_tiff_image, width, height)
    # print(tensor)

    # #
    # # run inference on the 256x256 patch image get the predicted mitochandria mask
    # #
    output = pipeline.infer(tensor)
    # print(output)

    # # Obtain the flattened predicted mitochandria mask result
    list1d = output.loc[0]["out.conv2d_37"]
    np1d = np.array(list1d)
    
    # # unflatten it
    predicted_mask = np1d.reshape(1,width,height,1)
    
    # # perform the element-wise comaprison operation using the threshold provided
    predicted_mask = (predicted_mask[0,:,:,0] > threshold).astype(np.uint8)
    
    # return predicted_mask
    return predicted_mask

Infer and Display Results

We will now perform our inferences and display the results. This results in a predicted mask showing us where the mitochondria cells are located.

  1. The first image is the input image.
  2. The 2nd image is the ground truth. The mask was created by a human who identified the mitochondria cells in the input image
  3. The 3rd image is the predicted mask after running inference on the Wallaroo pipeline.

We’ll perform this 10 times to show how quickly the inferences can be submitted.

for x in range(10):     
    # get a sample 256x256 mitochondria image
    random_patch = tiff_utils.get_random_patch_sample(patches)

    # build the path to the image
    patch_image_path = sample_mitochondria_patches_path + "/images/" + random_patch['patch_image_file']

    # run inference in order to get the predicted 256x256 mask
    predicted_mask = run_semantic_segmentation_inference(pipeline, patch_image_path, 256,256, 0.2)

    # # plot the results
    test_image = random_patch['patch_image'][:,:,0]
    test_image_title = f"Testing Image - {random_patch['index']}"

    ground_truth_image = random_patch['patch_mask'][:,:,0]
    ground_truth_image_title = "Ground Truth Mask"

    predicted_mask_title = 'Predicted Mask'

    tiff_utils.plot_test_results(test_image, test_image_title, \
                            ground_truth_image, ground_truth_image_title, \
                            predicted_mask, predicted_mask_title)

Complete Tutorial

With the demonstration complete, the pipeline is undeployed and the resources returned back to the Wallaroo instance.

pipeline.undeploy()
namebiolabsconnectionpipelinepdrv
created2023-07-14 15:29:16.969523+00:00
last_updated2023-07-14 15:29:23.004729+00:00
deployedFalse
tags
versions0daae212-f578-4723-a0e5-edbcfa548976, 143eb35c-8606-471c-8053-230640249ad6, 79d3b6ca-823f-406d-bf23-88a23b49ff1b
stepsbiolabsconnectionmodelpdrv

5 - CLIP ViT-B/32 Transformer Demonstration with Wallaroo

The CLIP ViT-B/32 Transformer Demonstration with Wallaroo deployment.

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

CLIP ViT-B/32 Transformer Demonstration with Wallaroo

The following tutorial demonstrates deploying and performing sample inferences with the Hugging Face CLIP ViT-B/32 Transformer model.

Prerequisites

This tutorial is geared towards the Wallaroo version 2023.2.1 and above. The model clip-vit-base-patch-32.zip must be downloaded and placed into the ./models directory. This is available from the following URL:

https://storage.googleapis.com/wallaroo-public-data/hf-clip-vit-b32/clip-vit-base-patch-32.zip

If performing this tutorial from outside the Wallaroo JupyterHub environment, install the Wallaroo SDK.

Steps

Imports

The first step is to import the libraries used for the example.

import json
import os
import requests

import wallaroo
from wallaroo.pipeline   import Pipeline
from wallaroo.deployment_config import DeploymentConfigBuilder
from wallaroo.framework import Framework
from wallaroo.object import EntityNotFoundError

import pyarrow as pa
import numpy as np
import pandas as pd

from PIL import Image

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

wl = wallaroo.Client()

Set Workspace and Pipeline

The next step is to create the Wallaroo workspace and pipeline used for the inference requests.

# return the workspace called <name> through the Wallaroo client.

def get_workspace(name, client):
    workspace = None
    for ws in client.list_workspaces():
        if ws.name() == name:
            workspace= ws
            return workspace
    # if no workspaces were found
    if workspace==None:
        workspace = wl.create_workspace(name)
    return workspace
# create the workspace and pipeline

workspace_name = 'clip-demo'
pipeline_name = 'clip-demo'

workspace = get_workspace(workspace_name, wl)

wl.set_current_workspace(workspace)
display(wl.get_current_workspace())

pipeline = wl.build_pipeline(pipeline_name)
pipeline
{'name': 'clip-demo', 'id': 8, 'archived': False, 'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4', 'created_at': '2023-11-20T18:57:53.667873+00:00', 'models': [], 'pipelines': [{'name': 'clip-demo', 'create_time': datetime.datetime(2023, 11, 20, 18, 57, 55, 550690, tzinfo=tzutc()), 'definition': '[]'}]}
nameclip-demo
created2023-11-20 18:57:55.550690+00:00
last_updated2023-11-20 18:57:55.550690+00:00
deployed(none)
archNone
tags
versionsff0e4846-f0f9-4fb0-bc06-d6daa21797bf
steps
publishedFalse

Configure and Upload Model

The 🤗 Hugging Face model is uploaded to Wallaroo by defining the input and output schema, and specifying the model’s framework as wallaroo.framework.Framework.HUGGING_FACE_ZERO_SHOT_IMAGE_CLASSIFICATION.

The data schemas are defined in Apache PyArrow Schema format.

The model is converted to the Wallaroo Containerized runtime after the upload is complete.

input_schema = pa.schema([
    pa.field('inputs', # required, fixed image dimensions
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=3
                ),
                list_size=640 
            ),
        list_size=480
    )),
    pa.field('candidate_labels', pa.list_(pa.string(), list_size=4)), # required, equivalent to `options` in the provided demo
]) 

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64(), list_size=4)), # has to be same as number of candidate labels
    pa.field('label', pa.list_(pa.string(), list_size=4)), # has to be same as number of candidate labels
])

Upload Model

model = wl.upload_model('clip-vit', './models/clip-vit-base-patch-32.zip', 
                        framework=Framework.HUGGING_FACE_ZERO_SHOT_IMAGE_CLASSIFICATION, 
                        input_schema=input_schema, 
                        output_schema=output_schema)
model
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime............................................................successful

Ready

Nameclip-vit
Version57594cc7-7db1-43c3-b21a-6dcaba846f26
File Nameclip-vit-base-patch-32.zip
SHA4efc24685a14e1682301cc0085b9db931aeb5f3f8247854bedc6863275ed0646
Statusready
Image Pathproxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/mlflow-deploy:v2023.4.0-4103
ArchitectureNone
Updated At2023-20-Nov 20:51:44

Deploy Pipeline

With the model uploaded and prepared, we add the model as a pipeline step and deploy it. For this example, we will allocate 4 Gi of RAM and 1 CPU to the model’s use through the pipeline deployment configuration.

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .cpus(.25).memory('1Gi') \
    .sidekick_memory(model, '4Gi') \
    .sidekick_cpus(model, 1.0) \
    .build()

The pipeline is deployed with the specified engine deployment.

Because the model is converted to the Wallaroo Containerized Runtime, the deployment step may timeout with the status still as Starting. If this occurs, wait an additional 60 seconds, then run the pipeline.status() cell. Once the status is Running, the rest of the tutorial can proceed.

pipeline.clear()
pipeline.add_model_step(model)
pipeline.deploy(deployment_config=deployment_config)
nameclip-demo
created2023-11-20 18:57:55.550690+00:00
last_updated2023-11-20 20:51:59.304917+00:00
deployedTrue
archNone
tags
versions7a6168c2-f807-465f-a122-33fe7b5d8a3d, ff0e4846-f0f9-4fb0-bc06-d6daa21797bf
stepsclip-vit
publishedFalse
pipeline.status()
{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.140',
   'name': 'engine-bb48bb959-2xctn',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'clip-demo',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'clip-vit',
      'version': '57594cc7-7db1-43c3-b21a-6dcaba846f26',
      'sha': '4efc24685a14e1682301cc0085b9db931aeb5f3f8247854bedc6863275ed0646',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.152',
   'name': 'engine-lb-584f54c899-gcbpj',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': [{'ip': '10.244.3.139',
   'name': 'engine-sidekick-clip-vit-11-75c5666b69-tqfbk',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'}]}

Run Inference

We verify the pipeline is deployed by checking the status().

The sample images in the ./data directory are converted into numpy arrays, and the candidate labels added as inputs. Both are set as DataFrame arrays where the field inputs are the image values, and candidate_labels the labels.

image_paths = [
    "./data/bear-in-tree.jpg",
    "./data/elephant-and-zebras.jpg",
    "./data/horse-and-dogs.jpg",
    "./data/kittens.jpg",
    "./data/remote-monitor.jpg"
]
images = []

for iu in image_paths:
    image = Image.open(iu)
    image = image.resize((640, 480)) # fixed image dimensions
    images.append(np.array(image))

dataframe = pd.DataFrame({"images": images})
input_data = {
        "inputs": images,
        "candidate_labels": [["cat", "dog", "horse", "elephant"]] * 5,
}
dataframe = pd.DataFrame(input_data)
dataframe
inputscandidate_labels
0[[[60, 62, 61], [62, 64, 63], [67, 69, 68], [7...[cat, dog, horse, elephant]
1[[[228, 235, 241], [229, 236, 242], [230, 237,...[cat, dog, horse, elephant]
2[[[177, 177, 177], [177, 177, 177], [177, 177,...[cat, dog, horse, elephant]
3[[[140, 25, 56], [144, 25, 67], [146, 24, 73],...[cat, dog, horse, elephant]
4[[[24, 20, 11], [22, 18, 9], [18, 14, 5], [21,...[cat, dog, horse, elephant]

Inference Outputs

The inference is run, and the labels with their corresponding confidence values for each label are mapped to out.label and out.score for each image.

results = pipeline.infer(dataframe,timeout=600,dataset=["in", "out", "metadata.elapsed", "time", "check_failures"])
pd.set_option('display.max_colwidth', None)
display(results.loc[:, ['out.label', 'out.score']])
out.labelout.score
0[elephant, dog, horse, cat][0.4146825075149536, 0.3483847379684448, 0.1285749077796936, 0.10835790634155273]
1[elephant, horse, dog, cat][0.9981434345245361, 0.001765866531059146, 6.823801231803373e-05, 2.2441448891186155e-05]
2[horse, dog, elephant, cat][0.7596800923347473, 0.217111736536026, 0.020392831414937973, 0.0028152535669505596]
3[cat, dog, elephant, horse][0.9870228171348572, 0.0066468678414821625, 0.0032716328278183937, 0.003058753442019224]
4[dog, horse, cat, elephant][0.5713965892791748, 0.17229433357715607, 0.15523898601531982, 0.1010700911283493]

Undeploy Pipelines

With the tutorial complete, the pipeline is undeployed and the resources returned back to the cluster.

pipeline.undeploy()
nameclip-demo
created2023-11-20 18:57:55.550690+00:00
last_updated2023-11-20 20:51:59.304917+00:00
deployedFalse
archNone
tags
versions7a6168c2-f807-465f-a122-33fe7b5d8a3d, ff0e4846-f0f9-4fb0-bc06-d6daa21797bf
stepsclip-vit
publishedFalse

6 - Containerized MLFlow Tutorial

The Containerized MLFlow Tutorial demonstrates how to use Wallaroo with containerized MLFlow models.

Setting Private Registry Configuration in Wallaroo

Configure Via Kots

If Wallaroo was installed via kots, use the following procedure to add the private model registry information.

  1. Launch the Wallaroo Administrative Dashboard through a terminal linked to the Kubernetes cluster. Replace the namespace with the one used in your installation.

    kubectl kots admin-console --namespace wallaroo
    
  2. Launch the dashboard, by default at http://localhost:8800.

  3. From the admin dashboard, select Config -> Private Model Container Registry.

  4. Enable Provide private container registry credentials for model images.

  5. Provide the following:

    1. Registry URL: The URL of the Containerized Model Container Registry. Typically in the format host:port. In this example, the registry for GitHub is used. NOTE: When setting the URL for the Containerized Model Container Registry, only the actual service address is needed. For example: with the full URL of the model as ghcr.io/wallaroolabs/wallaroo_tutorials/mlflow-statsmodels-example:2022.4, the URL would be ghcr.io/wallaroolabs.
    2. email: The email address of the user authenticating to the registry service.
    3. username: The username of the user authentication to the registry service.
    4. password: The password of the user authentication or token to the registry service.
  6. Scroll down and select Save config.

  7. Deploy the new version.

Once complete, the Wallaroo instance will be able to authenticate to the Containerized Model Container Registry and retrieve the images.

Configure via Helm

  1. During either the installation process or updates, set the following in the local-values.yaml file:

    1. privateModelRegistry:

      1. enabled: true
      2. secretName: model-registry-secret
      3. registry: The URL of the private registry.
      4. email: The email address of the user authenticating to the registry service.
      5. username: The username of the user authentication to the registry service.
      6. password: The password of the user authentication to the registry service.

      For example:

      
      # Other settings - DNS entries, etc.
      
      # The private registry settings
      privateModelRegistry:
        enabled: true
        secretName: model-registry-secret
        registry: "YOUR REGISTRY URL:YOUR REGISTRY PORT"
        email: "YOUR EMAIL ADDRESS"
        username: "YOUR USERNAME"
        password: "Your Password here"
      
  2. Install or update the Wallaroo instance via Helm as per the Wallaroo Helm Install instructions.

Once complete, the Wallaroo instance will be able to authenticate to the registry service and retrieve the images.

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

MLFlow Inference with Wallaroo Tutorial

Wallaroo users can register their trained MLFlow ML Models from a containerized model container registry into their Wallaroo instance and perform inferences with it through a Wallaroo pipeline.

As of this time, Wallaroo only supports MLFlow 1.3.0 containerized models. For information on how to containerize an MLFlow model, see the MLFlow Documentation.

This tutorial assumes that you have a Wallaroo instance, and have either your own containerized model or use the one from the reference and are running this Notebook from the Wallaroo Jupyter Hub service.

See the Wallaroo Private Containerized Model Container Registry Guide for details on how to configure a Wallaroo instance with a private model registry.

MLFlow Data Formats

When using containerized MLFlow models with Wallaroo, the inputs and outputs must be named. For example, the following output:

[-12.045839810372835]

Would need to be wrapped with the data values named:

[{"prediction": -12.045839810372835}]

A short sample code for wrapping data may be:

output_df = pd.DataFrame(prediction, columns=["prediction"])
return output_df

MLFlow Models and Wallaroo

MLFlow models are composed of two parts: the model, and the flavors. When submitting a MLFlow model to Wallaroo, both aspects must be part of the ML Model included in the container. For full information about MLFlow model structure, see the MLFlow Documentation.

Wallaroo registers the models from container registries. Organizations will either have to make their containers available in a public or through a private Containerized Model Container Registry service. For examples on setting up a private container registry service, see the Docker Documentation “Deploy a registry server”. For more details on setting up a container registry in a cloud environment, see the related documentation for your preferred cloud provider:

For this example, we will be using the MLFlow containers that was registered in a GitHub container registry service in MLFlow Creation Tutorial Part 03: Container Registration. The address of those containers are:

  • postprocess: ghcr.io/johnhansarickwallaroo/mlflowtests/mlflow-postprocess-example . Used for format data after the statsmodel inferences.
  • statsmodel: ghcr.io/johnhansarickwallaroo/mlflowtests/mlflow-statsmodels-example . The statsmodel generated in MLFlow Creation Tutorial Part 01: Model Creation.

Prerequisites

Before uploading and running an inference with a MLFlow model in Wallaroo the following will be required:

  • MLFlow Input Schema: The input schema with the fields and data types for each MLFlow model type uploaded to Wallaroo. In the examples below, the data types are imported using the pyarrow library.
  • An installed Wallaroo instance.
  • The following Python libraries installed:
    • os
    • wallaroo: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.

IMPORTANT NOTE: Wallaroo supports MLFlow 1.3.0. Please ensure the MLFlow models used in Wallaroo meet this specification.

MLFlow Inference Steps

To register a containerized MLFlow ML Model into Wallaroo, use the following general step:

  • Import Libraries
  • Connect to Wallaroo
  • Set MLFlow Input Schemas
  • Register MLFlow Model
  • Create Pipeline and Add Model Steps
  • Run Inference

Import Libraries

We start by importing the libraries we will need to connect to Wallaroo and use our MLFlow models. This includes the wallaroo libraries, pyarrow for data types, and the json library for handling JSON data.

import wallaroo
from wallaroo.object import EntityNotFoundError
import pyarrow as pa
import pandas as pd

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

def get_pipeline(name):
    try:
        pipeline = wl.pipelines_by_name(name)[0]
    except EntityNotFoundError:
        pipeline = wl.build_pipeline(name)
    return pipeline
prefix = 'mlflow'
workspace_name= f"{prefix}statsmodelworkspace"
pipeline_name = f"{prefix}statsmodelpipeline"

mlflowworkspace = get_workspace(workspace_name)
wl.set_current_workspace(mlflowworkspace)

pipeline = get_pipeline(pipeline_name)

Set MLFlow Input Schemas

Set the MLFlow input schemas through the pyarrow library. In the examples below, the input schemas for both the MLFlow model statsmodels-test and the statsmodels-test-postprocess model.

sm_input_schema = pa.schema([
  pa.field('temp', pa.float32()),
  pa.field('holiday', pa.uint8()),
  pa.field('workingday', pa.uint8()),
  pa.field('windspeed', pa.float32())
])

pp_input_schema = pa.schema([
    pa.field('predicted_mean', pa.float32())
])

Register MLFlow Model

Use the register_model_image method to register the Docker container containing the MLFlow models.

statsmodelUrl = "ghcr.io/wallaroolabs/wallaroo_tutorials/mlflow-statsmodels-example:2023.1"
postprocessUrl = "ghcr.io/wallaroolabs/wallaroo_tutorials/mlflow-postprocess-example:2023.1"

sm_model = wl.register_model_image(
    name=f"{prefix}statmodels",
    image=f"{statsmodelUrl}"
).configure("mlflow", input_schema=sm_input_schema, output_schema=pp_input_schema)
pp_model = wl.register_model_image(
    name=f"{prefix}postprocess",
    image=f"{postprocessUrl}"
).configure("mlflow", input_schema=pp_input_schema, output_schema=pp_input_schema)

Create Pipeline and Add Model Steps

With the models registered, we can add the MLFlow models as steps in the pipeline. Once ready, we will deploy the pipeline so it is available for submitting data for running inferences.

pipeline.add_model_step(sm_model)
pipeline.add_model_step(pp_model)
namemlflowstatsmodelpipeline
created2023-07-14 15:32:52.546581+00:00
last_updated2023-07-14 15:32:52.546581+00:00
deployed(none)
tags
versionsefa60e66-b2b7-4bb2-bc37-dd9ea7377467
steps
from wallaroo.deployment_config import DeploymentConfigBuilder

deployment_config = DeploymentConfigBuilder() \
    .cpus(0.25).memory('1Gi') \
    .sidekick_env(sm_model, {"GUNICORN_CMD_ARGS": "--timeout=180 --workers=1"}) \
    .sidekick_env(pp_model, {"GUNICORN_CMD_ARGS": "--timeout=180 --workers=1"}) \
    .build()

pipeline.deploy(deployment_config=deployment_config)
namemlflowstatsmodelpipeline
created2023-07-14 15:32:52.546581+00:00
last_updated2023-07-14 15:35:09.901078+00:00
deployedTrue
tags
versions2f8e9483-a9ad-436e-9068-7826cd8166c4, 539aae6b-4437-4f8c-8be0-db9ef11e80fb, efa60e66-b2b7-4bb2-bc37-dd9ea7377467
stepsmlflowstatmodels
pipeline.status()
{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.142',
   'name': 'engine-c55d468-qgk48',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'mlflowstatsmodelpipeline',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'mlflowpostprocess',
      'version': 'b7cd3426-825a-467d-8616-011ae8901d23',
      'sha': '825ebae48014d297134930028ab0e823bc0d9551334b9a4402c87a714e8156b2',
      'status': 'Running'},
     {'name': 'mlflowstatmodels',
      'version': '1a547bbb-36c7-4b5c-93a8-f357e3542d6f',
      'sha': '3afd13d9c5070679e284050cd099e84aa2e5cb7c08a788b21d6cb2397615d018',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.188',
   'name': 'engine-lb-584f54c899-xhdmq',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': [{'ip': '10.244.0.72',
   'name': 'engine-sidekick-mlflowstatmodels-27-786dc6b47c-bprqb',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'},
  {'ip': '10.244.0.73',
   'name': 'engine-sidekick-mlflowpostprocess-28-9fff54995-2n2nn',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'}]}

Run Inference

Once the pipeline is running, we can submit our data to the pipeline and return our results. Once finished, we will undeploy the pipeline to return the resources back to the cluster.

# Initial container run may need extra time to finish deploying - adding 90 second timeout to compensate
results = pipeline.infer_from_file('./resources/bike_day_eval_engine.df.json', timeout=90)
display(results.loc[:,["out.predicted_mean"]])
out.predicted_mean
00.281983
10.658847
20.572368
30.619873
4-1.217801
5-1.849156
60.933885
pipeline.undeploy()
namemlflowstatsmodelpipeline
created2023-07-14 15:32:52.546581+00:00
last_updated2023-07-14 15:32:56.591792+00:00
deployedFalse
tags
versions539aae6b-4437-4f8c-8be0-db9ef11e80fb, efa60e66-b2b7-4bb2-bc37-dd9ea7377467
stepsmlflowstatmodels

7 - Custom Inference Computer Vision Upload, Auto Packaging, and Deploy Tutorial

The uploading and auto packaging a Custom Inference aka Arbitrary Python computer vision model in Wallaroo.

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

Custom Inference Computer Vision Upload, Auto Packaging, and Deploy Tutorial

The following tutorial demonstrates using Wallaroo deploying Bring Your Own Predict aka Arbitrary Python in a Wallaroo instance. The model is auto packaged upon uploading. Once ready, the model is deployed and sample inferences performed on it.

Introduction

This tutorial focuses on the Resnet50 computer vision model. By default, this provides the following outputs from receiving an image converted to tensor values:

  • boxes: The bounding boxes for detected objects.
  • classes: The class of the detected object (bottle, coat, person, etc).
  • confidences: The confidence the model has that the detected model is the class.

For this demonstration, the model is modified with Wallaroo Bring Your Own Predict (BYOP) to add two additional fields:

  • avg_px_intensity: the average pixel intensity checks the input to determine the average value of the Red/Green/Blue pixel values. This is used as a benchmark to determine if the two images are significantly different. For example, an all white photo would have an avg_px_intensity of 1, while an all blank photo would have a value of 0.
  • avg_confidence: The average confidence of all detected objects.

Prerequisites

  • A Wallaroo Ops instance 2023.4 and above

To download the Wallaroo Computer Vision models, use the following link:

https://storage.googleapis.com/wallaroo-public-data/cv-demo-models/cv-retail-models.zip

Unzip the contents into the directory models.

The following libraries are required to run the tutorial:

onnx==1.12.0
onnxruntime==1.12.1
torchvision
torch
matplotlib==3.5.0
opencv-python
imutils
pytz
ipywidgets

To run this tutorial outside of a Wallaroo Ops center, the Wallaroo SDK is available and is installed via pip with:

pip install wallaroo==2023.4.1

Steps

Import Libraries

The following libraries are used to execute this tutorial. The utils.py provides additional helper methods for rendering the images into tensor fields and other useful tasks.

# preload needed libraries 

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework
from IPython.display import display
from IPython.display import Image
import pandas as pd
import json
import datetime
import time
import cv2
import matplotlib.pyplot as plt
import string
import random
import pyarrow as pa
import sys
import asyncio
import numpy as np

import utils
pd.set_option('display.max_colwidth', None)

import datetime

# api based inference request
import requests

import utils

# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

The option request_timeout provides additional time for the Wallaroo model upload process to complete.

wl = wallaroo.Client(request_timeout=600)

Create Workspace

We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace.

Workspace names must be unique. The following helper function will either create a new workspace, or retrieve an existing one with the same name. Verify that a pre-existing workspace has been shared with the targeted user.

Set the variables workspace_name to ensure a unique workspace name if required.

The workspace will then be set as the Current Workspace. Model uploads and pipeline creation through the SDK are set in the current workspace.

def get_workspace(name, client):
    workspace = None
    for ws in client.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = client.create_workspace(name)
    return workspace

workspace_name = "cv-retail-edge-observability"
model_name = "resnet-with-intensity"
model_file_name = "./models/model-with-pixel-intensity.zip"
pipeline_name = "retail-inv-tracker-edge-obs"

workspace = get_workspace(workspace_name, wl)
wl.set_current_workspace(workspace)
{'name': 'cv-retail-edge-observability', 'id': 12, 'archived': False, 'created_by': 'a762b468-4c2d-49dd-a9c7-7335e531741c', 'created_at': '2024-02-27T16:51:56.017038+00:00', 'models': [{'name': 'resnet-with-intensity', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2024, 2, 27, 16, 53, 13, 935887, tzinfo=tzutc()), 'created_at': datetime.datetime(2024, 2, 27, 16, 53, 13, 935887, tzinfo=tzutc())}], 'pipelines': [{'name': 'retail-inv-tracker-edge-obs', 'create_time': datetime.datetime(2024, 2, 27, 16, 56, 20, 567978, tzinfo=tzutc()), 'definition': '[]'}]}

Upload Model

The model is uploaded as a BYOP (Bring Your Own Predict) aka Aribitrary model, where the model, Python script and other artifacts are included in a .zip file. This requires the input and output schemas for the model specified in Apache Arrow Schema format.

The contents of the Wallaroo BYOP script includes the following to load the model, and add additional fields. By default the frcnn-resnet model outputs the following fields:

  • boxes: A List of the the bounding boxes of each detected object.
  • classes: A List of the class of each detected object.
  • confidences: A confidence score of how likely the object is what class was detected.

The model-with-pixel-intensity.zip BYOP model uses that class, then adds in the following fields.

  • avg_px_intensity: The average pixel intensidy of the image. This provides a way of determining if the same pictures from the same camera has significant changes. For example, a security camera would have a different pixel intensity in one picture where the light was on, versus one one when the lights were off.
  • avg_confidence: An average of all of the confidences output by the model. This is used for observability and track model drift.

The following predict method is used in the BYOP model.

def _predict(self, input_data: InferenceData):
        # Parse inputs
        inputs = input_data["tensor"]

        # Pass to onnx model
        # `dynamic_axes` hasn't been set in torch.onnx.export()
        # that is used in CVDemoUtils.loadPytorchAndConvertToOnnx()
        # therefore we cannot do batch inference
        ort_sess = ort.InferenceSession(self._model.SerializeToString())
        outputs = ort_sess.run(None, {"data": inputs.astype(np.float32)})

        boxes, classes, confidences = outputs

        # Calculate input derivatives
        avg_px_intensity = np.mean(inputs[0])
        
        # Calculate output derivatives
        avg_confidence = np.mean(confidences)

        # batch size isn't specified in the onnx session output
        # but we need to return a batch of outputs
        return {
            "boxes": np.array([boxes]),
            "classes": np.array([classes]),
            "confidences": np.array([confidences]),
            "avg_px_intensity": np.array([[avg_px_intensity]]),
            "avg_confidence": np.array([[avg_confidence]]),
        }

For full details on Wallaroo BYOP models, see Wallaroo SDK Essentials Guide: Model Uploads and Registrations: Arbitrary Python.

The following sets the input/output schemas in Apache pyarrow format, and uploads the models. Once uploaded, it is automatically

input_schema = pa.schema([
    pa.field('tensor', pa.list_(
        pa.list_(
            pa.list_(
                pa.float32(), # images are normalized
                list_size=640
            ),
            list_size=480
        ),
        list_size=3
    )),
])

output_schema = pa.schema([
    pa.field('boxes', pa.list_(pa.list_(pa.float32(), list_size=4))),
    pa.field('classes', pa.list_(pa.int64())),
    pa.field('confidences', pa.list_(pa.float32())),
    pa.field('avg_px_intensity', pa.list_(pa.float32())),
    pa.field('avg_confidence', pa.list_(pa.float32())),
])

model = wl.upload_model(model_name, 
                        model_file_name, 
                        framework=Framework.CUSTOM,
                        input_schema=input_schema, 
                        output_schema=output_schema)
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime........................successful

Ready

Deploy Pipeline

Next we configure the hardware we want to use for deployment. If we plan on eventually deploying to edge, this is a good way to simulate edge hardware conditions. The BYOP model is deployed as a Wallaroo Containerized Runtime, so the hardware allocation is performed through the sidekick options.

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .replica_count(1) \
    .cpus(1) \
    .memory("2Gi") \
    .sidekick_cpus(model, 1) \
    .sidekick_memory(model, '6Gi') \
    .build()

We create the pipeline with the wallaroo.client.build_pipeline method, and assign our model as a model pipeline step. Once complete, we will deploy the pipeline to allocate resources from the Kuberntes cluster hosting the Wallaroo Ops to the pipeline.

pipeline = wl.build_pipeline(pipeline_name)
pipeline.clear()
pipeline.add_model_step(model)
pipeline.deploy(deployment_config = deployment_config)
Waiting for deployment - this will take up to 600s ................... ok
nameretail-inv-tracker-edge-obs
created2024-02-27 16:56:20.567978+00:00
last_updated2024-02-27 17:55:07.307761+00:00
deployedTrue
archNone
tags
versions0e234ea5-ca90-40ea-abfb-a295941c606d, ecf277ed-d20f-481e-8dfe-9dc6937071f9, 21a90743-e121-485e-934f-754f335f4ea3, b4c9256f-719e-4d4c-b167-df32decec0aa
stepsresnet-with-intensity
publishedFalse

Sample Inferences

We’ll perform some sample inferences and display the outputs. For each image, we’ll use the inference result to draw the bounding boxes for all objects with a confidence value higher than 50%. We’ll display the average confidence across all detected objects after the bounding box image.

For more details on performing inferences, see Wallaroo SDK Essentials Guide: Inference Management.

width, height = 640, 480

baseline_images = [
    "./data/images/input/example/dairy_bottles.png",
    "./data/images/input/example/dairy_products.png",
    "./data/images/input/example/product_cheeses.png"
]

for sample_image in baseline_images:
    dfImage, resizedImage = utils.loadImageAndConvertToDataframe(sample_image, width, height)
    startTime = time.time()
    infResults = pipeline.infer(dfImage, timeout=300)
    endTime = time.time()

    elapsed = 1.0
    results = {
        'model_name' : model_name,
        'pipeline_name' : pipeline_name,
        'width': width,
        'height': height,
        'image' : resizedImage,
        'inf-results' : infResults,
        'confidence-target' : 0.50,
        'inference-time': (endTime-startTime),
        'onnx-time' : int(elapsed) / 1e+9,
        'classes_file': "./models/coco_classes.pickle",                 
        'color': 'BLUE'
    }

    image = utils.drawDetectedObjectsFromInference(results)
    
    display(infResults.loc[:, ['out.avg_confidence']])
out.avg_confidence
0[0.3588039]
out.avg_confidence
0[0.2970887]
out.avg_confidence
0[0.1656933]

The Wallaroo SDK is capable of using numpy arrays in a pandas DataFrame for inference requests. Our demonstration will focus on using API calls for inference requests, so we will flatten the numpy array and use that value for our inference inputs. The following examples show using a baseline and blurred image for inference requests and the sample outputs.

pipeline.undeploy()
Waiting for undeployment - this will take up to 600s ...................................... ok
nameretail-inv-tracker-edge-obs
created2024-02-27 16:56:20.567978+00:00
last_updated2024-02-27 17:55:07.307761+00:00
deployedFalse
archNone
tags
versions0e234ea5-ca90-40ea-abfb-a295941c606d, ecf277ed-d20f-481e-8dfe-9dc6937071f9, 21a90743-e121-485e-934f-754f335f4ea3, b4c9256f-719e-4d4c-b167-df32decec0aa
stepsresnet-with-intensity
publishedFalse

8 - Demand Curve Quick Start Guide

The Demand Curve Quick Start Guide demonstrates how to use Wallaroo to chart a demand curve based on submitted data. This example uses a model plus preprocess and postprocessing steps.

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

Demand Curve Pipeline Tutorial

This worksheet demonstrates a Wallaroo pipeline with data preprocessing, a model, and data postprocessing.

The model is a “demand curve” that predicts the expected number of units of a product that will be sold to a customer as a function of unit price and facts about the customer. Such models can be used for price optimization or sales volume forecasting. This is purely a “toy” demonstration, but is useful for detailing the process of working with models and pipelines.

Data preprocessing is required to create the features used by the model. Simple postprocessing prevents nonsensical estimates (e.g. negative units sold).

Prerequisites

  • An installed Wallaroo instance.
  • The following Python libraries installed:
    • os
    • wallaroo: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.
import json
import wallaroo
from wallaroo.object import EntityNotFoundError
import pandas
import numpy
import conversion
from wallaroo.object import EntityNotFoundError
import pyarrow as pa

# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()

Now that the Wallaroo client has been initialized, we can create the workspace and call it demandcurveworkspace, then set it as our current workspace. We’ll also create our pipeline so it’s ready when we add our models to it.

We’ll set some variables and methods to create our workspace, pipelines and models. Note that as of the July 2022 release of Wallaroo, workspace names must be unique. Pipelines with the same name will be created as a new version when built.

workspace_name = 'demandcurveworkspace'
pipeline_name = 'demandcurvepipeline'
model_name = 'demandcurvemodel'
model_file_name = './models/demand_curve_v1.onnx'
def get_workspace(name, client):
    workspace = None
    for ws in client.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = client.create_workspace(name)
    return workspace
workspace = get_workspace(workspace_name, wl)

wl.set_current_workspace(workspace)

demandcurve_pipeline = wl.build_pipeline(pipeline_name)
demandcurve_pipeline
namedemandcurvepipeline
created2024-03-13 19:22:55.459643+00:00
last_updated2024-03-13 19:23:42.945824+00:00
deployed(none)
archNone
tags
versionscf5f73c5-9725-49e7-a55c-530737653959, 5be24e45-8175-4b73-a091-d397d7bc5514
steps
publishedFalse

With our workspace established, we’ll upload three models:

  • ./models/preprocess_dc_byop.zip: A preprocess model step that formats the data into a tensor that the model can inference from.
  • ./models/demand_curve_v1.onnx: Our demand_curve model. We’ll store the upload configuration into demand_curve_model.
  • ./models/postprocess_dc_byop.zip: A postprocess model step that will zero out any negative values and return the output variable as “prediction”.

Note that the order we upload our models isn’t important - we’ll be establishing the actual process of moving data from one model to the next when we set up our pipeline.

demand_curve_model = wl.upload_model(model_name, 
                                     model_file_name, 
                                     framework=wallaroo.framework.Framework.ONNX).configure(tensor_fields=["tensor"])                   
input_schema = pa.schema([
    pa.field('Date', pa.string()),
    pa.field('cust_known', pa.bool_()),
    pa.field('StockCode', pa.int32()),
    pa.field('UnitPrice', pa.float32()),
    pa.field('UnitsSold', pa.int32())
])

output_schema = pa.schema([
    pa.field('tensor', pa.list_(pa.float64()))
])

preprocess_step = wl.upload_model('curve-preprocess', 
                                  './models/preprocess_dc_byop.zip', 
                                  framework=wallaroo.framework.Framework.CUSTOM, 
                                  input_schema=input_schema, 
                                  output_schema=output_schema)
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime........successful

Ready

input_schema = pa.schema([
    pa.field('variable', pa.list_(pa.float64()))
])

output_schema = pa.schema([
    pa.field('prediction', pa.list_(pa.float64()))
])

postprocess_step = wl.upload_model('curve-postprocess', 
                                   './models/postprocess_dc_byop.zip', 
                                   framework=wallaroo.framework.Framework.CUSTOM, 
                                   input_schema=input_schema, 
                                   output_schema=output_schema)
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime........successful

Ready

With our models uploaded, we’re going to create our own pipeline and give it three steps:

  • The preprocess step to put the data into a tensor format.
  • Then we apply the data to our demand_curve_model.
  • And finally, we prepare our data for output with the module_post.
# now make a pipeline
demandcurve_pipeline.undeploy()
demandcurve_pipeline.clear()
demandcurve_pipeline.add_model_step(preprocess_step)
demandcurve_pipeline.add_model_step(demand_curve_model)
demandcurve_pipeline.add_model_step(postprocess_step)
namedemandcurvepipeline
created2024-03-13 19:22:55.459643+00:00
last_updated2024-03-13 19:23:42.945824+00:00
deployed(none)
archNone
tags
versionscf5f73c5-9725-49e7-a55c-530737653959, 5be24e45-8175-4b73-a091-d397d7bc5514
steps
publishedFalse

And with that - let’s deploy our model pipeline. This usually takes about 45 seconds for the deployment to finish.

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(1).memory("1Gi").build()
demandcurve_pipeline.deploy(deployment_config=deploy_config)
Waiting for deployment - this will take up to 45s ............. ok
namedemandcurvepipeline
created2024-03-13 19:22:55.459643+00:00
last_updated2024-03-13 19:25:24.927731+00:00
deployedTrue
archNone
tags
versionsc2baa959-f50c-468e-875e-b3d14972d400, cf5f73c5-9725-49e7-a55c-530737653959, 5be24e45-8175-4b73-a091-d397d7bc5514
stepscurve-preprocess
publishedFalse

We can check the status of our pipeline to make sure everything was set up correctly:

demandcurve_pipeline.status()
{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.28.0.139',
   'name': 'engine-7b76dd59dd-kjvhm',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'demandcurvepipeline',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'curve-postprocess',
      'version': '9fd8b767-943e-477a-8a7f-f0424eb7a438',
      'sha': 'cf4cb335761e2bd5f238bd13f70e777f1fcc1eb31837ebea9cf3eb55c8faeb2f',
      'status': 'Running'},
     {'name': 'demandcurvemodel',
      'version': 'df1251a5-3fa3-4aff-9633-5a5577f40a3f',
      'sha': '2820b42c9e778ae259918315f25afc8685ecab9967bad0a3d241e6191b414a0d',
      'status': 'Running'},
     {'name': 'curve-preprocess',
      'version': 'e82604bb-5e6e-45a7-9147-6e7eb209b8ef',
      'sha': '22d6886115cbf667cfb7dbd394730625e09d0f8a8ff853848a7edebdb3c26f01',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.28.2.108',
   'name': 'engine-lb-d7cc8fc9c-2l2l2',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': [{'ip': '10.28.0.138',
   'name': 'engine-sidekick-curve-preprocess-6-86cdc949f7-2khcl',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'},
  {'ip': '10.28.2.107',
   'name': 'engine-sidekick-curve-postprocess-7-55c99ff755-c5w25',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'}]}

Everything is ready. Let’s feed our pipeline some data. We have some information prepared with the daily_purchasses.csv spreadsheet. We’ll start with just one row to make sure that everything is working correctly.

# read in some purchase data
purchases = pandas.read_csv('daily_purchases.csv')

# start with a one-row data frame for testing
subsamp_raw = purchases.iloc[0:1,: ]
subsamp_raw
Datecust_knownStockCodeUnitPriceUnitsSold
02010-12-01False219284.211
result = demandcurve_pipeline.infer(subsamp_raw)
display(result)
timein.Datein.StockCodein.UnitPricein.UnitsSoldin.cust_knownout.predictionanomaly.count
02024-03-13 19:25:39.1522010-12-01219284.211False[6.680255142999893]0

We can see from the out.prediction field that the demand curve has a predicted slope of 6.68 from our sample data. We can isolate that by specifying just the data output below.

display(result.loc[0, ['out.prediction']][0])
[6.680255142999893]

Bulk Inference

The initial test went perfectly. Now let’s throw some more data into our pipeline. We’ll draw 10 random rows from our spreadsheet, perform an inference from that, and then display the results and the logs showing the pipeline’s actions.

ix = numpy.random.choice(purchases.shape[0], size=10, replace=False)
converted = conversion.pandas_to_dict(purchases.iloc[ix,: ])
test_df = pd.DataFrame(converted['query'], columns=converted['colnames'])
display(test_df)

output = demandcurve_pipeline.infer(test_df)
display(output)
Datecust_knownStockCodeUnitPriceUnitsSold
02011-02-15True85099C1.9515
12011-04-14True232012.0820
22011-04-20True85099B2.08110
32011-08-19True219312.0820
42011-09-21True223862.0823
52011-09-12True85099C2.0810
62011-11-30False219292.0810
72011-01-21True85099C1.956
82011-11-25True232022.082
92011-09-29True223862.0852
timein.Datein.StockCodein.UnitPricein.UnitsSoldin.cust_knownout.predictionanomaly.count
02024-03-13 19:25:39.2582011-02-1585099C1.9515True[40.57067616108544]0
12024-03-13 19:25:39.2582011-04-14232012.0820True[33.125327529877765]0
22024-03-13 19:25:39.2582011-04-2085099B2.08110True[33.125327529877765]0
32024-03-13 19:25:39.2582011-08-19219312.0820True[33.125327529877765]0
42024-03-13 19:25:39.2582011-09-21223862.0823True[33.125327529877765]0
52024-03-13 19:25:39.2582011-09-1285099C2.0810True[33.125327529877765]0
62024-03-13 19:25:39.2582011-11-30219292.0810False[9.110871233285868]0
72024-03-13 19:25:39.2582011-01-2185099C1.956True[40.57067616108544]0
82024-03-13 19:25:39.2582011-11-25232022.082True[33.125327529877765]0
92024-03-13 19:25:39.2582011-09-29223862.0852True[33.125327529877765]0

Undeploy the Pipeline

Once we’ve finished with our demand curve demo, we’ll undeploy the pipeline and give the resources back to our Kubernetes cluster.

demandcurve_pipeline.undeploy()
Waiting for undeployment - this will take up to 45s .................................... ok
namedemandcurvepipeline
created2024-03-13 19:22:55.459643+00:00
last_updated2024-03-13 19:25:24.927731+00:00
deployedFalse
archNone
tags
versionsc2baa959-f50c-468e-875e-b3d14972d400, cf5f73c5-9725-49e7-a55c-530737653959, 5be24e45-8175-4b73-a091-d397d7bc5514
stepscurve-preprocess
publishedFalse

Thank you for being a part of this demonstration. If you have additional questions, please feel free to contact us at Wallaroo.

9 - IMDB Tutorial

The IMDB Tutorial demonstrates how to use Wallaroo to determine if reviews are positive or negative.

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

IMDB Sample

The following example demonstrates how to use Wallaroo with chained models. In this example, we will be using information from the IMDB (Internet Movie DataBase) with a sentiment model to detect whether a given review is positive or negative. Imagine using this to automatically scan Tweets regarding your product and finding either customers who need help or have nice things to say about your product.

Note that this example is considered a “toy” model - only the first 100 words in the review were tokenized, and the embedding is very small.

The following example is based on the Large Movie Review Dataset, and sample data can be downloaded from the aclIMDB dataset.

Prerequisites

  • An installed Wallaroo instance.
  • The following Python libraries installed:
    • os
    • wallaroo: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.
    • pandas: Pandas, mainly used for Pandas DataFrame
    • pyarrow: PyArrow for Apache Arrow support
    • polars: Polars for DataFrame with native Apache Arrow support
import wallaroo
from wallaroo.object import EntityNotFoundError

# to display dataframe tables
from IPython.display import display
# used to display dataframe information without truncating
import pandas as pd
pd.set_option('display.max_colwidth', None)
import pyarrow as pa
print(wallaroo.__version__)
2023.2.1rc2

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()

To test this model, we will perform the following:

  • Create a workspace for our models.
  • Upload two models:
    • embedder: Takes pre-tokenized text documents (model input: 100 integers/datum; output 800 numbers/datum) and creates an embedding from them.
    • sentiment: The second model classifies the resulting embeddings from 0 to 1, which 0 being an unfavorable review, 1 being a favorable review.
  • Create a pipeline that will take incoming data and pass it to the embedder, which will pass the output to the sentiment model, and then export the final result.
  • To test it, we will use information that has already been tokenized and submit it to our pipeline and gauge the results.

Just for the sake of this tutorial, we’ll use the SDK below to create our workspace , assign as our current workspace, then display all of the workspaces we have at the moment. We’ll also set up for our models and pipelines down the road, so we have one spot to change names to whatever fits your organization’s standards best.

To allow this tutorial to be run multiple times or by multiple users in the same Wallaroo instance, a random 4 character prefix will be added to the workspace, pipeline, and model.

When we create our new workspace, we’ll save it in the Python variable workspace so we can refer to it as needed.

First we’ll create a workspace for our environment, and call it imdbworkspace. We’ll also set up our pipeline so it’s ready for our models.

import string
import random

# make a random 4 character prefix
prefix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
workspace_name = f'{prefix}imdbworkspace'
pipeline_name = f'{prefix}imdbpipeline'
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

def get_pipeline(name):
    try:
        pipeline = wl.pipelines_by_name(name)[0]
    except EntityNotFoundError:
        pipeline = wl.build_pipeline(name)
    return pipeline
workspace = get_workspace(workspace_name)

wl.set_current_workspace(workspace)

imdb_pipeline = get_pipeline(pipeline_name)
imdb_pipeline
namebggqimdbpipeline
created2023-07-14 15:29:46.732165+00:00
last_updated2023-07-14 15:29:46.732165+00:00
deployed(none)
tags
versions844d5e89-1cb8-44a3-b9ff-2dbf1e75db27
steps

Just to make sure, let’s list our current workspace. If everything is going right, it will show us we’re in the imdb-workspace.

wl.get_current_workspace()
{'name': 'bggqimdbworkspace', 'id': 21, 'archived': False, 'created_by': '4e296632-35b3-460e-85fe-565e311bc566', 'created_at': '2023-07-14T15:29:45.662776+00:00', 'models': [], 'pipelines': [{'name': 'bggqimdbpipeline', 'create_time': datetime.datetime(2023, 7, 14, 15, 29, 46, 732165, tzinfo=tzutc()), 'definition': '[]'}]}

Now we’ll upload our two models:

  • embedder.onnx: This will be used to embed the tokenized documents for evaluation.
  • sentiment_model.onnx: This will be used to analyze the review and determine if it is a positive or negative review. The closer to 0, the more likely it is a negative review, while the closer to 1 the more likely it is to be a positive review.
embedder = (wl.upload_model(f'{prefix}embedder-o', 
                            './embedder.onnx', 
                            framework=wallaroo.framework.Framework.ONNX)
                            .configure(tensor_fields=["tensor"])
            )
smodel = (wl.upload_model(f'{prefix}smodel-o', 
                          './sentiment_model.onnx', 
                          framework=wallaroo.framework.Framework.ONNX)
                          .configure(runtime="onnx", tensor_fields=["flatten_1"])
        )

With our models uploaded, now we’ll create our pipeline that will contain two steps:

  • First, it runs the data through the embedder.
  • Second, it applies it to our sentiment model.
# now make a pipeline
imdb_pipeline.add_model_step(embedder)
imdb_pipeline.add_model_step(smodel)
namebggqimdbpipeline
created2023-07-14 15:29:46.732165+00:00
last_updated2023-07-14 15:29:46.732165+00:00
deployed(none)
tags
versions844d5e89-1cb8-44a3-b9ff-2dbf1e75db27
steps

Now that we have our pipeline set up with the steps, we can deploy the pipeline.

imdb_pipeline.deploy()
namebggqimdbpipeline
created2023-07-14 15:29:46.732165+00:00
last_updated2023-07-14 15:29:52.678281+00:00
deployedTrue
tags
versions8fd7b44a-f5d3-4291-b741-db398c478537, 844d5e89-1cb8-44a3-b9ff-2dbf1e75db27
stepsbggqembedder-o

We’ll check the pipeline status to verify it’s deployed and the models are ready.

imdb_pipeline.status()
{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.140',
   'name': 'engine-6cb454f6bb-4fz9b',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'bggqimdbpipeline',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'bggqsmodel-o',
      'version': 'aaca7ea2-b6d6-471f-9c5f-ea194de3112b',
      'sha': '3473ea8700fbf1a1a8bfb112554a0dde8aab36758030dcde94a9357a83fd5650',
      'status': 'Running'},
     {'name': 'bggqembedder-o',
      'version': '2ef7af8b-1ad6-4ae9-b126-f836a05e9e37',
      'sha': 'd083fd87fa84451904f71ab8b9adfa88580beb92ca77c046800f79780a20b7e4',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.186',
   'name': 'engine-lb-584f54c899-5zpkl',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': []}

To test this out, we’ll start with a single piece of information from our data directory.

singleton = pd.DataFrame.from_records(
    [
    {
        "tensor":[
            1607.0,
            2635.0,
            5749.0,
            199.0,
            49.0,
            351.0,
            16.0,
            2919.0,
            159.0,
            5092.0,
            2457.0,
            8.0,
            11.0,
            1252.0,
            507.0,
            42.0,
            287.0,
            316.0,
            15.0,
            65.0,
            136.0,
            2.0,
            133.0,
            16.0,
            4311.0,
            131.0,
            286.0,
            153.0,
            5.0,
            2826.0,
            175.0,
            54.0,
            548.0,
            48.0,
            1.0,
            17.0,
            9.0,
            183.0,
            1.0,
            111.0,
            15.0,
            1.0,
            17.0,
            284.0,
            982.0,
            18.0,
            28.0,
            211.0,
            1.0,
            1382.0,
            8.0,
            146.0,
            1.0,
            19.0,
            12.0,
            9.0,
            13.0,
            21.0,
            1898.0,
            122.0,
            14.0,
            70.0,
            14.0,
            9.0,
            97.0,
            25.0,
            74.0,
            1.0,
            189.0,
            12.0,
            9.0,
            6.0,
            31.0,
            3.0,
            244.0,
            2497.0,
            3659.0,
            2.0,
            665.0,
            2497.0,
            63.0,
            180.0,
            1.0,
            17.0,
            6.0,
            287.0,
            3.0,
            646.0,
            44.0,
            15.0,
            161.0,
            50.0,
            71.0,
            438.0,
            351.0,
            31.0,
            5749.0,
            2.0,
            0.0,
            0.0
        ]
    }
]
)
results = imdb_pipeline.infer(singleton)
display(results)
timein.tensorout.dense_1check_failures
02023-07-14 15:30:09.930[1607.0, 2635.0, 5749.0, 199.0, 49.0, 351.0, 16.0, 2919.0, 159.0, 5092.0, 2457.0, 8.0, 11.0, 1252.0, 507.0, 42.0, 287.0, 316.0, 15.0, 65.0, 136.0, 2.0, 133.0, 16.0, 4311.0, 131.0, 286.0, 153.0, 5.0, 2826.0, 175.0, 54.0, 548.0, 48.0, 1.0, 17.0, 9.0, 183.0, 1.0, 111.0, 15.0, 1.0, 17.0, 284.0, 982.0, 18.0, 28.0, 211.0, 1.0, 1382.0, 8.0, 146.0, 1.0, 19.0, 12.0, 9.0, 13.0, 21.0, 1898.0, 122.0, 14.0, 70.0, 14.0, 9.0, 97.0, 25.0, 74.0, 1.0, 189.0, 12.0, 9.0, 6.0, 31.0, 3.0, 244.0, 2497.0, 3659.0, 2.0, 665.0, 2497.0, 63.0, 180.0, 1.0, 17.0, 6.0, 287.0, 3.0, 646.0, 44.0, 15.0, 161.0, 50.0, 71.0, 438.0, 351.0, 31.0, 5749.0, 2.0, 0.0, 0.0][0.37142318]0

Since that works, let’s load up all 50,000 rows and do a full inference on each of them via an Apache Arrow file. Wallaroo pipeline inferences use Apache Arrow as their core data type, making this inference fast.

We’ll do a demonstration with a pandas DataFrame and display the first 5 results.

results = imdb_pipeline.infer_from_file('./data/test_data_50K.arrow')
# using pandas DataFrame

outputs = results.to_pandas()
display(outputs.loc[:5, ["time","out.dense_1"]])
timeout.dense_1
02023-07-14 15:30:17.404[0.8980188]
12023-07-14 15:30:17.404[0.056596935]
22023-07-14 15:30:17.404[0.9260802]
32023-07-14 15:30:17.404[0.926919]
42023-07-14 15:30:17.404[0.6618577]
52023-07-14 15:30:17.404[0.48736304]

Undeploy

With our pipeline’s work done, we’ll undeploy it and give our Kubernetes environment back its resources.

imdb_pipeline.undeploy()
namebggqimdbpipeline
created2023-07-14 15:29:46.732165+00:00
last_updated2023-07-14 15:29:52.678281+00:00
deployedFalse
tags
versions8fd7b44a-f5d3-4291-b741-db398c478537, 844d5e89-1cb8-44a3-b9ff-2dbf1e75db27
stepsbggqembedder-o

And there is our example. Please feel free to contact us at Wallaroo for if you have any questions.

10 - whisper-large-v2 Demonstration with Wallaroo

The whisper-large-v2 Demonstration with Wallaroo deployment.

This tutorial can be downloaded as part of the Wallaroo Tutorials repository.

Whisper Demo

The following tutorial demonstrates deploying the openai/whisper-large-v2 on a Wallaroo pipeline and performing inferences on it using the BYOP feature.

Data Prepartions

For this example, the following Python libraries were used:

These can be installed with the following command:

pip install librosa datasets --user

For these libraries, a sample of audio files was retrieved and converted using the following code.

import librosa
from datasets import load_dataset

# load the sample dataset and retrieve the audio files
dataset = load_dataset("Narsil/asr_dummy")

# the following is used to play them
audio_1, sr_1 = librosa.load(dataset["test"][0]["file"])
audio_2, sr_2 = librosa.load(dataset["test"][1]["file"])

audio_files = [(audio_1, sr_1), (audio_2, sr_2)]

# convert the audio files to numpy values in a DataFrame
input_data = {
        "inputs": [audio_1, audio_2],
        "return_timestamps": ["word", "word"],
}
dataframe = pd.DataFrame(input_data)

# the following will provide a UI to play the audio file samples

def display_audio(audio: np.array, sr: int) -> None:
    IPython.display.display(Audio(data=audio, rate=sr))

for audio, sr in audio_files:
    display_audio(audio, sr)

The resulting pandas DataFrame can either be submitted directly to a deployed Wallaroo pipeline using wallaroo.pipeline.infer, or the DataFrame exported to a pandas Record file in pandas JSON format, and used for an inference request using wallaroo.pipeline.infer_from_file.

For this example, the audio files are pre-converted to a JSON pandas Record table file, and used for the inference result. This removes the requirements to add additional Python libraries to a virtual environment or Wallaroo JupyterHub service. The code above is provided as an example of converting the dataset audio into values for inference requests.

Tutorial Steps

Import Libraries

The first step is to import the libraries we’ll be using. These are included by default in the Wallaroo instance’s JupyterHub service or are installed with the Wallaroo SDK.

import json
import os

import wallaroo
from wallaroo.pipeline   import Pipeline
from wallaroo.deployment_config import DeploymentConfigBuilder
from wallaroo.framework import Framework

import pyarrow as pa
import numpy as np
import pandas as pd

# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')
wallaroo.__version__
'2023.4.0+5d935fefc'

Open a Connection to Wallaroo

The next step is connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). If logging in externally, update the wallarooPrefix and wallarooSuffix variables with the proper DNS information. For more information on Wallaroo DNS settings, see the Wallaroo DNS Integration Guide.

For this tutorial, the request_timeout option is increased to allow the model conversion and pipeline deployment to proceed without any warning messages.

wl = wallaroo.Client(request_timeout=600)

Set Variables and Helper Functions

We’ll set the name of our workspace, pipeline, models and files. Workspace names must be unique across the Wallaroo workspace. For this, we’ll add in a randomly generated 4 characters to the workspace name to prevent collisions with other users’ workspaces. If running this tutorial, we recommend hard coding the workspace name so it will function in the same workspace each time it’s run.

We’ll set up some helper functions that will either use existing workspaces and pipelines, or create them if they do not already exist.

def get_workspace(name, client):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

The names for our workspace, pipeline, model, and model files are set here to make updating this tutorial easier.

  • IMPORTANT NOTE: Workspace names must be unique across the Wallaroo instance. To verify unique names, the randomization code below is provided to allow the workspace name to be unique. If this is not required, set suffix to ''.
import string
import random

# make a random 4 character suffix to prevent overwriting other user's workspaces
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))

suffix=''

workspace_name = f'whisper-tiny-demo{suffix}'
pipeline_name = 'whisper-hf-byop'
model_name = 'whisper-byop'
model_file_name = './models/model-auto-conversion_hugging-face_complex-pipelines_asr-whisper-tiny.zip'

Create Workspace and Pipeline

We will now create the Wallaroo workspace to store our model and set it as the current workspace. Future commands will default to this workspace for pipeline creation, model uploads, etc. We’ll create our Wallaroo pipeline that is used to deploy our arbitrary Python model.

workspace = get_workspace(workspace_name, wl)
wl.set_current_workspace(workspace)

pipeline = wl.build_pipeline(pipeline_name)

display(wl.get_current_workspace())
{'name': 'whisper-tiny-demojch', 'id': 32, 'archived': False, 'created_by': '9aa81a1f-952f-435e-b77d-504dd0215914', 'created_at': '2023-12-19T15:51:28.027575+00:00', 'models': [], 'pipelines': [{'name': 'whisper-hf-byop', 'create_time': datetime.datetime(2023, 12, 19, 15, 51, 28, 50715, tzinfo=tzutc()), 'definition': '[]'}]}

Configure & Upload Model

For this example, we will use the openai/whisper-tiny model for the automatic-speech-recognition pipeline task from the official 🤗 Hugging Face hub.

To manually create an automatic-speech-recognition pipeline from the 🤗 Hugging Face hub link above:

  1. Download the original model from the the official 🤗 Hugging Face hub.
from transformers import pipeline

pipe = pipeline("automatic-speech-recognition", model="openai/whisper-tiny")
pipe.save_pretrained("asr-whisper-tiny/")

As a last step, you can zip the folder containing all needed files as follows:

zip -r asr-whisper-tiny.zip asr-whisper-tiny/

Configure PyArrow Schema

You can find more info on the available inputs for the automatic-speech-recognition pipeline under the official source code from 🤗 Hugging Face.

The input and output schemas are defined in Apache pyarrow Schema format.

The model is then uploaded with the wallaroo.client.model_upload method, where we define:

  • The name to assign the model.
  • The model file path.
  • The input and output schemas.

The model is uploaded to the Wallaroo instance, where it is containerized to run with the Wallaroo Inference Engine.

input_schema = pa.schema([
    pa.field('inputs', pa.list_(pa.float32())), # required: the audio stored in numpy arrays of shape (num_samples,) and data type `float32`
    pa.field('return_timestamps', pa.string()) # optional: return start & end times for each predicted chunk
]) 

output_schema = pa.schema([
    pa.field('text', pa.string()), # required: the output text corresponding to the audio input
    pa.field('chunks', pa.list_(pa.struct([('text', pa.string()), ('timestamp', pa.list_(pa.float32()))]))), # required (if `return_timestamps` is set), start & end times for each predicted chunk
])
model = wl.upload_model(model_name, 
                        model_file_name, 
                        framework=Framework.HUGGING_FACE_AUTOMATIC_SPEECH_RECOGNITION, 
                        input_schema=input_schema, 
                        output_schema=output_schema)
model
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime............................................successful

Ready

Namewhisper-byop
Version6c36e84e-297c-4eec-9c38-9d7fb4e4e9db
File Namemodel-auto-conversion_hugging-face_complex-pipelines_asr-whisper-tiny.zip
SHAddd57c9c8d3ed5417783ebb7101421aa1e79429365d20326155c9c02ae1e8a13
Statusready
Image Pathproxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/mlflow-deploy:v2023.4.0-4297
ArchitectureNone
Updated At2023-19-Dec 15:55:27

Deploy Pipeline

The model is deployed with the wallaroo.pipeline.deploy(deployment_config) command. For the deployment configuration, we set the containerized aka sidekick memory to 8 GB to accommodate the size of the model, and CPUs to at least 4. To optimize performance, a GPU could be assigned to the containerized model.

deployment_config = DeploymentConfigBuilder() \
    .cpus(0.25).memory('1Gi') \
    .sidekick_memory(model, '8Gi') \
    .sidekick_cpus(model, 4.0) \
    .build()
pipeline = wl.build_pipeline(pipeline_name)
pipeline.add_model_step(model)

pipeline.deploy(deployment_config=deployment_config)
 ok
namewhisper-hf-byop
created2023-12-19 15:51:28.050715+00:00
last_updated2023-12-19 16:03:12.950146+00:00
deployedTrue
archNone
tags
versionsf03d5844-0a71-4791-bac9-b265680d0239, c9492252-e9c1-4e16-af96-4a9c9be2690a, 7579cda5-b36e-4b9b-a30e-1d544f03b9f8, 510b93fa-8389-4055-ae64-3e0e52016f88, 2e035d66-81d4-40f3-8e23-0b711953a176
stepswhisper-byop
publishedFalse

After a couple of minutes we verify the pipeline deployment was successful.

pipeline.status()
{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.9.7',
   'name': 'engine-65499bc68b-v2vbk',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'whisper-hf-byop',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'whisper-byop',
      'version': '6c36e84e-297c-4eec-9c38-9d7fb4e4e9db',
      'sha': 'ddd57c9c8d3ed5417783ebb7101421aa1e79429365d20326155c9c02ae1e8a13',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.2.112',
   'name': 'engine-lb-584f54c899-zlmqq',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': [{'ip': '10.244.9.6',
   'name': 'engine-sidekick-whisper-byop-16-6d5c4cd7f7-mfcpg',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'}]}

Run inference on the example dataset

We perform a sample inference with the provided DataFrame, and display the results.

%%time
result = pipeline.infer_from_file('./data/sound-examples.df.json', timeout=10000)
CPU times: user 138 ms, sys: 12.4 ms, total: 150 ms
Wall time: 6.02 s
display(result)
timein.inputsin.return_timestampsout.chunksout.textcheck_failures
02023-12-19 16:03:18.686[0.0003229662, 0.0003370901, 0.0002854846, 0.0...word[{'text': ' He', 'timestamp': [0.0, 1.08]}, {'...He hoped there would be Stu for dinner, turni...0
12023-12-19 16:03:18.686[0.0010076478, 0.0012469155, 0.0008045971, 0.0...word[{'text': ' Stuff', 'timestamp': [29.78, 29.78...Stuff it into you. His belly calcled him.0

Evaluate results

Let’s compare the results side by side with the audio inputs.

for transcription in result['out.text'].values:
    print(f"Transcription: {transcription}\n")
Transcription:  He hoped there would be Stu for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick, peppered, flour-fat and sauce.

Transcription: Stuff it into you. His belly calcled him.

Undeploy Pipelines

With the demonstration complete, we undeploy the pipelines to return the resources back to the Wallaroo instance.

pipeline.undeploy()
 ok
namewhisper-hf-byop
created2023-12-19 15:51:28.050715+00:00
last_updated2023-12-19 16:03:12.950146+00:00
deployedFalse
archNone
tags
versionsf03d5844-0a71-4791-bac9-b265680d0239, c9492252-e9c1-4e16-af96-4a9c9be2690a, 7579cda5-b36e-4b9b-a30e-1d544f03b9f8, 510b93fa-8389-4055-ae64-3e0e52016f88, 2e035d66-81d4-40f3-8e23-0b711953a176
stepswhisper-byop
publishedFalse