Recipes for deploying common models in Wallaroo.
Model Cookbooks
- 1: Aloha Quick Tutorial
- 2: Computer Vision Tutorials
- 2.1: Step 01: Detecting Objects Using mobilenet
- 2.2: Step 02: Detecting Objects Using resnet50
- 2.3: Step 03: mobilenet and resnet50 Shadow Deploy
- 3: Computer Vision: Yolov8n Demonstration
- 4: Computer Vision: Image Detection for Health Care
- 4.1: Computer Vision: Image Detection for Health Care
- 4.2: Computer Vision: Image Detection for Health Care
- 5: CLIP ViT-B/32 Transformer Demonstration with Wallaroo
- 6: Containerized MLFlow Tutorial
- 7: Custom Inference Computer Vision Upload, Auto Packaging, and Deploy Tutorial
- 8: Demand Curve Quick Start Guide
- 9: IMDB Tutorial
- 10: whisper-large-v2 Demonstration with Wallaroo
1 - Aloha Quick Tutorial
This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.
Aloha Demo
In this notebook we will walk through a simple pipeline deployment to inference on a model. For this example we will be using an open source model that uses an Aloha CNN LSTM model for classifying Domain names as being either legitimate or being used for nefarious purposes such as malware distribution.
Prerequisites
- An installed Wallaroo instance.
- The following Python libraries installed:
Tutorial Goals
For our example, we will perform the following:
- Create a workspace for our work.
- Upload the Aloha model.
- Create a pipeline that can ingest our submitted data, submit it to the model, and export the results
- Run a sample inference through our pipeline by loading a file
- Run a sample inference through our pipeline’s URL and store the results in a file.
All sample data and models are available through the Wallaroo Quick Start Guide Samples repository.
Connect to the Wallaroo Instance
The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. For more information on Wallaroo Client settings, see the Client Connection guide.e/wallaroo-sdk-essentials-client/).
import wallaroo
from wallaroo.object import EntityNotFoundError
# to display dataframe tables
from IPython.display import display
# used to display dataframe information without truncating
import pandas as pd
pd.set_option('display.max_colwidth', None)
import pyarrow as pa
# Login through local Wallaroo instance
wl = wallaroo.Client()
Create the Workspace
We will create a workspace to work in and call it the “alohaworkspace”, then set it as current workspace environment. We’ll also create our pipeline in advance as alohapipeline
. The model name and the model file will be specified for use in later steps.
To allow this tutorial to be run multiple times or by multiple users in the same Wallaroo instance, a random 4 character prefix will be added to the workspace, pipeline, and model.
import string
import random
# make a random 4 character suffix to verify uniqueness in tutorials
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
workspace_name = f'alohaworkspace{suffix}'
pipeline_name = f'alohapipeline{suffix}'
model_name = f'alohamodel{suffix}'
model_file_name = './alohacnnlstm.zip'
def get_workspace(name):
workspace = None
for ws in wl.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = wl.create_workspace(name)
return workspace
def get_pipeline(name):
try:
pipeline = wl.pipelines_by_name(name)[0]
except EntityNotFoundError:
pipeline = wl.build_pipeline(name)
return pipeline
wl.list_workspaces()[0:5]
[{'name': 'john.hummel@wallaroo.ai - Default Workspace', 'id': 1, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-24T16:55:38.589652+00:00', 'models': [{'name': 'hf-summarization-demoyns2', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 25, 15, 49, 36, 877913, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 25, 15, 49, 36, 877913, tzinfo=tzutc())}], 'pipelines': [{'name': 'hf-summarization-pipeline-edge', 'create_time': datetime.datetime(2023, 8, 25, 15, 52, 2, 329988, tzinfo=tzutc()), 'definition': '[]'}]},
{'name': 'edge-publish-demojohn', 'id': 5, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-24T16:57:29.048825+00:00', 'models': [{'name': 'ccfraud', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 24, 16, 57, 29, 410708, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 24, 16, 57, 29, 410708, tzinfo=tzutc())}], 'pipelines': [{'name': 'edge-pipeline', 'create_time': datetime.datetime(2023, 8, 24, 16, 57, 29, 486951, tzinfo=tzutc()), 'definition': '[]'}]},
{'name': 'workshop-workspace-sample', 'id': 6, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-24T21:01:46.66355+00:00', 'models': [{'name': 'house-price-prime', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 24, 21, 9, 13, 887924, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 24, 21, 8, 35, 654241, tzinfo=tzutc())}], 'pipelines': [{'name': 'houseprice-estimator', 'create_time': datetime.datetime(2023, 8, 24, 21, 16, 0, 405934, tzinfo=tzutc()), 'definition': '[]'}]},
{'name': 'john-is-nice', 'id': 7, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-24T21:05:57.618824+00:00', 'models': [], 'pipelines': []},
{'name': 'edge-hf-summarization', 'id': 8, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-25T18:43:02.41099+00:00', 'models': [{'name': 'hf-summarization', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 25, 18, 49, 26, 195772, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 25, 18, 49, 26, 195772, tzinfo=tzutc())}], 'pipelines': [{'name': 'edge-hf-summarization', 'create_time': datetime.datetime(2023, 8, 25, 18, 53, 19, 465175, tzinfo=tzutc()), 'definition': '[]'}]}]
workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
aloha_pipeline = get_pipeline(pipeline_name)
aloha_pipeline
name | alohapipelineudjo |
---|---|
created | 2023-08-28 17:10:05.043499+00:00 |
last_updated | 2023-08-28 17:10:05.043499+00:00 |
deployed | (none) |
tags | |
versions | e3f6ddff-9a25-48cc-9f94-fbc7bd67db29 |
steps | |
published | False |
We can verify the workspace is created the current default workspace with the get_current_workspace()
command.
wl.get_current_workspace()
{'name': 'alohaworkspaceudjo', 'id': 13, 'archived': False, 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384', 'created_at': '2023-08-28T17:10:04.351972+00:00', 'models': [], 'pipelines': [{'name': 'alohapipelineudjo', 'create_time': datetime.datetime(2023, 8, 28, 17, 10, 5, 43499, tzinfo=tzutc()), 'definition': '[]'}]}
Upload the Models
Now we will upload our models. Note that for this example we are applying the model from a .ZIP file. The Aloha model is a protobuf file that has been defined for evaluating web pages, and we will configure it to use data in the tensorflow
format.
from wallaroo.framework import Framework
model = wl.upload_model(model_name,
model_file_name,
framework=Framework.TENSORFLOW
)
Deploy a model
Now that we have a model that we want to use we will create a deployment for it.
We will tell the deployment we are using a tensorflow model and give the deployment name and the configuration we want for the deployment.
To do this, we’ll create our pipeline that can ingest the data, pass the data to our Aloha model, and give us a final output. We’ll call our pipeline aloha-test-demo
, then deploy it so it’s ready to receive data. The deployment process usually takes about 45 seconds.
- Note: If you receive an error that the pipeline could not be deployed because there are not enough resources, undeploy any other pipelines and deploy this one again. This command can quickly undeploy all pipelines to regain resources. We recommend not running this command in a production environment since it will cancel any running pipelines:
for p in wl.list_pipelines(): p.undeploy()
aloha_pipeline.add_model_step(model)
name | alohapipelineudjo |
---|---|
created | 2023-08-28 17:10:05.043499+00:00 |
last_updated | 2023-08-28 17:10:05.043499+00:00 |
deployed | (none) |
tags | |
versions | e3f6ddff-9a25-48cc-9f94-fbc7bd67db29 |
steps | |
published | False |
aloha_pipeline.deploy()
name | alohapipelineudjo |
---|---|
created | 2023-08-28 17:10:05.043499+00:00 |
last_updated | 2023-08-28 17:10:10.176032+00:00 |
deployed | True |
tags | |
versions | 750fec3b-f178-4abd-9643-2208e0fd1fbd, e3f6ddff-9a25-48cc-9f94-fbc7bd67db29 |
steps | alohamodeludjo |
published | False |
We can verify that the pipeline is running and list what models are associated with it.
aloha_pipeline.status()
{'status': 'Running',
'details': [],
'engines': [{'ip': '10.244.3.99',
'name': 'engine-7f6ccf6879-hmw2p',
'status': 'Running',
'reason': None,
'details': [],
'pipeline_statuses': {'pipelines': [{'id': 'alohapipelineudjo',
'status': 'Running'}]},
'model_statuses': {'models': [{'name': 'alohamodeludjo',
'version': 'b99dd290-870d-425e-8835-953decd402be',
'sha': 'd71d9ffc61aaac58c2b1ed70a2db13d1416fb9d3f5b891e5e4e2e97180fe22f8',
'status': 'Running'}]}}],
'engine_lbs': [{'ip': '10.244.4.114',
'name': 'engine-lb-584f54c899-4g6rh',
'status': 'Running',
'reason': None,
'details': []}],
'sidekicks': []}
Interferences
Infer 1 row
Now that the pipeline is deployed and our Aloha model is in place, we’ll perform a smoke test to verify the pipeline is up and running properly. We’ll use the infer_from_file
command to load a single encoded URL into the inference engine and print the results back out.
The result should tell us that the tokenized URL is legitimate (0) or fraud (1). This sample data should return close to 1 in out.main
.
smoke_test = pd.DataFrame.from_records(
[
{
"text_input":[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
28,
16,
32,
23,
29,
32,
30,
19,
26,
17
]
}
]
)
result = aloha_pipeline.infer(smoke_test)
display(result.loc[:, ["time","out.main"]])
time | out.main | |
---|---|---|
0 | 2023-08-28 17:10:27.482 | [0.997564] |
Infer From File
This time, we’ll give it a bigger set of data to infer. ./data/data_1k.arrow
is an Apache Arrow table with 1,000 records in it. Once submitted, we’ll turn the result into a DataFrame and display the first five results.
result = aloha_pipeline.infer_from_file('./data/data_1k.arrow')
display(result.to_pandas().loc[:, ["time","out.main"]])
time | out.main | |
---|---|---|
0 | 2023-08-28 17:10:28.399 | [0.997564] |
1 | 2023-08-28 17:10:28.399 | [0.9885122] |
2 | 2023-08-28 17:10:28.399 | [0.9993358] |
3 | 2023-08-28 17:10:28.399 | [0.99999857] |
4 | 2023-08-28 17:10:28.399 | [0.9984837] |
... | ... | ... |
995 | 2023-08-28 17:10:28.399 | [0.9999754] |
996 | 2023-08-28 17:10:28.399 | [0.9999727] |
997 | 2023-08-28 17:10:28.399 | [0.66066873] |
998 | 2023-08-28 17:10:28.399 | [0.9998954] |
999 | 2023-08-28 17:10:28.399 | [0.99999803] |
1000 rows × 2 columns
outputs = result.to_pandas()
display(outputs.loc[:5, ["time","out.main"]])
time | out.main | |
---|---|---|
0 | 2023-08-28 17:10:28.399 | [0.997564] |
1 | 2023-08-28 17:10:28.399 | [0.9885122] |
2 | 2023-08-28 17:10:28.399 | [0.9993358] |
3 | 2023-08-28 17:10:28.399 | [0.99999857] |
4 | 2023-08-28 17:10:28.399 | [0.9984837] |
5 | 2023-08-28 17:10:28.399 | [1.0] |
Batch Inference
Now that our smoke test is successful, let’s really give it some data. We have two inference files we can use:
data_1k.arrow
: Contains 10,000 inferencesdata_25k.arrow
: Contains 25,000 inferences
When Apache Arrow tables are submitted to a Wallaroo Pipeline, the inference is processed natively as an Arrow table, and the results are returned as an Arrow table. This allows for faster data processing than with JSON files or DataFrame objects.
We’ll pipe the data_25k.arrow
file through the aloha_pipeline
deployment URL, and place the results in a file named response.arrow
. We’ll also display the time this takes. Note that for larger batches of 50,000 inferences or more can be difficult to view in Jupyter Hub because of its size, so we’ll only display the first five rows.
- IMPORTANT NOTE: The
_deployment._url()
method will return an internal URL when using Python commands from within the Wallaroo instance - for example, the Wallaroo JupyterHub service. When connecting via an external connection,_deployment._url()
returns an external URL. External URL connections requires the authentication be included in the HTTP request, and that Model Endpoints Guide external endpoints are enabled in the Wallaroo configuration options.
inference_url = aloha_pipeline._deployment._url()
inference_url
'https://doc-test.api.wallarooexample.ai/v1/api/pipelines/infer/alohapipelineudjo-15/alohapipelineudjo'
connection =wl.mlops().__dict__
token = connection['token']
dataFile="./data/data_25k.arrow"
contentType="application/vnd.apache.arrow.file"
!curl -X POST {inference_url} -H "Authorization: Bearer {token}" -H "Content-Type:{contentType}" --data-binary @{dataFile} > curl_response.df
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 25.9M 100 21.1M 100 4874k 1740k 391k 0:00:12 0:00:12 --:--:-- 4647k
cc_data_from_file = pd.read_json('./curl_response.df', orient="records")
display(cc_data_from_file.head(5).loc[:5, ["time","out"]])
time | out | |
---|---|---|
0 | 1693242632007 | {'banjori': [0.0015195821], 'corebot': [0.9829147500000001], 'cryptolocker': [0.012099549000000001], 'dircrypt': [4.7591115e-05], 'gozi': [2.0289428e-05], 'kraken': [0.00031977256999999996], 'locky': [0.011029262000000001], 'main': [0.997564], 'matsnu': [0.010341609], 'pykspa': [0.008038961], 'qakbot': [0.016155055], 'ramdo': [0.00623623], 'ramnit': [0.0009985747000000001], 'simda': [1.7933434e-26], 'suppobox': [1.388995e-27]} |
1 | 1693242632007 | {'banjori': [7.447196e-18], 'corebot': [6.7359245e-08], 'cryptolocker': [0.1708199], 'dircrypt': [1.3220122000000002e-09], 'gozi': [1.2758705999999999e-24], 'kraken': [0.22559543], 'locky': [0.34209849999999997], 'main': [0.99999994], 'matsnu': [0.3080186], 'pykspa': [0.1828217], 'qakbot': [3.8022549999999994e-11], 'ramdo': [0.2062254], 'ramnit': [0.15215826], 'simda': [1.1701982e-30], 'suppobox': [3.1514454e-38]} |
2 | 1693242632007 | {'banjori': [2.8598648999999997e-21], 'corebot': [9.302004000000001e-08], 'cryptolocker': [0.04445298], 'dircrypt': [6.1637580000000004e-09], 'gozi': [8.3496755e-23], 'kraken': [0.48234479999999996], 'locky': [0.26332903], 'main': [1.0], 'matsnu': [0.29800338], 'pykspa': [0.22361776], 'qakbot': [1.5238920999999999e-06], 'ramdo': [0.32820392], 'ramnit': [0.029332489000000003], 'simda': [1.1995622e-31], 'suppobox': [0.0]} |
3 | 1693242632007 | {'banjori': [2.1387213e-15], 'corebot': [3.8817485e-10], 'cryptolocker': [0.045599736], 'dircrypt': [1.9090386e-07], 'gozi': [1.3140123e-25], 'kraken': [0.59542626], 'locky': [0.17374137], 'main': [0.9999996999999999], 'matsnu': [0.23151578], 'pykspa': [0.17591679999999998], 'qakbot': [1.0876152e-09], 'ramdo': [0.21832279999999998], 'ramnit': [0.0128692705], 'simda': [6.1588803e-28], 'suppobox': [1.4386237e-35]} |
4 | 1693242632007 | {'banjori': [9.453342500000001e-15], 'corebot': [7.091151e-10], 'cryptolocker': [0.049815163], 'dircrypt': [5.2914135e-09], 'gozi': [7.4132087e-19], 'kraken': [1.5504574999999998e-13], 'locky': [1.079181e-15], 'main': [0.9999988999999999], 'matsnu': [1.5003075e-15], 'pykspa': [0.33075705], 'qakbot': [2.6258850000000004e-07], 'ramdo': [0.5036279], 'ramnit': [0.020393765], 'simda': [0.0], 'suppobox': [2.3292326e-38]} |
Undeploy Pipeline
When finished with our tests, we will undeploy the pipeline so we have the Kubernetes resources back for other tasks. Note that if the deployment variable is unchanged aloha_pipeline.deploy() will restart the inference engine in the same configuration as before.
aloha_pipeline.undeploy()
name | alohapipelineudjo |
---|---|
created | 2023-08-28 17:10:05.043499+00:00 |
last_updated | 2023-08-28 17:10:10.176032+00:00 |
deployed | False |
tags | |
versions | 750fec3b-f178-4abd-9643-2208e0fd1fbd, e3f6ddff-9a25-48cc-9f94-fbc7bd67db29 |
steps | alohamodeludjo |
published | False |
2 - Computer Vision Tutorials
This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.
Step 00: Introduction and Setup
This tutorial demonstrates how to use the Wallaroo to detect objects in images through the following models:
- rnn mobilenet: A single stage object detector that performs fast inferences. Mobilenet is typically good at identifying objects at a distance.
- resnet50: A dual stage object detector with slower inferencing but but is able to detect objects that are closer to each other.
This tutorial series will demonstrate the following:
- How to deploy a Wallaroo pipeline with trained rnn mobilenet model and perform sample inferences to detect objects in pictures, then display those objects.
- How to deploy a Wallaroo pipeline with a trained resnet50 model and perform sample inferences to detect objects in pictures, then display those objects.
- Use the Wallaroo feature shadow deploy to have both models perform inferences, then select the inference result with the higher confidence and show the objects detected.
This tutorial assumes that users have installed the Wallaroo SDK or are running these tutorials from within their Wallaroo instance’s JupyterHub service.
This demonstration should be run within a Wallaroo JupyterHub instance for best results.
Prerequisites
The included OpenCV class is included in this demonstration as CVDemoUtils.py
, and requires the following dependencies:
- ffmpeg
- libsm
- libxext
Internal JupyterHub Service
To install these dependencies in the Wallaroo JupyterHub service, use the following commands from a terminal shell via the following procedure:
Launch the JupyterHub Service within the Wallaroo install.
Select File->New->Terminal.
Enter the following:
sudo apt-get update
sudo apt-get install ffmpeg libsm6 libxext6 -y
External SDK Users
For users using the Wallaroo SDK to connect with a remote Wallaroo instance, the following commands will install the required dependancies:
For Linux users, this can be installed with:
sudo apt-get update
sudo apt-get install ffmpeg libsm6 libxext6 -y
MacOS users can prepare their environments using a package manager such as Brew with the following:
brew install ffmpeg libsm libxext
Libraries and Dependencies
- This repository may use large file sizes for the models. Use the Wallaroo Tutorials Releases to download a .zip file of the most recent computer vision tutorial that includes the models.
- Import the following Python libraries into your environment:
These can be installed by running the command below in the Wallaroo JupyterHub service. Note the use of pip install torch --no-cache-dir
for low memory environments.
!pip install torchvision
!pip install torch --no-cache-dir
!pip install opencv-python
!pip install onnx
!pip install onnxruntime
!pip install imutils
!pip install pytz
!pip install ipywidgets
Requirement already satisfied: torchvision in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (0.14.1)
Requirement already satisfied: typing-extensions in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torchvision) (4.5.0)
Requirement already satisfied: numpy in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torchvision) (1.22.3)
Requirement already satisfied: requests in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torchvision) (2.25.1)
Requirement already satisfied: torch in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torchvision) (1.13.1)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torchvision) (10.1.0)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from requests->torchvision) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from requests->torchvision) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from requests->torchvision) (1.26.18)
Requirement already satisfied: certifi>=2017.4.17 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from requests->torchvision) (2023.7.22)
Requirement already satisfied: torch in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (1.13.1)
Requirement already satisfied: typing-extensions in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from torch) (4.5.0)
Requirement already satisfied: opencv-python in /Users/johnhansarick/.local/lib/python3.8/site-packages (4.8.1.78)
Requirement already satisfied: numpy>=1.21.0 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from opencv-python) (1.22.3)
Requirement already satisfied: onnx in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (1.15.0)
Requirement already satisfied: numpy in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnx) (1.22.3)
Requirement already satisfied: protobuf>=3.20.2 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnx) (4.24.4)
Collecting onnxruntime
Using cached onnxruntime-1.16.1-cp38-cp38-macosx_11_0_arm64.whl.metadata (4.1 kB)
Collecting coloredlogs (from onnxruntime)
Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
Requirement already satisfied: flatbuffers in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnxruntime) (1.12)
Requirement already satisfied: numpy>=1.21.6 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnxruntime) (1.22.3)
Requirement already satisfied: packaging in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnxruntime) (23.1)
Requirement already satisfied: protobuf in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from onnxruntime) (4.24.4)
Collecting sympy (from onnxruntime)
Using cached sympy-1.12-py3-none-any.whl (5.7 MB)
Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime)
Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
Collecting mpmath>=0.19 (from sympy->onnxruntime)
Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)
Using cached onnxruntime-1.16.1-cp38-cp38-macosx_11_0_arm64.whl (6.1 MB)
Installing collected packages: mpmath, sympy, humanfriendly, coloredlogs, onnxruntime
Successfully installed coloredlogs-15.0.1 humanfriendly-10.0 mpmath-1.3.0 onnxruntime-1.16.1 sympy-1.12
Collecting imutils
Using cached imutils-0.5.4-py3-none-any.whl
Installing collected packages: imutils
Successfully installed imutils-0.5.4
Requirement already satisfied: pytz in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (2023.3.post1)
Collecting ipywidgets
Using cached ipywidgets-8.1.1-py3-none-any.whl.metadata (2.4 kB)
Collecting comm>=0.1.3 (from ipywidgets)
Using cached comm-0.1.4-py3-none-any.whl.metadata (4.2 kB)
Requirement already satisfied: ipython>=6.1.0 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipywidgets) (7.24.1)
Requirement already satisfied: traitlets>=4.3.1 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipywidgets) (5.12.0)
Collecting widgetsnbextension~=4.0.9 (from ipywidgets)
Using cached widgetsnbextension-4.0.9-py3-none-any.whl.metadata (1.6 kB)
Collecting jupyterlab-widgets~=3.0.9 (from ipywidgets)
Using cached jupyterlab_widgets-3.0.9-py3-none-any.whl.metadata (4.1 kB)
Requirement already satisfied: setuptools>=18.5 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (68.0.0)
Requirement already satisfied: jedi>=0.16 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.18.1)
Requirement already satisfied: decorator in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1)
Requirement already satisfied: pickleshare in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.36)
Requirement already satisfied: pygments in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (2.15.1)
Requirement already satisfied: backcall in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.2.0)
Requirement already satisfied: matplotlib-inline in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.6)
Requirement already satisfied: pexpect>4.3 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (4.8.0)
Requirement already satisfied: appnope in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.2)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.3)
Requirement already satisfied: ptyprocess>=0.5 in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0)
Requirement already satisfied: wcwidth in /opt/homebrew/anaconda3/envs/wallaroosdk.2023.4.0/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=6.1.0->ipywidgets) (0.2.5)
Using cached ipywidgets-8.1.1-py3-none-any.whl (139 kB)
Using cached comm-0.1.4-py3-none-any.whl (6.6 kB)
Using cached jupyterlab_widgets-3.0.9-py3-none-any.whl (214 kB)
Using cached widgetsnbextension-4.0.9-py3-none-any.whl (2.3 MB)
Installing collected packages: widgetsnbextension, jupyterlab-widgets, comm, ipywidgets
Attempting uninstall: comm
Found existing installation: comm 0.1.2
Uninstalling comm-0.1.2:
Successfully uninstalled comm-0.1.2
Successfully installed comm-0.1.4 ipywidgets-8.1.1 jupyterlab-widgets-3.0.9 widgetsnbextension-4.0.9
The rest of the tutorials will rely on these libraries and applications, so finish their installation before running the tutorials in this series.
Models for Wallaroo Computer Vision Tutorials
In order for the wallaroo tutorial notebooks to run properly, the videos directory must contain these models in the models directory.
To download the Wallaroo Computer Vision models, use the following link:
https://storage.googleapis.com/wallaroo-public-data/cv-demo-models/cv-retail-models.zip
Unzip the contents into the directory models
.
Directory contents
- coco_classes.pickle - contain the 80 COCO classifications used by resnet50 and mobilenet object detectors.
- frcnn-resent.pt - PyTorch resnet50 model
- frcnn-resnet.pt.onnx - PyTorch resnet50 model converted to onnx
- mobilenet.pt - PyTorch mobilenet model
- mobilenet.pt.onnx - PyTorch mobilenet model converted to onnx
2.1 - Step 01: Detecting Objects Using mobilenet
This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.
Step 01: Detecting Objects Using mobilenet
The following tutorial demonstrates how to use a trained mobilenet model deployed in Wallaroo to detect objects. This process will use the following steps:
- Create a Wallaroo workspace and pipeline.
- Upload a trained mobilenet ML model and add it as a pipeline step.
- Deploy the pipeline.
- Perform an inference on a sample image.
- Draw the detected objects, their bounding boxes, their classifications, and the confidence of the classifications on the provided image.
- Review our results.
Steps
Import Libraries
The first step will be to import our libraries. Please check with Step 00: Introduction and Setup and verify that the necessary libraries and applications are added to your environment.
import torch
import pickle
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework
import numpy as np
import json
import requests
import time
import pandas as pd
from CVDemoUtils import CVDemo
# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
# used for unique connection names
import string
import random
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
suffix=''
Connect to the Wallaroo Instance
The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. For more information on Wallaroo Client settings, see the Client Connection guide.
# Login through local service
wl = wallaroo.Client()
Set Variables
The following variables and methods are used later to create or connect to an existing workspace, pipeline, and model.
workspace_name = f'mobilenetworkspacetest{suffix}'
pipeline_name = f'mobilenetpipeline{suffix}'
model_name = f'mobilenet{suffix}'
model_file_name = 'models/mobilenet.pt.onnx'
def get_workspace(name):
workspace = None
for ws in wl.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = wl.create_workspace(name)
return workspace
def get_pipeline(name):
try:
pipeline = wl.pipelines_by_name(name)[0]
except EntityNotFoundError:
pipeline = wl.build_pipeline(name)
return pipeline
Create Workspace
The workspace will be created or connected to, and set as the default workspace for this session. Once that is done, then all models and pipelines will be set in that workspace.
workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
wl.get_current_workspace()
{'name': 'mobilenetworkspacetest', 'id': 8, 'archived': False, 'created_by': '1b26fc10-d0f9-4f92-a1a2-ab342b3d069e', 'created_at': '2023-10-30T18:15:15.756292+00:00', 'models': [], 'pipelines': []}
Create Pipeline and Upload Model
We will now create or connect to an existing pipeline as named in the variables above.
pipeline = get_pipeline(pipeline_name)
mobilenet_model = wl.upload_model(model_name, model_file_name, framework=Framework.ONNX).configure(batch_config="single", tensor_fields=["tensor"])
Deploy Pipeline
With the model uploaded, we can add it is as a step in the pipeline, then deploy it. Once deployed, resources from the Wallaroo instance will be reserved and the pipeline will be ready to use the model to perform inference requests.
pipeline.add_model_step(mobilenet_model)
pipeline.deploy()
name | mobilenetpipeline |
---|---|
created | 2023-10-30 18:15:17.969988+00:00 |
last_updated | 2023-10-30 18:15:25.186553+00:00 |
deployed | True |
arch | None |
tags | |
versions | 7a1d84f9-13f7-405a-a4d0-5ec7de59f03e, b236b83b-1755-41d9-a2ae-804c040b9038 |
steps | mobilenet |
published | False |
Prepare input image
Next we will load a sample image and resize it to the width and height required for the object detector. Once complete, it the image will be converted to a numpy ndim array and added to a dictionary.
# The size the image will be resized to
width = 640
height = 480
# Only objects that have a confidence > confidence_target will be displayed on the image
cvDemo = CVDemo()
imagePath = 'data/images/input/example/dairy_bottles.png'
# The image width and height needs to be set to what the model was trained for. In this case 640x480.
tensor, resizedImage = cvDemo.loadImageAndResize(imagePath, width, height)
# get npArray from the tensorFloat
npArray = tensor.cpu().numpy()
#creates a dictionary with the wallaroo "tensor" key and the numpy ndim array representing image as the value.
dictData = {"tensor":[npArray]}
dataframedata = pd.DataFrame(dictData)
Run Inference
With that done, we can have the model detect the objects on the image by running an inference through the pipeline, and storing the results for the next step.
startTime = time.time()
# pass the dataframe in
#infResults = pipeline.infer(dataframedata, dataset=["*", "metadata.elapsed"])
infResults = pipeline.infer_from_file('./data/dairy_bottles.df.json', dataset=["*", "metadata.elapsed"])
endTime = time.time()
Draw the Inference Results
With our inference results, we can take them and use the Wallaroo CVDemo class and draw them onto the original image. The bounding boxes and the confidence value will only be drawn on images where the model returned a 90% confidence rate in the object’s identity.
df = pd.DataFrame(columns=['classification','confidence','x','y','width','height'])
pd.options.mode.chained_assignment = None # default='warn'
pd.options.display.float_format = '{:.2%}'.format
# Points to where all the inference results are
boxList = infResults.loc[0]["out.boxes"]
# # reshape this to an array of bounding box coordinates converted to ints
boxA = np.array(boxList)
boxes = boxA.reshape(-1, 4)
boxes = boxes.astype(int)
df[['x', 'y','width','height']] = pd.DataFrame(boxes)
classes = infResults.loc[0]["out.classes"]
confidences = infResults.loc[0]["out.confidences"]
infResults = {
'model_name' : model_name,
'pipeline_name' : pipeline_name,
'width': width,
'height': height,
'image' : resizedImage,
'boxes' : boxes,
'classes' : classes,
'confidences' : confidences,
'confidence-target' : 0.90,
'inference-time': (endTime-startTime),
'onnx-time' : int(infResults.loc[0]["metadata.elapsed"][1]) / 1e+9,
'color':(255,0,0)
}
image = cvDemo.drawAndDisplayDetectedObjectsWithClassification(infResults)
Extract the Inference Information
To show what is going on in the background, we’ll extract the inference results create a dataframe with columns representing the classification, confidence, and bounding boxes of the objects identified.
idx = 0
for idx in range(0,len(classes)):
df['classification'][idx] = cvDemo.CLASSES[classes[idx]] # Classes contains the 80 different COCO classificaitons
df['confidence'][idx] = confidences[idx]
df
classification | confidence | x | y | width | height | |
---|---|---|---|---|---|---|
0 | bottle | 98.65% | 0 | 210 | 85 | 479 |
1 | bottle | 90.12% | 72 | 197 | 151 | 468 |
2 | bottle | 60.78% | 211 | 184 | 277 | 420 |
3 | bottle | 59.22% | 143 | 203 | 216 | 448 |
4 | refrigerator | 53.73% | 13 | 41 | 640 | 480 |
5 | bottle | 45.13% | 106 | 206 | 159 | 463 |
6 | bottle | 43.73% | 278 | 1 | 321 | 93 |
7 | bottle | 43.09% | 462 | 104 | 510 | 224 |
8 | bottle | 40.85% | 310 | 1 | 352 | 94 |
9 | bottle | 39.19% | 528 | 268 | 636 | 475 |
10 | bottle | 35.76% | 220 | 0 | 258 | 90 |
11 | bottle | 31.81% | 552 | 96 | 600 | 233 |
12 | bottle | 26.45% | 349 | 0 | 404 | 98 |
13 | bottle | 23.06% | 450 | 264 | 619 | 472 |
14 | bottle | 20.48% | 261 | 193 | 307 | 408 |
15 | bottle | 17.46% | 509 | 101 | 544 | 235 |
16 | bottle | 17.31% | 592 | 100 | 633 | 239 |
17 | bottle | 16.00% | 475 | 297 | 551 | 468 |
18 | bottle | 14.91% | 368 | 163 | 423 | 362 |
19 | book | 13.66% | 120 | 0 | 175 | 81 |
20 | book | 13.32% | 72 | 0 | 143 | 85 |
21 | bottle | 12.22% | 271 | 200 | 305 | 274 |
22 | book | 12.13% | 161 | 0 | 213 | 85 |
23 | bottle | 11.96% | 162 | 0 | 214 | 83 |
24 | bottle | 11.53% | 310 | 190 | 367 | 397 |
25 | bottle | 9.62% | 396 | 166 | 441 | 360 |
26 | cake | 8.65% | 439 | 256 | 640 | 473 |
27 | bottle | 7.84% | 544 | 375 | 636 | 472 |
28 | vase | 7.23% | 272 | 2 | 306 | 96 |
29 | bottle | 6.28% | 453 | 303 | 524 | 463 |
30 | bottle | 5.28% | 609 | 94 | 635 | 211 |
Undeploy the Pipeline
With the inference complete, we can undeploy the pipeline and return the resources back to the Wallaroo instance.
pipeline.undeploy()
name | mobilenetpipeline |
---|---|
created | 2023-10-30 18:15:17.969988+00:00 |
last_updated | 2023-10-30 18:15:25.186553+00:00 |
deployed | False |
arch | None |
tags | |
versions | 7a1d84f9-13f7-405a-a4d0-5ec7de59f03e, b236b83b-1755-41d9-a2ae-804c040b9038 |
steps | mobilenet |
published | False |
2.2 - Step 02: Detecting Objects Using resnet50
This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.
Step 02: Detecting Objects Using resnet50
The following tutorial demonstrates how to use a trained mobilenet model deployed in Wallaroo to detect objects. This process will use the following steps:
- Create a Wallaroo workspace and pipeline.
- Upload a trained resnet50 ML model and add it as a pipeline step.
- Deploy the pipeline.
- Perform an inference on a sample image.
- Draw the detected objects, their bounding boxes, their classifications, and the confidence of the classifications on the provided image.
- Review our results.
Steps
Import Libraries
The first step will be to import our libraries. Please check with Step 00: Introduction and Setup and verify that the necessary libraries and applications are added to your environment.
import torch
import pickle
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework
import numpy as np
import json
import requests
import time
import pandas as pd
from CVDemoUtils import CVDemo
# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
# used for unique connection names
import string
import random
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
Connect to the Wallaroo Instance
The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. For more information on Wallaroo Client settings, see the Client Connection guide.
# Login through local service
wl = wallaroo.Client()
wl = wallaroo.Client()
Set Variables
The following variables and methods are used later to create or connect to an existing workspace, pipeline, and model. This example has both the resnet model, and a post process script.
workspace_name = f'resnetworkspace{suffix}'
pipeline_name = f'resnetnetpipeline{suffix}'
model_name = f'resnet50{suffix}'
model_file_name = 'models/frcnn-resnet.pt.onnx'
def get_workspace(name):
workspace = None
for ws in wl.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = wl.create_workspace(name)
return workspace
def get_pipeline(name):
try:
pipeline = wl.pipelines_by_name(name)[0]
except EntityNotFoundError:
pipeline = wl.build_pipeline(name)
return pipeline
Create Workspace
The workspace will be created or connected to, and set as the default workspace for this session. Once that is done, then all models and pipelines will be set in that workspace.
workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
wl.get_current_workspace()
{'name': 'resnetworkspacestck', 'id': 9, 'archived': False, 'created_by': '6236ad2a-7eb8-4bbc-a8c9-39ce92767bad', 'created_at': '2023-10-24T16:52:27.179962+00:00', 'models': [], 'pipelines': []}
Create Pipeline and Upload Model
We will now create or connect to an existing pipeline as named in the variables above.
pipeline = get_pipeline(pipeline_name)
resnet_model = wl.upload_model(model_name, model_file_name, framework=Framework.ONNX).configure(batch_config="single", tensor_fields=["tensor"])
Deploy Pipeline
With the model uploaded, we can add it is as a step in the pipeline, then deploy it. Once deployed, resources from the Wallaroo instance will be reserved and the pipeline will be ready to use the model to perform inference requests.
pipeline.add_model_step(resnet_model)
name | resnetnetpipelinestck |
---|---|
created | 2023-10-24 16:52:29.027087+00:00 |
last_updated | 2023-10-24 16:52:29.027087+00:00 |
deployed | (none) |
tags | |
versions | 6e3874c5-e1bb-4847-a60d-2c42b7bc0a7e |
steps | |
published | False |
pipeline.deploy()
name | resnetnetpipelinestck |
---|---|
created | 2023-10-24 16:52:29.027087+00:00 |
last_updated | 2023-10-24 16:52:40.680056+00:00 |
deployed | True |
tags | |
versions | 52cb9d23-29cc-431f-9b08-d95d337b5bea, 6e3874c5-e1bb-4847-a60d-2c42b7bc0a7e |
steps | resnet50stck |
published | False |
Test the pipeline by running inference on a sample image
Prepare input image
Next we will load a sample image and resize it to the width and height required for the object detector.
We will convert the image to a numpy ndim array and add it do a dictionary
#The size the image will be resized to
width = 640
height = 480
cvDemo = CVDemo()
imagePath = 'data/images/input/example/dairy_bottles.png'
# The image width and height needs to be set to what the model was trained for. In this case 640x480.
tensor, resizedImage = cvDemo.loadImageAndResize(imagePath, width, height)
# get npArray from the tensorFloat
npArray = tensor.cpu().numpy()
dictData = {"tensor":[npArray]}
dataframedata = pd.DataFrame(dictData)
Run Inference
With that done, we can have the model detect the objects on the image by running an inference through the pipeline, and storing the results for the next step.
IMPORTANT NOTE: If necessary, add timeout=60
to the infer
method if more time is needed to upload the data file for the inference request.
startTime = time.time()
# pass the dataframe in
# infResults = pipeline.infer(dataframedata, dataset=["*", "metadata.elapsed"])
infResults = pipeline.infer_from_file('./data/dairy_bottles.df.json', dataset=["*", "metadata.elapsed"])
endTime = time.time()
Draw the Inference Results
With our inference results, we can take them and use the Wallaroo CVDemo class and draw them onto the original image. The bounding boxes and the confidence value will only be drawn on images where the model returned a 50% confidence rate in the object’s identity.
df = pd.DataFrame(columns=['classification','confidence','x','y','width','height'])
pd.options.mode.chained_assignment = None # default='warn'
pd.options.display.float_format = '{:.2%}'.format
# Points to where all the inference results are
boxList = infResults.loc[0]["out.boxes"]
# # reshape this to an array of bounding box coordinates converted to ints
boxA = np.array(boxList)
boxes = boxA.reshape(-1, 4)
boxes = boxes.astype(int)
df[['x', 'y','width','height']] = pd.DataFrame(boxes)
classes = infResults.loc[0]["out.classes"]
confidences = infResults.loc[0]["out.confidences"]
infResults = {
'model_name' : model_name,
'pipeline_name' : pipeline_name,
'width': width,
'height': height,
'image' : resizedImage,
'boxes' : boxes,
'classes' : classes,
'confidences' : confidences,
'confidence-target' : 0.90,
'inference-time': (endTime-startTime),
'onnx-time' : int(infResults.loc[0]["metadata.elapsed"][1]) / 1e+9,
'color':(255,0,0)
}
image = cvDemo.drawAndDisplayDetectedObjectsWithClassification(infResults)
Extract the Inference Information
To show what is going on in the background, we’ll extract the inference results create a dataframe with columns representing the classification, confidence, and bounding boxes of the objects identified.
idx = 0
for idx in range(0,len(classes)):
cocoClasses = cvDemo.getCocoClasses()
df['classification'][idx] = cocoClasses[classes[idx]] # Classes contains the 80 different COCO classificaitons
df['confidence'][idx] = confidences[idx]
df
classification | confidence | x | y | width | height | |
---|---|---|---|---|---|---|
0 | bottle | 99.65% | 2 | 193 | 76 | 475 |
1 | bottle | 98.83% | 610 | 98 | 639 | 232 |
2 | bottle | 97.00% | 544 | 98 | 581 | 230 |
3 | bottle | 96.96% | 454 | 113 | 484 | 210 |
4 | bottle | 96.48% | 502 | 331 | 551 | 476 |
... | ... | ... | ... | ... | ... | ... |
95 | bottle | 5.72% | 556 | 287 | 580 | 322 |
96 | refrigerator | 5.66% | 80 | 161 | 638 | 480 |
97 | bottle | 5.60% | 455 | 334 | 480 | 349 |
98 | bottle | 5.46% | 613 | 267 | 635 | 375 |
99 | bottle | 5.37% | 345 | 2 | 395 | 99 |
100 rows × 6 columns
Undeploy the Pipeline
With the inference complete, we can undeploy the pipeline and return the resources back to the Wallaroo instance.
pipeline.undeploy()
name | resnetnetpipelinestck |
---|---|
created | 2023-10-24 16:52:29.027087+00:00 |
last_updated | 2023-10-24 16:52:40.680056+00:00 |
deployed | False |
tags | |
versions | 52cb9d23-29cc-431f-9b08-d95d337b5bea, 6e3874c5-e1bb-4847-a60d-2c42b7bc0a7e |
steps | resnet50stck |
published | False |
2.3 - Step 03: mobilenet and resnet50 Shadow Deploy
This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.
Step 03: Detecting Objects Using Shadow Deploy
The following tutorial demonstrates how to use two trained models, one based on the resnet50, the other on mobilenet, deployed in Wallaroo to detect objects. This builds on the previous tutorials in this series, Step 01: Detecting Objects Using mobilenet" and “Step 02: Detecting Objects Using resnet50”.
For this tutorial, the Wallaroo feature Shadow Deploy will be used to submit inference requests to both models at once. The mobilnet object detector is the control and the faster-rcnn object detector is the challenger. The results between the two will be compared for their confidence, and that confidence will be used to draw bounding boxes around identified objects.
This process will use the following steps:
- Create a Wallaroo workspace and pipeline.
- Upload a trained resnet50 ML model and trained mobilenet model and add them as a shadow deployed step with the mobilenet as the control model.
- Deploy the pipeline.
- Perform an inference on a sample image.
- Based on the
- Draw the detected objects, their bounding boxes, their classifications, and the confidence of the classifications on the provided image.
- Review our results.
Steps
Import Libraries
The first step will be to import our libraries. Please check with Step 00: Introduction and Setup and verify that the necessary libraries and applications are added to your environment.
import torch
import pickle
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework
import numpy as np
import json
import requests
import time
import pandas as pd
from CVDemoUtils import CVDemo
# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
# used for unique connection names
import string
import random
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
Connect to the Wallaroo Instance
The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. For more information on Wallaroo Client settings, see the Client Connection guide.
# Login through local service
wl = wallaroo.Client()
wl = wallaroo.Client()
Set Variables
The following variables and methods are used later to create or connect to an existing workspace, pipeline, and model. This example has both the resnet model, and a post process script.
workspace_name = f'shadowimageworkspacetest{suffix}'
pipeline_name = f'shadowimagepipelinetest{suffix}'
control_model_name = f'mobilenet{suffix}'
control_model_file_name = 'models/mobilenet.pt.onnx'
challenger_model_name = f'resnet50{suffix}'
challenger_model_file_name = 'models/frcnn-resnet.pt.onnx'
def get_workspace(name):
workspace = None
for ws in wl.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = wl.create_workspace(name)
return workspace
def get_pipeline(name):
try:
pipeline = wl.pipelines_by_name(name)[0]
except EntityNotFoundError:
pipeline = wl.build_pipeline(name)
return pipeline
Create Workspace
The workspace will be created or connected to, and set as the default workspace for this session. Once that is done, then all models and pipelines will be set in that workspace.
workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
wl.get_current_workspace()
{'name': 'shadowimageworkspacetestkpld', 'id': 10, 'archived': False, 'created_by': '6236ad2a-7eb8-4bbc-a8c9-39ce92767bad', 'created_at': '2023-10-24T17:05:03.040658+00:00', 'models': [], 'pipelines': []}
Create Pipeline and Upload Model
We will now create or connect to an existing pipeline as named in the variables above, then upload each of the models.
pipeline = get_pipeline(pipeline_name)
control = wl.upload_model(control_model_name, control_model_file_name, framework=Framework.ONNX).configure(batch_config="single", tensor_fields=["tensor"])
challenger = wl.upload_model(challenger_model_name, challenger_model_file_name, framework=Framework.ONNX).configure(batch_config="single", tensor_fields=["tensor"])
Shadow Deploy Pipeline
For this step, rather than deploying each model into a separate step, both will be deployed into a single step as a Shadow Deploy step. This will take the inference input data and process it through both pipelines at the same time. The inference results for the control will be stored in it’s ['outputs']
array, while the results for the challenger are stored the ['shadow_data']
array.
pipeline.add_shadow_deploy(control, [challenger])
name | shadowimagepipelinetestkpld |
---|---|
created | 2023-10-24 17:05:04.926467+00:00 |
last_updated | 2023-10-24 17:05:04.926467+00:00 |
deployed | (none) |
tags | |
versions | 2f6bee8b-bf31-41cf-b7d2-d4912bfdcca8 |
steps | |
published | False |
pipeline.deploy()
name | shadowimagepipelinetestkpld |
---|---|
created | 2023-10-24 17:05:04.926467+00:00 |
last_updated | 2023-10-24 17:05:22.613731+00:00 |
deployed | True |
tags | |
versions | 433aa539-cb64-4733-85d3-68f6f769dd36, 2f6bee8b-bf31-41cf-b7d2-d4912bfdcca8 |
steps | mobilenetkpld |
published | False |
pipeline.status()
{'status': 'Running',
'details': [],
'engines': [{'ip': '10.244.3.35',
'name': 'engine-5888d5b5d6-2dvlb',
'status': 'Running',
'reason': None,
'details': [],
'pipeline_statuses': {'pipelines': [{'id': 'shadowimagepipelinetestkpld',
'status': 'Running'}]},
'model_statuses': {'models': [{'name': 'mobilenetkpld',
'version': 'be97c0ef-afb1-4835-9f94-ec6534fa9c07',
'sha': '9044c970ee061cc47e0c77e20b05e884be37f2a20aa9c0c3ce1993dbd486a830',
'status': 'Running'},
{'name': 'resnet50kpld',
'version': 'f0967b5f-4b17-4dbd-b1d6-49b3339f041b',
'sha': '43326e50af639105c81372346fb9ddf453fea0fe46648b2053c375360d9c1647',
'status': 'Running'}]}}],
'engine_lbs': [{'ip': '10.244.4.58',
'name': 'engine-lb-584f54c899-hl4lx',
'status': 'Running',
'reason': None,
'details': []}],
'sidekicks': []}
Prepare input image
Next we will load a sample image and resize it to the width and height required for the object detector.
We will convert the image to a numpy ndim array and add it do a dictionary
imagePath = 'data/images/input/example/store-front.png'
# The image width and height needs to be set to what the model was trained for. In this case 640x480.
cvDemo = CVDemo()
# The size the image will be resized to meet the input requirements of the object detector
width = 640
height = 480
tensor, controlImage = cvDemo.loadImageAndResize(imagePath, width, height)
challengerImage = controlImage.copy()
# get npArray from the tensorFloat
npArray = tensor.cpu().numpy()
#creates a dictionary with the wallaroo "tensor" key and the numpy ndim array representing image as the value.
# dictData = {"tensor": npArray.tolist()}
dictData = {"tensor":[npArray]}
dataframedata = pd.DataFrame(dictData)
Run Inference using Shadow Deployment
Now lets have the model detect the objects on the image by running inference and extracting the results
startTime = time.time()
infResults = pipeline.infer_from_file('./data/dairy_bottles.df.json', dataset=["*", "metadata.elapsed"])
#infResults = pipeline.infer(dataframedata, dataset=["*", "metadata.elapsed"])
endTime = time.time()
Extract Control Inference Results
First we’ll extract the inference result data for the control model and map it onto the image.
df = pd.DataFrame(columns=['classification','confidence','x','y','width','height'])
pd.options.mode.chained_assignment = None # default='warn'
pd.options.display.float_format = '{:.2%}'.format
# Points to where all the inference results are
# boxList = infResults[0]["out.output"]
boxList = infResults.loc[0]["out.boxes"]
# # reshape this to an array of bounding box coordinates converted to ints
boxA = np.array(boxList)
controlBoxes = boxA.reshape(-1, 4)
controlBoxes = controlBoxes.astype(int)
df[['x', 'y','width','height']] = pd.DataFrame(controlBoxes)
controlClasses = infResults.loc[0]["out.classes"]
controlConfidences = infResults.loc[0]["out.confidences"]
results = {
'model_name' : control.name(),
'pipeline_name' : pipeline.name(),
'width': width,
'height': height,
'image' : controlImage,
'boxes' : controlBoxes,
'classes' : controlClasses,
'confidences' : controlConfidences,
'confidence-target' : 0.9,
'color':CVDemo.RED, # color to draw bounding boxes and the text in the statistics
'inference-time': (endTime-startTime),
'onnx-time' : 0,
}
cvDemo.drawAndDisplayDetectedObjectsWithClassification(results)
Display the Control Results
Here we will use the Wallaroo CVDemo helper class to draw the control model results on the image.
The full results will be displayed in a dataframe with columns representing the classification, confidence, and bounding boxes of the objects identified.
Once extracted from the results we will want to reshape the flattened array into an array with 4 elements (x,y,width,height).
idx = 0
cocoClasses = cvDemo.getCocoClasses()
for idx in range(0,len(controlClasses)):
df['classification'][idx] = cocoClasses[controlClasses[idx]] # Classes contains the 80 different COCO classificaitons
df['confidence'][idx] = controlConfidences[idx]
df
classification | confidence | x | y | width | height | |
---|---|---|---|---|---|---|
0 | bottle | 98.65% | 0 | 210 | 85 | 479 |
1 | bottle | 90.12% | 72 | 197 | 151 | 468 |
2 | bottle | 60.78% | 211 | 184 | 277 | 420 |
3 | bottle | 59.22% | 143 | 203 | 216 | 448 |
4 | refrigerator | 53.73% | 13 | 41 | 640 | 480 |
5 | bottle | 45.13% | 106 | 206 | 159 | 463 |
6 | bottle | 43.73% | 278 | 1 | 321 | 93 |
7 | bottle | 43.09% | 462 | 104 | 510 | 224 |
8 | bottle | 40.85% | 310 | 1 | 352 | 94 |
9 | bottle | 39.19% | 528 | 268 | 636 | 475 |
10 | bottle | 35.76% | 220 | 0 | 258 | 90 |
11 | bottle | 31.81% | 552 | 96 | 600 | 233 |
12 | bottle | 26.45% | 349 | 0 | 404 | 98 |
13 | bottle | 23.06% | 450 | 264 | 619 | 472 |
14 | bottle | 20.48% | 261 | 193 | 307 | 408 |
15 | bottle | 17.46% | 509 | 101 | 544 | 235 |
16 | bottle | 17.31% | 592 | 100 | 633 | 239 |
17 | bottle | 16.00% | 475 | 297 | 551 | 468 |
18 | bottle | 14.91% | 368 | 163 | 423 | 362 |
19 | book | 13.66% | 120 | 0 | 175 | 81 |
20 | book | 13.32% | 72 | 0 | 143 | 85 |
21 | bottle | 12.22% | 271 | 200 | 305 | 274 |
22 | book | 12.13% | 161 | 0 | 213 | 85 |
23 | bottle | 11.96% | 162 | 0 | 214 | 83 |
24 | bottle | 11.53% | 310 | 190 | 367 | 397 |
25 | bottle | 9.62% | 396 | 166 | 441 | 360 |
26 | cake | 8.65% | 439 | 256 | 640 | 473 |
27 | bottle | 7.84% | 544 | 375 | 636 | 472 |
28 | vase | 7.23% | 272 | 2 | 306 | 96 |
29 | bottle | 6.28% | 453 | 303 | 524 | 463 |
30 | bottle | 5.28% | 609 | 94 | 635 | 211 |
Display the Challenger Results
Here we will use the Wallaroo CVDemo helper class to draw the challenger model results on the input image.
challengerDf = pd.DataFrame(columns=['classification','confidence','x','y','width','height'])
pd.options.mode.chained_assignment = None # default='warn'
pd.options.display.float_format = '{:.2%}'.format
# Points to where all the inference results are
boxList = infResults.loc[0][f"out_{challenger_model_name}.boxes"]
# outputs = results['outputs']
# boxes = outputs[0]
# # reshape this to an array of bounding box coordinates converted to ints
# boxList = boxes['Float']['data']
boxA = np.array(boxList)
challengerBoxes = boxA.reshape(-1, 4)
challengerBoxes = challengerBoxes.astype(int)
challengerDf[['x', 'y','width','height']] = pd.DataFrame(challengerBoxes)
challengerClasses = infResults.loc[0][f"out_{challenger_model_name}.classes"]
challengerConfidences = infResults.loc[0][f"out_{challenger_model_name}.confidences"]
results = {
'model_name' : challenger.name(),
'pipeline_name' : pipeline.name(),
'width': width,
'height': height,
'image' : challengerImage,
'boxes' : challengerBoxes,
'classes' : challengerClasses,
'confidences' : challengerConfidences,
'confidence-target' : 0.9,
'color':CVDemo.RED, # color to draw bounding boxes and the text in the statistics
'inference-time': (endTime-startTime),
'onnx-time' : 0,
}
cvDemo.drawAndDisplayDetectedObjectsWithClassification(results)
Display Challenger Results
The inference results for the objects detected by the challenger model will be displayed including the confidence values. Once extracted from the results we will want to reshape the flattened array into an array with 4 elements (x,y,width,height).
idx = 0
for idx in range(0,len(challengerClasses)):
challengerDf['classification'][idx] = cvDemo.CLASSES[challengerClasses[idx]] # Classes contains the 80 different COCO classificaitons
challengerDf['confidence'][idx] = challengerConfidences[idx]
challengerDf
classification | confidence | x | y | width | height | |
---|---|---|---|---|---|---|
0 | bottle | 99.65% | 2 | 193 | 76 | 475 |
1 | bottle | 98.83% | 610 | 98 | 639 | 232 |
2 | bottle | 97.00% | 544 | 98 | 581 | 230 |
3 | bottle | 96.96% | 454 | 113 | 484 | 210 |
4 | bottle | 96.48% | 502 | 331 | 551 | 476 |
... | ... | ... | ... | ... | ... | ... |
95 | bottle | 5.72% | 556 | 287 | 580 | 322 |
96 | refrigerator | 5.66% | 80 | 161 | 638 | 480 |
97 | bottle | 5.60% | 455 | 334 | 480 | 349 |
98 | bottle | 5.46% | 613 | 267 | 635 | 375 |
99 | bottle | 5.37% | 345 | 2 | 395 | 99 |
100 rows × 6 columns
pipeline.undeploy()
name | shadowimagepipelinetestkpld |
---|---|
created | 2023-10-24 17:05:04.926467+00:00 |
last_updated | 2023-10-24 17:05:22.613731+00:00 |
deployed | False |
tags | |
versions | 433aa539-cb64-4733-85d3-68f6f769dd36, 2f6bee8b-bf31-41cf-b7d2-d4912bfdcca8 |
steps | mobilenetkpld |
published | False |
Conclusion
Notice the difference in the control confidence and the challenger confidence. Clearly we can see in this example the challenger resnet50 model is performing better than the control mobilenet model. This is likely due to the fact that frcnn resnet50 model is a 2 stage object detector vs the frcnn mobilenet is a single stage detector.
This completes using Wallaroo’s shadow deployment feature to compare different computer vision models.
3 - Computer Vision: Yolov8n Demonstration
The following tutorial is available on the Wallaroo Github Repository.
Computer Vision Yolov8n Deployment in Wallaroo
The Yolov8 computer vision model is used for fast recognition of objects in images. This tutorial demonstrates how to deploy a Yolov8n pre-trained model into a Wallaroo Ops server and perform inferences on it.
For this tutorial, the helper module CVDemoUtils
and WallarooUtils
are used to transform a sample image into a pandas DataFrame. This DataFrame is then submitted to the Yolov8n model deployed in Wallaroo.
This demonstration follows these steps:
- Upload the Yolo8 model to Wallaroo
- Add the Yolo8 model as a Wallaroo pipeline step
- Deploy the Wallaroo pipeline and allocate cluster resources to the pipeline
- Perform sample inferences
- Undeploy and return the resources
References
- Wallaroo Workspaces: Workspaces are environments were users upload models, create pipelines and other artifacts. The workspace should be considered the fundamental area where work is done. Workspaces are shared with other users to give them access to the same models, pipelines, etc.
- Wallaroo Model Upload and Registration: ML Models are uploaded to Wallaroo through the SDK or the MLOps API to a workspace. ML models include default runtimes (ONNX, Python Step, and TensorFlow) that are run directly through the Wallaroo engine, and containerized runtimes (Hugging Face, PyTorch, etc) that are run through in a container through the Wallaroo engine.
- Wallaroo Pipelines: Pipelines are used to deploy models for inferencing. Each model is a pipeline step in a pipelines, where the inputs of the previous step are fed into the next. Pipeline steps can be ML models, Python scripts, or Arbitrary Python (these contain necessary models and artifacts for running a model).
Steps
Load Libraries
The first step is loading the required libraries including the Wallaroo Python module.
# Import Wallaroo Python SDK
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework
from CVDemoUtils import CVDemo
from WallarooUtils import Util
cvDemo = CVDemo()
util = Util()
# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
Connect to the Wallaroo Instance through the User Interface
The next step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. For more information on Wallaroo Client settings, see the Client Connection guide.
wl = wallaroo.Client()
Create a New Workspace
We’ll use the SDK below to create our workspace , assign as our current workspace, then display all of the workspaces we have at the moment. We’ll also set up variables for our models and pipelines down the road, so we have one spot to change names to whatever fits your organization’s standards best.
To allow this tutorial to be run by multiple users in the same Wallaroo instance, a random 4 character prefix will be added to the workspace, pipeline, and model. Feel free to set suffix=''
if this is not required.
# import string
# import random
# # make a random 4 character suffix to verify uniqueness in tutorials
# suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
suffix = ''
model_name = 'yolov8n'
model_filename = 'models/yolov8n.onnx'
pipeline_name = 'yolo8demonstration'
workspace_name = f'yolo8-demonstration{suffix}'
def get_workspace(name):
workspace = None
for ws in wl.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = wl.create_workspace(name)
return workspace
Upload the Model
When a model is uploaded to a Wallaroo cluster, it is optimized and packaged to make it ready to run as part of a pipeline. In many times, the Wallaroo Server can natively run a model without any Python overhead. In other cases, such as a Python script, a custom Python environment will be automatically generated. This is comparable to the process of “containerizing” a model by adding a small HTTP server and other wrapping around it.
Our pretrained model is in ONNX format, which is specified in the framework
parameter. For this model, the tensor fields are set to images
to match the input parameters, and the batch configuration is set to single
- only one record will be submitted at a time.
# Upload Retrained Yolo8 Model
yolov8_model = (wl.upload_model(model_name,
model_filename,
framework=Framework.ONNX)
.configure(tensor_fields=['images'],
batch_config="single"
)
)
Pipeline Deployment Configuration
For our pipeline we set the deployment configuration to only use 1 cpu and 1 GiB of RAM.
deployment_config = wallaroo.DeploymentConfigBuilder() \
.replica_count(1) \
.cpus(1) \
.memory("1Gi") \
.build()
Build and Deploy the Pipeline
Now we build our pipeline and set our Yolo8 model as a pipeline step, then deploy the pipeline using the deployment configuration above.
pipeline = wl.build_pipeline(pipeline_name) \
.add_model_step(yolov8_model)
pipeline.deploy(deployment_config=deployment_config)
name | yolo8demonstration |
---|---|
created | 2023-10-19 15:33:26.144685+00:00 |
last_updated | 2023-10-19 15:33:27.154726+00:00 |
deployed | True |
tags | |
versions | 39b02242-de8b-46cd-849e-8a896226a84a, bed60f2a-ddd6-4a48-a9fd-debe6b1e1bca |
steps | yolov8n |
published | False |
Convert Image to DataFrame
The sample image dogbike.png
was converted to a DataFrame using the cvDemo
helper modules. The converted DataFrame is stored as ./data/dogbike.df.json
to save time.
The code sample below demonstrates how to use this module to convert the sample image to a DataFrame.
# convert the image to a tensor
width, height = 640, 640
tensor1, resizedImage1 = cvDemo.loadImageAndResize('dogbike.png', width, height)
tensor1.flatten()
# add the tensor to a DataFrame and save the DataFrame in pandas record format
df = util.convert_data(tensor1,'images')
df.to_json("data.json", orient = 'records')
# convert the image to a tensor
width, height = 640, 640
tensor1, resizedImage1 = cvDemo.loadImageAndResize('./data/dogbike.png', width, height)
tensor1.flatten()
# add the tensor to a DataFrame and save the DataFrame in pandas record format
df = util.convert_data(tensor1,'images')
df.to_json("data.json", orient = 'records')
Inference Request
We submit the DataFrame to the pipeline using wallaroo.pipeline.infer_from_file
, and store the results in the variable inf1
.
inf1 = pipeline.infer_from_file('./data/dogbike.df.json')
Display Bounding Boxes
Using our helper method cvDemo
we’ll identify the objects detected in the photo and their bounding boxes. Only objects with a confidence threshold of 50% or more are shown.
inf1.loc[:, ['out.output0']]
out.output0 | |
---|---|
0 | [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...] |
confidence_thres = 0.50
iou_thres = 0.25
cvDemo.drawYolo8Boxes(inf1, resizedImage1, width, height, confidence_thres, iou_thres, draw=True)
Score: 86.47% | Class: Dog | Bounding Box: [108, 250, 149, 356]
Score: 81.13% | Class: Bicycle | Bounding Box: [97, 149, 375, 323]
Score: 63.16% | Class: Car | Bounding Box: [390, 85, 186, 108]
array([[[ 34, 34, 34],
[ 35, 35, 35],
[ 33, 33, 33],
...,
[ 33, 33, 33],
[ 33, 33, 33],
[ 35, 35, 35]],
[[ 33, 33, 33],
[ 34, 34, 34],
[ 34, 34, 34],
...,
[ 34, 34, 34],
[ 33, 33, 33],
[ 34, 34, 34]],
[[ 53, 54, 48],
[ 54, 55, 49],
[ 54, 55, 49],
...,
[153, 178, 111],
[151, 183, 108],
[159, 176, 99]],
...,
[[159, 167, 178],
[159, 165, 177],
[158, 163, 175],
...,
[126, 127, 121],
[127, 125, 120],
[128, 120, 117]],
[[160, 168, 179],
[156, 162, 174],
[152, 157, 169],
...,
[126, 127, 121],
[129, 127, 122],
[127, 118, 116]],
[[155, 163, 174],
[155, 162, 174],
[152, 158, 170],
...,
[127, 127, 121],
[130, 126, 122],
[128, 119, 116]]], dtype=uint8)
Inference Through Pipeline API
Another method of performing an inference using the pipeline’s deployment url.
Performing an inference through an API requires the following:
- The authentication token to authorize the connection to the pipeline.
- The pipeline’s inference URL.
- Inference data to sent to the pipeline - in JSON, DataFrame records format, or Apache Arrow.
Full details are available through the Wallaroo API Connection Guide on how retrieve an authorization token and perform inferences through the pipeline’s API.
For this demonstration we’ll submit the pandas record, request a pandas record as the return, and set the authorization header. The results will be stored in the file curl_response.df
.
deploy_url = pipeline._deployment._url()
headers = wl.auth.auth_header()
headers['Content-Type']='application/json; format=pandas-records'
headers['Accept']='application/json; format=pandas-records'
!curl -X POST {deploy_url} \
-H "Authorization:{headers['Authorization']}" \
-H "Content-Type:application/json; format=pandas-records" \
-H "Accept:application/json; format=pandas-records" \
--data @./data/dogbike.df.json > curl_response.df
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 38.0M 100 22.9M 100 15.0M 5624k 3701k 0:00:04 0:00:04 --:--:-- 9334k
pipeline.undeploy()
name | yolo8demonstration |
---|---|
created | 2023-10-11 14:37:32.252497+00:00 |
last_updated | 2023-10-11 14:59:03.213137+00:00 |
deployed | False |
tags | |
versions | 2a3933c4-52db-40c6-b80f-9031664fd08a, 95bbabf1-1f15-4e4b-9e67-f7730c2b2cbd, 6c672144-ed4f-4505-97eb-a5b1763af847, 7149e0bc-089b-4d57-9a0b-5d4f4a9a4097, 329e394b-5105-4dc3-b0ff-5411623fc139, 7acaea4e-6ae3-426b-9f97-5e3dcc39c48e, a8b2c009-e7b5-4b96-81b9-40447797a05f, 09952a45-2401-4ebd-8e85-c678365b64a7, d870a558-10ef-448e-b00d-068c10c7e82b, fa531e16-1706-43c4-98d9-e0dd6355fe6f, 4c0b535e-b39b-40f4-82a7-34965b2f7c2a, 3507964d-382f-4e1c-84c7-64c5e27f819c, 9971f8dd-a17b-4d6a-ab72-d786d4990fab, b92a035f-903c-4039-8303-8ceb979a53c2 |
steps | yolov8n |
published | False |
4 - Computer Vision: Image Detection for Health Care
This tutorial can be found on the Wallaroo Tutorials Github Repository.
Image Detection for Health Care Computer Vision Tutorial Part 00: Prerequisites
The following tutorial demonstrates how to use Wallaroo to detect mitochondria from high resolution images. For this example we will be using a high resolution 1536x2048 image that is broken down into “patches” of 256x256 images that can be quickly analyzed.
Mitochondria are known as the “powerhouse” of the cell, and having a healthy amount of mitochondria indicates that a patient has enough energy to live a healthy life, or may have underlying issues that a doctor can check for.
Scanning high resolution images of patient cells can be used to count how many mitochondria a patient has, but the process is laborious. The following ML Model is trained to examine an image of cells, then detect which structures are mitochondria. This is used to speed up the process of testing patients and determining next steps.
Prerequisites
The included TiffImagesUtils class is included in this demonstration as CVDemoUtils.py
, and requires the following dependencies:
- ffmpeg
- libsm
- libxext
Internal JupyterHub Service
To install these dependencies in the Wallaroo JupyterHub service, use the following commands from a terminal shell via the following procedure:
Launch the JupyterHub Service within the Wallaroo install.
Select File->New->Terminal.
Enter the following:
sudo apt-get update
sudo apt-get install ffmpeg libsm6 libxext6 -y
External SDK Users
For users using the Wallaroo SDK to connect with a remote Wallaroo instance, the following commands will install the required dependancies:
For Linux users, this can be installed with:
sudo apt-get update
sudo apt-get install ffmpeg libsm6 libxext6 -y
MacOS users can prepare their environments using a package manager such as Brew with the following:
brew install ffmpeg libsm libxext
### Libraries and Dependencies
1. This repository may use large file sizes for the models. If necessary, install [Git Large File Storage (LFS)](https://git-lfs.com) or use the [Wallaroo Tutorials Releases](https://github.com/WallarooLabs/Wallaroo_Tutorials/releases) to download a .zip file of the most recent computer vision tutorial that includes the models.
1. Import the following Python libraries into your environment:
1. [torch](https://pypi.org/project/torch/)
1. [wallaroo](https://pypi.org/project/wallaroo/)
1. [torchvision](https://pypi.org/project/torchvision/)
1. [opencv-python](https://pypi.org/project/opencv-python/)
1. [onnx](https://pypi.org/project/onnx/)
1. [onnxruntime](https://pypi.org/project/onnxruntime/)
1. [imutils](https://pypi.org/project/imutils/)
1. [pytz](https://pypi.org/project/pytz/)
1. [ipywidgets](https://pypi.org/project/ipywidgets/)
These can be installed by running the command below in the Wallaroo JupyterHub service. Note the use of `pip install torch --no-cache-dir` for low memory environments.
```python
!pip install torchvision==0.15.2
!pip install torch==2.0.1 --no-cache-dir
!pip install opencv-python==4.7.0.72
!pip install onnx==1.12.0
!pip install onnxruntime==1.15.0
!pip install imutils==0.5.4
!pip install pytz
!pip install ipywidgets==8.0.6
!pip install patchify==0.2.3
!pip install tifffile==2023.4.12
!pip install piexif==1.1.3
Requirement already satisfied: torchvision==0.15.2 in /opt/conda/lib/python3.9/site-packages (0.15.2)
Requirement already satisfied: requests in /opt/conda/lib/python3.9/site-packages (from torchvision==0.15.2) (2.25.1)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/lib/python3.9/site-packages (from torchvision==0.15.2) (9.2.0)
Requirement already satisfied: torch==2.0.1 in /opt/conda/lib/python3.9/site-packages (from torchvision==0.15.2) (2.0.1)
Requirement already satisfied: numpy in /opt/conda/lib/python3.9/site-packages (from torchvision==0.15.2) (1.22.3)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.7.99)
Requirement already satisfied: filelock in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (3.12.0)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (3.1.2)
Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (10.9.0.58)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (8.5.0.96)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.10.3.66)
Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.7.4.91)
Requirement already satisfied: networkx in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (3.1)
Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (2.14.3)
Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (10.2.10.91)
Requirement already satisfied: triton==2.0.0 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (2.0.0)
Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.7.91)
Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (4.3.0)
Requirement already satisfied: sympy in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (1.12)
Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.4.0.1)
Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.7.101)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1->torchvision==0.15.2) (11.7.99)
Requirement already satisfied: wheel in /opt/conda/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1->torchvision==0.15.2) (0.37.1)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1->torchvision==0.15.2) (62.3.2)
Requirement already satisfied: cmake in /opt/conda/lib/python3.9/site-packages (from triton==2.0.0->torch==2.0.1->torchvision==0.15.2) (3.26.4)
Requirement already satisfied: lit in /opt/conda/lib/python3.9/site-packages (from triton==2.0.0->torch==2.0.1->torchvision==0.15.2) (16.0.5.post0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.9/site-packages (from requests->torchvision==0.15.2) (1.26.9)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.9/site-packages (from requests->torchvision==0.15.2) (2022.5.18.1)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.9/site-packages (from requests->torchvision==0.15.2) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.9/site-packages (from requests->torchvision==0.15.2) (4.0.0)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.9/site-packages (from jinja2->torch==2.0.1->torchvision==0.15.2) (2.1.1)
Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.9/site-packages (from sympy->torch==2.0.1->torchvision==0.15.2) (1.3.0)
Requirement already satisfied: torch==2.0.1 in /opt/conda/lib/python3.9/site-packages (2.0.1)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (3.1.2)
Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.7.4.91)
Requirement already satisfied: networkx in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (3.1)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.7.99)
Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (10.2.10.91)
Requirement already satisfied: triton==2.0.0 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (2.0.0)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.7.99)
Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (10.9.0.58)
Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (2.14.3)
Requirement already satisfied: filelock in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (3.12.0)
Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.7.101)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.10.3.66)
Requirement already satisfied: sympy in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (1.12)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (8.5.0.96)
Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.4.0.1)
Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (11.7.91)
Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.9/site-packages (from torch==2.0.1) (4.3.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (62.3.2)
Requirement already satisfied: wheel in /opt/conda/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (0.37.1)
Requirement already satisfied: cmake in /opt/conda/lib/python3.9/site-packages (from triton==2.0.0->torch==2.0.1) (3.26.4)
Requirement already satisfied: lit in /opt/conda/lib/python3.9/site-packages (from triton==2.0.0->torch==2.0.1) (16.0.5.post0)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.9/site-packages (from jinja2->torch==2.0.1) (2.1.1)
Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.9/site-packages (from sympy->torch==2.0.1) (1.3.0)
Requirement already satisfied: opencv-python==4.7.0.72 in /opt/conda/lib/python3.9/site-packages (4.7.0.72)
Requirement already satisfied: numpy>=1.19.3 in /opt/conda/lib/python3.9/site-packages (from opencv-python==4.7.0.72) (1.22.3)
Requirement already satisfied: onnx==1.12.0 in /opt/conda/lib/python3.9/site-packages (1.12.0)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /opt/conda/lib/python3.9/site-packages (from onnx==1.12.0) (4.3.0)
Requirement already satisfied: protobuf<=3.20.1,>=3.12.2 in /opt/conda/lib/python3.9/site-packages (from onnx==1.12.0) (3.19.5)
Requirement already satisfied: numpy>=1.16.6 in /opt/conda/lib/python3.9/site-packages (from onnx==1.12.0) (1.22.3)
Requirement already satisfied: onnxruntime==1.15.0 in /opt/conda/lib/python3.9/site-packages (1.15.0)
Requirement already satisfied: packaging in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (21.3)
Requirement already satisfied: flatbuffers in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (1.12)
Requirement already satisfied: protobuf in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (3.19.5)
Requirement already satisfied: numpy>=1.21.6 in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (1.22.3)
Requirement already satisfied: coloredlogs in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (15.0.1)
Requirement already satisfied: sympy in /opt/conda/lib/python3.9/site-packages (from onnxruntime==1.15.0) (1.12)
Requirement already satisfied: humanfriendly>=9.1 in /opt/conda/lib/python3.9/site-packages (from coloredlogs->onnxruntime==1.15.0) (10.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.9/site-packages (from packaging->onnxruntime==1.15.0) (3.0.9)
Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.9/site-packages (from sympy->onnxruntime==1.15.0) (1.3.0)
Requirement already satisfied: imutils==0.5.4 in /opt/conda/lib/python3.9/site-packages (0.5.4)
Requirement already satisfied: pytz in /opt/conda/lib/python3.9/site-packages (2022.1)
Requirement already satisfied: ipywidgets==8.0.6 in /opt/conda/lib/python3.9/site-packages (8.0.6)
Requirement already satisfied: jupyterlab-widgets~=3.0.7 in /opt/conda/lib/python3.9/site-packages (from ipywidgets==8.0.6) (3.0.7)
Requirement already satisfied: ipython>=6.1.0 in /opt/conda/lib/python3.9/site-packages (from ipywidgets==8.0.6) (7.24.1)
Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.9/site-packages (from ipywidgets==8.0.6) (6.13.0)
Requirement already satisfied: traitlets>=4.3.1 in /opt/conda/lib/python3.9/site-packages (from ipywidgets==8.0.6) (5.2.1.post0)
Requirement already satisfied: widgetsnbextension~=4.0.7 in /opt/conda/lib/python3.9/site-packages (from ipywidgets==8.0.6) (4.0.7)
Requirement already satisfied: psutil in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (5.9.1)
Requirement already satisfied: matplotlib-inline>=0.1 in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (0.1.3)
Requirement already satisfied: debugpy>=1.0 in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (1.6.0)
Requirement already satisfied: nest-asyncio in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (1.5.5)
Requirement already satisfied: packaging in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (21.3)
Requirement already satisfied: tornado>=6.1 in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (6.1)
Requirement already satisfied: jupyter-client>=6.1.12 in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets==8.0.6) (7.3.1)
Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (4.8.0)
Requirement already satisfied: backcall in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (0.2.0)
Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (0.18.1)
Requirement already satisfied: pickleshare in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (0.7.5)
Requirement already satisfied: decorator in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (5.1.1)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (3.0.29)
Requirement already satisfied: setuptools>=18.5 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (62.3.2)
Requirement already satisfied: pygments in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets==8.0.6) (2.12.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.9/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets==8.0.6) (0.8.3)
Requirement already satisfied: entrypoints in /opt/conda/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets==8.0.6) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets==8.0.6) (2.8.2)
Requirement already satisfied: jupyter-core>=4.9.2 in /opt/conda/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets==8.0.6) (4.10.0)
Requirement already satisfied: pyzmq>=22.3 in /opt/conda/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets==8.0.6) (23.0.0)
Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.9/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets==8.0.6) (0.7.0)
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=6.1.0->ipywidgets==8.0.6) (0.2.5)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.9/site-packages (from packaging->ipykernel>=4.5.1->ipywidgets==8.0.6) (3.0.9)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.9/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets==8.0.6) (1.16.0)
Requirement already satisfied: patchify==0.2.3 in /opt/conda/lib/python3.9/site-packages (0.2.3)
Requirement already satisfied: numpy<2,>=1 in /opt/conda/lib/python3.9/site-packages (from patchify==0.2.3) (1.22.3)
Requirement already satisfied: tifffile==2023.4.12 in /opt/conda/lib/python3.9/site-packages (2023.4.12)
Requirement already satisfied: numpy in /opt/conda/lib/python3.9/site-packages (from tifffile==2023.4.12) (1.22.3)
Requirement already satisfied: piexif==1.1.3 in /opt/conda/lib/python3.9/site-packages (1.1.3)
Wallaroo SDK
The Wallaroo SDK is provided by default with the Wallaroo instance’s JupyterHub service. To install the Wallaroo SDK manually, it is provided from the Python Package Index and is installed with pip
. Verify that the same version of the Wallaroo SDK is the same version as the Wallaroo instance. For example for Wallaroo release 2023.2, the SDK install command is:
pip install wallaroo==2023.4.1
See the Wallaroo SDK Install Guides for full details.
4.1 - Computer Vision: Image Detection for Health Care
This tutorial can be found on the Wallaroo Tutorials Github Repository.
Image Detection for Health Care Computer Vision Tutorial Part 01: Mitochondria Detection
The following tutorial demonstrates how to use Wallaroo to detect mitochondria from high resolution images. For this example we will be using a high resolution 1536x2048 image that is broken down into “patches” of 256x256 images that can be quickly analyzed.
Mitochondria are known as the “powerhouse” of the cell, and having a healthy amount of mitochondria indicates that a patient has enough energy to live a healthy life, or may have underling issues that a doctor can check for.
Scanning high resolution images of patient cells can be used to count how many mitochondria a patient has, but the process is laborious. The following ML Model is trained to examine an image of cells, then detect which structures are mitochondria. This is used to speed up the process of testing patients and determining next steps.
Tutorial Goals
This tutorial will perform the following:
- Upload and deploy the
mitochondria_epochs_15.onnx
model to a Wallaroo pipeline. - Randomly select from from a selection of 256x256 images that were originally part of a larger 1536x2048 image.
- Convert the images into a numpy array inserted into a pandas DataFrame.
- Submit the DataFrame to the Wallaroo pipeline and use the results to create a mask image of where the model detects mitochondria.
- Compare the original image against a map of “ground truth” and the model’s mask image.
- Undeploy the pipeline and return the resources back to the Wallaroo instance.
Prerequisites
Complete the steps from Mitochondria Detection Computer Vision Tutorial Part 00: Prerequisites.
Mitochondria Computer Vision Detection Steps
Import Libraries
The first step is to import the necessary libraries. Included with this tutorial are the following custom modules:
tiff_utils
: Organizes the tiff images to perform random image selections and other tasks.
Note that tensorflow may return warnings depending on the environment.
import json
import IPython.display as display
import time
import matplotlib.pyplot as plt
from IPython.display import clear_output, display
from lib.TiffImageUtils import TiffUtils
import tifffile as tiff
import pandas as pd
import wallaroo
from wallaroo.object import EntityNotFoundError
import numpy as np
from matplotlib import pyplot as plt
import cv2
from keras.utils import normalize
tiff_utils = TiffUtils()
# used for unique connection names
import string
import random
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')
Open a Connection to Wallaroo
The next step is connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. If logging in externally, update the wallarooPrefix
and wallarooSuffix
variables with the proper DNS information. For more information on Wallaroo DNS settings, see the Wallaroo DNS Integration Guide.
wl = wallaroo.Client()
Create Workspace and Pipeline
We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace.
Workspace, pipeline, and model names should be unique to each Wallaroo instance, so we’ll add in a randomly generated suffix so multiple people can run this tutorial in a Wallaroo instance without affecting each other.
workspace_name = f'biolabsworkspace{suffix}'
pipeline_name = f'biolabspipeline{suffix}'
model_name = f'biolabsmodel{suffix}'
model_file_name = 'models/mitochondria_epochs_15.onnx'
def get_workspace(name):
workspace = None
for ws in wl.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = wl.create_workspace(name)
return workspace
def get_pipeline(name):
try:
pipeline = wl.pipelines_by_name(name)[0]
except EntityNotFoundError:
pipeline = wl.build_pipeline(name)
return pipeline
workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
pipeline = get_pipeline(pipeline_name)
pipeline
name | biolabspipelinebspy |
---|---|
created | 2023-07-14 15:28:32.639523+00:00 |
last_updated | 2023-07-14 15:28:32.639523+00:00 |
deployed | (none) |
tags | |
versions | c70dbdfe-e380-41b0-9da6-97bbfae90554 |
steps |
Upload the Models
Now we will:
- Upload our model.
- Apply it as a step in our pipeline.
- Create a pipeline deployment with enough memory to perform the inferences.
- Deploy the pipeline.
deployment_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(1).memory("2Gi").build()
model = (wl.upload_model(model_name,
model_file_name,
framework=wallaroo.framework.Framework.ONNX)
.configure(tensor_fields=["tensor"])
)
pipeline = wl.build_pipeline(pipeline_name) \
.add_model_step(model) \
.deploy(deployment_config = deployment_config)
Retrieve Image and Convert to Data
The next step is to process the image into a numpy array that the model is trained to detect from.
We start by retrieving all the patch images from a recorded time series tiff recorded on one of our microscopes.
sample_mitochondria_patches_path = "./patches/ms-01-atl-3-22-23_9-50"
patches = tiff_utils.get_all_patches(sample_mitochondria_patches_path)
Randomly we will retrieve a 256x256 patch image and use it to do our semantic segmentation prediction.
We’ll then convert it into a numpy array and insert into a DataFrame for a single inference.
The following helper function loadImageAndConvertTiff
is used to convert the image into a numpy, then insert that into the DataFrame. This allows a later command to take the randomly grabbed image perform the process on other images.
def loadImageAndConvertTiff(imagePath, width, height):
img = cv2.imread(imagePath, 0)
imgNorm = np.expand_dims(normalize(np.array(img), axis=1),2)
imgNorm=imgNorm[:,:,0][:,:,None]
imgNorm=np.expand_dims(imgNorm, 0)
resizedImage = None
#creates a dictionary with the wallaroo "tensor" key and the numpy ndim array representing image as the value.
dictData = {"tensor":[imgNorm]}
dataframedata = pd.DataFrame(dictData)
# display(dataframedata)
return dataframedata, resizedImage
def run_semantic_segmentation_inference(pipeline, input_tiff_image, width, height, threshold):
tensor, resizedImage = loadImageAndConvertTiff(input_tiff_image, width, height)
# print(tensor)
# #
# # run inference on the 256x256 patch image get the predicted mitochandria mask
# #
output = pipeline.infer(tensor)
# print(output)
# # Obtain the flattened predicted mitochandria mask result
list1d = output.loc[0]["out.conv2d_37"]
np1d = np.array(list1d)
# # unflatten it
predicted_mask = np1d.reshape(1,width,height,1)
# # perform the element-wise comaprison operation using the threshold provided
predicted_mask = (predicted_mask[0,:,:,0] > threshold).astype(np.uint8)
# return predicted_mask
return predicted_mask
Infer and Display Results
We will now perform our inferences and display the results. This results in a predicted mask showing us where the mitochondria cells are located.
- The first image is the input image.
- The 2nd image is the ground truth. The mask was created by a human who identified the mitochondria cells in the input image
- The 3rd image is the predicted mask after running inference on the Wallaroo pipeline.
We’ll perform this 10 times to show how quickly the inferences can be submitted.
for x in range(10):
# get a sample 256x256 mitochondria image
random_patch = tiff_utils.get_random_patch_sample(patches)
# build the path to the image
patch_image_path = sample_mitochondria_patches_path + "/images/" + random_patch['patch_image_file']
# run inference in order to get the predicted 256x256 mask
predicted_mask = run_semantic_segmentation_inference(pipeline, patch_image_path, 256,256, 0.2)
# # plot the results
test_image = random_patch['patch_image'][:,:,0]
test_image_title = f"Testing Image - {random_patch['index']}"
ground_truth_image = random_patch['patch_mask'][:,:,0]
ground_truth_image_title = "Ground Truth Mask"
predicted_mask_title = 'Predicted Mask'
tiff_utils.plot_test_results(test_image, test_image_title, \
ground_truth_image, ground_truth_image_title, \
predicted_mask, predicted_mask_title)
Complete Tutorial
With the demonstration complete, the pipeline is undeployed and the resources returned back to the Wallaroo instance.
pipeline.undeploy()
name | biolabspipelinebspy |
---|---|
created | 2023-07-14 15:28:32.639523+00:00 |
last_updated | 2023-07-14 15:28:38.163950+00:00 |
deployed | False |
tags | |
versions | 0460ef47-3de3-43b2-8f62-16e76be8ce93, ef41bc7b-8213-4dd4-a1b9-54ac1253c652, c70dbdfe-e380-41b0-9da6-97bbfae90554 |
steps | biolabsmodelbspy |
4.2 - Computer Vision: Image Detection for Health Care
This tutorial can be found on the Wallaroo Tutorials Github Repository.
Image Detection for Health Care Computer Vision Tutorial Part 02: External Data Connection Stores
The first example in this tutorial series showed selecting from 10 random images of cells and detecting mitochondria from within them.
This tutorial will expand on that by using a Wallaroo connection to retrieve a high resolution 1536x2048 image, break it down into 256x256 “patches” that can be quickly analyzed.
Wallaroo connections are definitions set by MLOps engineers that are used by other Wallaroo users for connection information to a data source. For this example, the data source will be a GitHub URL, but they could be Google BigQuery databases, Kafka topics, or any number of user defined examples.
Tutorial Goals
This tutorial will perform the following:
- Upload and deploy the
mitochondria_epochs_15.onnx
model to a Wallaroo pipeline. - Create a Wallaroo connection pointing to a location for a high resolution image of cells.
- Break down the image into 256x256 images based on the how the model was trained to detect mitochondria.
- Convert the images into a numpy array inserted into a pandas DataFrame.
- Submit the DataFrame to the Wallaroo pipeline and use the results to create a mask image of where the model detects mitochondria.
- Compare the original image against a map of “ground truth” and the model’s mask image.
- Undeploy the pipeline and return the resources back to the Wallaroo instance.
Prerequisites
Prerequisites
Complete the steps from Mitochondria Detection Computer Vision Tutorial Part 00: Prerequisites.
Mitochondria Computer Vision Detection Steps
Import Libraries
The first step is to import the necessary libraries. Included with this tutorial are the following custom modules:
tiff_utils
: Organizes the tiff images to perform random image selections and other tasks.
import json
import IPython.display as display
import time
import matplotlib.pyplot as plt
from IPython.display import clear_output, display
from lib.TiffImageUtils import TiffUtils
import tifffile as tiff
import pandas as pd
import wallaroo
from wallaroo.object import EntityNotFoundError, RequiredAttributeMissing
import numpy as np
from matplotlib import pyplot as plt
import cv2
from keras.utils import normalize
tiff_utils = TiffUtils()
# used for unique connection names
import string
import random
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')
Open a Connection to Wallaroo
The next step is connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. If logging in externally, update the wallarooPrefix
and wallarooSuffix
variables with the proper DNS information. For more information on Wallaroo DNS settings, see the Wallaroo DNS Integration Guide.
wl = wallaroo.Client()
Create Workspace and Pipeline
We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace.
Workspace, pipeline, and model names should be unique to each Wallaroo instance, so we’ll add in a randomly generated suffix so multiple people can run this tutorial in a Wallaroo instance without effecting each other.
workspace_name = f'biolabsconnectionworkspace{suffix}'
pipeline_name = f'biolabsconnectionpipeline{suffix}'
model_name = f'biolabsconnectionmodel{suffix}'
model_file_name = 'models/mitochondria_epochs_15.onnx'
def get_workspace(name):
workspace = None
for ws in wl.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = wl.create_workspace(name)
return workspace
def get_pipeline(name):
try:
pipeline = wl.pipelines_by_name(name)[0]
except EntityNotFoundError:
pipeline = wl.build_pipeline(name)
return pipeline
workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
pipeline = get_pipeline(pipeline_name)
pipeline
name | biolabsconnectionpipelinepdrv |
---|---|
created | 2023-07-14 15:29:16.969523+00:00 |
last_updated | 2023-07-14 15:29:16.969523+00:00 |
deployed | (none) |
tags | |
versions | 79d3b6ca-823f-406d-bf23-88a23b49ff1b |
steps |
Upload the Models
Now we will:
- Upload our model.
- Apply it as a step in our pipeline.
- Create a pipeline deployment with enough memory to perform the inferences.
- Deploy the pipeline.
deployment_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(1).memory("2Gi").build()
model = (wl.upload_model(model_name,
model_file_name,
framework=wallaroo.framework.Framework.ONNX)
.configure(tensor_fields=["tensor"])
)
pipeline = wl.build_pipeline(pipeline_name) \
.add_model_step(model) \
.deploy(deployment_config = deployment_config)
Create the Wallaroo Connection
The Wallaroo connection will point to the location of our high resolution images:
- One the cell photos to be retrieved
- The other “ground truth” masks that are mapped compared against the model’s predictions.
The images will be retrieved, then parsed into a series of 256x256 images.
image_connection_name = f'mitochondria_image_source{suffix}'
image_connection_type = "HTTP"
image_connection_argument = {
'cell_images':'https://storage.googleapis.com/wallaroo-public-data/csa_demo/computer-vision/examples/medical/bio-labs/atl-lab/images/ms-01-atl-3-22-23_9-50.tiff',
'ground_truth_masks': 'https://storage.googleapis.com/wallaroo-public-data/csa_demo/computer-vision/examples/medical/bio-labs/atl-lab/masks/ms-01-atl-3-22-23_9-50-masks.tiff'
}
connection = wl.create_connection(image_connection_name, image_connection_type, image_connection_argument)
Retrieve Images
We’ll use our new connection to reach out, retrieve the images and store them locally for processing. The information is stored in the details()
method for the connection, which is hidden by default when showing the connection, but the data can be retrieved when necessary.
inference_source_connection = wl.get_connection(name=image_connection_name)
display(inference_source_connection)
Field | Value |
---|---|
Name | mitochondria_image_sourcepdrv |
Connection Type | HTTP |
Details | ***** |
Created At | 2023-07-14T15:32:00.930954+00:00 |
Linked Workspaces | [] |
patches_dict = tiff_utils.build_patches("downloaded_patches",
(256,256),
256,
inference_source_connection.details()['cell_images'],
inference_source_connection.details()['ground_truth_masks'] )
created dir downloaded_patches/ms-01-atl-3-22-23_9-50
saving file downloaded_patches/ms-01-atl-3-22-23_9-50/ms-01-atl-3-22-23_9-50.tiff
Retrieve Image and Convert to Data
The next step is to process the image into a numpy array that the model is trained to detect from.
We start by retrieving all the patch images from a recorded time series tiff recorded on one of our microscopes.
sample_mitochondria_patches_path = "./downloaded_patches/ms-01-atl-3-22-23_9-50"
patches = tiff_utils.get_all_patches(sample_mitochondria_patches_path)
Randomly we will retrieve a 256x256 patch image and use it to do our semantic segmentation prediction.
We’ll then convert it into a numpy array and insert into a DataFrame for a single inference.
The following helper function loadImageAndConvertTiff
is used to convert the image into a numpy, then insert that into the DataFrame. This allows a later command to take the randomly grabbed image perform the process on other images.
def loadImageAndConvertTiff(imagePath, width, height):
img = cv2.imread(imagePath, 0)
imgNorm = np.expand_dims(normalize(np.array(img), axis=1),2)
imgNorm=imgNorm[:,:,0][:,:,None]
imgNorm=np.expand_dims(imgNorm, 0)
resizedImage = None
#creates a dictionary with the wallaroo "tensor" key and the numpy ndim array representing image as the value.
dictData = {"tensor":[imgNorm]}
dataframedata = pd.DataFrame(dictData)
# display(dataframedata)
return dataframedata, resizedImage
def run_semantic_segmentation_inference(pipeline, input_tiff_image, width, height, threshold):
tensor, resizedImage = loadImageAndConvertTiff(input_tiff_image, width, height)
# print(tensor)
# #
# # run inference on the 256x256 patch image get the predicted mitochandria mask
# #
output = pipeline.infer(tensor)
# print(output)
# # Obtain the flattened predicted mitochandria mask result
list1d = output.loc[0]["out.conv2d_37"]
np1d = np.array(list1d)
# # unflatten it
predicted_mask = np1d.reshape(1,width,height,1)
# # perform the element-wise comaprison operation using the threshold provided
predicted_mask = (predicted_mask[0,:,:,0] > threshold).astype(np.uint8)
# return predicted_mask
return predicted_mask
Infer and Display Results
We will now perform our inferences and display the results. This results in a predicted mask showing us where the mitochondria cells are located.
- The first image is the input image.
- The 2nd image is the ground truth. The mask was created by a human who identified the mitochondria cells in the input image
- The 3rd image is the predicted mask after running inference on the Wallaroo pipeline.
We’ll perform this 10 times to show how quickly the inferences can be submitted.
for x in range(10):
# get a sample 256x256 mitochondria image
random_patch = tiff_utils.get_random_patch_sample(patches)
# build the path to the image
patch_image_path = sample_mitochondria_patches_path + "/images/" + random_patch['patch_image_file']
# run inference in order to get the predicted 256x256 mask
predicted_mask = run_semantic_segmentation_inference(pipeline, patch_image_path, 256,256, 0.2)
# # plot the results
test_image = random_patch['patch_image'][:,:,0]
test_image_title = f"Testing Image - {random_patch['index']}"
ground_truth_image = random_patch['patch_mask'][:,:,0]
ground_truth_image_title = "Ground Truth Mask"
predicted_mask_title = 'Predicted Mask'
tiff_utils.plot_test_results(test_image, test_image_title, \
ground_truth_image, ground_truth_image_title, \
predicted_mask, predicted_mask_title)
Complete Tutorial
With the demonstration complete, the pipeline is undeployed and the resources returned back to the Wallaroo instance.
pipeline.undeploy()
name | biolabsconnectionpipelinepdrv |
---|---|
created | 2023-07-14 15:29:16.969523+00:00 |
last_updated | 2023-07-14 15:29:23.004729+00:00 |
deployed | False |
tags | |
versions | 0daae212-f578-4723-a0e5-edbcfa548976, 143eb35c-8606-471c-8053-230640249ad6, 79d3b6ca-823f-406d-bf23-88a23b49ff1b |
steps | biolabsconnectionmodelpdrv |
5 - CLIP ViT-B/32 Transformer Demonstration with Wallaroo
This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.
CLIP ViT-B/32 Transformer Demonstration with Wallaroo
The following tutorial demonstrates deploying and performing sample inferences with the Hugging Face CLIP ViT-B/32 Transformer model.
Prerequisites
This tutorial is geared towards the Wallaroo version 2023.2.1 and above. The model clip-vit-base-patch-32.zip
must be downloaded and placed into the ./models
directory. This is available from the following URL:
https://storage.googleapis.com/wallaroo-public-data/hf-clip-vit-b32/clip-vit-base-patch-32.zip
If performing this tutorial from outside the Wallaroo JupyterHub environment, install the Wallaroo SDK.
Steps
Imports
The first step is to import the libraries used for the example.
import json
import os
import requests
import wallaroo
from wallaroo.pipeline import Pipeline
from wallaroo.deployment_config import DeploymentConfigBuilder
from wallaroo.framework import Framework
from wallaroo.object import EntityNotFoundError
import pyarrow as pa
import numpy as np
import pandas as pd
from PIL import Image
Connect to the Wallaroo Instance
The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. For more information on Wallaroo Client settings, see the Client Connection guide.
wl = wallaroo.Client()
Set Workspace and Pipeline
The next step is to create the Wallaroo workspace and pipeline used for the inference requests.
- References
# return the workspace called <name> through the Wallaroo client.
def get_workspace(name, client):
workspace = None
for ws in client.list_workspaces():
if ws.name() == name:
workspace= ws
return workspace
# if no workspaces were found
if workspace==None:
workspace = wl.create_workspace(name)
return workspace
# create the workspace and pipeline
workspace_name = 'clip-demo'
pipeline_name = 'clip-demo'
workspace = get_workspace(workspace_name, wl)
wl.set_current_workspace(workspace)
display(wl.get_current_workspace())
pipeline = wl.build_pipeline(pipeline_name)
pipeline
{'name': 'clip-demo', 'id': 8, 'archived': False, 'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4', 'created_at': '2023-11-20T18:57:53.667873+00:00', 'models': [], 'pipelines': [{'name': 'clip-demo', 'create_time': datetime.datetime(2023, 11, 20, 18, 57, 55, 550690, tzinfo=tzutc()), 'definition': '[]'}]}
name | clip-demo |
---|---|
created | 2023-11-20 18:57:55.550690+00:00 |
last_updated | 2023-11-20 18:57:55.550690+00:00 |
deployed | (none) |
arch | None |
tags | |
versions | ff0e4846-f0f9-4fb0-bc06-d6daa21797bf |
steps | |
published | False |
Configure and Upload Model
The 🤗 Hugging Face model is uploaded to Wallaroo by defining the input and output schema, and specifying the model’s framework as wallaroo.framework.Framework.HUGGING_FACE_ZERO_SHOT_IMAGE_CLASSIFICATION
.
The data schemas are defined in Apache PyArrow Schema format.
The model is converted to the Wallaroo Containerized runtime after the upload is complete.
input_schema = pa.schema([
pa.field('inputs', # required, fixed image dimensions
pa.list_(
pa.list_(
pa.list_(
pa.int64(),
list_size=3
),
list_size=640
),
list_size=480
)),
pa.field('candidate_labels', pa.list_(pa.string(), list_size=4)), # required, equivalent to `options` in the provided demo
])
output_schema = pa.schema([
pa.field('score', pa.list_(pa.float64(), list_size=4)), # has to be same as number of candidate labels
pa.field('label', pa.list_(pa.string(), list_size=4)), # has to be same as number of candidate labels
])
Upload Model
model = wl.upload_model('clip-vit', './models/clip-vit-base-patch-32.zip',
framework=Framework.HUGGING_FACE_ZERO_SHOT_IMAGE_CLASSIFICATION,
input_schema=input_schema,
output_schema=output_schema)
model
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime............................................................successful
Ready
Name | clip-vit |
Version | 57594cc7-7db1-43c3-b21a-6dcaba846f26 |
File Name | clip-vit-base-patch-32.zip |
SHA | 4efc24685a14e1682301cc0085b9db931aeb5f3f8247854bedc6863275ed0646 |
Status | ready |
Image Path | proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/mlflow-deploy:v2023.4.0-4103 |
Architecture | None |
Updated At | 2023-20-Nov 20:51:44 |
Deploy Pipeline
With the model uploaded and prepared, we add the model as a pipeline step and deploy it. For this example, we will allocate 4 Gi of RAM and 1 CPU to the model’s use through the pipeline deployment configuration.
deployment_config = wallaroo.DeploymentConfigBuilder() \
.cpus(.25).memory('1Gi') \
.sidekick_memory(model, '4Gi') \
.sidekick_cpus(model, 1.0) \
.build()
The pipeline is deployed with the specified engine deployment.
Because the model is converted to the Wallaroo Containerized Runtime, the deployment step may timeout with the status
still as Starting
. If this occurs, wait an additional 60 seconds, then run the pipeline.status()
cell. Once the status is Running
, the rest of the tutorial can proceed.
pipeline.clear()
pipeline.add_model_step(model)
pipeline.deploy(deployment_config=deployment_config)
name | clip-demo |
---|---|
created | 2023-11-20 18:57:55.550690+00:00 |
last_updated | 2023-11-20 20:51:59.304917+00:00 |
deployed | True |
arch | None |
tags | |
versions | 7a6168c2-f807-465f-a122-33fe7b5d8a3d, ff0e4846-f0f9-4fb0-bc06-d6daa21797bf |
steps | clip-vit |
published | False |
pipeline.status()
{'status': 'Running',
'details': [],
'engines': [{'ip': '10.244.3.140',
'name': 'engine-bb48bb959-2xctn',
'status': 'Running',
'reason': None,
'details': [],
'pipeline_statuses': {'pipelines': [{'id': 'clip-demo',
'status': 'Running'}]},
'model_statuses': {'models': [{'name': 'clip-vit',
'version': '57594cc7-7db1-43c3-b21a-6dcaba846f26',
'sha': '4efc24685a14e1682301cc0085b9db931aeb5f3f8247854bedc6863275ed0646',
'status': 'Running'}]}}],
'engine_lbs': [{'ip': '10.244.4.152',
'name': 'engine-lb-584f54c899-gcbpj',
'status': 'Running',
'reason': None,
'details': []}],
'sidekicks': [{'ip': '10.244.3.139',
'name': 'engine-sidekick-clip-vit-11-75c5666b69-tqfbk',
'status': 'Running',
'reason': None,
'details': [],
'statuses': '\n'}]}
Run Inference
We verify the pipeline is deployed by checking the status()
.
The sample images in the ./data
directory are converted into numpy arrays, and the candidate labels added as inputs. Both are set as DataFrame arrays where the field inputs
are the image values, and candidate_labels
the labels.
image_paths = [
"./data/bear-in-tree.jpg",
"./data/elephant-and-zebras.jpg",
"./data/horse-and-dogs.jpg",
"./data/kittens.jpg",
"./data/remote-monitor.jpg"
]
images = []
for iu in image_paths:
image = Image.open(iu)
image = image.resize((640, 480)) # fixed image dimensions
images.append(np.array(image))
dataframe = pd.DataFrame({"images": images})
input_data = {
"inputs": images,
"candidate_labels": [["cat", "dog", "horse", "elephant"]] * 5,
}
dataframe = pd.DataFrame(input_data)
dataframe
inputs | candidate_labels | |
---|---|---|
0 | [[[60, 62, 61], [62, 64, 63], [67, 69, 68], [7... | [cat, dog, horse, elephant] |
1 | [[[228, 235, 241], [229, 236, 242], [230, 237,... | [cat, dog, horse, elephant] |
2 | [[[177, 177, 177], [177, 177, 177], [177, 177,... | [cat, dog, horse, elephant] |
3 | [[[140, 25, 56], [144, 25, 67], [146, 24, 73],... | [cat, dog, horse, elephant] |
4 | [[[24, 20, 11], [22, 18, 9], [18, 14, 5], [21,... | [cat, dog, horse, elephant] |
Inference Outputs
The inference is run, and the labels with their corresponding confidence values for each label are mapped to out.label
and out.score
for each image.
results = pipeline.infer(dataframe,timeout=600,dataset=["in", "out", "metadata.elapsed", "time", "check_failures"])
pd.set_option('display.max_colwidth', None)
display(results.loc[:, ['out.label', 'out.score']])
out.label | out.score | |
---|---|---|
0 | [elephant, dog, horse, cat] | [0.4146825075149536, 0.3483847379684448, 0.1285749077796936, 0.10835790634155273] |
1 | [elephant, horse, dog, cat] | [0.9981434345245361, 0.001765866531059146, 6.823801231803373e-05, 2.2441448891186155e-05] |
2 | [horse, dog, elephant, cat] | [0.7596800923347473, 0.217111736536026, 0.020392831414937973, 0.0028152535669505596] |
3 | [cat, dog, elephant, horse] | [0.9870228171348572, 0.0066468678414821625, 0.0032716328278183937, 0.003058753442019224] |
4 | [dog, horse, cat, elephant] | [0.5713965892791748, 0.17229433357715607, 0.15523898601531982, 0.1010700911283493] |
Undeploy Pipelines
With the tutorial complete, the pipeline is undeployed and the resources returned back to the cluster.
pipeline.undeploy()
name | clip-demo |
---|---|
created | 2023-11-20 18:57:55.550690+00:00 |
last_updated | 2023-11-20 20:51:59.304917+00:00 |
deployed | False |
arch | None |
tags | |
versions | 7a6168c2-f807-465f-a122-33fe7b5d8a3d, ff0e4846-f0f9-4fb0-bc06-d6daa21797bf |
steps | clip-vit |
published | False |
6 - Containerized MLFlow Tutorial
Setting Private Registry Configuration in Wallaroo
Configure Via Kots
If Wallaroo was installed via kots
, use the following procedure to add the private model registry information.
Launch the Wallaroo Administrative Dashboard through a terminal linked to the Kubernetes cluster. Replace the namespace with the one used in your installation.
kubectl kots admin-console --namespace wallaroo
Launch the dashboard, by default at
http://localhost:8800
.From the admin dashboard, select Config -> Private Model Container Registry.
Enable Provide private container registry credentials for model images.
Provide the following:
- Registry URL: The URL of the Containerized Model Container Registry. Typically in the format
host:port
. In this example, the registry for GitHub is used. NOTE: When setting the URL for the Containerized Model Container Registry, only the actual service address is needed. For example: with the full URL of the model asghcr.io/wallaroolabs/wallaroo_tutorials/mlflow-statsmodels-example:2022.4
, the URL would beghcr.io/wallaroolabs
. - email: The email address of the user authenticating to the registry service.
- username: The username of the user authentication to the registry service.
- password: The password of the user authentication or token to the registry service.
- Registry URL: The URL of the Containerized Model Container Registry. Typically in the format
Scroll down and select Save config.
Deploy the new version.
Once complete, the Wallaroo instance will be able to authenticate to the Containerized Model Container Registry and retrieve the images.
Configure via Helm
During either the installation process or updates, set the following in the
local-values.yaml
file:privateModelRegistry
:enabled
: truesecretName
:model-registry-secret
registry
: The URL of the private registry.email
: The email address of the user authenticating to the registry service.username
: The username of the user authentication to the registry service.password
: The password of the user authentication to the registry service.
For example:
# Other settings - DNS entries, etc. # The private registry settings privateModelRegistry: enabled: true secretName: model-registry-secret registry: "YOUR REGISTRY URL:YOUR REGISTRY PORT" email: "YOUR EMAIL ADDRESS" username: "YOUR USERNAME" password: "Your Password here"
Install or update the Wallaroo instance via Helm as per the Wallaroo Helm Install instructions.
Once complete, the Wallaroo instance will be able to authenticate to the registry service and retrieve the images.
This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.
MLFlow Inference with Wallaroo Tutorial
Wallaroo users can register their trained MLFlow ML Models from a containerized model container registry into their Wallaroo instance and perform inferences with it through a Wallaroo pipeline.
As of this time, Wallaroo only supports MLFlow 1.3.0 containerized models. For information on how to containerize an MLFlow model, see the MLFlow Documentation.
This tutorial assumes that you have a Wallaroo instance, and have either your own containerized model or use the one from the reference and are running this Notebook from the Wallaroo Jupyter Hub service.
See the Wallaroo Private Containerized Model Container Registry Guide for details on how to configure a Wallaroo instance with a private model registry.
MLFlow Data Formats
When using containerized MLFlow models with Wallaroo, the inputs and outputs must be named. For example, the following output:
[-12.045839810372835]
Would need to be wrapped with the data values named:
[{"prediction": -12.045839810372835}]
A short sample code for wrapping data may be:
output_df = pd.DataFrame(prediction, columns=["prediction"])
return output_df
MLFlow Models and Wallaroo
MLFlow models are composed of two parts: the model, and the flavors. When submitting a MLFlow model to Wallaroo, both aspects must be part of the ML Model included in the container. For full information about MLFlow model structure, see the MLFlow Documentation.
Wallaroo registers the models from container registries. Organizations will either have to make their containers available in a public or through a private Containerized Model Container Registry service. For examples on setting up a private container registry service, see the Docker Documentation “Deploy a registry server”. For more details on setting up a container registry in a cloud environment, see the related documentation for your preferred cloud provider:
- Google Cloud Platform Container Registry
- Amazon Web Services Elastic Container Registry
- Microsoft Azure Container Registry
For this example, we will be using the MLFlow containers that was registered in a GitHub container registry service in MLFlow Creation Tutorial Part 03: Container Registration. The address of those containers are:
- postprocess: ghcr.io/johnhansarickwallaroo/mlflowtests/mlflow-postprocess-example . Used for format data after the statsmodel inferences.
- statsmodel: ghcr.io/johnhansarickwallaroo/mlflowtests/mlflow-statsmodels-example . The statsmodel generated in MLFlow Creation Tutorial Part 01: Model Creation.
Prerequisites
Before uploading and running an inference with a MLFlow model in Wallaroo the following will be required:
- MLFlow Input Schema: The input schema with the fields and data types for each MLFlow model type uploaded to Wallaroo. In the examples below, the data types are imported using the
pyarrow
library. - An installed Wallaroo instance.
- The following Python libraries installed:
os
wallaroo
: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.
IMPORTANT NOTE: Wallaroo supports MLFlow 1.3.0. Please ensure the MLFlow models used in Wallaroo meet this specification.
MLFlow Inference Steps
To register a containerized MLFlow ML Model into Wallaroo, use the following general step:
- Import Libraries
- Connect to Wallaroo
- Set MLFlow Input Schemas
- Register MLFlow Model
- Create Pipeline and Add Model Steps
- Run Inference
Import Libraries
We start by importing the libraries we will need to connect to Wallaroo and use our MLFlow models. This includes the wallaroo
libraries, pyarrow
for data types, and the json
library for handling JSON data.
import wallaroo
from wallaroo.object import EntityNotFoundError
import pyarrow as pa
import pandas as pd
Connect to the Wallaroo Instance
The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. For more information on Wallaroo Client settings, see the Client Connection guide.
# Login through local Wallaroo instance
wl = wallaroo.Client()
def get_workspace(name):
workspace = None
for ws in wl.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = wl.create_workspace(name)
return workspace
def get_pipeline(name):
try:
pipeline = wl.pipelines_by_name(name)[0]
except EntityNotFoundError:
pipeline = wl.build_pipeline(name)
return pipeline
prefix = 'mlflow'
workspace_name= f"{prefix}statsmodelworkspace"
pipeline_name = f"{prefix}statsmodelpipeline"
mlflowworkspace = get_workspace(workspace_name)
wl.set_current_workspace(mlflowworkspace)
pipeline = get_pipeline(pipeline_name)
Set MLFlow Input Schemas
Set the MLFlow input schemas through the pyarrow
library. In the examples below, the input schemas for both the MLFlow model statsmodels-test
and the statsmodels-test-postprocess
model.
sm_input_schema = pa.schema([
pa.field('temp', pa.float32()),
pa.field('holiday', pa.uint8()),
pa.field('workingday', pa.uint8()),
pa.field('windspeed', pa.float32())
])
pp_input_schema = pa.schema([
pa.field('predicted_mean', pa.float32())
])
Register MLFlow Model
Use the register_model_image
method to register the Docker container containing the MLFlow models.
statsmodelUrl = "ghcr.io/wallaroolabs/wallaroo_tutorials/mlflow-statsmodels-example:2023.1"
postprocessUrl = "ghcr.io/wallaroolabs/wallaroo_tutorials/mlflow-postprocess-example:2023.1"
sm_model = wl.register_model_image(
name=f"{prefix}statmodels",
image=f"{statsmodelUrl}"
).configure("mlflow", input_schema=sm_input_schema, output_schema=pp_input_schema)
pp_model = wl.register_model_image(
name=f"{prefix}postprocess",
image=f"{postprocessUrl}"
).configure("mlflow", input_schema=pp_input_schema, output_schema=pp_input_schema)
Create Pipeline and Add Model Steps
With the models registered, we can add the MLFlow models as steps in the pipeline. Once ready, we will deploy the pipeline so it is available for submitting data for running inferences.
pipeline.add_model_step(sm_model)
pipeline.add_model_step(pp_model)
name | mlflowstatsmodelpipeline |
---|---|
created | 2023-07-14 15:32:52.546581+00:00 |
last_updated | 2023-07-14 15:32:52.546581+00:00 |
deployed | (none) |
tags | |
versions | efa60e66-b2b7-4bb2-bc37-dd9ea7377467 |
steps |
from wallaroo.deployment_config import DeploymentConfigBuilder
deployment_config = DeploymentConfigBuilder() \
.cpus(0.25).memory('1Gi') \
.sidekick_env(sm_model, {"GUNICORN_CMD_ARGS": "--timeout=180 --workers=1"}) \
.sidekick_env(pp_model, {"GUNICORN_CMD_ARGS": "--timeout=180 --workers=1"}) \
.build()
pipeline.deploy(deployment_config=deployment_config)
name | mlflowstatsmodelpipeline |
---|---|
created | 2023-07-14 15:32:52.546581+00:00 |
last_updated | 2023-07-14 15:35:09.901078+00:00 |
deployed | True |
tags | |
versions | 2f8e9483-a9ad-436e-9068-7826cd8166c4, 539aae6b-4437-4f8c-8be0-db9ef11e80fb, efa60e66-b2b7-4bb2-bc37-dd9ea7377467 |
steps | mlflowstatmodels |
pipeline.status()
{'status': 'Running',
'details': [],
'engines': [{'ip': '10.244.3.142',
'name': 'engine-c55d468-qgk48',
'status': 'Running',
'reason': None,
'details': [],
'pipeline_statuses': {'pipelines': [{'id': 'mlflowstatsmodelpipeline',
'status': 'Running'}]},
'model_statuses': {'models': [{'name': 'mlflowpostprocess',
'version': 'b7cd3426-825a-467d-8616-011ae8901d23',
'sha': '825ebae48014d297134930028ab0e823bc0d9551334b9a4402c87a714e8156b2',
'status': 'Running'},
{'name': 'mlflowstatmodels',
'version': '1a547bbb-36c7-4b5c-93a8-f357e3542d6f',
'sha': '3afd13d9c5070679e284050cd099e84aa2e5cb7c08a788b21d6cb2397615d018',
'status': 'Running'}]}}],
'engine_lbs': [{'ip': '10.244.4.188',
'name': 'engine-lb-584f54c899-xhdmq',
'status': 'Running',
'reason': None,
'details': []}],
'sidekicks': [{'ip': '10.244.0.72',
'name': 'engine-sidekick-mlflowstatmodels-27-786dc6b47c-bprqb',
'status': 'Running',
'reason': None,
'details': [],
'statuses': '\n'},
{'ip': '10.244.0.73',
'name': 'engine-sidekick-mlflowpostprocess-28-9fff54995-2n2nn',
'status': 'Running',
'reason': None,
'details': [],
'statuses': '\n'}]}
Run Inference
Once the pipeline is running, we can submit our data to the pipeline and return our results. Once finished, we will undeploy the pipeline to return the resources back to the cluster.
# Initial container run may need extra time to finish deploying - adding 90 second timeout to compensate
results = pipeline.infer_from_file('./resources/bike_day_eval_engine.df.json', timeout=90)
display(results.loc[:,["out.predicted_mean"]])
out.predicted_mean | |
---|---|
0 | 0.281983 |
1 | 0.658847 |
2 | 0.572368 |
3 | 0.619873 |
4 | -1.217801 |
5 | -1.849156 |
6 | 0.933885 |
pipeline.undeploy()
name | mlflowstatsmodelpipeline |
---|---|
created | 2023-07-14 15:32:52.546581+00:00 |
last_updated | 2023-07-14 15:32:56.591792+00:00 |
deployed | False |
tags | |
versions | 539aae6b-4437-4f8c-8be0-db9ef11e80fb, efa60e66-b2b7-4bb2-bc37-dd9ea7377467 |
steps | mlflowstatmodels |
7 - Custom Inference Computer Vision Upload, Auto Packaging, and Deploy Tutorial
This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.
Custom Inference Computer Vision Upload, Auto Packaging, and Deploy Tutorial
The following tutorial demonstrates using Wallaroo deploying Bring Your Own Predict aka Arbitrary Python in a Wallaroo instance. The model is auto packaged upon uploading. Once ready, the model is deployed and sample inferences performed on it.
Introduction
This tutorial focuses on the Resnet50 computer vision model. By default, this provides the following outputs from receiving an image converted to tensor values:
boxes
: The bounding boxes for detected objects.classes
: The class of the detected object (bottle, coat, person, etc).confidences
: The confidence the model has that the detected model is the class.
For this demonstration, the model is modified with Wallaroo Bring Your Own Predict (BYOP) to add two additional fields:
avg_px_intensity
: the average pixel intensity checks the input to determine the average value of the Red/Green/Blue pixel values. This is used as a benchmark to determine if the two images are significantly different. For example, an all white photo would have anavg_px_intensity
of1
, while an all blank photo would have a value of0
.avg_confidence
: The average confidence of all detected objects.
Prerequisites
- A Wallaroo Ops instance 2023.4 and above
To download the Wallaroo Computer Vision models, use the following link:
https://storage.googleapis.com/wallaroo-public-data/cv-demo-models/cv-retail-models.zip
Unzip the contents into the directory models
.
The following libraries are required to run the tutorial:
onnx==1.12.0
onnxruntime==1.12.1
torchvision
torch
matplotlib==3.5.0
opencv-python
imutils
pytz
ipywidgets
To run this tutorial outside of a Wallaroo Ops center, the Wallaroo SDK is available and is installed via pip
with:
pip install wallaroo==2023.4.1
Steps
Import Libraries
The following libraries are used to execute this tutorial. The utils.py
provides additional helper methods for rendering the images into tensor fields and other useful tasks.
# preload needed libraries
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework
from IPython.display import display
from IPython.display import Image
import pandas as pd
import json
import datetime
import time
import cv2
import matplotlib.pyplot as plt
import string
import random
import pyarrow as pa
import sys
import asyncio
import numpy as np
import utils
pd.set_option('display.max_colwidth', None)
import datetime
# api based inference request
import requests
import utils
# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')
Connect to the Wallaroo Instance
The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. For more information on Wallaroo Client settings, see the Client Connection guide.
The option request_timeout
provides additional time for the Wallaroo model upload process to complete.
wl = wallaroo.Client(request_timeout=600)
Create Workspace
We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace.
Workspace names must be unique. The following helper function will either create a new workspace, or retrieve an existing one with the same name. Verify that a pre-existing workspace has been shared with the targeted user.
Set the variables workspace_name
to ensure a unique workspace name if required.
The workspace will then be set as the Current Workspace. Model uploads and pipeline creation through the SDK are set in the current workspace.
def get_workspace(name, client):
workspace = None
for ws in client.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = client.create_workspace(name)
return workspace
workspace_name = "cv-retail-edge-observability"
model_name = "resnet-with-intensity"
model_file_name = "./models/model-with-pixel-intensity.zip"
pipeline_name = "retail-inv-tracker-edge-obs"
workspace = get_workspace(workspace_name, wl)
wl.set_current_workspace(workspace)
{'name': 'cv-retail-edge-observability', 'id': 12, 'archived': False, 'created_by': 'a762b468-4c2d-49dd-a9c7-7335e531741c', 'created_at': '2024-02-27T16:51:56.017038+00:00', 'models': [{'name': 'resnet-with-intensity', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2024, 2, 27, 16, 53, 13, 935887, tzinfo=tzutc()), 'created_at': datetime.datetime(2024, 2, 27, 16, 53, 13, 935887, tzinfo=tzutc())}], 'pipelines': [{'name': 'retail-inv-tracker-edge-obs', 'create_time': datetime.datetime(2024, 2, 27, 16, 56, 20, 567978, tzinfo=tzutc()), 'definition': '[]'}]}
Upload Model
The model is uploaded as a BYOP (Bring Your Own Predict) aka Aribitrary model, where the model, Python script and other artifacts are included in a .zip file. This requires the input and output schemas for the model specified in Apache Arrow Schema format.
The contents of the Wallaroo BYOP script includes the following to load the model, and add additional fields. By default the frcnn-resnet
model outputs the following fields:
- boxes: A List of the the bounding boxes of each detected object.
- classes: A List of the class of each detected object.
- confidences: A confidence score of how likely the object is what class was detected.
The model-with-pixel-intensity.zip
BYOP model uses that class, then adds in the following fields.
- avg_px_intensity: The average pixel intensidy of the image. This provides a way of determining if the same pictures from the same camera has significant changes. For example, a security camera would have a different pixel intensity in one picture where the light was on, versus one one when the lights were off.
- avg_confidence: An average of all of the confidences output by the model. This is used for observability and track model drift.
The following predict
method is used in the BYOP model.
def _predict(self, input_data: InferenceData):
# Parse inputs
inputs = input_data["tensor"]
# Pass to onnx model
# `dynamic_axes` hasn't been set in torch.onnx.export()
# that is used in CVDemoUtils.loadPytorchAndConvertToOnnx()
# therefore we cannot do batch inference
ort_sess = ort.InferenceSession(self._model.SerializeToString())
outputs = ort_sess.run(None, {"data": inputs.astype(np.float32)})
boxes, classes, confidences = outputs
# Calculate input derivatives
avg_px_intensity = np.mean(inputs[0])
# Calculate output derivatives
avg_confidence = np.mean(confidences)
# batch size isn't specified in the onnx session output
# but we need to return a batch of outputs
return {
"boxes": np.array([boxes]),
"classes": np.array([classes]),
"confidences": np.array([confidences]),
"avg_px_intensity": np.array([[avg_px_intensity]]),
"avg_confidence": np.array([[avg_confidence]]),
}
For full details on Wallaroo BYOP models, see Wallaroo SDK Essentials Guide: Model Uploads and Registrations: Arbitrary Python.
The following sets the input/output schemas in Apache pyarrow
format, and uploads the models. Once uploaded, it is automatically
input_schema = pa.schema([
pa.field('tensor', pa.list_(
pa.list_(
pa.list_(
pa.float32(), # images are normalized
list_size=640
),
list_size=480
),
list_size=3
)),
])
output_schema = pa.schema([
pa.field('boxes', pa.list_(pa.list_(pa.float32(), list_size=4))),
pa.field('classes', pa.list_(pa.int64())),
pa.field('confidences', pa.list_(pa.float32())),
pa.field('avg_px_intensity', pa.list_(pa.float32())),
pa.field('avg_confidence', pa.list_(pa.float32())),
])
model = wl.upload_model(model_name,
model_file_name,
framework=Framework.CUSTOM,
input_schema=input_schema,
output_schema=output_schema)
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime........................successful
Ready
Deploy Pipeline
Next we configure the hardware we want to use for deployment. If we plan on eventually deploying to edge, this is a good way to simulate edge hardware conditions. The BYOP model is deployed as a Wallaroo Containerized Runtime, so the hardware allocation is performed through the sidekick
options.
deployment_config = wallaroo.DeploymentConfigBuilder() \
.replica_count(1) \
.cpus(1) \
.memory("2Gi") \
.sidekick_cpus(model, 1) \
.sidekick_memory(model, '6Gi') \
.build()
We create the pipeline with the wallaroo.client.build_pipeline
method, and assign our model as a model pipeline step. Once complete, we will deploy the pipeline to allocate resources from the Kuberntes cluster hosting the Wallaroo Ops to the pipeline.
pipeline = wl.build_pipeline(pipeline_name)
pipeline.clear()
pipeline.add_model_step(model)
pipeline.deploy(deployment_config = deployment_config)
Waiting for deployment - this will take up to 600s ................... ok
name | retail-inv-tracker-edge-obs |
---|---|
created | 2024-02-27 16:56:20.567978+00:00 |
last_updated | 2024-02-27 17:55:07.307761+00:00 |
deployed | True |
arch | None |
tags | |
versions | 0e234ea5-ca90-40ea-abfb-a295941c606d, ecf277ed-d20f-481e-8dfe-9dc6937071f9, 21a90743-e121-485e-934f-754f335f4ea3, b4c9256f-719e-4d4c-b167-df32decec0aa |
steps | resnet-with-intensity |
published | False |
Sample Inferences
We’ll perform some sample inferences and display the outputs. For each image, we’ll use the inference result to draw the bounding boxes for all objects with a confidence value higher than 50%. We’ll display the average confidence across all detected objects after the bounding box image.
For more details on performing inferences, see Wallaroo SDK Essentials Guide: Inference Management.
width, height = 640, 480
baseline_images = [
"./data/images/input/example/dairy_bottles.png",
"./data/images/input/example/dairy_products.png",
"./data/images/input/example/product_cheeses.png"
]
for sample_image in baseline_images:
dfImage, resizedImage = utils.loadImageAndConvertToDataframe(sample_image, width, height)
startTime = time.time()
infResults = pipeline.infer(dfImage, timeout=300)
endTime = time.time()
elapsed = 1.0
results = {
'model_name' : model_name,
'pipeline_name' : pipeline_name,
'width': width,
'height': height,
'image' : resizedImage,
'inf-results' : infResults,
'confidence-target' : 0.50,
'inference-time': (endTime-startTime),
'onnx-time' : int(elapsed) / 1e+9,
'classes_file': "./models/coco_classes.pickle",
'color': 'BLUE'
}
image = utils.drawDetectedObjectsFromInference(results)
display(infResults.loc[:, ['out.avg_confidence']])
out.avg_confidence | |
---|---|
0 | [0.3588039] |
out.avg_confidence | |
---|---|
0 | [0.2970887] |
out.avg_confidence | |
---|---|
0 | [0.1656933] |
The Wallaroo SDK is capable of using numpy arrays in a pandas DataFrame for inference requests. Our demonstration will focus on using API calls for inference requests, so we will flatten the numpy array and use that value for our inference inputs. The following examples show using a baseline and blurred image for inference requests and the sample outputs.
pipeline.undeploy()
Waiting for undeployment - this will take up to 600s ...................................... ok
name | retail-inv-tracker-edge-obs |
---|---|
created | 2024-02-27 16:56:20.567978+00:00 |
last_updated | 2024-02-27 17:55:07.307761+00:00 |
deployed | False |
arch | None |
tags | |
versions | 0e234ea5-ca90-40ea-abfb-a295941c606d, ecf277ed-d20f-481e-8dfe-9dc6937071f9, 21a90743-e121-485e-934f-754f335f4ea3, b4c9256f-719e-4d4c-b167-df32decec0aa |
steps | resnet-with-intensity |
published | False |
8 - Demand Curve Quick Start Guide
This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.
Demand Curve Pipeline Tutorial
This worksheet demonstrates a Wallaroo pipeline with data preprocessing, a model, and data postprocessing.
The model is a “demand curve” that predicts the expected number of units of a product that will be sold to a customer as a function of unit price and facts about the customer. Such models can be used for price optimization or sales volume forecasting. This is purely a “toy” demonstration, but is useful for detailing the process of working with models and pipelines.
Data preprocessing is required to create the features used by the model. Simple postprocessing prevents nonsensical estimates (e.g. negative units sold).
Prerequisites
- An installed Wallaroo instance.
- The following Python libraries installed:
os
wallaroo
: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.
import json
import wallaroo
from wallaroo.object import EntityNotFoundError
import pandas
import numpy
import conversion
from wallaroo.object import EntityNotFoundError
import pyarrow as pa
# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')
Connect to the Wallaroo Instance
The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. For more information on Wallaroo Client settings, see the Client Connection guide.
# Login through local Wallaroo instance
wl = wallaroo.Client()
Now that the Wallaroo client has been initialized, we can create the workspace and call it demandcurveworkspace
, then set it as our current workspace. We’ll also create our pipeline so it’s ready when we add our models to it.
We’ll set some variables and methods to create our workspace, pipelines and models. Note that as of the July 2022 release of Wallaroo, workspace names must be unique. Pipelines with the same name will be created as a new version when built.
workspace_name = 'demandcurveworkspace'
pipeline_name = 'demandcurvepipeline'
model_name = 'demandcurvemodel'
model_file_name = './models/demand_curve_v1.onnx'
def get_workspace(name, client):
workspace = None
for ws in client.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = client.create_workspace(name)
return workspace
workspace = get_workspace(workspace_name, wl)
wl.set_current_workspace(workspace)
demandcurve_pipeline = wl.build_pipeline(pipeline_name)
demandcurve_pipeline
name | demandcurvepipeline |
---|---|
created | 2024-03-13 19:22:55.459643+00:00 |
last_updated | 2024-03-13 19:23:42.945824+00:00 |
deployed | (none) |
arch | None |
tags | |
versions | cf5f73c5-9725-49e7-a55c-530737653959, 5be24e45-8175-4b73-a091-d397d7bc5514 |
steps | |
published | False |
With our workspace established, we’ll upload three models:
./models/preprocess_dc_byop.zip
: A preprocess model step that formats the data into a tensor that the model can inference from../models/demand_curve_v1.onnx
: Our demand_curve model. We’ll store the upload configuration intodemand_curve_model
../models/postprocess_dc_byop.zip
: A postprocess model step that will zero out any negative values and return the output variable as “prediction”.
Note that the order we upload our models isn’t important - we’ll be establishing the actual process of moving data from one model to the next when we set up our pipeline.
demand_curve_model = wl.upload_model(model_name,
model_file_name,
framework=wallaroo.framework.Framework.ONNX).configure(tensor_fields=["tensor"])
input_schema = pa.schema([
pa.field('Date', pa.string()),
pa.field('cust_known', pa.bool_()),
pa.field('StockCode', pa.int32()),
pa.field('UnitPrice', pa.float32()),
pa.field('UnitsSold', pa.int32())
])
output_schema = pa.schema([
pa.field('tensor', pa.list_(pa.float64()))
])
preprocess_step = wl.upload_model('curve-preprocess',
'./models/preprocess_dc_byop.zip',
framework=wallaroo.framework.Framework.CUSTOM,
input_schema=input_schema,
output_schema=output_schema)
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime........successful
Ready
input_schema = pa.schema([
pa.field('variable', pa.list_(pa.float64()))
])
output_schema = pa.schema([
pa.field('prediction', pa.list_(pa.float64()))
])
postprocess_step = wl.upload_model('curve-postprocess',
'./models/postprocess_dc_byop.zip',
framework=wallaroo.framework.Framework.CUSTOM,
input_schema=input_schema,
output_schema=output_schema)
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime........successful
Ready
With our models uploaded, we’re going to create our own pipeline and give it three steps:
- The preprocess step to put the data into a tensor format.
- Then we apply the data to our
demand_curve_model
. - And finally, we prepare our data for output with the
module_post
.
# now make a pipeline
demandcurve_pipeline.undeploy()
demandcurve_pipeline.clear()
demandcurve_pipeline.add_model_step(preprocess_step)
demandcurve_pipeline.add_model_step(demand_curve_model)
demandcurve_pipeline.add_model_step(postprocess_step)
name | demandcurvepipeline |
---|---|
created | 2024-03-13 19:22:55.459643+00:00 |
last_updated | 2024-03-13 19:23:42.945824+00:00 |
deployed | (none) |
arch | None |
tags | |
versions | cf5f73c5-9725-49e7-a55c-530737653959, 5be24e45-8175-4b73-a091-d397d7bc5514 |
steps | |
published | False |
And with that - let’s deploy our model pipeline. This usually takes about 45 seconds for the deployment to finish.
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(1).memory("1Gi").build()
demandcurve_pipeline.deploy(deployment_config=deploy_config)
Waiting for deployment - this will take up to 45s ............. ok
name | demandcurvepipeline |
---|---|
created | 2024-03-13 19:22:55.459643+00:00 |
last_updated | 2024-03-13 19:25:24.927731+00:00 |
deployed | True |
arch | None |
tags | |
versions | c2baa959-f50c-468e-875e-b3d14972d400, cf5f73c5-9725-49e7-a55c-530737653959, 5be24e45-8175-4b73-a091-d397d7bc5514 |
steps | curve-preprocess |
published | False |
We can check the status of our pipeline to make sure everything was set up correctly:
demandcurve_pipeline.status()
{'status': 'Running',
'details': [],
'engines': [{'ip': '10.28.0.139',
'name': 'engine-7b76dd59dd-kjvhm',
'status': 'Running',
'reason': None,
'details': [],
'pipeline_statuses': {'pipelines': [{'id': 'demandcurvepipeline',
'status': 'Running'}]},
'model_statuses': {'models': [{'name': 'curve-postprocess',
'version': '9fd8b767-943e-477a-8a7f-f0424eb7a438',
'sha': 'cf4cb335761e2bd5f238bd13f70e777f1fcc1eb31837ebea9cf3eb55c8faeb2f',
'status': 'Running'},
{'name': 'demandcurvemodel',
'version': 'df1251a5-3fa3-4aff-9633-5a5577f40a3f',
'sha': '2820b42c9e778ae259918315f25afc8685ecab9967bad0a3d241e6191b414a0d',
'status': 'Running'},
{'name': 'curve-preprocess',
'version': 'e82604bb-5e6e-45a7-9147-6e7eb209b8ef',
'sha': '22d6886115cbf667cfb7dbd394730625e09d0f8a8ff853848a7edebdb3c26f01',
'status': 'Running'}]}}],
'engine_lbs': [{'ip': '10.28.2.108',
'name': 'engine-lb-d7cc8fc9c-2l2l2',
'status': 'Running',
'reason': None,
'details': []}],
'sidekicks': [{'ip': '10.28.0.138',
'name': 'engine-sidekick-curve-preprocess-6-86cdc949f7-2khcl',
'status': 'Running',
'reason': None,
'details': [],
'statuses': '\n'},
{'ip': '10.28.2.107',
'name': 'engine-sidekick-curve-postprocess-7-55c99ff755-c5w25',
'status': 'Running',
'reason': None,
'details': [],
'statuses': '\n'}]}
Everything is ready. Let’s feed our pipeline some data. We have some information prepared with the daily_purchasses.csv
spreadsheet. We’ll start with just one row to make sure that everything is working correctly.
# read in some purchase data
purchases = pandas.read_csv('daily_purchases.csv')
# start with a one-row data frame for testing
subsamp_raw = purchases.iloc[0:1,: ]
subsamp_raw
Date | cust_known | StockCode | UnitPrice | UnitsSold | |
---|---|---|---|---|---|
0 | 2010-12-01 | False | 21928 | 4.21 | 1 |
result = demandcurve_pipeline.infer(subsamp_raw)
display(result)
time | in.Date | in.StockCode | in.UnitPrice | in.UnitsSold | in.cust_known | out.prediction | anomaly.count | |
---|---|---|---|---|---|---|---|---|
0 | 2024-03-13 19:25:39.152 | 2010-12-01 | 21928 | 4.21 | 1 | False | [6.680255142999893] | 0 |
We can see from the out.prediction
field that the demand curve has a predicted slope of 6.68 from our sample data. We can isolate that by specifying just the data output below.
display(result.loc[0, ['out.prediction']][0])
[6.680255142999893]
Bulk Inference
The initial test went perfectly. Now let’s throw some more data into our pipeline. We’ll draw 10 random rows from our spreadsheet, perform an inference from that, and then display the results and the logs showing the pipeline’s actions.
ix = numpy.random.choice(purchases.shape[0], size=10, replace=False)
converted = conversion.pandas_to_dict(purchases.iloc[ix,: ])
test_df = pd.DataFrame(converted['query'], columns=converted['colnames'])
display(test_df)
output = demandcurve_pipeline.infer(test_df)
display(output)
Date | cust_known | StockCode | UnitPrice | UnitsSold | |
---|---|---|---|---|---|
0 | 2011-02-15 | True | 85099C | 1.95 | 15 |
1 | 2011-04-14 | True | 23201 | 2.08 | 20 |
2 | 2011-04-20 | True | 85099B | 2.08 | 110 |
3 | 2011-08-19 | True | 21931 | 2.08 | 20 |
4 | 2011-09-21 | True | 22386 | 2.08 | 23 |
5 | 2011-09-12 | True | 85099C | 2.08 | 10 |
6 | 2011-11-30 | False | 21929 | 2.08 | 10 |
7 | 2011-01-21 | True | 85099C | 1.95 | 6 |
8 | 2011-11-25 | True | 23202 | 2.08 | 2 |
9 | 2011-09-29 | True | 22386 | 2.08 | 52 |
time | in.Date | in.StockCode | in.UnitPrice | in.UnitsSold | in.cust_known | out.prediction | anomaly.count | |
---|---|---|---|---|---|---|---|---|
0 | 2024-03-13 19:25:39.258 | 2011-02-15 | 85099C | 1.95 | 15 | True | [40.57067616108544] | 0 |
1 | 2024-03-13 19:25:39.258 | 2011-04-14 | 23201 | 2.08 | 20 | True | [33.125327529877765] | 0 |
2 | 2024-03-13 19:25:39.258 | 2011-04-20 | 85099B | 2.08 | 110 | True | [33.125327529877765] | 0 |
3 | 2024-03-13 19:25:39.258 | 2011-08-19 | 21931 | 2.08 | 20 | True | [33.125327529877765] | 0 |
4 | 2024-03-13 19:25:39.258 | 2011-09-21 | 22386 | 2.08 | 23 | True | [33.125327529877765] | 0 |
5 | 2024-03-13 19:25:39.258 | 2011-09-12 | 85099C | 2.08 | 10 | True | [33.125327529877765] | 0 |
6 | 2024-03-13 19:25:39.258 | 2011-11-30 | 21929 | 2.08 | 10 | False | [9.110871233285868] | 0 |
7 | 2024-03-13 19:25:39.258 | 2011-01-21 | 85099C | 1.95 | 6 | True | [40.57067616108544] | 0 |
8 | 2024-03-13 19:25:39.258 | 2011-11-25 | 23202 | 2.08 | 2 | True | [33.125327529877765] | 0 |
9 | 2024-03-13 19:25:39.258 | 2011-09-29 | 22386 | 2.08 | 52 | True | [33.125327529877765] | 0 |
Undeploy the Pipeline
Once we’ve finished with our demand curve demo, we’ll undeploy the pipeline and give the resources back to our Kubernetes cluster.
demandcurve_pipeline.undeploy()
Waiting for undeployment - this will take up to 45s .................................... ok
name | demandcurvepipeline |
---|---|
created | 2024-03-13 19:22:55.459643+00:00 |
last_updated | 2024-03-13 19:25:24.927731+00:00 |
deployed | False |
arch | None |
tags | |
versions | c2baa959-f50c-468e-875e-b3d14972d400, cf5f73c5-9725-49e7-a55c-530737653959, 5be24e45-8175-4b73-a091-d397d7bc5514 |
steps | curve-preprocess |
published | False |
Thank you for being a part of this demonstration. If you have additional questions, please feel free to contact us at Wallaroo.
9 - IMDB Tutorial
This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.
IMDB Sample
The following example demonstrates how to use Wallaroo with chained models. In this example, we will be using information from the IMDB (Internet Movie DataBase) with a sentiment model to detect whether a given review is positive or negative. Imagine using this to automatically scan Tweets regarding your product and finding either customers who need help or have nice things to say about your product.
Note that this example is considered a “toy” model - only the first 100 words in the review were tokenized, and the embedding is very small.
The following example is based on the Large Movie Review Dataset, and sample data can be downloaded from the aclIMDB dataset.
Prerequisites
- An installed Wallaroo instance.
- The following Python libraries installed:
import wallaroo
from wallaroo.object import EntityNotFoundError
# to display dataframe tables
from IPython.display import display
# used to display dataframe information without truncating
import pandas as pd
pd.set_option('display.max_colwidth', None)
import pyarrow as pa
print(wallaroo.__version__)
2023.2.1rc2
Connect to the Wallaroo Instance
The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. For more information on Wallaroo Client settings, see the Client Connection guide.
# Login through local Wallaroo instance
wl = wallaroo.Client()
To test this model, we will perform the following:
- Create a workspace for our models.
- Upload two models:
embedder
: Takes pre-tokenized text documents (model input: 100 integers/datum; output 800 numbers/datum) and creates an embedding from them.sentiment
: The second model classifies the resulting embeddings from 0 to 1, which 0 being an unfavorable review, 1 being a favorable review.
- Create a pipeline that will take incoming data and pass it to the embedder, which will pass the output to the sentiment model, and then export the final result.
- To test it, we will use information that has already been tokenized and submit it to our pipeline and gauge the results.
Just for the sake of this tutorial, we’ll use the SDK below to create our workspace , assign as our current workspace, then display all of the workspaces we have at the moment. We’ll also set up for our models and pipelines down the road, so we have one spot to change names to whatever fits your organization’s standards best.
To allow this tutorial to be run multiple times or by multiple users in the same Wallaroo instance, a random 4 character prefix will be added to the workspace, pipeline, and model.
When we create our new workspace, we’ll save it in the Python variable workspace
so we can refer to it as needed.
First we’ll create a workspace for our environment, and call it imdbworkspace
. We’ll also set up our pipeline so it’s ready for our models.
import string
import random
# make a random 4 character prefix
prefix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
workspace_name = f'{prefix}imdbworkspace'
pipeline_name = f'{prefix}imdbpipeline'
def get_workspace(name):
workspace = None
for ws in wl.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = wl.create_workspace(name)
return workspace
def get_pipeline(name):
try:
pipeline = wl.pipelines_by_name(name)[0]
except EntityNotFoundError:
pipeline = wl.build_pipeline(name)
return pipeline
workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
imdb_pipeline = get_pipeline(pipeline_name)
imdb_pipeline
name | bggqimdbpipeline |
---|---|
created | 2023-07-14 15:29:46.732165+00:00 |
last_updated | 2023-07-14 15:29:46.732165+00:00 |
deployed | (none) |
tags | |
versions | 844d5e89-1cb8-44a3-b9ff-2dbf1e75db27 |
steps |
Just to make sure, let’s list our current workspace. If everything is going right, it will show us we’re in the imdb-workspace
.
wl.get_current_workspace()
{'name': 'bggqimdbworkspace', 'id': 21, 'archived': False, 'created_by': '4e296632-35b3-460e-85fe-565e311bc566', 'created_at': '2023-07-14T15:29:45.662776+00:00', 'models': [], 'pipelines': [{'name': 'bggqimdbpipeline', 'create_time': datetime.datetime(2023, 7, 14, 15, 29, 46, 732165, tzinfo=tzutc()), 'definition': '[]'}]}
Now we’ll upload our two models:
embedder.onnx
: This will be used to embed the tokenized documents for evaluation.sentiment_model.onnx
: This will be used to analyze the review and determine if it is a positive or negative review. The closer to 0, the more likely it is a negative review, while the closer to 1 the more likely it is to be a positive review.
embedder = (wl.upload_model(f'{prefix}embedder-o',
'./embedder.onnx',
framework=wallaroo.framework.Framework.ONNX)
.configure(tensor_fields=["tensor"])
)
smodel = (wl.upload_model(f'{prefix}smodel-o',
'./sentiment_model.onnx',
framework=wallaroo.framework.Framework.ONNX)
.configure(runtime="onnx", tensor_fields=["flatten_1"])
)
With our models uploaded, now we’ll create our pipeline that will contain two steps:
- First, it runs the data through the embedder.
- Second, it applies it to our sentiment model.
# now make a pipeline
imdb_pipeline.add_model_step(embedder)
imdb_pipeline.add_model_step(smodel)
name | bggqimdbpipeline |
---|---|
created | 2023-07-14 15:29:46.732165+00:00 |
last_updated | 2023-07-14 15:29:46.732165+00:00 |
deployed | (none) |
tags | |
versions | 844d5e89-1cb8-44a3-b9ff-2dbf1e75db27 |
steps |
Now that we have our pipeline set up with the steps, we can deploy the pipeline.
imdb_pipeline.deploy()
name | bggqimdbpipeline |
---|---|
created | 2023-07-14 15:29:46.732165+00:00 |
last_updated | 2023-07-14 15:29:52.678281+00:00 |
deployed | True |
tags | |
versions | 8fd7b44a-f5d3-4291-b741-db398c478537, 844d5e89-1cb8-44a3-b9ff-2dbf1e75db27 |
steps | bggqembedder-o |
We’ll check the pipeline status to verify it’s deployed and the models are ready.
imdb_pipeline.status()
{'status': 'Running',
'details': [],
'engines': [{'ip': '10.244.3.140',
'name': 'engine-6cb454f6bb-4fz9b',
'status': 'Running',
'reason': None,
'details': [],
'pipeline_statuses': {'pipelines': [{'id': 'bggqimdbpipeline',
'status': 'Running'}]},
'model_statuses': {'models': [{'name': 'bggqsmodel-o',
'version': 'aaca7ea2-b6d6-471f-9c5f-ea194de3112b',
'sha': '3473ea8700fbf1a1a8bfb112554a0dde8aab36758030dcde94a9357a83fd5650',
'status': 'Running'},
{'name': 'bggqembedder-o',
'version': '2ef7af8b-1ad6-4ae9-b126-f836a05e9e37',
'sha': 'd083fd87fa84451904f71ab8b9adfa88580beb92ca77c046800f79780a20b7e4',
'status': 'Running'}]}}],
'engine_lbs': [{'ip': '10.244.4.186',
'name': 'engine-lb-584f54c899-5zpkl',
'status': 'Running',
'reason': None,
'details': []}],
'sidekicks': []}
To test this out, we’ll start with a single piece of information from our data directory.
singleton = pd.DataFrame.from_records(
[
{
"tensor":[
1607.0,
2635.0,
5749.0,
199.0,
49.0,
351.0,
16.0,
2919.0,
159.0,
5092.0,
2457.0,
8.0,
11.0,
1252.0,
507.0,
42.0,
287.0,
316.0,
15.0,
65.0,
136.0,
2.0,
133.0,
16.0,
4311.0,
131.0,
286.0,
153.0,
5.0,
2826.0,
175.0,
54.0,
548.0,
48.0,
1.0,
17.0,
9.0,
183.0,
1.0,
111.0,
15.0,
1.0,
17.0,
284.0,
982.0,
18.0,
28.0,
211.0,
1.0,
1382.0,
8.0,
146.0,
1.0,
19.0,
12.0,
9.0,
13.0,
21.0,
1898.0,
122.0,
14.0,
70.0,
14.0,
9.0,
97.0,
25.0,
74.0,
1.0,
189.0,
12.0,
9.0,
6.0,
31.0,
3.0,
244.0,
2497.0,
3659.0,
2.0,
665.0,
2497.0,
63.0,
180.0,
1.0,
17.0,
6.0,
287.0,
3.0,
646.0,
44.0,
15.0,
161.0,
50.0,
71.0,
438.0,
351.0,
31.0,
5749.0,
2.0,
0.0,
0.0
]
}
]
)
results = imdb_pipeline.infer(singleton)
display(results)
time | in.tensor | out.dense_1 | check_failures | |
---|---|---|---|---|
0 | 2023-07-14 15:30:09.930 | [1607.0, 2635.0, 5749.0, 199.0, 49.0, 351.0, 16.0, 2919.0, 159.0, 5092.0, 2457.0, 8.0, 11.0, 1252.0, 507.0, 42.0, 287.0, 316.0, 15.0, 65.0, 136.0, 2.0, 133.0, 16.0, 4311.0, 131.0, 286.0, 153.0, 5.0, 2826.0, 175.0, 54.0, 548.0, 48.0, 1.0, 17.0, 9.0, 183.0, 1.0, 111.0, 15.0, 1.0, 17.0, 284.0, 982.0, 18.0, 28.0, 211.0, 1.0, 1382.0, 8.0, 146.0, 1.0, 19.0, 12.0, 9.0, 13.0, 21.0, 1898.0, 122.0, 14.0, 70.0, 14.0, 9.0, 97.0, 25.0, 74.0, 1.0, 189.0, 12.0, 9.0, 6.0, 31.0, 3.0, 244.0, 2497.0, 3659.0, 2.0, 665.0, 2497.0, 63.0, 180.0, 1.0, 17.0, 6.0, 287.0, 3.0, 646.0, 44.0, 15.0, 161.0, 50.0, 71.0, 438.0, 351.0, 31.0, 5749.0, 2.0, 0.0, 0.0] | [0.37142318] | 0 |
Since that works, let’s load up all 50,000 rows and do a full inference on each of them via an Apache Arrow file. Wallaroo pipeline inferences use Apache Arrow as their core data type, making this inference fast.
We’ll do a demonstration with a pandas DataFrame and display the first 5 results.
results = imdb_pipeline.infer_from_file('./data/test_data_50K.arrow')
# using pandas DataFrame
outputs = results.to_pandas()
display(outputs.loc[:5, ["time","out.dense_1"]])
time | out.dense_1 | |
---|---|---|
0 | 2023-07-14 15:30:17.404 | [0.8980188] |
1 | 2023-07-14 15:30:17.404 | [0.056596935] |
2 | 2023-07-14 15:30:17.404 | [0.9260802] |
3 | 2023-07-14 15:30:17.404 | [0.926919] |
4 | 2023-07-14 15:30:17.404 | [0.6618577] |
5 | 2023-07-14 15:30:17.404 | [0.48736304] |
Undeploy
With our pipeline’s work done, we’ll undeploy it and give our Kubernetes environment back its resources.
imdb_pipeline.undeploy()
name | bggqimdbpipeline |
---|---|
created | 2023-07-14 15:29:46.732165+00:00 |
last_updated | 2023-07-14 15:29:52.678281+00:00 |
deployed | False |
tags | |
versions | 8fd7b44a-f5d3-4291-b741-db398c478537, 844d5e89-1cb8-44a3-b9ff-2dbf1e75db27 |
steps | bggqembedder-o |
And there is our example. Please feel free to contact us at Wallaroo for if you have any questions.
10 - whisper-large-v2 Demonstration with Wallaroo
This tutorial can be downloaded as part of the Wallaroo Tutorials repository.
Whisper Demo
The following tutorial demonstrates deploying the openai/whisper-large-v2 on a Wallaroo
pipeline and performing inferences on it using the BYOP feature.
Data Prepartions
For this example, the following Python libraries were used:
These can be installed with the following command:
pip install librosa datasets --user
For these libraries, a sample of audio files was retrieved and converted using the following code.
import librosa
from datasets import load_dataset
# load the sample dataset and retrieve the audio files
dataset = load_dataset("Narsil/asr_dummy")
# the following is used to play them
audio_1, sr_1 = librosa.load(dataset["test"][0]["file"])
audio_2, sr_2 = librosa.load(dataset["test"][1]["file"])
audio_files = [(audio_1, sr_1), (audio_2, sr_2)]
# convert the audio files to numpy values in a DataFrame
input_data = {
"inputs": [audio_1, audio_2],
"return_timestamps": ["word", "word"],
}
dataframe = pd.DataFrame(input_data)
# the following will provide a UI to play the audio file samples
def display_audio(audio: np.array, sr: int) -> None:
IPython.display.display(Audio(data=audio, rate=sr))
for audio, sr in audio_files:
display_audio(audio, sr)
The resulting pandas DataFrame can either be submitted directly to a deployed Wallaroo pipeline using wallaroo.pipeline.infer
, or the DataFrame exported to a pandas Record file in pandas JSON format, and used for an inference request using wallaroo.pipeline.infer_from_file
.
For this example, the audio files are pre-converted to a JSON pandas Record table file, and used for the inference result. This removes the requirements to add additional Python libraries to a virtual environment or Wallaroo JupyterHub service. The code above is provided as an example of converting the dataset audio into values for inference requests.
Tutorial Steps
Import Libraries
The first step is to import the libraries we’ll be using. These are included by default in the Wallaroo instance’s JupyterHub service or are installed with the Wallaroo SDK.
- References
import json
import os
import wallaroo
from wallaroo.pipeline import Pipeline
from wallaroo.deployment_config import DeploymentConfigBuilder
from wallaroo.framework import Framework
import pyarrow as pa
import numpy as np
import pandas as pd
# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')
wallaroo.__version__
'2023.4.0+5d935fefc'
Open a Connection to Wallaroo
The next step is connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. If logging in externally, update the wallarooPrefix
and wallarooSuffix
variables with the proper DNS information. For more information on Wallaroo DNS settings, see the Wallaroo DNS Integration Guide.
For this tutorial, the request_timeout
option is increased to allow the model conversion and pipeline deployment to proceed without any warning messages.
wl = wallaroo.Client(request_timeout=600)
Set Variables and Helper Functions
We’ll set the name of our workspace, pipeline, models and files. Workspace names must be unique across the Wallaroo workspace. For this, we’ll add in a randomly generated 4 characters to the workspace name to prevent collisions with other users’ workspaces. If running this tutorial, we recommend hard coding the workspace name so it will function in the same workspace each time it’s run.
We’ll set up some helper functions that will either use existing workspaces and pipelines, or create them if they do not already exist.
def get_workspace(name, client):
workspace = None
for ws in wl.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = wl.create_workspace(name)
return workspace
The names for our workspace, pipeline, model, and model files are set here to make updating this tutorial easier.
- IMPORTANT NOTE: Workspace names must be unique across the Wallaroo instance. To verify unique names, the randomization code below is provided to allow the workspace name to be unique. If this is not required, set
suffix
to''
.
import string
import random
# make a random 4 character suffix to prevent overwriting other user's workspaces
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
suffix=''
workspace_name = f'whisper-tiny-demo{suffix}'
pipeline_name = 'whisper-hf-byop'
model_name = 'whisper-byop'
model_file_name = './models/model-auto-conversion_hugging-face_complex-pipelines_asr-whisper-tiny.zip'
Create Workspace and Pipeline
We will now create the Wallaroo workspace to store our model and set it as the current workspace. Future commands will default to this workspace for pipeline creation, model uploads, etc. We’ll create our Wallaroo pipeline that is used to deploy our arbitrary Python model.
workspace = get_workspace(workspace_name, wl)
wl.set_current_workspace(workspace)
pipeline = wl.build_pipeline(pipeline_name)
display(wl.get_current_workspace())
{'name': 'whisper-tiny-demojch', 'id': 32, 'archived': False, 'created_by': '9aa81a1f-952f-435e-b77d-504dd0215914', 'created_at': '2023-12-19T15:51:28.027575+00:00', 'models': [], 'pipelines': [{'name': 'whisper-hf-byop', 'create_time': datetime.datetime(2023, 12, 19, 15, 51, 28, 50715, tzinfo=tzutc()), 'definition': '[]'}]}
Configure & Upload Model
For this example, we will use the openai/whisper-tiny
model for the automatic-speech-recognition
pipeline task from the official 🤗 Hugging Face
hub.
To manually create an automatic-speech-recognition
pipeline from the 🤗 Hugging Face
hub link above:
- Download the original model from the the official
🤗 Hugging Face
hub.
from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model="openai/whisper-tiny")
pipe.save_pretrained("asr-whisper-tiny/")
As a last step, you can zip
the folder containing all needed files as follows:
zip -r asr-whisper-tiny.zip asr-whisper-tiny/
Configure PyArrow Schema
You can find more info on the available inputs for the automatic-speech-recognition
pipeline under the official source code from 🤗 Hugging Face
.
The input and output schemas are defined in Apache pyarrow Schema format.
The model is then uploaded with the wallaroo.client.model_upload
method, where we define:
- The name to assign the model.
- The model file path.
- The input and output schemas.
The model is uploaded to the Wallaroo instance, where it is containerized to run with the Wallaroo Inference Engine.
- References
input_schema = pa.schema([
pa.field('inputs', pa.list_(pa.float32())), # required: the audio stored in numpy arrays of shape (num_samples,) and data type `float32`
pa.field('return_timestamps', pa.string()) # optional: return start & end times for each predicted chunk
])
output_schema = pa.schema([
pa.field('text', pa.string()), # required: the output text corresponding to the audio input
pa.field('chunks', pa.list_(pa.struct([('text', pa.string()), ('timestamp', pa.list_(pa.float32()))]))), # required (if `return_timestamps` is set), start & end times for each predicted chunk
])
model = wl.upload_model(model_name,
model_file_name,
framework=Framework.HUGGING_FACE_AUTOMATIC_SPEECH_RECOGNITION,
input_schema=input_schema,
output_schema=output_schema)
model
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime............................................successful
Ready
Name | whisper-byop |
Version | 6c36e84e-297c-4eec-9c38-9d7fb4e4e9db |
File Name | model-auto-conversion_hugging-face_complex-pipelines_asr-whisper-tiny.zip |
SHA | ddd57c9c8d3ed5417783ebb7101421aa1e79429365d20326155c9c02ae1e8a13 |
Status | ready |
Image Path | proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/mlflow-deploy:v2023.4.0-4297 |
Architecture | None |
Updated At | 2023-19-Dec 15:55:27 |
Deploy Pipeline
The model is deployed with the wallaroo.pipeline.deploy(deployment_config)
command. For the deployment configuration, we set the containerized aka sidekick
memory to 8 GB to accommodate the size of the model, and CPUs to at least 4. To optimize performance, a GPU could be assigned to the containerized model.
- References
deployment_config = DeploymentConfigBuilder() \
.cpus(0.25).memory('1Gi') \
.sidekick_memory(model, '8Gi') \
.sidekick_cpus(model, 4.0) \
.build()
pipeline = wl.build_pipeline(pipeline_name)
pipeline.add_model_step(model)
pipeline.deploy(deployment_config=deployment_config)
ok
name | whisper-hf-byop |
---|---|
created | 2023-12-19 15:51:28.050715+00:00 |
last_updated | 2023-12-19 16:03:12.950146+00:00 |
deployed | True |
arch | None |
tags | |
versions | f03d5844-0a71-4791-bac9-b265680d0239, c9492252-e9c1-4e16-af96-4a9c9be2690a, 7579cda5-b36e-4b9b-a30e-1d544f03b9f8, 510b93fa-8389-4055-ae64-3e0e52016f88, 2e035d66-81d4-40f3-8e23-0b711953a176 |
steps | whisper-byop |
published | False |
After a couple of minutes we verify the pipeline deployment was successful.
pipeline.status()
{'status': 'Running',
'details': [],
'engines': [{'ip': '10.244.9.7',
'name': 'engine-65499bc68b-v2vbk',
'status': 'Running',
'reason': None,
'details': [],
'pipeline_statuses': {'pipelines': [{'id': 'whisper-hf-byop',
'status': 'Running'}]},
'model_statuses': {'models': [{'name': 'whisper-byop',
'version': '6c36e84e-297c-4eec-9c38-9d7fb4e4e9db',
'sha': 'ddd57c9c8d3ed5417783ebb7101421aa1e79429365d20326155c9c02ae1e8a13',
'status': 'Running'}]}}],
'engine_lbs': [{'ip': '10.244.2.112',
'name': 'engine-lb-584f54c899-zlmqq',
'status': 'Running',
'reason': None,
'details': []}],
'sidekicks': [{'ip': '10.244.9.6',
'name': 'engine-sidekick-whisper-byop-16-6d5c4cd7f7-mfcpg',
'status': 'Running',
'reason': None,
'details': [],
'statuses': '\n'}]}
Run inference on the example dataset
We perform a sample inference with the provided DataFrame, and display the results.
%%time
result = pipeline.infer_from_file('./data/sound-examples.df.json', timeout=10000)
CPU times: user 138 ms, sys: 12.4 ms, total: 150 ms
Wall time: 6.02 s
display(result)
time | in.inputs | in.return_timestamps | out.chunks | out.text | check_failures | |
---|---|---|---|---|---|---|
0 | 2023-12-19 16:03:18.686 | [0.0003229662, 0.0003370901, 0.0002854846, 0.0... | word | [{'text': ' He', 'timestamp': [0.0, 1.08]}, {'... | He hoped there would be Stu for dinner, turni... | 0 |
1 | 2023-12-19 16:03:18.686 | [0.0010076478, 0.0012469155, 0.0008045971, 0.0... | word | [{'text': ' Stuff', 'timestamp': [29.78, 29.78... | Stuff it into you. His belly calcled him. | 0 |
Evaluate results
Let’s compare the results side by side with the audio inputs.
for transcription in result['out.text'].values:
print(f"Transcription: {transcription}\n")
Transcription: He hoped there would be Stu for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick, peppered, flour-fat and sauce.
Transcription: Stuff it into you. His belly calcled him.
Undeploy Pipelines
With the demonstration complete, we undeploy the pipelines to return the resources back to the Wallaroo instance.
pipeline.undeploy()
ok
name | whisper-hf-byop |
---|---|
created | 2023-12-19 15:51:28.050715+00:00 |
last_updated | 2023-12-19 16:03:12.950146+00:00 |
deployed | False |
arch | None |
tags | |
versions | f03d5844-0a71-4791-bac9-b265680d0239, c9492252-e9c1-4e16-af96-4a9c9be2690a, 7579cda5-b36e-4b9b-a30e-1d544f03b9f8, 510b93fa-8389-4055-ae64-3e0e52016f88, 2e035d66-81d4-40f3-8e23-0b711953a176 |
steps | whisper-byop |
published | False |