Run Anywhere: Deploy and Publish Computer Vision Model Resnet50 with Post Processing for IBM Power10
This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.
Run Anywhere: Deploy and Publish Computer Vision Model Resnet50 with Post Processing for IBM Power10
The following tutorial demonstrates how to use a trained computer vision Resnet50 model and deploy in Wallaroo for inferencing on IBM Power10. The Resnet50 model is contained in the Wallaroo Arbitrary Python aka BYOP Framework. This provides additional output fields and features to the standard Resnet50 model.
The same model is then published and made available for deployment on edge and multi-cloud devices on the IBM Power10 architecture.
For access to these sample models and for a demonstration:
- Contact your Wallaroo Support Representative OR
- Schedule Your Wallaroo.AI Demo Today
Deploying on IBM Power10 is made easy with Wallaroo. Models are defined with an architecture at the upload stage, allowing the deployment of the same model on different architectures.
Tutorial Overview
This tutorial demonstrates using Wallaroo to:
- Upload a model and define the deployment architecture as
Power10. - Deploy the model on a node with the Power10 chips and perform a sample inference.
- Publish the model to an OCI registry for deployment on edge and multi-cloud environments with Power10 chips.
Requirements
The following tutorial requires the following:
- Wallaroo version 2024.4 and above.
- At least one Power10 node deployed in the cluster.
This process will use the following steps:
- Upload the model to Wallaroo with the architecture set to
Power10. - Deploy the model on Power10 nodes and perform sample computer vision detection inferences.
- Publish the model to an OCI (Open Container Initiative) Registry for deployment on edge and multi-cloud environments.
Steps
Import Libraries
The first step will be to import our libraries.
# preload needed libraries
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework
from wallaroo.engine_config import Architecture
from IPython.display import display
from IPython.display import Image
import pandas as pd
import json
import datetime
import time
#import cv2
import matplotlib.pyplot as plt
import string
import random
import pyarrow as pa
import sys
import asyncio
pd.set_option('display.max_colwidth', None)
import sys
import utils
Connect to the Wallaroo Instance
The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.
# Login through local service
wl = wallaroo.Client()
Set Variables
The following variables and methods are used later to create or connect to an existing workspace, pipeline, and model. This example has both the resnet model, and a post process script.
workspace_name = f'resnetworkspace-power'
pipeline_name = f'resnetnetpipeline-p10'
model_name = f'resnet50p10'
model_file_name = 'models/frcnn-resnet.pt.onnx'
Create Workspace
The workspace will be created or connected to, and set as the default workspace for this session. Once that is done, then all models and pipelines will be set in that workspace.
workspace = wl.get_workspace(name=workspace_name, create_if_not_exist=True)
wl.set_current_workspace(workspace)
wl.get_current_workspace()
{'name': 'resnetworkspace-power', 'id': 26, 'archived': False, 'created_by': 'jeff.will@wallaroo.ai', 'created_at': '2024-12-24T19:08:14.581402+00:00', 'models': [{'name': 'resnet50', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2024, 12, 24, 19, 9, 11, 778678, tzinfo=tzutc()), 'created_at': datetime.datetime(2024, 12, 24, 19, 9, 11, 778678, tzinfo=tzutc())}, {'name': 'cv-post-process-drift-detection', 'versions': 4, 'owner_id': '""', 'last_update_time': datetime.datetime(2024, 12, 24, 21, 16, 58, 850242, tzinfo=tzutc()), 'created_at': datetime.datetime(2024, 12, 24, 19, 9, 25, 884423, tzinfo=tzutc())}], 'pipelines': [{'name': 'resnetnetpipeline-p10', 'create_time': datetime.datetime(2025, 1, 2, 20, 15, 48, 786752, tzinfo=tzutc()), 'definition': '[]'}, {'name': 'resnetnetpipeline-jcw', 'create_time': datetime.datetime(2024, 12, 24, 19, 8, 31, 201189, tzinfo=tzutc()), 'definition': '[]'}]}
Create Pipeline and Upload Model
We will now create or connect to an existing pipeline as named in the variables above.
pipeline = wl.build_pipeline(pipeline_name)
resnet_model = wl.upload_model(model_name,
model_file_name,
framework=Framework.ONNX,
arch=Architecture.Power10).configure(batch_config="single",
tensor_fields=["tensor"])
Upload Post Processing Module
The following module takes the results from the CV model output, and averages the confidence values from all detected objects. The avg_confidence is a new field created through the Wallaroo Custom Model from the confidences list field. This is used for observability and model drift tracking.
input_schema = pa.schema([
pa.field('boxes', pa.list_(pa.float32())),
pa.field('classes', pa.list_(pa.int64())),
pa.field('confidences', pa.list_(pa.float32()))
]
)
output_schema = pa.schema([
pa.field('boxes', pa.list_(pa.float32())),
pa.field('classes', pa.list_(pa.int64())),
pa.field('confidences', pa.list_(pa.float32())),
pa.field('avg_confidence', pa.float32()),
])
module_post_process_model = wl.upload_model("cv-post-process-drift-detectionp10",
"./models/post-process-drift-detection.zip",
framework=Framework.PYTHON,
arch=Architecture.Power10,
input_schema=input_schema,
output_schema=output_schema
)
Deploy Pipeline
With the model uploaded, we can add it is as a step in the pipeline, then deploy it. Once deployed, resources from the Wallaroo instance will be reserved and the pipeline will be ready to use the model to perform inference requests.
module_post_process_model=wl.get_model("cv-post-process-drift-detectionp10")
module_post_process_model
| Name | cv-post-process-drift-detectionp10 |
| Version | 32764894-ce86-4b32-9586-efbbc46a2aaf |
| File Name | post-process-drift-detection.zip |
| SHA | eefc55277b091dd90c45704ff51bbd68dbc0f0f7e686930c5409a606659cefcc |
| Status | ready |
| Image Path | proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/mac-deploy:v2025.1.0-6142 |
| Architecture | power10 |
| Acceleration | none |
| Updated At | 2025-02-Jan 20:59:04 |
| Workspace id | 26 |
| Workspace name | resnetworkspace-power |
pipeline.undeploy()
pipeline.clear()
pipeline.add_model_step(resnet_model)
pipeline.add_model_step(module_post_process_model)
deploy_config = (wallaroo.DeploymentConfigBuilder()
.replica_count(1)
.cpus(1)
.memory("1Gi")
.build())
pipeline.deploy(deployment_config=deploy_config)
pipeline.status()
{'status': 'Running',
'details': [],
'engines': [{'ip': '10.244.1.220',
'name': 'engine-6b8cd974b7-c67f7',
'status': 'Running',
'reason': None,
'details': [],
'pipeline_statuses': {'pipelines': [{'id': 'resnetnetpipeline-p10',
'status': 'Running',
'version': 'b2dc98c2-4666-435e-83c5-46ce40115838'}]},
'model_statuses': {'models': [{'model_version_id': 70,
'name': 'cv-post-process-drift-detectionp10',
'sha': 'eefc55277b091dd90c45704ff51bbd68dbc0f0f7e686930c5409a606659cefcc',
'status': 'Running',
'version': '32764894-ce86-4b32-9586-efbbc46a2aaf'},
{'model_version_id': 69,
'name': 'resnet50p10',
'sha': '43326e50af639105c81372346fb9ddf453fea0fe46648b2053c375360d9c1647',
'status': 'Running',
'version': '204d45b4-bb78-48ac-a72d-5855437c84c3'}]}}],
'engine_lbs': [{'ip': '10.244.1.221',
'name': 'engine-lb-7455f865d5-6t2rh',
'status': 'Running',
'reason': None,
'details': []}],
'sidekicks': [{'ip': '10.244.1.219',
'name': 'engine-sidekick-cv-post-process-drift-detectionp10-70-5555zswnc',
'status': 'Running',
'reason': None,
'details': [],
'statuses': '\n'}]}
Test the pipeline by running inference on a sample image
Prepare input image
Next we will load a sample image and resize it to the width and height required for the object detector.
We will convert the image to a numpy ndim array and add it do a dictionary
import cv2
image = cv2.imread('./data/images/input/example/dairy_bottles.png')
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
plt.figure(figsize=(12,8))
plt.grid(False)
plt.imshow(image)
plt.show()
width, height = 640, 480
dfImage, resizedImage = utils.loadImageAndConvertToDataframe('./data/images/input/example/dairy_bottles.png', width, height)
Run Inference
With that done, we can have the model detect the objects on the image by running an inference through the pipeline, and storing the results for the next step.
IMPORTANT NOTE: If necessary, add timeout=60 to the infer method if more time is needed to upload the data file for the inference request.
startTime = time.time()
infResults = pipeline.infer(dfImage, timeout=300)
endTime = time.time()
infResults.loc[:, ['time', 'out.avg_confidence']]
| time | out.avg_confidence | |
|---|---|---|
| 0 | 2025-01-02 21:04:38.114 | 0.358804 |
Draw the Inference Results
With our inference results, we can take them and use the Wallaroo CVDemo class and draw them onto the original image. The bounding boxes and the confidence value will only be drawn on images where the model returned a 50% confidence rate in the object’s identity.
elapsed = 1.0
results = {
'model_name' : model_name,
'pipeline_name' : pipeline_name,
'width': width,
'height': height,
'image' : resizedImage,
'inf-results' : infResults,
'confidence-target' : 0.50,
'inference-time': (endTime-startTime),
'onnx-time' : int(elapsed) / 1e+9,
'classes_file': "./models/coco_classes.pickle",
'color': 'BLUE'
}
image = utils.drawDetectedObjectsFromInference(results)
Undeploy the Pipeline
With the inference complete, we can undeploy the pipeline and return the resources back to the Wallaroo instance.
pipeline.undeploy()
Publish the Model
Publishing the model takes the model, pipeline, and Wallaroo engine and puts them into an OCI compliant registry for later deployment on Edge and multi-cloud environments.
Publishing the pipeline uses the pipeline wallaroo.pipeline.Pipeline.publish() command. This requires that the Wallaroo Ops instance have Edge Registry Services enabled.
When publishing, we specify the pipeline deployment configuration through the wallaroo.DeploymentConfigBuilder. For our example, we do not specify the architecture; the architecture and is inherited from the model.
The following publishes the pipeline to the OCI registry and displays the container details. For more information, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.
pipeline.publish(deployment_config=deploy_config)
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing.......... Published.
| ID | 3 | |
| Pipeline Name | resnetnetpipeline-p10 | |
| Pipeline Version | 9e6a6ac9-793c-4edd-8adf-ef0ce53618f6 | |
| Status | Published | |
| Workspace Id | 26 | |
| Workspace Name | resnetworkspace-power | |
| Edges | ||
| Engine URL | sample.registry.example.com/uat/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-ppc64le:v2025.1.0-6142 | |
| Pipeline URL | sample.registry.example.com/uat/pipelines/resnetnetpipeline-p10:9e6a6ac9-793c-4edd-8adf-ef0ce53618f6 | |
| Helm Chart URL | oci://sample.registry.example.com/uat/charts/resnetnetpipeline-p10 | |
| Helm Chart Reference | sample.registry.example.com/uat/charts@sha256:b33f5b97767dab6cdca8469c55d3584bb11f0cb14bd38a68440ebc7b02a3f866 | |
| Helm Chart Version | 0.0.1-9e6a6ac9-793c-4edd-8adf-ef0ce53618f6 | |
| Engine Config | {'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '1Gi'}, 'requests': {'cpu': 1.0, 'memory': '1Gi'}, 'accel': 'none', 'arch': 'power10', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': {}}} | |
| User Images | [] | |
| Created By | jeff.will@wallaroo.ai | |
| Created At | 2025-01-02 21:05:28.310283+00:00 | |
| Updated At | 2025-01-02 21:05:28.310283+00:00 | |
| Replaces | ||
| Docker Run Command |
Note: Please set the EDGE_PORT, OCI_USERNAME, and OCI_PASSWORD environment variables. | |
| Podman Run Command |
Note: Please set the EDGE_PORT, OCI_USERNAME, and OCI_PASSWORD environment variables. | |
| Helm Install Command |
Note: Please set the HELM_INSTALL_NAME, HELM_INSTALL_NAMESPACE,
OCI_USERNAME, and OCI_PASSWORD environment variables. |
Edge Deployment
Included with the publish details include instructions on deploying the model via Docker Run and Helm Install commands with the defined deployment configuration on the Power10 architecture.
Contact Us
For access to these sample models and for a demonstration:
- Contact your Wallaroo Support Representative OR
- Schedule Your Wallaroo.AI Demo Today