This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

Run Anywhere for ARM Architecture Tutorial: Custom Inference Computer Vision with Resnet50

Wallaroo Run Anywhere provides model deployment in any device, any cloud, and any architecture. Models uploaded to Wallaroo are set to their targeted architecture.

Organizations can deploy uploaded models to clusters that have nodes with the provisioned architecture. The following architectures are supported:

• X86: The standard X86 architecture.
• ARM: For more details on cloud providers and their ARM offerings, see Create ARM Nodepools for Kubernetes Clusters.

Model Architecture Inheritance

The model’s deployment configuration inherits its architecture. Models automatically deploy in the target architecture provided nodepools with the architecture are available. For information on setting up nodepools with specific architectures, see Infrastructure Configuration Guides.

That deployment configuration is carried over to the models’ publication in an Open Container Initiative (OCI) Registries, which allows edge model deployments on X64 and ARM architectures. More details on deploying models on edge devices is available with the Wallaroo Run Anywhere Guides.

The deployment configuration can be overridden for either model deployment in the Wallaroo Ops instance, or in the Edge devices.

This tutorial demonstrates deploying a ML model trained to predict house prices to ARM edge locations through the following steps.

• Upload a model with the architecture set to ARM.
• Create a pipeline with the uploaded model as a model step.
• Publish the pipeline model to an Open Container Initiative (OCI) Registry for both X64 and ARM deployments.

In this notebook, we use a Computer Vision Resnet-50 model wrapped in a BYOP container to add additional enhancements. For more information on custom inference models using BYOP (Bring Your Own Predict), see Model Uploads and Registrations: Arbitrary Python.

Goal

Demonstrate publishing a pipeline with model steps to various architectures.

Resources

This tutorial provides the following:

• Models:
  • model-with-pixel-intensity.zip: A custom inference model combined with the Resnet50 model. This is downloaded from the Wallaroo Public Data bucket from the following URL. Unzip the following .zip file into the ./models folder:
    • https://storage.googleapis.com/wallaroo-public-data/cv-demo-models/cv-retail-models.zip

Prerequisites

• A deployed Wallaroo instance with Edge Registry Services and Edge Observability enabled.
• The following Python libraries installed:
  • wallaroo: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.
  • pandas: Pandas, mainly used for Pandas DataFrame
  • json: Used for format input data for inference requests.
• A X64 Docker deployment to deploy the model on an edge location.

Model Upload and Pipeline Steps

The following process will prepare for the pipeline publish.

Import Libraries

The first step will be to import our libraries, and set variables used through this tutorial.

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework
from wallaroo.engine_config import Architecture
import pyarrow as pa

from IPython.display import display

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

import datetime
import time

workspace_name = f'run-anywhere-architecture-cv-byop'
arm_pipeline_name = f'architecture-demonstration-arm'
model_name_arm = f'computer-vision-with-pixel-intensity-arm'
model_file_name = './models/model-with-pixel-intensity.zip'

# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()

Create Workspace

We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace.

Workspace, pipeline, and model names should be unique to each user, so we’ll add in a randomly generated suffix so multiple people can run this tutorial in a Wallaroo instance without effecting each other.

workspace = wl.get_workspace(name=workspace_name, create_if_not_exist=True)

wl.set_current_workspace(workspace)

{'name': 'run-anywhere-architecture-cv-byop', 'id': 55, 'archived': False, 'created_by': 'eed2002f-769f-4cbd-a189-8ca1e9bf496c', 'created_at': '2024-04-22T15:01:56.542427+00:00', 'models': [], 'pipelines': []}

Upload Models and Set ARM Target Architecture

For our example, we will upload the champion model that has been trained to detect objects from a photo. The BYOP version outputs additional fields including the average confidence and pixel intensity.

Models are uploaded to Wallaroo via the wallaroo.client.upload_model method which takes the following arguments:

We upload the model and set the architecture to ARM.

model_name_arm = 'computer-vision-resnet50-arm'
model_file_name = './models/frcnn-resnet.pt.onnx'

arm_model = wl.upload_model(model_name_arm, 
                        model_file_name, 
                        framework=Framework.ONNX,
                        arch=Architecture.ARM)

display(arm_model)

Build Pipeline

We build the pipeline with the wallaroo.client.build_pipeline(pipeline_name command, and set the model as a model step in the pipeline.

pipeline_arm = wl.build_pipeline('architecture-demonstration-arm')
pipeline_arm.clear()
pipeline_arm.add_model_step(arm_model)

For the deployment configuration, we specify the number of cpus, replicas, and memory. We do not specify an architecture, since this is inherited from the model arch setting.

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .replica_count(1) \
    .cpus(1) \
    .memory("2Gi") \
    .build()

pipeline_arm.deploy(deployment_config = deployment_config)

display(pipeline_arm)

pipeline_arm.undeploy()

Waiting for undeployment - this will take up to 45s ........... ok

Pipeline Publish for ARM Architecture via the Wallaroo SDK

We now publish our pipeline as two different versions.

• ARM: The model’s architecture was set to ARM, when when we publish the pipeline to the OCI registry, it will automatically inherit that architecture.
• X64: We override the model’s architecture to push a published version that can be deployed on X64 based devices.

Publish Pipeline for ARM

Publishing the pipeline uses the pipeline wallaroo.pipeline.Pipeline.publish() command. This requires that the Wallaroo Ops instance have Edge Registry Services enabled.

When publishing, we specify the pipeline deployment configuration through the wallaroo.DeploymentConnfigBuilder. For our example, we do not specify the architecture; the architecture and is inherited from the model.

The following publishes the pipeline to the OCI registry and displays the container details. For more information, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .replica_count(1) \
    .cpus(1) \
    .memory("2Gi") \
    .build()

pub_arm = pipeline_arm.publish(deployment_config=deployment_config)
display(pub_arm)

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing...... Published.

docker run \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=sample.registry.example.com/uat/pipelines/architecture-demonstration-arm:b653cfbe-d700-4393-b517-3e615dee7aa5 \
    -e CONFIG_CPUS=1 sample.registry.example.com/uat/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-aarch64:v2024.1.0-main-4963

helm install --atomic $HELM_INSTALL_NAME \
    oci://sample.registry.example.com/uat/charts/architecture-demonstration-arm \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-b653cfbe-d700-4393-b517-3e615dee7aa5 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD

For details on performing inference requests through an edge deployed model, see Edge Deployment Endpoints.