ML models can be deployed in centralized Wallaroo OPs instances and Edge devices on a variety of infrastructures and processors. The CPU infrastructure is set during the model upload and packaging stage.

Models specified with the ARM architecture during the upload and automated model packaging can be deployed on Wallaroo Ops instances or multicloud deployments.

ARM Support from Cloud Providers

ARM processors for Kubernetes clusters in Cloud environments are supported by the following providers.

• Amazon Web Services (AWS)
• Google Cloud Platform (GCP)
• Microsoft Azure
• Oracle Cloud Infrastructure (OCI)

Model Packaging and Deployments Prerequisites for ARM

To upload and package a model for Wallaroo Ops or multicloud edge deployments, the following prerequisites must be met.

• Wallaroo Ops
  • At least one node with ARM support deployed in the cluster.
    • For details on adding ARM nodes to a cluster, see Create ARM Nodepools for Kubernetes Clusters
• Edge Devices
  • Enable Edge Registry Services in the Wallaroo instance to publish the pipeline to an OCI registry for edge deployments.
  • ARM processor support for the edge device.

AI Workloads for ARM via the Wallaroo SDK

The Wallaroo SDK provides ARM support for models uploaded for Wallaroo Ops or multicloud edge deployments.

Upload Models for ARM via the Wallaroo SDK

Models are uploaded to Wallaroo via the wallaroo.client.upload_model method. The infrastructure is set with the optional arch parameter, which accepts the wallaroo.engine_config.Architecture object.

wallaroo.client.upload_model has the following parameters. For more details on model uploads, see Automated Model Packaging.

Deploy Models for ARM via the Wallaroo SDK

Models are added to pipeline as pipeline steps. Models are then deployed through the wallaroo.pipeline.Pipeline.deploy(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig] = None) method.

For full details, see Pipeline Deployment Configuration.

When deploying a model in a Wallaroo Ops instance, the deployment configurations inherits the model architecture setting. No additional changes are needed to set the architecture when deploying the model. Other settings, such as the number of CPUs, etc can be changed without modifying the architecture setting.

To change the architecture settings for model deployment, models should be re-uploaded as either a new model or a new model version for maximum compatibility with the hardware infrastructure. For more information on uploading models or new model versions, see Upload Models for ARM via the Wallaroo SDK.

Publish Pipeline for ARM via the Wallaroo SDK

Publishing the pipeline to uses the method wallaroo.pipeline.publish(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig]).

This requires that the Wallaroo Ops instance have Edge Registry Services enabled.

A deployment configuration must be included with the pipeline publish, even if no changes to the cpus, memory, etc are made. For more detail on deployment configurations, see Pipeline Deployment Configuration.

The deployment configuration for the pipeline publish inherits the model’s architecture. Options such as the number of cpus, amount of memory, etc can be adjusted without impacting the model’s architecture settings.

Pipelines do not need to be deployed in the Wallaroo Ops instance before publishing the pipeline.

For more information, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.

Model Deployment on ARM Via the Wallaroo SDK Examples

The following examples demonstrates:

• Uploading a model for packaging in the Wallaroo Ops instance with the arch set to ARM.
• Creating a pipeline and adding the model as a pipeline step.
• Deploying the pipeline and demonstrating the deployment configuration inherits the model’s architecture setting.
• Publishing the pipeline to an OCI registry and demonstrating the deployment configuration inherits the model’s architecture setting.

Note that the arch and accel deployment configuration settings are not specified, as the deployment configuration inherits the model’s architecture settings.

Model Deployment on ARM Examples: Hugging Face Summarization Model

First we demonstrate uploading a Hugging Face Summarization model for ARM processor deployment. Note the arch setting is set to wallaroo.engine_config.Architecture.ARM.

input_schema = pa.schema([
    pa.field('inputs', pa.string()),
    pa.field('return_text', pa.bool_()),
    pa.field('return_tensors', pa.bool_()),
    pa.field('clean_up_tokenization_spaces', pa.bool_()),
    # pa.field('generate_kwargs', pa.map_(pa.string(), pa.null())), # dictionaries are not currently supported by the engine
])

output_schema = pa.schema([
    pa.field('summary_text', pa.string()),
])


model_name_arm = f'hf-summarizer-arm'
model_file_name = './models/hf_summarization.zip'

model_arm = wl.upload_model(model_name_arm, 
                        model_file_name, 
                        framework=wallaroo.framework.Framework.HUGGING_FACE_SUMMARIZATION, 
                        input_schema=input_schema, 
                        output_schema=output_schema,
                        arch=wallaroo.engine_config.Architecture.ARM
                        )

Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime......................successful

Ready

We now create a pipeline and add the model as a pipeline step, then deploy the pipeline. The model was packaged as a Wallaroo Containerized Runtime, therefore the deployment configuration for the pipeline will specify the model deployment in the sidekick deployment. For more details on pipeline deployment configurations, see Pipeline Deployment Configuration.

Note that the arch and accel deployment configuration settings are not specified in the deployment configuration, as the deployment configuration inherits the model’s architecture settings.

# create the pipeline and add the model as a pipeline step
pipeline_arm = wl.build_pipeline('architecture-demonstration-arm')
pipeline_arm.add_model_step(model_arm)

# create the deployment configuration and specify 4 cpus with 8Gi RAM.  We do not have to specify the architecture;
# that is inherited from the model's `Architecture` setting.

from wallaroo.deployment_config import DeploymentConfigBuilder

deployment_config = DeploymentConfigBuilder() \
    .cpus(0.25).memory('1Gi') \
    .sidekick_cpus(model_arm, 4) \
    .sidekick_memory(model_arm, "8Gi") \
    .build()

pipeline_arm.deploy(deployment_config=deployment_config)

#display the pipeline details
display(pipeline_arm)

We now publish the pipeline to an OCI registry. It is not required that we deploy the pipeline before publishing it.

For our example, we will use the default deployment configuration. We note again that the Engine Config specifies the arm architecture, which was inherited from the model’s arch setting.

# default deployment configuration
pub_arm = pipeline_arm.publish(deployment_config=wallaroo.DeploymentConfigBuilder().build())
pub_arm

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing................ Published.

docker run \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=registry.example.com/uat/pipelines/architecture-demonstration-arm:fd5e3d64-9eea-492d-92b2-8bdb5b20ec83 \
    -e CONFIG_CPUS=1 registry.example.com/uat/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-aarch64:v2024.1.0-main-4870

helm install --atomic $HELM_INSTALL_NAME \
    oci://registry.example.com/uat/charts/architecture-demonstration-arm \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-fd5e3d64-9eea-492d-92b2-8bdb5b20ec83 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD

Model Deployment on ARM Examples: Resnet50 Computer Vision Model

For this example, we will use a Resnet50 Computer Vision model.

We upload the model and set the architecture to ARM.

model_name_arm = 'computer-vision-resnet50-arm'
model_file_name = './models/frcnn-resnet.pt.onnx'

arm_model = wl.upload_model(model_name_arm, 
                        model_file_name, 
                        framework=Framework.ONNX,
                        arch=Architecture.ARM)

display(arm_model)

We then build the pipeline, add the model as our model step, and deploy it with a deployment configuration that allocates 1 CPU and 2Gi of RAM. We then show the deployment configuration inherited the model’s architecture setting.

pipeline_arm = wl.build_pipeline('architecture-demonstration-arm')
pipeline_arm.clear()
pipeline_arm.add_model_step(arm_model)

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .replica_count(1) \
    .cpus(1) \
    .memory("2Gi") \
    .build()

pipeline_arm.deploy(deployment_config = deployment_config)
display(pipeline_arm)

We now publish the pipeline. Note that the Engine Config inherited the architecture from the model.

# use the default deployment configuration
pub_arm = pipeline_arm.publish(deployment_config=wallaroo.DeploymentConfigBuilder().build())
display(pub_arm)

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing....... Published.

docker run \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=registry.example.com/uat/pipelines/architecture-demonstration-arm:890b56ee-2a0e-4ed1-ae96-c021ca801a7e \
    -e CONFIG_CPUS=1 registry.example.com/uat/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-aarch64:v2024.1.0-main-4870

helm install --atomic $HELM_INSTALL_NAME \
    oci://registry.example.com/uat/charts/architecture-demonstration-arm \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-890b56ee-2a0e-4ed1-ae96-c021ca801a7e \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD

Tutorials

The following examples are available to demonstrate uploading and publishing models with ARM processor support.

• AI Workloads on ARM: Computer Vision Resnet50 Model on ARM Tutorial
• AI Workloads on ARM: House Price Prediction Model on ARM Tutorial