Wallaroo supports deploying models that run with hardware accelerators that increase inference speed and performance.

Deploying models with AI hardware accelerators through through Wallaroo uses the following process depending on whether the model is deployed in Wallaroo, or deployed in an edge or multi-cloud environment.

Prerequisites

The following prerequisites must be met before uploading and deploying models with hardware accelerators.

• Hardware Availability: Hardware accelerators must be available in the environment that the model is deployed in.
  • For instructions on adding GPU to Kubernetes clusters, see see Create GPU Nodepools for Kubernetes Clusters.
  • For details on adding ARM nodes to a cluster, see Create ARM Nodepools for Kubernetes Clusters.

Supported Accelerators

The following accelerators are supported:

Upload Model

Accelerators are set during model upload and packaging. To change the accelerator used with a model, re-upload the model with the new accelerator setting for maximum compatibility and support.

Models uploaded to Wallaroo have the accelerator set via the wallaroo.client.upload_model method’s Optional accel: wallaroo.engine_config.Acceleration parameter. For more details on model uploads and other parameters, see Automated Model Packaging.

The following is a generic template for uploading models with the Wallaroo SDK.

model = wl.upload_model(name={Model Name}
                        path={Model File Path}
                        framework=Framework.{Wallaroo Framework},
                        arch=wallaroo.engine_config.Architecture.{X86 | ARM}, # defaults to X86
                        accel=wallaroo.engine_config.Acceleration.{Accelerator}) # defaults to None

Deploy and Infer in the Cloud

Models deployed in the cloud in the Wallaroo Ops center take the following steps:

• Create a pipeline and add the model as a pipeline step.
• Deploy the model. This allocates resources for the models use.

When deploying a model, the deployment configurations inherits the model’s accelerator and architecture settings. Other settings, such as the number of CPUs, amount of RAM, etc can be changed without modifying the accelerator setting are modified with the deployment configuration; for full details, see Pipeline Deployment Configuration. If no deployment configuration is provided, then the default resource allocations are used.

To change the accelerator settings, models should be re-uploaded as either a new model or a new model version for maximum compatibility with the hardware infrastructure. For more information on uploading models or new model versions, see Model Accelerator at Model Upload.

The following is a generic template for deploying models in Wallaroo with a deployment configuration.

# create the pipeline

pipeline = wl.build_pipeline(pipeline_name={Pipeline Name})

# add the model as a pipeline step

pipeline.add_model_step(model)

# create a deployment configuration

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .cpus({Number of CPUs}) \
    .memory("{Amount of RAM}") \
    .build()

# deploy the model with the deployment configuration
pipeline.deploy(deployment_config = deployment_config)

Deploy in Edge and Multi-cloud Environments

Deploying a model in an Edge and Multi-cloud environment takes two steps after uploading the model:

• Publish the Model: The model with its AI Accelerator settings is published to an OCI (Open Container Initiative) Registry.
• Deploy the Model on Edge: The model is deployed in an environment with hardware matching the AI Accelerator and architecture via docker or helm.

Publish the Model

Publishing the pipeline to uses the method [wallaroo.pipeline.publish(deployment_config)].

This requires that the Wallaroo Ops has Edge Registry Services enabled.

The deployment configuration for the pipeline publish inherits the model’s accelerator. Options such as the number of cpus, amount of memory, etc can be adjusted without impacting the model’s accelerator settings.

A deployment configuration must be included with the pipeline publish, even if no changes to the cpus, memory, etc are made. For more detail on deployment configurations, see Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration.

Pipelines do not need to be deployed in the Wallaroo Ops Center before publishing the pipeline. This is useful in multicloud deployments to edge devices with different hardware accelerators than the Wallaroo Ops instance.

To change the model acceleration settings, upload the model as a new model or model version with the new acceleration settings.

For more information, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.

The Wallaroo SDK publish includes the docker run and the helm run fields. These provide “copy and paste” scripts for deploying the model in edge and multi-cloud environments.

The following template shows publishing the model to the OCI (Open Container Initiative) Registry associated with the Wallaroo Ops Center, and the abbreviated output.

# default deployment configuration
publish = pipeline.publish(deployment_config=deployment_config)
display(publish)

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing....... Published.

docker run \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL={PIPELINE URL} \
    -e CONFIG_CPUS=1 {ENGINE URL}

helm install --atomic $HELM_INSTALL_NAME \
    {HELM CHART REGISTRY URL} \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version {HELM VERSION} \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD

Model Deployment on Edge

Models published via the Wallaroo SDK return docker run and helm install template for deploying the model on edge and multi-cloud environments. The docker run commands require the following environmental variables set before executing:

• $PERSISTENT_VOLUME_DIR (Optional: Used with Edge and Multi-cloud Observability): The directory path for the model deployment’s persistent volume. This stores session and other data for connecting back to the Wallaroo instance for inference logs and other uses.
• $EDGE_PORT: The network port used to submit inference requests to the deployed model.
• $OCI_USERNAME: The user name or identifier to authenticate to the OCI (Open Container Initiative) Registry where the model was published.
• $OCI_PASSWORD: The password or token to authenticate to the OCI (Open Container Initiative) Registry where the model was published.

The following docker run template is returned by Wallaroo during pipeline publish for deploying the model in an edge and multi-cloud environment.

docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL={PIPELINE_URL}:{PIPELINE_VERSION}\
    {Wallaroo_Engine_URL}:{WALLAROO_ENGINE_VERSION}

The following helm install template is returned by Wallaroo during pipeline publish for deploying the model in an edge and multi-cloud environment.

helm install --atomic $HELM_INSTALL_NAME \
    {HELM CHART REGISTRY URL} \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version {HELM VERSION} \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD

--rm -it --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* ) --ulimit nofile=262144:262144 --cap-add=sys_nice

docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    --rm -it --device /dev/dri \
    --group-add=$(stat -c "%g" /dev/dri/render* ) \
    --ulimit nofile=262144:262144 --cap-add=sys_nice \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL={PIPELINE_URL}:{PIPELINE_VERSION} \
    {Wallaroo_Engine_URL}:{WALLAROO_ENGINE_VERSION}

sudo usermod -aG docker $USER
newgrp docker

docker run --runtime nvidia --privileged --gpus all \
    -v $PERSISTENT_VOLUME_DIR:/persist \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/sample-edge-deploy:446aeed9-2d52-47ae-9e5c-f2a05ef0d4d6\
    -e EDGE_BUNDLE=abc123 \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-jetson:v2024.4.0-5849

Model Accelerator Deployment Troubleshooting

If the specified hardware accelerator or infrastructure is not available in the Wallaroo Ops cluster during deployment, the following error message is displayed:

Tutorials

The following examples are available to demonstrate uploading and publishing models with hardware accelerator support.

• Deploy on AIO
• Deploy on CUDA
• Deploy on Jetson
• Deploy on OpenVINO
• Deploy on OpenVINO with Intel GPUs