Wallaroo supports deploying models that run with hardware accelerators that increase the inference speed and performance.

Hardware accelerators are set during model uploads, which set what hardware is allocated during model deployment in Wallaroo Ops and multicloud deployments.

Prerequisites

The following prerequisites must be met before uploading and deploying models with hardware accelerators.

• Hardware Availability: Hardware accelerators must be available in the environment that the model is deployed in.
  • For instructions on adding GPU to Kubernetes clusters, see see Create GPU Nodepools for Kubernetes Clusters.
  • For details on adding ARM nodes to a cluster, see Create ARM Nodepools for Kubernetes Clusters.

Supported Accelerators

The following accelerators are supported:

Set Accelerator via the Wallaroo SDK

Accelerators are set during model upload and packaging. To change the accelerator used with a model, re-upload the model with the new accelerator setting for maximum compatibility and support.

Model Accelerator at Model Upload

Models uploaded to Wallaroo have the accelerator set via the wallaroo.client.upload_model method’s Optional accel: wallaroo.engine_config.Acceleration parameter. For more details on model uploads and other parameters, see Automated Model Packaging.

Deploy Models with Accelerator Settings

Models are added to pipeline as pipeline steps. Models are then deployed through the wallaroo.pipeline.Pipeline.deploy(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig] = None) method.

For full details, see Pipeline Deployment Configuration.

When deploying a model, the deployment configurations inherits the model accelerator setting. No additional changes are needed to set the accelerator when deploying the model. Other settings, such as the number of CPUs, etc can be changed without modifying the accelerator setting.

To change the accelerator settings, models should be re-uploaded as either a new model or a new model version for maximum compatibility with the hardware infrastructure. For more information on uploading models or new model versions, see Model Accelerator at Model Upload.

Model Accelerator for Pipeline Publish

Publishing the pipeline to uses the method wallaroo.pipeline.publish(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig]).

This requires that the Wallaroo Ops has Edge Registry Services enabled.

The deployment configuration for the pipeline publish inherits the model’s accelerator. Options such as the number of cpus, amount of memory, etc can be adjusted without impacting the model’s accelerator settings.

A deployment configuration must be included with the pipeline publish, even if no changes to the cpus, memory, etc are made. For more detail on deployment configurations, see Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration.

Pipelines do not need to be deployed in the Wallaroo Ops instance before publishing the pipeline. This is useful in multicloud deployments to edge devices with different hardware accelerators than the Wallaroo Ops instance.

To change the model acceleration settings, upload the model as a new model or model version with the new acceleration settings.

For more information, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.

nvidia-docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/sample-edge-deploy:446aeed9-2d52-47ae-9e5c-f2a05ef0d4d6\
    -e EDGE_BUNDLE=abc123 \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:2024.1.0-5097

Model Deployment with Accelerators Via the Wallaroo SDK Examples

The following demonstrates the following:

• Upload two Resnet50 models.
  • One set to ARM with the AIO accelerator.
  • One set to ARM with the Jetson accelerator.
• Create two pipelines and add each model as a pipeline step. Deploy each pipeline, and demonstrate the deployment configuration inherits the model’s acceleration settings.
• Publish each pipeline and demonstrate the deployment configuration for the pipeline publish inherits the model’s acceleration settings.

Resnet 50 Model for ARM and AIO

For this example, we will use a Resnet50 Computer Vision model.

We upload the model and set the architecture to ARM and the accelerator to AIO.

model_name_arm = 'computer-vision-resnet50-arm'
model_file_name = './models/frcnn-resnet.pt.onnx'

arm_model = wl.upload_model(model_name_arm, 
                        model_file_name, 
                        framework=Framework.ONNX,
                        arch=wallaroo.engine_config.Architecture.ARM,
                        accel=wallaroo.engine_config.Acceleration.AIO)

display(arm_model)

We then build the pipeline, add the model as our model step, and deploy it with a deployment configuration that allocates 1 CPU and 2Gi of RAM. We then show the deployment configuration inherited the model’s acceleration settings.

pipeline_arm = wl.build_pipeline('acceleration-demonstration-arm')
pipeline_arm.clear()
pipeline_arm.add_model_step(arm_model)

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .replica_count(1) \
    .cpus(1) \
    .memory("2Gi") \
    .build()

pipeline_arm.deploy(deployment_config = deployment_config)
display(pipeline_arm)

We now publish the pipeline. Note that the Engine Config inherited the acceleration from the model.

# default deployment configuration
pub_arm = pipeline_arm.publish(deployment_config=wallaroo.DeploymentConfigBuilder().build())
display(pub_arm)

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing....... Published.

docker run \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=registry.example.com/uat/pipelines/acceleration-demonstration-arm:890b56ee-2a0e-4ed1-ae96-c021ca801a7e \
    -e CONFIG_CPUS=1 registry.example.com/uat/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-aarch64:v2024.1.0-main-4870

helm install --atomic $HELM_INSTALL_NAME \
    oci://registry.example.com/uat/charts/acceleration-demonstration-arm \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-890b56ee-2a0e-4ed1-ae96-c021ca801a7e \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD

HF Summarizer with CUDA

For this example, we will use a Hugging Face summarizer model. When uploaded, this model is converted to the Wallaroo Containerized Runtime.

We upload the model and set the accelerator to CUDA.

model = wl.upload_model('hf-summarization', 
                        './models/hf-summarisation-bart-large-samsun.zip', 
                        framework=Framework.HUGGING_FACE_SUMMARIZATION, 
                        input_schema=input_schema, 
                        output_schema=output_schema,
                        accel=wallaroo.engine_config.Acceleration.CUDA)
display(model)

We then build the pipeline, add the model as our model step, and deploy it with a deployment configuration that allocates 4 cpus, 4 GI RAM and 1 GPU to the Containerized Runtime. We then show the deployment configuration inherited the model’s acceleration settings from the model without being specified

deployment_config = DeploymentConfigBuilder() \
    .cpus(1).memory('1Gi') \
    .sidekick_gpus(model, 1) \
    .sidekick_cpus(model,4) \
    .sidekick_memory(model, '8Gi') \
    .deployment_label('wallaroo.ai/accelerator: a100') \
    .build()
display(deployment_config)

{'engine': {'cpu': 1,
  'resources': {'limits': {'cpu': 1, 'memory': '1Gi'},
   'requests': {'cpu': 1, 'memory': '1Gi'}},
  'node_selector': 'wallaroo.ai/accelerator: a100',
  'arch': 'x86',
  'accel': 'cuda'},
 'enginelb': {},
 'engineAux': {'images': {'hf-summarization-yns-65': {'resources': {'limits': {'nvidia.com/gpu': 1,
      'cpu': 4,
      'memory': '8Gi'},
     'requests': {'nvidia.com/gpu': 1, 'cpu': 4, 'memory': '8Gi'}}}}},
 'node_selector': {}}

We now publish the pipeline. Note that the Engine Config inherited the acceleration from the model.

# default deployment configuration
pub_arm = pipeline.publish(deployment_config)
display(pub_arm)

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing....... Published.

docker run \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/hf-summarization-pipeline:6d453276-a4cf-4b01-90d7-78e9da1dd72a \
    -e CONFIG_CPUS=1 ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-cuda:v2024.1.0-main-4921

helm install --atomic $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/hf-summarization-pipeline \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-6d453276-a4cf-4b01-90d7-78e9da1dd72a \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD

Model Accelerator Deployment Troubleshooting

If the specified hardware accelerator or infrastructure is not available in the Wallaroo Ops cluster during deployment, the following error message is displayed:

Tutorials

The following examples are available to demonstrate uploading and publishing models with hardware accelerator support.

• Run AI Workloads with Hardware Accelerators: Aloha Tutorial