ML models uploaded to Wallaroo can be deployed with or without GPUs by specifying the deployment configuration. The following is a condensed guide for deploying ML models with GPU support through the Wallaroo SDK. For full details on model deployment configurations, see the Model Deploy guide.

Model Deployments Prerequisites for GPU Support

Before deploying a ML model with GPU support, a nodepool with VMs with GPUs must be available. The Platform Administration Guide Create GPU Nodepools for Kubernetes Clusters details how to create nodepools with GPU support for different clouds.

The nodepool with GPU must include the following Kubernetes taints and labels.

The custom label is required for deploying ML models with GPU support; this allows Wallaroo to select the correct nodepool that has the GPU hardware the ML model requires. For more details on Kubernetes taints and labels for ML model deployment in Wallaroo, see the Taints and Tolerations Guide.

Model Deployment for GPU Support

ML model deployments in Wallaroo via the Wallaroo SDK use the following steps:

Create Deployment Configuration

Setting a deployment configuration follows this process:

1. Pipeline deployment configurations are created through the wallaroo.deployment_config.DeploymentConfigBuilder() class.
2. Once the configuration options are set, the deployment configuration is finalized with the wallaroo.deployment_config.DeploymentConfigBuilder().build() method.
3. The deployment configuration is applied when applied during model deployment via the wallaroo.pipeline.Pipeline.deploy method.

Create GPU Deployment Configuration

Deployment configurations with GPU support are created through the wallaroo.deployment_config.DeploymentConfigBuilder() class.

GPUs can only be allocated by entire integer units from the GPU enabled nodepools. gpus(1), gpus(2), etc are valid values, while gpus(0.25) are not.

Organizations should be aware of how many GPUs are allocated to the cluster. If all GPUs are already allocated to other deployments, or if there are not enough GPUs to fulfill the request, the deployment will fail and return an error message.

GPU Support

Wallaroo 2023.2.1 and above supports Kubernetes nodepools with Nvidia Cuda GPUs.

See the Create GPU Nodepools for Kubernetes Clusters guide for instructions on adding GPU enabled nodepools to a Kubernetes cluster.

Deployment configurations default to the following^*.

*: For Kubernetes limits and requests.
**: Resources are always allocated for the Wallaroo Native Runtime engine even if there are no Wallaroo Native Runtimes included in the deployment, so it is recommended to decrease these resources when pipelines use Containerized Runtimes.
***: Resources for Wallaroo Containerized Runtimes only apply with a Wallaroo Containerized Runtime is part of the deployment.

The DeploymentConfigBuilder takes the following methods as arguments to specify what resources are allocated. Note that there are two separate runtimes based on the model:

• Wallaroo Native Runtime: ML models including ONNX and TensorFlow that are deployed natively to Wallaroo. Models deployed in this runtime share all resources allocated to the runtime.
• Wallaroo Containerized Runtime: ML models including Hugging Face, BYOP (Bring Your Own Predict), etc. These models are specified with the number of GPUs, amount of RAM, and other values.

For more details on ML model upload and packaging with Wallaroo, see the Model Upload guide

The following represents the essential parameters for setting the deployment configuration with GPU enabled ML models. For full details on setting deployment configurations for ML mode deployment, see the Deployment Configuration guide.

Once the configuration options are set, the deployment configuration is finalized with the wallaroo.deployment_config.DeploymentConfigBuilder().build() method.

Deploy ML Model with GPUs Example

The following demonstrates deploying a ML model with GPU support through Wallaroo.

This assumes that the model has already been uploaded. We retrieve it via the wallaroo.client.Client.get_model method, which retrieves the model based on its name.

import wallaroo

# create and save the client connection to Wallaroo
wl = wallaroo.Client()

# retrieve the model reference
gpu_model = wl.get_model('sample-gpu-model')

The model is added as a pipeline step to a Wallaroo pipeline. For details, see the Model Deploy guide.

# create the Wallaroo pipeline

pipeline = wl.build_pipeline('sample-gpu-pipeline')

# add the ML model as a pipeline step

pipeline.add_model_step(gpu_model)

Depending on the Wallaroo Runtime the model is deployed in, the following deployments could be used. Each will assign a GPU to the appropriate runtime.

• GPU For Native Runtime Deployment
• GPU For Containerized Runtime Deployment

gpu_deployment_configuration = wallaroo.DeploymentConfigBuilder()
                                .cpus(4) \
                                .memory('3Gi') \
                                .gpus(1) \
                                .deployment_label('doc-gpu-label:true') \
                                .build()

gpu_deployment_configuration = wallaroo.DeploymentConfigBuilder() \
                                .cpus(0.25) \
                                .memory('1Gi') \
                                .gpus(0) \
                                .deployment_label('doc-gpu-label:true') \
                                .sidekick_gpus(gpu_model, 1) \
                                .sidekick_cpus(gpu_model, 4) \
                                .sidekick_memory(gpu_model, '3Gi') \
                                .build()

With the deployment configuration finalized, we deploy the model with the wallaroo.pipeline.Pipeline.deploy(deployment_configuration) method, using the deployment configuration gpu_deployment_configuration. This allocates the specified resources from the deployment configuration for the ML model. Note that if the resources are not available, this request will return an error.

pipeline.deploy(gpu_deployment_configuration)

Once the deployment is complete, the ML model is ready for inference requests. For more details on submitting inference requests on deployed ML Models, see Model Inference.