Kubernetes Nodepool Guides

The following are guides to assist organizations in creating the Kubernetes environments used to host a Wallaroo Ops installation.

ARM Architecture CPU Nodepools

The following guide demonstrates how to create nodepools with ARM architecture.

Wallaroo supports for ARM architecture CPUs. For example, Azure supports Ampere® Altra® Arm-based processor included with the following virtual machines:

The following templates demonstrate how to create an ARM nodepool, then assign that nodepool to an existing cluster. These steps can be used in conjunction with Wallaroo Enterprise Install Guides.

ARM nodes are required both for the Wallaroo inference engine, and for converting Wallaroo non-native runtimes. For standard installs of Wallaroo, the options are either:
- Create two nodepools:
  - One nodepool with the wallaroo.ai/pipelines=true:NoSchedule taint for the Wallaroo Engine
  - One nodepool without taints for Wallaroo non-native runtime conversions.
- Create one nodepool without taints used for both auto-conversion and engine deployments.

For custom tolerations, see Taints and Tolerations Guide.

The following scripts show examples of setting up an ARM nodepool for use with the Wallaroo instance.

IMPORTANT NOTE

When adding one or more new nodepools to an existing Kubernetes cluster hosting Wallaroo, verify that the nodepool's availability zone(s) match the existing one(s) used for the Wallaroo installation.

New nodepools must be the same as the ones already hosting the Wallaroo installation to allow pods to deploy.

References:

The following sample script for Microsoft Azure command line tool creates two nodepools into an existing cluster with the Standard_D4ps_v5 virtual machine, providing 4 cpus and 16 GB RAM under the the Ampere® Altra® Arm-based processors. For more details, see Azure Virtual Machines Dpsv5 and Dpdsv5-series.

The first nodepool demonstrated is for general use, and is used for packaging models for the architecture. For this example, one nodepool is created with no taints and the label wallaroo.ai/node-purpose: general.

RESOURCE_GROUP="YOUR RESOURCE GROUP"
CLUSTER_NAME="YOUR CLUSTER NAME"
ARM_NODEPOOL_NAME="YOUR ARM NODEPOOL NAME"

az aks nodepool add \                
    --resource-group $RESOURCE_GROUP \
    --cluster-name $CLUSTER_NAME \
    --name $ARM_NODEPOOL_NAME \
    --node-count 1 \
    --node-vm-size Standard_D4ps_v5 \
    --labels wallaroo.ai/node-purpose=general

For example, to create an ARM nodepool arm_node_general to the existing cluster wallaroo-cluster in the resource group sample-group, the following would be used:

RESOURCE_GROUP="YOUR RESOURCE GROUP"
CLUSTER_NAME="YOUR CLUSTER NAME"
ARM_NODEPOOL_NAME="YOUR ARM NODEPOOL NAME"

az aks nodepool add \                
    --resource-group sample-group \
    --cluster-name wallaroo-cluster \
    --name arm_node_general \
    --node-count 1 \
    --node-vm-size Standard_D4ps_v5 \
    --labels wallaroo.ai/node-purpose=general

The following creates a nodepool with 0-3 nodes in the nodepool for deploying models in the target architecture, allowing it to spin up or down VMs as required.

RESOURCE_GROUP="YOUR RESOURCE GROUP"
CLUSTER_NAME="YOUR CLUSTER NAME"
ARM_ENGINE_NODEPOOL_NAME="YOUR ARM NODEPOOL NAME"

az aks nodepool add \                
    --resource-group $RESOURCE_GROUP \
    --cluster-name $CLUSTER_NAME \
    --name $ARM_ENGINE_NODEPOOL_NAME \
    --node-count 0 \
    --node-vm-size Standard_D4ps_v5 \
    --node-taints wallaroo.ai/pipelines=true:NoSchedule \
    --labels wallaroo.ai/node-purpose=pipelines \
    --enable-cluster-autoscaler \
    --min-count 0 \
    --max-count 3

For example, to create an ARM nodepool arm_node_01 to the existing cluster wallaroo-cluster in the resource group sample-group, the following would be used:

RESOURCE_GROUP="sample-group"
CLUSTER_NAME="wallaroo-cluster"
ARM_NODEPOOL_NAME="arm_node_01"

az aks nodepool add \                
    --resource-group $RESOURCE_GROUP \
    --cluster-name $CLUSTER_NAME \
    --name $ARM_NODEPOOL_NAME \
    --node-count 0 \
    --node-vm-size Standard_D4ps_v5 \
    --node-taints wallaroo.ai/pipelines=true:NoSchedule \
    --enable-cluster-autoscaler \
    --min-count 0 \
    --max-count 3

The following sample script for Amazon Web Services tool eksctl creates a nodepool with the m6g.xlarge virtual machine, providing 4 cpus and 16 GB RAM under the the Arm-based AWS Graviton2 processors processors. For more details, see Amazon EC2 M6g Instances.

The following example template shows adding the nodepool to an existing cluster.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: YOUR CLUSTER NAME HERE # This must match the name of the existing cluster
  region: YOUR REGION HERE 

managedNodeGroups:

- name: YOUR NODEPOOL NAME HERE # Used for packaging models and general use
  instanceType: m6g.medium # the ARM based virtual machine to use
  minSize: 1
  maxSize: 1
  labels:
    wallaroo.ai/node-purpose: "general"
  tags:
    k8s.io/cluster-autoscaler/node-template/label/k8s.dask.org/node-purpose: general
  iam:
    withAddonPolicies:
      autoScaler: true # used to autoscale between the minimum and maximum nodes
  containerRuntime: containerd
  amiFamily: AmazonLinux2
  availabilityZones:
    - INSERT YOUR ZONE HERE # this should match the region for optimal performance
  volumeSize: 100
- name: YOUR NODEPOOL NAME HERE # Used for deploying models
  instanceType: m6g.medium # the ARM based virtual machine to use
  minSize: 1
  maxSize: 1
  labels:
    wallaroo.ai/node-purpose: "pipelines"
  taints:
    - key: wallaroo.ai/pipelines # allows the Wallaroo engine to run in this nodepool 
      value: "true"
      effect: NoSchedule
  tags:
    k8s.io/cluster-autoscaler/node-template/label/k8s.dask.org/node-purpose: pipelines
    k8s.io/cluster-autoscaler/node-template/taint/k8s.dask.org/dedicated: "true:NoSchedule"
  iam:
    withAddonPolicies:
      autoScaler: true # used to autoscale between the minimum and maximum nodes
  containerRuntime: containerd
  amiFamily: AmazonLinux2
  availabilityZones:
    - INSERT YOUR ZONE HERE # this should match the region for optimal performance
  volumeSize: 100

The following sample script for Google Cloud Platform command line tool creates a nodepool with the t2a-standard-4 virtual machine, providing 4 cpus and 16 GB RAM under the the Ampere® Altra® Arm-based processors. For more details, see Tau T2A machine series.

The following must be specified:

Cluster Name: The cluster the nodepool is attached to.
Zone and node location: The location the nodepool is hosted in. For optimal performance, this should match the zone the cluster is in.
Machine type: The virtual machine to use.
Number of nodes: The number of nodes used for the nodepool. This example uses 1.
Service account: The IAM service account. If none is specified, then the default is used.

The following demonstrates adding nodes used for general purpose and packaging models.

CLUSTER_NAME="YOUR CLUSTER NAME"
ARM_NODEPOOL_NAME="YOUR NODEPOOL NAME"
ZONE="YOUR ZONE"
NODE_LOCATIONS="YOUR LOCATIONS"
NUM_NODES=YOUR NUMBER OF NODES


gcloud container node-pools create $ARM_NODEPOOL_NAME \
    --cluster $CLUSTER_NAME \
    --zone $ZONE \
    --node-locations NODE_LOCATIONS \
    --labels=wallaroo.ai/node-purpose=general \
    --machine-type T2A_MACHINE_TYPE \
    --num-nodes NUM_NODES

For example, to create an ARM nodepool arm_node_general to the existing cluster wallaroo-cluster in the resource group sample-group, the following would be used:

CLUSTER_NAME="wallaroo-cluster"
ZONE="us-west1-a"
NODE_LOCATIONS="us-west1-a"
ARM_NODEPOOL_NAME="arm_node_general"
NUM_NODES=1

gcloud container node-pools create $ARM_NODEPOOL_NAME \
    --cluster $CLUSTER_NAME \
    --zone $ZONE \
    --node-locations NODE_LOCATIONS \
    --labels=wallaroo.ai/node-purpose=general \
    --machine-type t2a-standard-4 \
    --num-nodes NUM_NODES

The following demonstrates adding nodes used for deploying models.

CLUSTER_NAME="YOUR CLUSTER NAME"
ARM_NODEPOOL_NAME="YOUR NODEPOOL NAME"
ZONE="YOUR ZONE"
NODE_LOCATIONS="YOUR LOCATIONS"
NUM_NODES=YOUR NUMBER OF NODES


gcloud container node-pools create $ARM_NODEPOOL_NAME \
    --cluster $CLUSTER_NAME \
    --zone $ZONE \
    --node-locations NODE_LOCATIONS \
    --node-taints=wallaroo.ai/pipelines=true:NoSchedule \
    --labels=wallaroo.ai/node-purpose=pipelines \
    --machine-type T2A_MACHINE_TYPE \
    --num-nodes NUM_NODES

For example, to create an ARM nodepool arm_node_01 to the existing cluster wallaroo-cluster in the resource group sample-group, the following would be used:

CLUSTER_NAME="wallaroo-cluster"
ZONE="us-west1-a"
NODE_LOCATIONS="us-west1-a"
ARM_NODEPOOL_NAME="arm_node_01"
NUM_NODES=1

gcloud container node-pools create $ARM_NODEPOOL_NAME \
    --cluster $CLUSTER_NAME \
    --zone $ZONE \
    --node-locations NODE_LOCATIONS \
    --node-taints=wallaroo.ai/pipelines=true:NoSchedule \
    --labels=wallaroo.ai/node-purpose=pipelines \
    --machine-type t2a-standard-4 \
    --num-nodes NUM_NODES

ARM Deployment Tutorials

The following tutorials demonstrate deploying a pipeline with the specified architecture.

Wallaroo Pipeline Architecture

GPU Nodepools

Wallaroo provides support for ML models that use GPUs. The following templates demonstrate how to create a nodepool in different cloud providers, then assign that nodepool to an existing cluster. These steps can be used in conjunction with Wallaroo Enterprise Install Guides.

Note that deploying pipelines with GPU support is only available for Wallaroo Enterprise.

For standard Wallaroo installations, GPU nodepools must include the following taints and labels:

Taint	Label
`wallaroo.ai/pipelines=true:NoSchedule`	`wallaroo.ai/node-purpose: pipelines` `{custom label}`, for example: `wallaroo/gpu:true`

For custom tolerations and labels, see the Taints and Tolerations Guide.

IMPORTANT NOTE

When adding one or more new nodepools to an existing Kubernetes cluster hosting Wallaroo, verify that the nodepool's availability zone(s) match the existing one(s) used for the Wallaroo installation.

New nodepools must be the same as the ones already hosting the Wallaroo installation to allow pods to deploy.

References:

The following script creates a nodepool with NVidia Tesla K80 gpu using the Standard_NC6 machine type and autoscales from 0-3 nodes. Each node has one GPU in this example so the max .gpu() that can be requested by a pipeline step is 1.

For detailed steps on adding GPU to a cluster, see Microsoft Azure Use GPUs for compute-intensive workloads on Azure Kubernetes Service (AKS) guide.

Note that the labels are required as part of the Wallaroo pipeline deployment with GPU support. The label below is an example, but a label must be provided.

RESOURCE_GROUP="YOUR RESOURCE GROUP"
CLUSTER_NAME="YOUR CLUSTER NAME"
GPU_NODEPOOL_NAME="YOUR GPU NODEPOOL NAME"

az extension add --name aks-preview

az extension update --name aks-preview

az feature register --namespace "Microsoft.ContainerService" --name "GPUDedicatedVHDPreview"

az provider register -n Microsoft.ContainerService

az aks nodepool add \                
    --resource-group $RESOURCE_GROUP \
    --cluster-name $CLUSTER_NAME \
    --name $GPU_NODEPOOL_NAME \
    --node-count 0 \
    --node-vm-size Standard_NC6 \
    --node-taints "wallaroo.ai/pipelines=true:NoSchedule" \
    --labels wallaroo.ai/node-purpose=pipelines \
    --aks-custom-headers UseGPUDedicatedVHD=true \
    --enable-cluster-autoscaler \
    --min-count 0 \
    --max-count 3 \

The following script creates a nodepool uses NVidia T4 GPUs and autoscales from 0-3 nodes. Each node has one GPU in this example so the max .gpu() that can be requested by a pipeline step is 1.

Google GKE automatically adds the following taint to the created nodepool.

NO_SCHEDULE nvidia.com/gpu present

Note that the labels are required as part of the Wallaroo pipeline deployment with GPU support. The label below is an example, but a label must be provided.

gcloud container \
    --project $GCP_PROJECT \
    node-pools create $GPU_NODEPOOL_NAME \
    --cluster $GCP_CLUSTER \
    --region $REGION \
    --machine-type "a2-ultragpu-1g" \
    --accelerator "type=nvidia-tesla-a100,count=1,gpu-driver-version=default" \
    --image-type "COS_CONTAINERD" \
    --disk-type "pd-balanced" \
    --disk-size "100" \
    --node-labels wallaroo.ai/node-purpose=pipelines {add custom label here} \ # node-purpose is a required label; custom label - at least one custom label is required
    --node-taints=wallaroo.ai/pipelines=true:NoSchedule \
    --metadata disable-legacy-endpoints=true \
    --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
    --num-nodes "1" \
    --enable-autoscaling \
    --min-nodes "0" \
    --max-nodes "1" \
    --location-policy "BALANCED" \
    --enable-autoupgrade \
    --enable-autorepair \
    --max-surge-upgrade 1 \
    --max-unavailable-upgrade 0

The following steps are used to create a AWS EKS Nodepool with GPU nodes.

Prerequisites: An existing AWS (Amazon Web Service) EKS (Elastic Kubernetes Service) cluster. See Wallaroo Enterprise Comprehensive Install Guide: Environment Setup Guides for a sample creation of an AWS EKS cluster for hosting a Wallaroo Enterprise instance.
eksctl: Command line tool for installating and updating EKS clusters.
Administrator access to the EKS cluster and capabilty of running kubectl commands.

Create the nodepool with the following configuration file. Note that the labels are required as part of the Wallaroo pipeline deployment with GPU support. The label below is an example, but a label must be provided.
1. The sample configuration file below uses the AWS instance type g5.2xlarge. Modify as required.

eksctl create nodegroup --config-file=<path>

Sample config file:

# aws-gpu-nodepool.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: YOUR CLUSTER NAME HERE # This must match the name of the existing cluster
  region: YOUR REGION HERE

managedNodeGroups:
- name: YOUR NODEPOOL NAME HERE
  instanceType: g5.2xlarge
  minSize: 1
  maxSize: 3
  labels:
    wallaroo.ai/node-purpose: "pipelines"
    wallaroo-gpu-label: "true"
  taints:
    - key: wallaroo.ai/pipelines
      value: "true"
      effect: NoSchedule
  tags:
    k8s.io/cluster-autoscaler/node-template/label/k8s.dask.org/node-purpose: pipelines
    k8s.io/cluster-autoscaler/node-template/taint/k8s.dask.org/dedicated: "true:NoSchedule"
  iam:
    withAddonPolicies:
      autoScaler: true
  containerRuntime: containerd
  amiFamily: AmazonLinux2
  availabilityZones:
    - INSERT YOUR ZONE HERE
  volumeSize: 100

References:
- eksctl Managing nodegroups

GPU Deployment Tutorials

The following tutorials demonstrate deploying a pipeline with the specified architecture.

Large Language Model with GPU Pipeline Deployment in Wallaroo Demonstration

Kubernetes Nodepool Guides

Table of Contents

ARM Architecture CPU Nodepools

IMPORTANT NOTE

ARM Deployment Tutorials

GPU Nodepools

IMPORTANT NOTE

GPU Deployment Tutorials