This guide is targeted towards system administrators and data scientists who want to work with the easiest, fastest, and free method of running your own machine learning models. Some knowledge of the following will be useful in working with this guide:
Working knowledge of Linux distributions, particularly Ubuntu.
Either Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure experience.
Working knowledge of Kubernetes, mainly kubectl and kots.
Desire to see your models working in the cloud.
Select either Wallaroo Community or Enterprise for the general steps on how to install Wallaroo. Organizations that already have a prepared environment can skip directly to the respective installation guide for their edition of Wallaroo.
Note the differences between the Wallaroo Community and Wallaroo Enterprise. Wallaroo Community is limited to a maximum of 32 cores and 2 pipelines. For organizations that require more resources, the Wallaroo Enterprise Edition may be more appropriate.
For more information, Contact Us so we can help you find out which is better for your needs.
1 - Wallaroo Prerequisites Guide
Software and other local system requirements before installing Wallaroo
General Time to Completion: 30 minutes.
Before installing Wallaroo version, verify that the following hardware and software requirements are met.
Environment Hardware Requirements
The following system requirements are required for the minimum settings for running Wallaroo in a Kubernetes cloud cluster.
Minimum number of nodes: 4
Minimum Number of CPU Cores: 8
Minimum RAM per node: 16 GB
Minimum Storage: A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.
Wallaroo recommends at least 16 cores total to enable all services. At less than 16 cores, services will have to be disabled to allow basic functionality as detailed in this table.
The following network requirements are required for the minimum settings for running Wallaroo:
For Wallaroo Enterprise users: 200 IP addresses are required to be allocated per cloud environment.
For Wallaroo Community users: 98 IP addresses are required to be allocated per cloud environment.
DNS services integration is required for Wallaroo Enterprise edition. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.
DNS services integration is required to provide access to the various supporting services that are part of the Wallaroo instance. These include:
Simplified user authentication and management.
Centralized services for accessing the Wallaroo Dashboard, Wallaroo SDK and Authentication.
Collaboration features allowing teams to work together.
Managed security, auditing and traceability.
Environment Software Requirements
Running Wallaroo has two sets of software requirements:
Environment Requirements: The following software must be installed on the environment that will be running Wallaroo version. Most are automatically available through the supported cloud providers.
Kubernetes Admin Requirements: The following software must be installed in the system where the Kubernetes environment is being managed - aka where kubectl and kots will be installed. This requires the following software be installed to manage the Kubernetes environment:
Running Wallaroo has two sets of software requirements:
Environment Requirements: The following software must be installed on the environment that will be running Wallaroo version. Most are automatically available through the supported cloud providers.
Kubernetes Admin Requirements: The following software must be installed in the system where the Kubernetes environment is being managed - aka where kubectl and either kots or helm will be installed. This requires the following software be installed to manage the Kubernetes environment:
Wallaroo uses different nodes for various services, which can be assigned to a different node pool to contain resources separate from other nodes. The following nodes selectors can be configured:
ML Engine node selector
ML Engine Load Balance node selector
Database Node Selector
Grafana node selector
Prometheus node selector
Each Lab * Node Selector
Cost Calculators
Organizations that intend to install Wallaroo into a Cloud environment can obtain an estimate of environment costs. The Wallaroo Install Guides list recommended virtual machine types and other settings that can be used to calculate costs for the environment.
For more information, see the pricing calculators for the following cloud services:
Before starting to install Wallaroo, the following requirements must be met at the local system level where Kubernetes commands are issued to the cluster:
kubectl: This interfaces with the Kubernetes server created in the Wallaroo environment.
Version 1.91.3: Preferred for Wallaroo versions 2022.4 and later.
Version 1.81.0: Preferred for Wallaroo versions before 2022.4 and after the August 2022 release.
Version 1.70.1: Preferred for versions of Wallaroo before the August 2022 release.
The following are quick guides on how to install the required software to install and perform updates to Wallaroo. See the respective web sites for full installation information.
kubectl Quick Install Guide
The following are quick guides for installing kubectl for different operating systems. For more details, see the instructions for your specific environment.
kubectl Install For Deb Package based Linux Systems
For users running a deb based package system such as Ubuntu Linux, the following commands will install kubectl and kots into the local system. They assume the user has sudo level access to the system.
Update the apt-get repository:
sudo apt-get update
Install the prerequisite software apt-transport-https, ca-certificates, and curl.
Install the Google Cloud repository into the local repository configuration:
echo"deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main"\
| sudo tee /etc/apt/sources.list.d/kubernetes.list
Update the apt-get repository, then install kubectl:
sudo apt-get update
sudo apt-get install -y kubectl
Verify the kubectl installation:
kubectl version --client
kubectl Install For macOS Using Homebrew
To install kubectl on a macOS system using Homebrew:
Issue the brew install command:
brew install kubectl
Verify the installation:
kubectl version --client
kots Quick Install Guide
The following are quick guides for installing kots for different operating systems. For more details, see the instructions for your specific environment.
IMPORTANT NOTE
As of this time, Wallaroo requireskots version 1.91.3. Please verify that version is installed before starting the Wallaroo installation process.
Install curl.
For deb based Linux systems, update the apt-get repository and install curl:
sudo apt-get update
sudo apt-get install curl
For macOS based systems curl is installed by default.
Install kots by downloading the script and piping it into the bash shell:
Copy the renamed kubectl-kots to anywhere on the PATH.
Next, verify successful installation.
~ kubectl kots version
Replicated KOTS 1.91.3
2 - Wallaroo Community Setup Guides
How to set up Wallaroo Community in various environments in the minimum number of steps
Welcome to Wallaroo! If this is your first time installing Wallaroo, or if you’re performing a fresh installation in a new system, this guide is made to help get you from an empty environment to working with your own machine language models as quickly as possible.
A typical installation of Wallaroo follows this pattern:
2.1.1 - Wallaroo Community AWS EC2 Setup Instructions
How to set up your Wallaroo Community Edition AWS Environment with EC2
Setup AWS EC2 Environment for Wallaroo
The following instructions are made to assist users set up their Amazon Web Services (AWS) environment for running Wallaroo using AWS virtual servers with EC2. This allows organizations to stand a single virtual machine and used a pre-made Amazon Machine Images (AMIs) to quickly stand up an environment that can be used to install Wallaroo.
AWS Prerequisites
To install Wallaroo in your AWS environment based on these instructions, the following prerequisites must be met:
Register an AWS account: https://aws.amazon.com/ and assign the proper permissions according to your organization’s needs. This must be a paid AWS account - Wallaroo will not operate on the free tier level of virtual machines.
Steps
Create the EC2 VM
To create your Wallaroo instance using a pre-made AMI:
Log into AWS cloud console.
Set the region to N. Virginia. Other regions will be added over time.
Select Services -> EC2.
Select Instances, then from the upper right hand section Launch Instances->Launch Instances.
Set the Name and any additional tags.
In Application and OS Images, enter Wallaroo Install and press Enter.
From the search results, select Community AMIs and select Wallaroo Installer 3a.
Set the Instance Type as c6i.8xlarge or c6a.8xlarge as the minimum machine type. This provides 32 cores with 60 GB memory.
For Key pair (login) select one of the following:
Select an existing Key pair name
Select Create new key pair and set the following:
Name: The name of the new key pair.
Key pair type: Select either RSA or ED25519.
Private key file format: Select either .pem or .ppk. These instructions are based on the .pem file.
Select Create key pair when complete.
Set the following for Network settings:
Firewall: Select Create security group or select from an existing one that best fits your organization.
Allow SSH traffic from: Set to Enabled and Anywhere 0.0.0.0/0.
Allow HTTPs traffic from the internet: Set to Enabled.
Set the following for Configure Storage:
Set Root volume to at least 400 GiB, type standard.
Review the Summary and verify the following:
Number of instances: 1
Virtual server type: Matches the minimum requirement listed above.
Verify the other settings are accurate.
Select Launch Instance.
It is recommended to give the instance time to complete its setup process. This typically takes 20 minutes.
Verify the Setup
To verify the environment is setup for Wallaroo:
From the EC2 Dashboard, select the virtual machine created for your Wallaroo instance.
Note the Public IPv4 DNS address.
From a terminal, run ssh to connect to your virtual machine. The installation requires access to port 8800 and the private key selected or created in the instructions above.
The ssh command format for connecting to your virtual machine uses the following format, replacing the $keyfile, $VM_DNS with your private key file and the DNS address to your Amazon VM:
If the Kubernetes setup is still installing, wait until complete and when prompted select EXIT to complete the process. This process may take up to 20 to 30 minutes.
Install Wallaroo
With your environment ready, it’s time to install Wallaroo.
The following tips can be used to save costs on your AWS EC2 instance.
Stop Instances When Not In Use
One cost saving measure is to stop instances when not in use. If you intend to stop an instance, register it with static IP address so when it is turned back on your services will continue to function without interruption.
I keep seeing the errors such as connect failed. Is this a problem?
Sometimes you may see an error such as channel 3: open failed: connect failed: Connection refused. This is the ssh port forwarding attempting to connect to port 8800 during the installation, and can be ignored.
When Launching JupyterHub, I get a Server 500 error
If you shut down and restart a Wallaroo instance in a new environment or change the IP address, some settings may not be updated. Run the following command to restart the deployment process and update the settings to match the current environment:
kubectl rollout restart deployment hub
2.1.2 - Wallaroo Community AWS EKS Setup Instructions
How to set up your Wallaroo Community AWS Environment with EKS
The following instructions are made to assist users set up their Amazon Web Services (AWS) environment for running Wallaroo using AWS Elastic Kubernetes Service (EKS).
These represent a recommended setup, but can be modified to fit your specific needs.
If the prerequisites are already met, skip ahead to Install Wallaroo.
The following video demonstrates this process:
AWS Prerequisites
To install Wallaroo in your AWS environment based on these instructions, the following prerequisites must be met:
Register an AWS account: https://aws.amazon.com/ and assign the proper permissions according to your organization’s needs.
The Kubernetes cluster must include the following minimum settings:
Nodes must be OS type Linux with using the containerd driver.
Role-based access control (RBAC) must be enabled.
Minimum of 4 nodes, each node with a minimum of 8 CPU cores and 16 GB RAM. 50 GB will be allocated per node for a total of 625 GB for the entire cluster.
RBAC is enabled.
Recommended Aws Machine type: c5.4xlarge. For more information, see the AWS Instance Types.
Running Wallaroo has two sets of software requirements:
Environment Requirements: The following software must be installed on the environment that will be running Wallaroo version. Most are automatically available through the supported cloud providers.
Kubernetes Admin Requirements: The following software must be installed in the system where the Kubernetes environment is being managed - aka where kubectl and kots will be installed. This requires the following software be installed to manage the Kubernetes environment:
The following recommendations will assist in reducing the cost of a cloud based Kubernetes Wallaroo cluster.
Turn off the cluster when not in use. An AWS EKS (Elastic Kubernetes Services) cluster can be turn off when not in use, then turned back on again when needed. If organizations adopt this process, be aware of the following issues:
IP Address Reassignment: The load balancer public IP address may be reassigned when the cluster is restarted by the cloud service unless a static IP address is assigned. For more information in Amazon Web Services see the Associate Elastic IP addresses with resources in your VPC user guide.
Assign to a Single Availability Zone: Clusters that span multiple availability zones may have issues accessing persistent volumes that were provisioned in another availability zone from the node when the node is restarted. The simple solution is to assign the entire cluster into a single availability zone. For more information in Amazon Web Services see the Regions and Zones guide.
The scripts and configuration files are set up to create the AWS environment for a Wallaroo instance are based on a single availability zone. Modify the script as required for your organization.
Community Cluster Setup Instructions
The following is based on the requirements for Wallaroo Community. Note that Wallaroo Community does not use adaptive nodepools. Adapt the settings as required for your organization’s needs, as long as they meet the prerequisites listed above.
IMPORTANT NOTE
This sample YAML file can be downloaded from here:
eksctl: A command line tool for managing Amazon EKS clusters from a configured YAML file. See the EKSCTL Install Guide for more details.
kubectl: This interfaces with the Kubernetes server created in the Wallaroo environment.
kots Version: Used to manage software installed in a Kubernetes environment.
Create the Cluster
Create the cluster with the following command, which creates the environment and sets the correct Kubernetes version.
eksctl create cluster -f aws.yaml
During the process the Kuberntes credentials will be copied into the local environment. To verify the setup is complete, use the kubectl get nodes command to display the available nodes as in the following example:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-21-253.us-east-2.compute.internal Ready <none> 13m v1.23.8-eks-9017834
ip-192-168-30-36.us-east-2.compute.internal Ready <none> 13m v1.23.8-eks-9017834
ip-192-168-55-123.us-east-2.compute.internal Ready <none> 12m v1.23.8-eks-9017834
ip-192-168-79-70.us-east-2.compute.internal Ready <none> 13m v1.23.8-eks-9017834
Wallaroo can now be installed.
Install Wallaroo
With your environment ready, it’s time to install Wallaroo.
2.1.3 - Wallaroo Community Azure Setup Instructions
How to set up your Wallaroo Community Azure Environment
The following instructions are made to assist users set up their Microsoft Azure Kubernetes environment for running Wallaroo Community. These represent a recommended setup, but can be modified to fit your specific needs.
The Kubernetes cluster must include the following minimum settings:
Nodes must be OS type Linux with using the containerd driver.
Role-based access control (RBAC) must be enabled.
Minimum of 4 nodes, each node with a minimum of 8 CPU cores and 16 GB RAM. 50 GB will be allocated per node for a total of 625 GB for the entire cluster.
RBAC is enabled.
Minimum machine type is set to to Standard_D8s_v4.
The following recommendations will assist in reducing the cost of a cloud based Kubernetes Wallaroo cluster.
Turn off the cluster when not in use. An Azure Kubernetes Service (AKS) cluster can be turn off when not in use, then turned back on again when needed to save on costs. For more information on starting and stopping an AKS cluster, see the Stop and Start an Azure Kubernetes Service (AKS) cluster guide.
If organizations adopt this process, be aware of the following issues:
Assign to a Single Availability Zone: Clusters that span multiple availability zones may have issues accessing persistent volumes that were provisioned in another availability zone from the node when the cluster is restarted. The simple solution is to assign the entire cluster into a single availability zone. For more information in Microsoft Azure see the Create an Azure Kubernetes Service (AKS) cluster that uses availability zones guide.
The scripts and configuration files are set up to create the Azure environment for a Wallaroo instance are based on a single availability zone. Modify the script as required for your organization.
The Azure Resource Group used for the Kubernetes environment.
WALLAROO_GROUP_LOCATION
eastus
The region that the Kubernetes environment will be installed to.
WALLAROO_CONTAINER_REGISTRY
wallarooceacr
The Azure Container Registry used for the Kubernetes environment.
WALLAROO_CLUSTER
wallarooceaks
The name of the Kubernetes cluster that Wallaroo is installed to.
WALLAROO_SKU_TYPE
Basic
The Azure Kubernetes Service SKU type.
WALLAROO_NODEPOOL
wallaroocepool
The main nodepool for the Kubernetes cluster.
WALLAROO_VM_SIZE
Standard_D8s_v4
The VM type used for the standard Wallaroo cluster nodes.
WALLAROO_CLUSTER_SIZE
4
The number of nodes in the cluster.
Quick Setup Script
The following sample script creates an Azure Kubernetes environment ready for use with Wallaroo Community. This script requires the following prerequisites listed above.
Modify the installation file to fit for your organization. The only parts that require modification are the variables listed in the beginning as follows:
The following steps are geared towards a standard Linux or macOS system that supports the prerequisites listed above. Modify these steps based on your local environment.
Download the script above.
In a terminal window set the script status as execute with the command chmod +x wallaroo_community_azure_install.bash.
Modify the script variables listed above based on your requirements.
Run the script with either bash wallaroo_community_azure_install.bash or ./wallaroo_community_azure_install.bash from the same directory as the script.
Manual Setup Guide
The following steps are guidelines to assist new users in setting up their Azure environment for Wallaroo. Feel free to replace these with commands with ones that match your needs.
The following are used for the example commands below. Replace them with your specific environment settings:
Azure Resource Group: wallarooCEGroup
Azure Resource Group Location: eastus
Azure Container Registry: wallarooCEAcr
Azure Kubernetes Cluster: wallarooCEAKS
Azure Container SKU type: Basic
Azure Nodepool Name: wallarooCEPool
Setting up an Azure AKS environment is based on the Azure Kubernetes Service tutorial, streamlined to show the minimum steps in setting up your own Wallaroo environment in Azure.
Manual Guide
This follows these major steps:
Create an Azure Resource Group
Create an Azure Container Registry
Create the Azure Kubernetes Environment
Set Variables
The following are the variables used in the environment setup process. Modify them as best fits your organization’s needs.
To create an Azure Resource Group for Wallaroo in Microsoft Azure, use the following template:
az group create --name $WALLAROO_RESOURCE_GROUP --location $WALLAROO_GROUP_LOCATION
(Optional): Set the default Resource Group to the one recently created. This allows other Azure commands to automatically select this group for commands such as az aks list, etc.
az configure --defaults group=$WALLAROO_RESOURCE_GROUP
Create an Azure Container Registry
An Azure Container Registry(ACR) manages the container images for services includes Kubernetes. The template for setting up an Azure ACR that supports Wallaroo is the following:
And now we can create our Kubernetes service in Azure that will host our Wallaroo that meet the prerequisites. Modify the settings to meet your organization’s needs. This creates a 4 node cluster with a total of 32 cores.
Once the Kubernetes environment is complete, associate it with the local Kubernetes configuration by importing the credentials through the following template command:
az aks get-credentials --resource-group $WALLAROO_RESOURCE_GROUP --name $WALLAROO_CLUSTER
Verify the cluster is available through the kubectl get nodes command.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-mainpool-37402055-vmss000000 Ready agent 81m v1.23.8
aks-mainpool-37402055-vmss000001 Ready agent 81m v1.23.8
aks-mainpool-37402055-vmss000002 Ready agent 81m v1.23.8
aks-mainpool-37402055-vmss000003 Ready agent 81m v1.23.8
Install Wallaroo
With your environment ready, it’s time to install Wallaroo.
How to set up your Wallaroo Community GCP Environment
The following instructions are made to assist users set up their Google Cloud Platform (GCP) Kubernetes environment for running Wallaroo. These represent a recommended setup, but can be modified to fit your specific needs. In particular, these instructions will provision a GKE cluster with 32 CPUs in total. Please ensure that your project’s resource limits support that.
Quick Setup Guide: Download a bash script to automatically set up the GCP environment through the Google Cloud Platform command line interface gcloud.
Manual Setup Guide: A list of the gcloud commands used to create the environment through manual commands.
The following video demonstrates the manual guide:
GCP Prerequisites
Organizations that wish to run Wallaroo in their Google Cloud Platform environment must complete the following prerequisites:
The following recommendations will assist in reducing the cost of a cloud based Kubernetes Wallaroo cluster.
Turn off the cluster when not in use. A GCP Google Kubernetes Engine (GKE) cluster can be turn off when not in use, then turned back on again when needed. If organizations adopt this process, be aware of the following issues:
IP Address Reassignment: The load balancer public IP address may be reassigned when the cluster is restarted by the cloud service unless a static IP address is assigned. For more information in Google Cloud Platform see the Configuring domain names with static IP addresses user guide.
Assign to a Single Availability Zone: Clusters that span multiple availability zones may have issues accessing persistent volumes that were provisioned in another availability zone from the node when the cluster is restarted. The simple solution is to assign the entire cluster into a single availability zone. For more information in Google Cloud Platform see the Regions and zones guide.
The scripts and configuration files are set up to create the GCP environment for a Wallaroo instance are based on a single availability zone. Modify the script as required for your organization.
The name of the Google Project used for the Wallaroo instance.
WALLAROO_CLUSTER
wallaroo-ce
The name of the Kubernetes cluster for the Wallaroo instance.
WALLAROO_GCP_REGION
us-central1
The region the Kubernetes environment is installed to. Update this to your GCP Computer Engine region.
WALLAROO_NODE_LOCATION
us-central1-f
The location the Kubernetes nodes are installed to. Update this to your GCP Compute Engine Zone.
WALLAROO_GCP_NETWORK_NAME
wallaroo-network
The Google network used with the Kubernetes environment.
WALLAROO_GCP_SUBNETWORK_NAME
wallaroo-subnet-1
The Google network subnet used with the Kubernetes environment.
WALLAROO_GCP_MACHINE_TYPE
e2-standard-8
Recommended VM size per GCP node.
WALLAROO_CLUSTER_SIZE
4
Number of nodes installed into the cluster. 4 nodes will create a 32 core cluster.
Quick Setup Script
A sample script is available here, and creates a Google Kubernetes Engine cluster ready for use with Wallaroo Community. This script requires the prerequisites listed above, and uses the variables as listed in Standard Setup Variables.
The following steps are geared towards a standard Linux or macOS system that supports the prerequisites listed above. Modify these steps based on your local environment.
Download the script above.
In a terminal window set the script status as execute with the command chmod +x bash wallaroo_community_gcp_install.bash.
Modify the script variables listed above based on your requirements.
Run the script with either bash wallaroo_community_gcp_install.bash or ./wallaroo_community_gcp_install.bash from the same directory as the script.
Manual Setup Guide
The following steps are guidelines to assist new users in setting up their GCP environment for Wallaroo. Feel free to replace these with commands with ones that match your needs.
See the Google Cloud SDK for full details on commands and settings.
The commands below are set to meet the prerequisites listed above, and uses the variables as listed in Standard Setup Variables. Modify them as best fits your organization’s needs.
Set Variables
The following are the variables used in the environment setup process. Modify them as best fits your organization’s needs.
First create a GCP network that is used to connect to the cluster with the gcloud compute networks create command. For more information, see the gcloud compute networks create page.
Once the network is created, the gcloud container clusters create command is used to create a cluster. For more information see the gcloud container clusters create page.
The command can take several minutes to complete based on the size and complexity of the clusters. Verify the process is complete with the clusters list command:
To verify the Kubernetes credentials for your cluster have been installed locally, use the kubectl get nodes command. This will display the nodes in the cluster as demonstrated below:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-wallaroo-ce-default-pool-863f02db-7xd4 Ready <none> 39m v1.21.6-gke.1503
gke-wallaroo-ce-default-pool-863f02db-8j2d Ready <none> 39m v1.21.6-gke.1503
gke-wallaroo-ce-default-pool-863f02db-hn06 Ready <none> 39m v1.21.6-gke.1503
gke-wallaroo-ce-default-pool-3946eaca-4l3s Ready <none> 39m v1.21.6-gke.1503
Install Wallaroo
With your environment ready, it’s time to install Wallaroo.
What does the error ‘Insufficient project quota to satisfy request: resource “CPUS_ALL_REGIONS”’ mean?
Make sure that the Compute Engine Zone and Region are properly set based on your organization’s requirements. The instructions above default to us-central1, so change that zone to install your Wallaroo instance in the correct location.
In the case of the script, this would mean changing the region and location from:
How to set up Wallaroo Community in various environments in the minimum number of steps
Welcome to Wallaroo! If this is your first time installing Wallaroo, or if you’re performing a fresh installation in a new system, this guide is made to help get you from an empty environment to working with your own machine language models as quickly as possible.
A typical installation of Wallaroo follows this pattern:
Step
Description
Average Setup Time
Setup Environment
Create an environment that meets the Wallaroo prerequisites
30 minutes
Current Step: Install Wallaroo
Install Wallaroo into a prepared environment
15 minutes
Once the environment is prepared, installing Wallaroo Community follows this process:
2.2.1 - How to Download a Wallaroo Community License
Register for your Wallaroo Community and retrieve your Wallaroo Community License
Register Your Wallaroo Community Account
Completion Time
General completion time: 5 minutes
The first step to installing Wallaroo CE is to set up your Wallaroo Community Account at the web site https://portal.wallaroo.community. This process typically takes about 5 minutes.
Once you’ve submitted your credentials, you’ll be sent an email with a link to your license file.
Follow the link and download your license file. Store it in a secure location.
Wallaroo Community can be installed into a Kubernetes cloud environment, or into a Kubernetes environment that meets the Wallaroo Prerequisites Guide. Organizations that use the Wallaroo Community AWS EC2 Setup Instructions procedure do not have to set up a Kubernetes environment, as it is already configured for them.
If the prerequisites are already configured, jump to Install Wallaroo to start installing.
This video demonstrates that procedure:
The procedure assumes at least a basic knowledge of Kubernetes and how to use the kubectl and kots version 1.91.3 applications.
The environment is ready, the tools are installed - let’s install Wallaroo! The following will use kubectl and kots through the following procedure:
Install the Wallaroo Community Edition using kots install wallaroo/ce, specifying the namespace to install. For example, if wallaroo is the namespace, then the command is:
Wallaroo Community Edition will be downloaded and installed into your Kubernetes environment in the namespace specified. When prompted, set the default password for the Wallaroo environment. When complete, Wallaroo Community Edition will display the URL for the Admin Console, and how to end the Admin Console from running.
• Deploying Admin Console
• Creating namespace ✓
• Waiting for datastore to be ready ✓
Enter a new password to be used for the Admin Console: •••••••••••••
• Waiting for Admin Console to be ready ✓
• Press Ctrl+C to exit• Go to http://localhost:8800 to access the Admin Console
Wallaroo Community edition will continue to run until terminated. To relaunch in the future, use the following command:
kubectl-kots admin-console --namespace wallaroo
Initial Configuration and License Upload Procedure
Once Wallaroo Community edition has been installed for the first time, we can perform initial configuration and load our Wallaroo Community license file through the following process:
If Wallaroo Community Edition has not started, launch it with the following command:
❯ kubectl-kots admin-console --namespace wallaroo
• Press Ctrl+C to exit • Go to http://localhost:8800 to access the Admin Console
Enter the Wallaroo Community Admin Console address into a browser. You will be prompted for the default password as set in the step above. Enter it and select Log in.
Upload your license file.
The Configure Wallaroo Community page will be displayed which allows you to customize your Wallaroo environment. For now, scroll to the bottom and select Continue. These settings can be customized at a later date.
The Wallaroo Community Admin Console will run the preflight checks to verify that all of the minimum requirements are not met. This may take a few minutes. If there are any issues, Wallaroo can still be launched but may not function properly. When ready, select Continue.
The Wallaroo Community Dashboard will be displayed. There may be additional background processes that are completing their setup procedures, so there may be a few minute wait until those are complete. If everything is ready, then the Wallaroo Dashboard will show a green Ready.
Under the license information is the DNS entry for your Wallaroo instance. This is where you and other users of your Wallaroo instance can log in. In this example, the URL will be https://beautiful-horse-9537.wallaroo.community. Note that it may take a few minutes for the DNS entries to propagate and this URL to be available.
You will receive an email invitation for the email address connected to this URL with a temporary password and a link to this Wallaroo instance’s URL. Either enter the URL for your Wallaroo instance or use the link in the email.
To login to your new Wallaroo instance, enter the email address and temporary password associated with the license.
With that, Wallaroo Community edition is launched and ready for use! You can end the Admin Console from your terminal session above. From this point on you can just use the Wallaroo instance URL.
Now that your Wallaroo Community edition has been installed, let’s work with some sample models to show off what you can do. Check out either the Wallaroo 101 if this is your first time using Wallaroo, or for more examples of how to use Wallaroo see the Wallaroo Tutorials.
If you see an error similar to ‘failed to deploy admin console: failed to wait for {some node}: timeout waiting for {some node}’, it may be because the connection between your local system and the cloud service is slow, or related issues.
If this occurs, adding the option --wait-duration 5m give enough time for nodes to finish starting.
3 - Wallaroo Enterprise Setup Guides
How to set up Wallaroo Enterprise
This guide is targeted towards system administrators and data scientists who want to work with the easiest, fastest, and free method of running your own machine learning models.
A typical installation of Wallaroo follows this process:
Step
Description
Average Setup Time
Setup Environment
Create an environment that meets the Wallaroo prerequisites
30 minutes
Install Wallaroo
Install Wallaroo into a prepared environment
15 minutes
Configure Wallaroo
Update Wallaroo post-install
Variable
Some knowledge of the following will be useful in working with this guide:
Working knowledge of Linux distributions, particularly Ubuntu.
Either Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure experience. For more information, see the Wallaroo Enterprise Environment Setup Guides guides.
Working knowledge of Kubernetes, mainly kubectl and kots.
For more information, Contact Us for additional details.
Taints and Tolerances Guide: How to configure Wallaroo for specific taints and tolerances so ensure that only Wallaroo services are running in specific nodes.
How to set up your Wallaroo Enterprise Edition AWS Environment with EKS
The following instructions are made to assist users set up their Amazon Web Services (AWS) environment for running Wallaroo Enterprise using AWS Elastic Kubernetes Service (EKS).
These represent a recommended setup, but can be modified to fit your specific needs.
Installation Flow
A typical installation of Wallaroo Enterprise follows this flow:
Create Environment: Create the environment to install Wallaroo that meets the system prerequisites.
Configure DNS: Configure DNS services and the Wallaroo instance for your organization’s use.
AWS Prerequisites
To install Wallaroo in your AWS environment based on these instructions, the following prerequisites must be met:
Register an AWS account: https://aws.amazon.com/ and assign the proper permissions according to your organization’s needs.
The Kubernetes cluster must include the following minimum settings:
Nodes must be OS type Linux with using the containerd driver.
Role-based access control (RBAC) must be enabled.
Minimum of 4 nodes, each node with a minimum of 8 CPU cores and 16 GB RAM. 50 GB will be allocated per node for a total of 625 GB for the entire cluster.
RBAC is enabled.
Recommended Aws Machine type: c5.4xlarge. For more information, see the AWS Instance Types.
Running Wallaroo has two sets of software requirements:
Environment Requirements: The following software must be installed on the environment that will be running Wallaroo version. Most are automatically available through the supported cloud providers.
Kubernetes Admin Requirements: The following software must be installed in the system where the Kubernetes environment is being managed - aka where kubectl and kots will be installed. This requires the following software be installed to manage the Kubernetes environment:
Running Wallaroo has two sets of software requirements:
Environment Requirements: The following software must be installed on the environment that will be running Wallaroo version. Most are automatically available through the supported cloud providers.
Kubernetes Admin Requirements: The following software must be installed in the system where the Kubernetes environment is being managed - aka where kubectl and kots will be installed. This requires the following software be installed to manage the Kubernetes environment:
Organizations that intend to stop and restart their Kubernetes environment on an intentional or regular basis are recommended to use a single availability zone for their nodes. This minimizes issues such as persistent volumes in different availability zones, etc.
Organizations that intend to use Wallaroo Enterprise in a high availability cluster are encouraged to follow best practices including using separate availability zones for redundancy, etc.
Custom Configurations
Wallaroo can be configured with custom installations depending on your organization’s needs. The following options are available:
Taints and Tolerances Guide: How to configure Wallaroo for specific taints and tolerances so ensure that only Wallaroo services are running in specific nodes.
AWS Environment Setup Steps
The following steps are guidelines to assist new users in setting up their AWS environment for Wallaroo. Feel free to replace these with commands with ones that match your needs.
These commands make use of the command line tool eksctl which streamlines the process in creating Amazon Elastic Kubernetes Service clusters for our Wallaroo environment.
The following are used for the example commands below. Replace them with your specific environment settings:
AWS Cluster Name: wallarooAWS
Create an AWS EKS Cluster
The following eksctl configuration file is an example of setting up the AWS environment for a Wallaroo cluster, including the static and adaptive nodepools. Adjust these names and settings based on your organizations requirements.
IMPORTANT NOTE
This sample YAML file can be downloaded from here:
During the process the Kubernetes credentials will be copied into the local environment. To verify the setup is complete, use the kubectl get nodes command to display the available nodes as in the following example:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-21-253.us-east-2.compute.internal Ready <none> 13m v1.23.8-eks-9017834
ip-192-168-30-36.us-east-2.compute.internal Ready <none> 13m v1.23.8-eks-9017834
ip-192-168-38-31.us-east-2.compute.internal Ready <none> 9m46s v1.23.8-eks-9017834
ip-192-168-55-123.us-east-2.compute.internal Ready <none> 12m v1.23.8-eks-9017834
ip-192-168-79-70.us-east-2.compute.internal Ready <none> 13m v1.23.8-eks-9017834
ip-192-168-37-222.us-east-2.compute.internal Ready <none> 13m v1.23.8-eks-9017834
Install Wallaroo
With the environment prepared, Wallaroo can now be installed.
3.1.2 - Wallaroo Enterprise Single Node Linux Setup Instructions
How to prepare a single node Linux environment for Wallaroo Enterprise installations.
Organizations can run Wallaroo within a single node Linux environment that meet the prerequisites.
The following guide is based on installing Wallaroo Enterprise into virtual machines based on Ubuntu 22.04 hosted in Google Cloud Platform (GCP), Amazon Web Services (AWS) and Microsoft Azure. For other environments and configurations, consult your Wallaroo support representative.
Installation Flow
A typical installation of Wallaroo Enterprise follows this flow:
Create Environment: Create the environment to install Wallaroo that meets the system prerequisites.
650 GB allocated for the root partition, plus 50 GB allocated per node and another 50 GB for the JupyterHub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.
Ensure memory swapping is disabled by removing it from /etc/fstab if needed.
DNS services for integrating your Wallaroo Enterprise instance. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.
Note that if Wallaroo is being installed into a cloud environment such as Google Cloud Platform, Microsoft Azure, Amazon Web Services, etc, then additional considerations such as networking, DNS, certificates, and other considerations must be accounted for.
Template Single Node Scripts
The following template scripts are provided as examples on how to create single node virtual machines that meet the requirements listed above in AWS, GCP, and Microsoft Azure environments.
# Variables# The name of the virtual machineNAME=$USER-demo-vm # eg bob-demo-vm# The image used : ubuntu/images/current/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20230208IMAGE_ID=ami-0557a15b87f6559cf
# Instance type meeting the Wallaroo requirements.INSTANCE_TYPE=c6i.8xlarge # c6a.8xlarge is also acceptable# key name - generate keys using Amazon EC2 Key Pairs# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html# Wallaroo people: https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#KeyPairs:v=3 - MYKEY=DocNode
# We will whitelist the our source IP for maximum security -- just use 0.0.0.0/0 if you don't care.MY_IP=$(curl -s https://checkip.amazonaws.com)/32
# Create security group in the Default VPCaws ec2 create-security-group --group-name $NAME --description "$USER demo" --no-cli-pager
# Open port 22 and 443aws ec2 authorize-security-group-ingress --group-name $NAME --protocol tcp --port 22 --cidr $MY_IP --no-cli-pager
aws ec2 authorize-security-group-ingress --group-name $NAME --protocol tcp --port 443 --cidr $MY_IP --no-cli-pager
# increase Boot device size to 650 GB# Change the location from `/tmp/device.json` as required.# cat <<EOF > /tmp/device.json # [{# "DeviceName": "/dev/sda1",# "Ebs": { # "VolumeSize": 650,# "VolumeType": "gp2"# }# }]# EOF# Launch instance with a 650 GB Boot device.aws ec2 run-instances --image-id $IMAGE_ID --count 1 --instance-type $INSTANCE_TYPE\
--no-cli-pager \
--key-name $MYKEY\
--block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":650,"VolumeType":"gp2"}}]'\
--tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=$NAME}]"\
--security-groups $NAME# Sample output:# {# "Instances": [# {# ...# "InstanceId": "i-0123456789abcdef", # Keep this instance-id for later# ...# }# ]# }#INSTANCEID=YOURINSTANCE# After several minutes, a public IP will be known. This command will retrieve it.# aws ec2 describe-instances --output text --instance-id $INSTANCEID \# --query 'Reservations[*].Instances[*].{ip:PublicIpAddress}'# Sample Output# 12.23.34.56# KEYFILE=KEYFILELOCATION #usually ~/.ssh/key.pem - verify this is the same as the key above.# SSH to the VM - replace $INSTANCEIP#ssh -i $KEYFILE ubuntu@$INSTANCEIP# Stop the VM - replace the $INSTANCEID#aws ec2 stop-instances --instance-id $INSTANCEID# Restart the VM#aws ec2 start-instances --instance-id $INSTANCEID# Clean up - destroy VM#aws ec2 terminate-instances --instance-id $INSTANCEID
#!/bin/bash
# Variables list. Update as per your organization's settingsNAME=$USER-demo-vm # eg bob-demo-vmRESOURCEGROUP=YOURRESOURCEGROUP
LOCATION=eastus
IMAGE=Canonical:0001-com-ubuntu-server-jammy:22_04-lts:22.04.202301140
# Pick a locationaz account list-locations -o table |egrep 'US|----|Name'# Create resource groupaz group create -l $LOCATION --name $USER-demo-$(date +%y%m%d)# Create VM. This will create ~/.ssh/id_rsa and id_rsa.pub - store these for later use.az vm create --resource-group $RESOURCEGROUP --name $NAME --image $IMAGE --generate-ssh-keys \
--size Standard_D32s_v4 --os-disk-size-gb 500 --public-ip-sku Standard
# Sample output# {# "location": "eastus",# "privateIpAddress": "10.0.0.4",# "publicIpAddress": "20.127.249.196", <-- Write this down as MYPUBIP# "resourceGroup": "mnp-demo-230213",# ...# }# SSH port is open by default. This adds an application port.az vm open-port --resource-group $RESOURCEGROUP --name $NAME --port 443# SSH to the VM - assumes that ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub from above are availble.# ssh $MYPUBIP# Use this Stop the VM ("deallocate" frees resources and billing; "stop" does not)# az vm deallocate --resource-group $RESOURCEGROUP --name $NAME# Restart the VM# az vm start --resource-group $RESOURCEGROUP --name $NAME
# SettingsNAME=$USER-demo-$(date +%y%m%d)# eg bob-demo-230210ZONE=us-west1-a # For a complete list, use `gcloud compute zones list | egrep ^us-`PROJECT=wallaroo-dev-253816 # Insert the GCP Project ID here. This is the one for Wallaroo.# Create VMIMAGE=projects/ubuntu-os-cloud/global/images/current/ubuntu-2204-jammy-v20230114
# Port 22 and 443 open by defaultgcloud compute instances create $NAME\
--project=$PROJECT\
--zone=$ZONE\
--machine-type=e2-standard-32 \
--network-interface=network-tier=STANDARD,subnet=default \
--maintenance-policy=MIGRATE \
--provisioning-model=STANDARD \
--no-service-account \
--no-scopes \
--tags=https-server \
--create-disk=boot=yes,image=${IMAGE},size=500,type=pd-standard \
--no-shielded-secure-boot \
--no-shielded-vtpm \
--no-shielded-integrity-monitoring \
--reservation-affinity=any
# Get the external IP addressgcloud compute instances describe $NAME --zone $ZONE --format='get(networkInterfaces[0].accessConfigs[0].natIP)'# SSH to the VM#gcloud compute ssh $NAME --zone $ZONE# SCP file to the instance - replace $FILE with the file path. Useful for copying up the license file up to the instance.#gcloud compute scp --zone $ZONE $FILE $NAME:~/# SSH port forward to the VM#gcloud compute ssh $NAME --zone $ZONE -- -NL 8800:localhost:8800# Suspend the VM#gcloud compute instances stop $NAME --zone $ZONE# Restart the VM#gcloud compute instances start $NAME --zone $ZONE
Kubernetes Installation Steps
The following script and steps will install the Kubernetes version and requirements into the Linux node that supports a Wallaroo single node installation.
The process includes these major steps:
Install Kubernetes
Install Kots Version
Install Kubernetes
curl is installed in the default scripts provided above. Verify that it is installed if using some other platform.
Verify that the Ubuntu distribution is up to date, and reboot if necessary after updating.
sudo apt update
sudo apt upgrade
Start the Kubernetes installation with the following script, substituting the URL path as appropriate for your license.
For Wallaroo versions 2022.4 and below:
curl https://kurl.sh/9398a3a | sudo bash
For Wallaroo versions 2023.1 and later, the install is based on the license channel. For example, if your license uses the EE channel, then the path is /wallaroo-ee; that is, /wallaroo- plus the lower-case channel name. Note that the Kubernetes install channel must match the License version. Check with your Wallaroo support representative with any questions about your version.
curl https://kurl.sh/wallaroo-ee | sudo bash
If prompted with This application is incompatible with memory swapping enabled. Disable swap to continue? (Y/n), reply Y.
Set up the Kubernetes configuration with the following commands:
Log out, and log back in as the same user. Verify the installation was successful with the following:
kubectl get nodes
It should return results similar to the following:
NAME STATUS ROLES AGE VERSION
wallux Ready control-plane,master 6m26s v1.23.6
Install Kots
Install kots with the following process.
Run the following script and provide your password for the sudo based commands when prompted.
curl https://kots.io/install/1.91.3 | sudo bash
Verify kots was installed with the following command:
kubectl kots version
It should return results similar to the following:
Replicated KOTS 1.91.3
Connection Options
Once Kubernetes has been set up on the Linux node, users can opt to copy the Kubernetes configuration to a local system, updating the IP address and other information as required. See the Configure Access to Multiple Clusters.
The easiest method is to create a SSH tunnel to the Linux node. Usually this will be in the format:
ssh $IP -L8800:localhost:8800
For example, in an AWS instance that may be as follows, replaying $KEYFILE with the link to the keyfile and $IP with the IP address of the Linux node.
ssh -i $KEYFILE ubuntu@$IP -L8800:localhost:8800
In a GCP instance, gcloud can be used as follows, replacing $NAME with the name of the GCP instance, $ZONE with the zone it was installed into.
How to set up your Wallaroo Enterprise Edition Azure Environment
The following instructions are made to assist users set up their Microsoft Azure Kubernetes environment for running Wallaroo Enterprise. These represent a recommended setup, but can be modified to fit your specific needs.
The Kubernetes cluster must include the following minimum settings:
Nodes must be OS type Linux with using the containerd driver.
Role-based access control (RBAC) must be enabled.
Minimum of 4 nodes, each node with a minimum of 8 CPU cores and 16 GB RAM. 50 GB will be allocated per node for a total of 625 GB for the entire cluster.
RBAC is enabled.
Minimum machine type is set to to Standard_D8s_v4.
Running Wallaroo has two sets of software requirements:
Environment Requirements: The following software must be installed on the environment that will be running Wallaroo version. Most are automatically available through the supported cloud providers.
Kubernetes Admin Requirements: The following software must be installed in the system where the Kubernetes environment is being managed - aka where kubectl and kots will be installed. This requires the following software be installed to manage the Kubernetes environment:
Organizations that intend to stop and restart their Kubernetes environment on an intentional or regular basis are recommended to use a single availability zone for their nodes. This minimizes issues such as persistent volumes in different availability zones, etc.
Organizations that intend to use Wallaroo Enterprise in a high availability cluster are encouraged to follow best practices including using separate availability zones for redundancy, etc.
Custom Configurations
Wallaroo can be configured with custom installations depending on your organization’s needs. The following options are available:
Taints and Tolerances Guide: How to configure Wallaroo for specific taints and tolerances so ensure that only Wallaroo services are running in specific nodes.
The Azure Resource Group used for the KUbernetes environment.
WALLAROO_GROUP_LOCATION
eastus
The region that the Kubernetes environment will be installed to.
WALLAROO_CONTAINER_REGISTRY
wallarooacr
The Azure Container Registry used for the Kubernetes environment.
WALLAROO_CLUSTER
wallarooaks
The name of the Kubernetes cluster that Wallaroo is installed to.
WALLAROO_SKU_TYPE
Basic
The Azure Kubernetes Service SKU type.
WALLAROO_VM_SIZE
Standard_D8s_v4
The VM type used for the standard Wallaroo cluster nodes.
POSTGRES_VM_SIZE
Standard_D8s_v4
The VM type used for the postgres nodepool.
ENGINELB_VM_SIZE
Standard_D8s_v4
The VM type used for the engine-lb nodepool.
ENGINE_VM_SIZE
Standard_F8s_v2
The VM type used for the engine nodepool.
Setup Environment Steps
Quick Setup Script
A sample script is available here, and creates an Azure Kubernetes environment ready for use with Wallaroo Enterprise. This script requires the following prerequisites listed above and uses the variables listed in Standard Setup Variables. Modify them as best fits your organization’s needs.
The following steps are geared towards a standard Linux or macOS system that supports the prerequisites listed above. Modify these steps based on your local environment.
Download the script above.
In a terminal window set the script status as execute with the command chmod +x wallaroo_enterprise_install_azure_expandable.bash.
Modify the script variables listed above based on your requirements.
Run the script with either bash wallaroo_enterprise_install_azure_expandable.bash or ./wallaroo_enterprise_install_azure_expandable.bash from the same directory as the script.
Manual Setup Guide
The following steps are guidelines to assist new users in setting up their Azure environment for Wallaroo.
The process uses the variables listed in Standard Setup Variables. Modify them as best fits your organization’s needs.
Setting up an Azure AKS environment is based on the Azure Kubernetes Service tutorial, streamlined to show the minimum steps in setting up your own Wallaroo environment in Azure.
Manual Guide
This follows these major steps:
Create an Azure Resource Group
Create an Azure Container Registry
Create the Azure Kubernetes Environment
Set Variables
The following are the variables used for the rest of the commands. Modify them as fits your organization’s needs.
To create an Azure Resource Group for Wallaroo in Microsoft Azure, use the following template:
az group create --name $WALLAROO_RESOURCE_GROUP --location $WALLAROO_GROUP_LOCATION
(Optional): Set the default Resource Group to the one recently created. This allows other Azure commands to automatically select this group for commands such as az aks list, etc.
az configure --defaults group={Resource Group Name}
For example:
az configure --defaults group=wallarooGroup
Create an Azure Container Registry
An Azure Container Registry(ACR) manages the container images for services includes Kubernetes. The template for setting up an Azure ACR that supports Wallaroo is the following:
Wallaroo Enterprise supports autoscaling and static nodepools. The following commands are used to create both to support the Wallaroo Enterprise cluster.
The following static nodepools are set up to support the Wallaroo cluster for postgres. Update the VM_SIZE based on your requirements.
For additional settings such as customizing the node pools for your Wallaroo Kubernetes cluster to customize the type of virtual machines used and other settings, see the Microsoft Azure documentation on using system node pools.
Download Wallaroo Kubernetes Configuration
Once the Kubernetes environment is complete, associate it with the local Kubernetes configuration by importing the credentials through the following template command:
az aks get-credentials --resource-group $WALLAROO_RESOURCE_GROUP --name $WALLAROO_CLUSTER
Verify the cluster is available through the kubectl get nodes command.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-engine-99896855-vmss000000 Ready agent 40m v1.23.8
aks-enginelb-54433467-vmss000000 Ready agent 48m v1.23.8
aks-mainpool-37402055-vmss000000 Ready agent 81m v1.23.8
aks-mainpool-37402055-vmss000001 Ready agent 81m v1.23.8
aks-mainpool-37402055-vmss000002 Ready agent 81m v1.23.8
aks-postgres-40215394-vmss000000 Ready agent 52m v1.23.8
Install Wallaroo
With the environment prepared, Wallaroo can now be installed.
How to set up your Wallaroo Enterprise GCP Environment
The following instructions are made to assist users set up their Google Cloud Platform (GCP) Kubernetes environment for running Wallaroo. These represent a recommended setup, but can be modified to fit your specific needs. In particular, these instructions will provision a GKE cluster with 56 CPUs in total. Please ensure that your project’s resource limits support that.
Quick Setup Script: Download a bash script to automatically set up the GCP environment through the Google Cloud Platform command line interface gcloud.
Manual Setup Guide: A list of the gcloud commands used to create the environment through manual commands.
Installation Flow
A typical installation of Wallaroo Enterprise follows this flow:
Create Environment: Create the environment to install Wallaroo that meets the system prerequisites.
Running Wallaroo has two sets of software requirements:
Environment Requirements: The following software must be installed on the environment that will be running Wallaroo version. Most are automatically available through the supported cloud providers.
Kubernetes Admin Requirements: The following software must be installed in the system where the Kubernetes environment is being managed - aka where kubectl and kots will be installed. This requires the following software be installed to manage the Kubernetes environment:
Organizations that intend to stop and restart their Kubernetes environment on an intentional or regular basis are recommended to use a single availability zone for their nodes. This minimizes issues such as persistent volumes in different availability zones, etc.
Organizations that intend to use Wallaroo Enterprise in a high availability cluster are encouraged to follow best practices including using separate availability zones for redundancy, etc.
Custom Configurations
Wallaroo can be configured with custom installations depending on your organization’s needs. The following options are available:
Taints and Tolerances Guide: How to configure Wallaroo for specific taints and tolerances so ensure that only Wallaroo services are running in specific nodes.
The name of the Google Project used for the Wallaroo instance.
WALLAROO_CLUSTER
wallaroo
The name of the Kubernetes cluster for the Wallaroo instance.
WALLAROO_GCP_REGION
us-central1
The region the Kubernetes environment is installed to. Update this to your GCP Computer Engine region.
WALLAROO_NODE_LOCATION
us-central1-f
The location the Kubernetes nodes are installed to. Update this to your GCP Compute Engine Zone.
WALLAROO_GCP_NETWORK_NAME
wallaroo-network
The Google network used with the Kubernetes environment.
WALLAROO_GCP_SUBNETWORK_NAME
wallaroo-subnet-1
The Google network subnet used with the Kubernets environment.
DEFAULT_VM_SIZE
e2-standard-8
The VM type used for the default nodepool.
POSTGRES_VM_SIZE
n2-standard-8
The VM type used for the postgres nodepool.
ENGINELB_VM_SIZE
c2-standard-8
The VM type used for the engine-lb nodepool.
ENGINE_VM_SIZE
c2-standard-8
The VM type used for the engine nodepool.
Quick Setup Script
A sample script is available here, and creates a Google Kubernetes Engine cluster ready for use with Wallaroo Enterprise. This script requires the prerequisites listed above and uses the variables as listed in Standard Setup Variables.
The following steps are geared towards a standard Linux or macOS system that supports the prerequisites listed above. Modify these steps based on your local environment.
Download the script above.
In a terminal window set the script status as execute with the command chmod +x bash wallaroo_enterprise_gcp_expandable.bash.
Modify the script variables listed above based on your requirements.
Run the script with either bash wallaroo_enterprise_gcp_expandable.bash or ./wallaroo_enterprise_gcp_expandable.bash from the same directory as the script.
Set Variables
The following are the variables used in the environment setup process. Modify them as best fits your organization’s needs.
The following steps are guidelines to assist new users in setting up their GCP environment for Wallaroo. The variables used in the commands are as listed in Standard Setup Variables. Feel free to replace these with ones that match your needs.
See the Google Cloud SDK for full details on commands and settings.
Create a GCP Network
First create a GCP network that is used to connect to the cluster with the gcloud compute networks create command. For more information, see the gcloud compute networks create page.
Once the network is created, the gcloud container clusters create command is used to create a cluster. For more information see the gcloud container clusters create page.
The following is a recommended format, replacing the {} listed variables based on your setup. For Google GKE containerd is enabled by default.
The command can take several minutes to complete based on the size and complexity of the clusters. Verify the process is complete with the clusters list command:
gcloud container clusters list
Wallaroo Enterprise Nodepools
The following static nodepools can be set based on your organizations requirements. Adjust the settings or names based on your requirements.
The following autoscaling nodepools are used for the engine load balancers and Wallaroo engine. Again, replace names and virtual machine types based on your organizations requirements.
To verify the Kubernetes credentials for your cluster have been installed locally, use the kubectl get nodes command. This will display the nodes in the cluster as demonstrated below:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-wallaroo-default-pool-863f02db-7xd4 Ready <none> 39m v1.21.6-gke.1503
gke-wallaroo-default-pool-863f02db-8j2d Ready <none> 39m v1.21.6-gke.1503
gke-wallaroo-default-pool-863f02db-hn06 Ready <none> 39m v1.21.6-gke.1503
gke-wallaroo-engine-3946eaca-4l3s Ready <none> 89s v1.21.6-gke.1503
gke-wallaroo-engine-lb-2e33a27f-64wb Ready <none> 26m v1.21.6-gke.1503
gke-wallaroo-postgres-d22d73d3-5qp5 Ready <none> 28m v1.21.6-gke.1503
Install Wallaroo
With the environment prepared, Wallaroo can now be installed.
What does the error ‘Insufficient project quota to satisfy request: resource “CPUS_ALL_REGIONS”’ mean?
Make sure that the Compute Engine Zone and Region are properly set based on your organization’s requirements. The instructions above default to us-central1, so change that zone to install your Wallaroo instance in the correct location.
3.2 - Wallaroo Enterprise Install Guides
How to set up Wallaroo Enterprise
This guide is targeted towards system administrators and data scientists who want to work with the easiest, fastest, and free method of running your own machine learning models.
A typical installation of Wallaroo follows this process:
Step
Description
Average Setup Time
Setup Environment
Create an environment that meets the Wallaroo prerequisites
30 minutes
Current Step:Install Wallaroo
Install Wallaroo into a prepared environment
15 minutes
Configure Wallaroo
Update Wallaroo post-install
Variable
Some knowledge of the following will be useful in working with this guide:
Working knowledge of Linux distributions, particularly Ubuntu.
Either Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure experience. For more information, see the Wallaroo Enterprise Environment Setup Guides guides.
Working knowledge of Kubernetes, mainly kubectl and kots.
For more information, Contact Us for additional details.
3.2.1 - How to Install Wallaroo Enterprise in an Air Gap Environment
Average Install Time
45 minutes depending on system performance and network connections.
Organizations that require Wallaroo be installed into an “air gap” environment - where the Wallaroo instance does not connect to the public Internet - can use these instructions to install Wallaroo into an existing Kubernetes cluster.
This guide assumes knowledge of how to use Kubernetes and work with internal clusters. The following conditions must be completed before starting an air gap installation of Wallaroo:
A Kubernetes cluster is installed and meets the prerequisites listed below.
A private container registry is available to the cluster, along with push and read credentials for that registry. This service is required for the air gap installation process to have images pushed and pulled for the installation. For examples on setting up a private container registry service, see the Docker Documentation “Deploy a registry server”. See Example Registry Service Install for an example of installing an unsecure private registry for testing purposes. For more details on setting up a container registry in a cloud environment, see the related documentation for your preferred cloud provider:
Before installing Wallaroo version, verify that the following hardware and software requirements are met.
Environment Hardware Requirements
The following system requirements are required for the minimum settings for running Wallaroo in a Kubernetes cloud cluster.
Minimum number of nodes: 4
Minimum Number of CPU Cores: 8
Minimum RAM per node: 16 GB
Minimum Storage: A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.
Wallaroo recommends at least 16 cores total to enable all services. At less than 16 cores, services will have to be disabled to allow basic functionality as detailed in this table.
The following network requirements are required for the minimum settings for running Wallaroo:
For Wallaroo Enterprise users: 200 IP addresses are required to be allocated per cloud environment.
For Wallaroo Community users: 98 IP addresses are required to be allocated per cloud environment.
DNS services integration is required for Wallaroo Enterprise edition. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.
DNS services integration is required to provide access to the various supporting services that are part of the Wallaroo instance. These include:
Simplified user authentication and management.
Centralized services for accessing the Wallaroo Dashboard, Wallaroo SDK and Authentication.
Collaboration features allowing teams to work together.
Managed security, auditing and traceability.
Environment Software Requirements
Running Wallaroo has two sets of software requirements:
Environment Requirements: The following software must be installed on the environment that will be running Wallaroo version. Most are automatically available through the supported cloud providers.
Kubernetes Admin Requirements: The following software must be installed in the system where the Kubernetes environment is being managed - aka where kubectl and kots will be installed. This requires the following software be installed to manage the Kubernetes environment:
Running Wallaroo has two sets of software requirements:
Environment Requirements: The following software must be installed on the environment that will be running Wallaroo version. Most are automatically available through the supported cloud providers.
Kubernetes Admin Requirements: The following software must be installed in the system where the Kubernetes environment is being managed - aka where kubectl and either kots or helm will be installed. This requires the following software be installed to manage the Kubernetes environment:
Wallaroo uses different nodes for various services, which can be assigned to a different node pool to contain resources separate from other nodes. The following nodes selectors can be configured:
ML Engine node selector
ML Engine Load Balance node selector
Database Node Selector
Grafana node selector
Prometheus node selector
Each Lab * Node Selector
Install Instructions
The installation is broken into the following major processes:
The Wallaroo delivery team the URL and password to your organization’s License and Air Gap Download page. The following links are provided:
(A)Wallaroo Enterprise License File: The Wallaroo enterprise license file for this account. This is downloaded as a yaml file.
(B)Wallaroo Airgap Installation File: The air gap installation file that includes the necessary containers for the Wallaroo installation. This is typically about 6 GB in size. By selecting the link icon, the Wallaroo Airgap Installation File URL will be copied to the clipboard that can be used for curl or similar download commands. This file is typically downloaded as wallaroo.airgap.
IMPORTANT NOTE
If the Wallaroo Air Gap Bundle link is not available, contact your Wallaroo support representative.
(C)KOTS CLI: The installation files to install kots into the node that manages the Kubernetes cluster. This file is typically downloaded as kots_linux_amd64.tar.gz.
(D)KOTS Airgap Bundle: A set of files required by the Kubernetes environment to install Wallaroo via the air gap method. This file is typically downloaded as kotsadm.tar.gz.
Download these files either through the provided License and Airgap Download page, or by copying the links from the page and using the following command line commands into node performing the air gap installation with curl as follows:
Wallaroo Enterprise License File:
curl -LO {Link to Wallaroo Enterprise License File}
Airgap Installation File. Note the use of the -Lo option to download the Wallaroo air gap file as wallaroo.airgap, and the use of the single quotes around the Wallaroo Air Gap Installation File URL.
Place these files onto the air gap server or node that administrates the Kubernetes cluster. Once these files are on the node, the cluster can be air gapped and the required software installed through the next steps.
Install Kots
Install kots into the node managing the Kubernetes cluster with the following commands:
Extract the archive:
tar zxvf kots_linux_amd64.tar.gz kots
Install kots to the /usr/local/bin directory. Adjust this directory to match the location of the kubectl command.
sudo mv kots /usr/local/bin/kubectl-kots
Verify the kots installation by checking the version. The result should be similar to the following:
kubectl kots version
Replicated KOTS 1.91.3
Install the Kots Admin Console
This step will Extract the KOTS Admin Console container images and push them into a private registry. Registry credentials provided in this step must have push access. These credentials will not be stored anywhere or reused later.
This requires the following:
Private Registry Host: The URL of the private registry host used by the Kubernetes cluster.
Private Registry Port: The port of the private registry used by the Kubernetes cluster (5000 by default).
KOTS Airgap Bundle (default: kotsadm.tar.gz): Downloaded as part of Download Assets step.
Registry Push Username: The username with push access to the private registry.
Registry Push Password: The password of the registry user with push access to the private registry.
Adjust the command based on your organizations registry setup.
Install Wallaroo Airgap
This step will install the Wallaroo air gap file into the Kubernetes cluster through the Kots Admin images.
Registry credentials provided in this step only need to have read access, and they will be stored in a Kubernetes secret in the same namespace where Admin Console will be installed. These credentials will be used to pull the images, and will be automatically created as an imagePullSecret on all of the Admin Console pods.
This requires the following:
Private Registry Host: The URL of the private registry host used by the Kubernetes cluster.
Private Registry Port: The port of the private registry used by the Kubernetes cluster (5000 by default).
Wallaroo Namespace (default: wallaroo): The kubernetes namespace used to install the Wallaroo isntance.
Wallaroo Airgap Installation File (default: wallaroo.airgap): Downloaded as part of Download Assets step.
Wallaroo License File: Downloaded as part of Download Assets step.
Registry Read Username: The username with read access to the private registry.
Registry Read Password: The password of the registry user with read access to the private registry.
The command will take the following format. Note that the option --license-file {Wallaroo License File} is required. This will point to the license REQUIRED for an air gap installation.
The following flags can be added to speed up the configuration process:
--shared-password {Wallaroo Admin Dashboard Password}: The password used to access the Wallaroo Admin Dashboard.
--config-values config.yaml: Sets up the Wallaroo instance configuration based on the supplied yaml file.
--no-port-forward: Does not forward port 8800 for use.
--skip-preflights: Skip the standard preflight checks and launch the Wallaroo instance.
For example, the following will install Wallaroo Enterprise into the namespace wallaroo using the provided license file, using the shared password wallaroo and skipping the preflight checks:
Preflight checks will verify that the Wallaroo instance meets the prerequisites. If any fail, check your Kubernetes environment and verify they are in alignment.
Preflight checks will be skipped if Wallaroo was installed with the --skip-preflights option.
Wallaroo Admin Console
If no license file was provided through the command line, it can be provided through the Wallaroo Admin Console on port 8800. To access the Wallaroo Admin Console, some method of port forwarding through the jump box will have to be configured to the air gapped cluster.
Status Checks
While the installer allocates resources and deploys workloads, the status page will show as Missing or Unavailable. If it stays in this state for more than twenty minutes, proceed to troubleshooting or contact Wallaroo technical support.
Once the application has become ready, the status indication will turn green and ready Ready.
Troubleshooting
At any time, the administration console can create troubleshooting bundles for Wallaroo technical support to assess product health and help with problems. Support bundles contain logs and configuration files which can be examined before downloading and transmitting to Wallaroo. The console also has a configurable redaction mechanism in cases where sensitive information such as passwords, tokens, or PII (Personally Identifiable Information) need to be removed from logs in the bundle.
To manage support bundles:
Log into the administration console.
Select the Troubleshoot tab.
Select Analyze Wallaroo.
Select Download bundle to save the bundle file as a compressed archive.
Send the file to Wallaroo technical support.
At any time, any existing bundle can be examined and downloaded from the Troubleshoot tab.
Example Registry Service Install
The following example demonstrates how to set up an unsecure local registry service that can be used for testing. This process is not advised for production systems, and it only provided as an example for testing the air gap install process. This example uses an Ubuntu 20.04 instance as the installation environment.
This example assumes that the containerd service is installed and used by the Kubernetes cluster.
Private Container Registry Service Install Process
To install a demo container registry service on an Ubuntu 20.04 instance:
Update the containerd service as follows, replacing YOUR-HOST-HERE with the hostname of the registry service configured above. Comment out any existing registry entries and replace with the new insecure registry service:
3.2.2 - How to Install Wallaroo Enterprise on Cloud or Single Node Linux Platforms
Organizations that use cloud services such as Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure can install Wallaroo Enterprise. These instructions also work with Single Node Linux based installations.
Installation Flow
A typical installation of Wallaroo Enterprise follows this flow:
Create Environment: Create the environment to install Wallaroo that meets the system prerequisites.
Configure DNS: Configure DNS services and the Wallaroo instance for your organization’s use.
Before installation, the following prerequisites must be met:
Have a Wallaroo Enterprise license file. For more information, you can request a demonstration.
Set up a cloud Kubernetes environment that meets the requirements. Clusters must meet the following minimum specifications:
Minimum number of nodes: 4
Minimum Number of CPU Cores: 8
Minimum RAM: 16 GB
A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.
Runtime: containerd is required.
DNS services for integrating your Wallaroo Enterprise instance. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.
Environment Software Requirements
Running Wallaroo has two sets of software requirements:
Environment Requirements: The following software must be installed on the environment that will be running Wallaroo version. Most are automatically available through the supported cloud providers.
Kubernetes Admin Requirements: The following software must be installed in the system where the Kubernetes environment is being managed - aka where kubectl and kots will be installed. This requires the following software be installed to manage the Kubernetes environment:
For details on setting up your Kubernetes cloud cluster, see the How to Set Up Your Wallaroo Cloud Environment guide. Users will have to manage the network settings, port forward, and other settings for their cloud environment.
Install Wallaroo
Wallaroo Enterprise can be installed either interactively or automatically through the kubectl and kots applications.
Automated Install
To automatically install Wallaroo into the namespace wallaroo, specify the administrative password and the license file during the installation as in the following format with the following variables:
NAMESPACE: The namespace for the Wallaroo Enterprise install, typically wallaroo.
LICENSEFILE: The location of the Wallaroo Enterprise license file.
SHAREDPASSWORD: The password of for the Wallaroo Administrative Dashboard.
The Interactive Install process allows users to adjust the configuration settings before Wallaroo is deployed. It requires users be able to access the Wallaroo Administrative Dashboard through a browser, typically on port 8080.
IMPORTANT NOTE: Users who install Wallaroo through another node such as in the single node installation can port use SSH tunneling to access the Wallaroo Administrative Dashboard. For example:
ssh IP -L8800:localhost:8800
Install the Wallaroo Enterprise Edition using kots install wallaroo/ee, specifying the namespace to install Wallaroo into. For example, if wallaroo is the namespace, then the command is:
Wallaroo Enterprise Edition will be downloaded and installed into your Kubernetes environment in the namespace specified. When prompted, set the default password for the Wallaroo environment. When complete, Wallaroo Enterprise Edition will display the URL for the Admin Console, and how to end the Admin Console from running.
• Deploying Admin Console
• Creating namespace ✓
• Waiting for datastore to be ready ✓
Enter a new password to be used for the Admin Console: •••••••••••••
• Waiting for Admin Console to be ready ✓
• Press Ctrl+C to exit• Go to http://localhost:8800 to access the Admin Console
Configuration
Once installed, Wallaroo will continue to run until terminated.
To relaunch the Wallaroo Administrative Dashboard and make changes or updates, use the following command:
kubectl-kots admin-console --namespace wallaroo
DNS Services
Wallaroo Enterprise requires integration into your organizations DNS services, and is completed in the next step.
Install Wallaroo
With the environment prepared, Wallaroo can now be installed.
3.2.3 - How to Install Wallaroo Enterprise via Helm
Organizations that prefer to use the Helm package manager for Kubernetes can install Wallaroo versions 2022.4 and above via Helm.
The following procedure demonstrates how to install Wallaroo using Helm. For more information on settings and options for a Helm based install, see the Wallaroo Helm Reference Guides.
3.2.3.1 - Wallaroo Helm Standard Cloud Based Install
The following instructions detail how to install Wallaroo Enterprise via Helm for Kubernetes cloud environments such as Microsoft Azure, Amazon Web Service, and Google Cloud Platform.
IMPORTANT NOTE
These instructions are for Wallaroo Enterprise only.
Before installing Wallaroo version, verify that the following hardware and software requirements are met.
Wallaroo Version
Helm installation is supported for Wallaroo version 2022.4 or later.
IMPORTANT NOTE
Wallaroo releases before version 2022.4 are not supported for Helm installations.
Environment Hardware Requirements
The following system requirements are required for the minimum settings for running Wallaroo in a Kubernetes cloud cluster.
Minimum number of nodes: 4
Minimum Number of CPU Cores: 8
Minimum RAM per node: 16 GB
Minimum Storage: A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.
Wallaroo recommends at least 16 cores total to enable all services. At less than 16 cores, services will have to be disabled to allow basic functionality as detailed in this table.
The following network requirements are required for the minimum settings for running Wallaroo:
For Wallaroo Enterprise users: 200 IP addresses are required to be allocated per cloud environment.
For Wallaroo Community users: 98 IP addresses are required to be allocated per cloud environment.
DNS services integration is required for Wallaroo Enterprise edition. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.
DNS services integration is required to provide access to the various supporting services that are part of the Wallaroo instance. These include:
Simplified user authentication and management.
Centralized services for accessing the Wallaroo Dashboard, Wallaroo SDK and Authentication.
Collaboration features allowing teams to work together.
Managed security, auditing and traceability.
Environment Software Requirements
Running Wallaroo has two sets of software requirements:
Environment Requirements: The following software must be installed on the environment that will be running Wallaroo version. Most are automatically available through the supported cloud providers.
Kubernetes Admin Requirements: The following software must be installed in the system where the Kubernetes environment is being managed - aka where kubectl will be installed. This requires the following software be installed to manage the Kubernetes environment:
Wallaroo uses different nodes for various services, which can be assigned to a different node pool to contain resources separate from other nodes. The following nodes selectors can be configured:
ML Engine node selector
ML Engine Load Balance node selector
Database Node Selector
Grafana node selector
Prometheus node selector
Each Lab * Node Selector
Kubernetes Installation Instructions
This sample Helm installation procedure has the following steps:
Preflight and Support Bundle configuration files: The files preflight.yaml and support-bundle.yaml are used in the commands below to complete the preflight process and generate the support bundle package as needed for troubleshooting needs.
Preflight verification command: The commands to verify that the Kubernetes environment meets the requirements for the Wallaroo install.
Install Wallaroo Command: Instructions on installations into the Kubernetes environment using Helm through the Wallaroo container registry.
The following steps are used with these command and configuration files to install Wallaroo Enterprise via Helm.
Registration Login
The first step in the Wallaroo installation process via Helm is to connect to the Kubernetes environment that will host the Wallaroo Enterprise instance and login into the Wallaroo container registry through the command provided by the Wallaroo support staff. The command will take the following format, replacing $YOURUSERNAME and $YOURPASSWORD with the respective username and password provided.
IMPORTANT NOTE
The preflight test is not programmatically enforced during installation via Helm and should be performed manually before installation. If the Kubernetes environment does not meet the requirements the Wallaroo installation may fail or perform erratically. Please verify that all preflight test run successfully before proceeding to install Wallaroo.
Preflight verification is performed with the following command, using the preflight.yaml configuration file provided by the Wallaroo support representative as listed above.
If successful, the tests will show PASS for each preflight requirement as in the following example:
name: cluster-resources status: running completed: 0 total: 2name: cluster-resources status: completed completed: 1 total: 2name: cluster-info status: running completed: 1 total: 2name: cluster-info status: completed completed: 2 total: 2 --- PASS Required Kubernetes Version
--- Your cluster meets the recommended and required versions of Kubernetes.
--- PASS Container Runtime
--- Containerd container runtime was found.
--- PASS Check Kubernetes environment.
--- KURL is a supported distribution
--- PASS Cluster Resources
--- Cluster resources are satisfactory
--- PASS Every node in the cluster must have at least 12Gi of memory
--- All nodes have at least 12 GB of memory capacity
--- PASS Every node in the cluster must have at least 8 cpus allocatable.
--- All nodes have at least 8 CPU capacity
--- PASS wallaroo
PASS
Install Wallaroo
With the preflight checks and prerequisites met, Wallaroo can be installed via Helm through the following process:
Create namespace. By default, the namespace wallaroo is used:
Set the TLS certificate secret in the Kubernetes environment:
Create the certificate and private key. It is recommended to name it after the domain name of your Wallaroo instance. For example: wallaroo.example.com. For production environments, organizations are recommended to use certificates from their certificate authority. Note that the Wallaroo SDK will not connect from an external connection without valid certificates. For more information on using DNS settings and certificates, see the Wallaroo DNS Integration Guide.
Create the Kubernetes secret from the certificates created in the previous step, replacing $TLSCONFIG with the name of the Kubernetes secret. Store the secret name for a the step Configure local values file.
Configure local values file: The default Helm install of Wallaroo contains various default settings. The local values file overwrites values based on the organization needs. The following represents the minimum mandatory values for a Wallaroo installation using certificates and the default LoadBalancer for a cloud Kubernetes cluster. The configuration details below is saved as local-values.yaml for these examples.
domainPrefix and domainSuffix: Used to set the DNS settings for the Wallaroo instance. For more information, see the Wallaroo DNS Integration Guide.
deploymentStage and custTlsSecretName: These are set for use with the Kubernetes secret created in the previous step. External connections through the Wallaroo SDK require valid certificates.
generate_secrets: Secrets for administrative and other users can be generated by the Helm install process, or set manually. This setting scrambles the passwords during installation.
apilb: Sets the apilb service options including the following:
serviceType: LoadBalancer: Uses the default LoadBalancer setting for the Kubernetes cloud service the Wallaroo instance is installed into. Replace with the specific service connection settings as required.
external_inference_endpoints_enabled: true: This setting is required for performing external SDK inferences to a Wallaroo instance. For more information, see the Wallaroo Model Endpoints Guide
domainPrefix: doc-test
domainSuffix: example.com # Ie, main URL is https://ds.big.corp, then https://ds.keycloak.big.corp etc# Provide a TLS secret for DNS domain.# Required for connecting the external SDK to a Wallaroo instancedeploymentStage: cust # Must be provided if `custTlsSecretName` is added.custTlsSecretName: my-tls-secrets
generate_secrets: trueapilb:
# Generic loadbalancer for the cluster. Replace with the specific notebalancer for the cloud service.serviceType: LoadBalancer
# Required to perform remote inferences either through the SDK or the APIexternal_inference_endpoints_enabled: truedashboard:
# Sets the Wallaroo instance nameclientName: "YOUR COMPANY NAME HERE"# Adds authentication to the Wallaroo Dashboardauth:
enabled: true
The resources used by the Wallaroo services can be modified. For full details, see the Wallaroo Helm References Guides. The following example shows limiting the apilb service:
domainPrefix: doc-test
domainSuffix: example.com # Ie, main URL is https://ds.big.corp, then https://ds.keycloak.big.corp etc# Provide a TLS secret for DNS domain.# Required for connecting the external SDK to a Wallaroo instancedeploymentStage: cust # Must be provided if `custTlsSecretName` is added.custTlsSecretName: my-tls-certs
generate_secrets: trueapilb:
# Generic loadbalancer for the cluster. Replace with the specific notebalancer for the cloud service.serviceType: LoadBalancer
# Required to perform remote inferences either through the SDK or the APIexternal_inference_endpoints_enabled: trueresources:
limits:
cpu: 0.5requests:
cpu: 0.1dashboard:
# Sets the Wallaroo instance nameclientName: "YOUR COMPANY NAME HERE"# Adds authentication to the Wallaroo Dashboardauth:
enabled: true
Install Wallaroo: The Wallaroo support representative will provide the installation command for the Helm install that will use the Wallaroo container registry. This assumes that the preflight checks were successful. This command uses the following format:
Verify the Installation: Once the installation is complete, verify the installation with the helm test $RELEASE command. With the settings above, this would be:
helm test wallaroo
A successful installation will resemble the following:
NAME: wallaroo
LAST DEPLOYED: Wed Dec 21 09:15:23 2022NAMESPACE: wallaroo
STATUS: deployed
REVISION: 1TEST SUITE: wallaroo-fluent-bit-test-connection
Last Started: Wed Dec 21 11:58:34 2022Last Completed: Wed Dec 21 11:58:37 2022Phase: Succeeded
TEST SUITE: wallaroo-test-connections-hook
Last Started: Wed Dec 21 11:58:37 2022Last Completed: Wed Dec 21 11:58:41 2022Phase: Succeeded
TEST SUITE: wallaroo-test-objects-hook
Last Started: Wed Dec 21 11:58:41 2022Last Completed: Wed Dec 21 11:58:53 2022Phase: Succeeded
At this point, the installation is complete and can be accessed through the fully qualified domain names set in the installation process above. Verify that the DNS settings are accurate before attempting to connect to the Wallaroo instance. For more information, see the Wallaroo DNS Integration Guide.
If issues are detected in the Wallaroo instance, a support bundle file can be generated using the support-bundle.yaml file provided by the Wallaroo support representative. This creates a collection of log files, configuration files and other details into a .zip file that can be submitted to the Wallaroo support team for review.
This support bundle is generated through the following command:
To uninstall Wallaroo via Helm, use the following command replacing the $RELEASE with the name of the release used to install Wallaroo. By default, this is wallaroo:
helm uninstall wallaroo
It is also recommended to remove the wallaroo namespace after the helm uninstall is complete.
IMPORTANT NOTE
Do not remove the Wallaroo namespace until after the helm uninstall is complete. Removing the namespace first can leave resources hanging and can cause issues when trying to reinstall Wallaroo via Helm.
kubectl delete namespace wallaroo
3.2.3.2 - Wallaroo Helm Reference Guides
The following guides include reference details related to installing Wallaroo via Helm.
3.2.3.2.1 - Wallaroo Helm Reference Table
Wallaroo
A Helm chart for the control plane for Wallaroo
The following table lists the configurable parameters of the Wallaroo chart and their default values.
Parameter
Description
Default
kubernetes_distribution
One of: aks, eks, gke, or kurl. May be safe to leave defaulted.
""
imageRegistry
imageRegistry where images are pulled from
"ghcr.io/wallaroolabs"
imageTag
imageTag that images default to - can be overridden for each component
"main"
assays.enabled
Controls the display of Assay data in the Dashboard
true
custTlsSecretName
Name of existing Kubernetes TLS type secret
""
deploymentStage
Deployment stage, must be set to “cust” when deployed
"dev"
custTlsCert
Customer provided certificate chain when deploymentStage is “cust”.
""
custTlsKey
Customer provided private key when deploymentStage is “cust”.
DNS prefix of Wallaroo endpoints, can be empty for none
"xxx"
domainSuffix
DNS suffix of Wallaroo endpoints, MUST be provided
"yyy"
externalIpOverride
Used in cases where we can’t accurately determine our external, inbound IP address. Normally “”.
""
imagePullPolicy
Global policy saying when K8s pulls images: Always, Never, or IfNotPresent.
"Always"
wallarooSecretName
Secret name for pulling Wallaroo images
"regcred"
apilb.nodeSelector
standard node selector for API-LB
{}
apilb.annotations
Annotations for api-lb service
{}
apilb.serviceType
Service type of api-lb service
"ClusterIP"
plateau.enabled
Enable Plateau deployment
true
plateau.diskSize
Disk space to allocate. Smaller than 100Gi is not recommended.
"100Gi"
telemetry.enabled
Used only for our CE product. Leave disabled for EE/Helm installs.
false
dashboard.enabled
Enable dashboard service
true
dashboard.clientName
Customer display name which appears at the top of the dashboard window.
"Fitzroy Macropods, LLC"
minio.persistence.size
Minio model storage disk size. Smaller than 10Gi is not recommended.
"10Gi"
helmTests.nodeSelector
When helm test is run, this selector places the test pods.
{}
pythonAPIServer.enabled
This service is used for model conversion.
false
explainabilityServer.enabled
Enable the model explainability service
false
3.2.3.2.2 - Wallaroo Helm Reference Details
post_delete_hook
This hook runs when you do helm uninstall unless … - you give –no-hooks to helm - you set the enable flag to False at INSTALL time.
imageRegistry
Registry and Tag portion of Wallaroo images. Third party images are not included. Tag is computed at runtime and overridden. In online Helm installs, these should not be touched; in airgap Helm installs imageRegistry must be overridden to local registry.
generate_secrets
If true, generate random secrets for several services at install time. If false, use the generic defaults listed here, which can also be overridden by caller.
assays
This is a (currently) Dashboard-specific feature flag to control the display of Assays.
custTlsSecretName
To provide TLS certificates, (1) set deploymentStage to “cust”, then (2) provide EITHER the name of an existing Kubernetes TLS secret in custTlsSecret OR provide base64 encoded secrets in custTlsCert and custTlsKey.
domainPrefix
DNS specification for our named external service endpoints. To form URLs, we concatenate the optional domainPrefix, the service name in question, and then the domainSuffix. Their values are based on license, type, and customer config inputs. They MUST be overriden per install via helm values, or by Replicated.
Community
prefix/suffix in license
domainPrefix
domainSuffix
dashboard_fqdn
thing_fqdn (thing = jup, kc, etc)
""
wallaroo.community
(never)
(never)
cust123
wallaroo.community
cust123.wallaroo.community
cust123.thing.wallaroo.community
Enterprise et al
prefix/suffix from config
domainPrefix
domainSuffix
dashboard_fqdn
thing_fqdn (thing = jup, kc, etc)
""
wl.bigco
wl.bigco
thing.wl.bigco
cust123
wl.bigco
cust123.wl.bigco
cust123.thing.wl.bigco
wallarooSecretName
In online Helm installs, an image pull secret is created and this is its name. The secret allows the Kubernetes node to pull images from proxy.replicated.com. In airgap Helm installs, a local Secret of type docker-registry must be created and this value set to its name.
privateModelRegistry
If the customer has specified a private model container registry, the enable flag will reflect and the secret will be populated. registry, username, and password are mandatory. email is optional. registry is of the form “hostname:port”.
apilb
Main ingress LB for Wallaroo services. The Kubernetes Ingress object is not used, instead we deploy a single Envoy load balancer with a single IP in all cases, which serves: TLS termination, authentication (JWT) checking, and both host based and path based application routing. Customer should be aware of two values in particular. api.serviceType defaults to ClusterIP. If api.serviceType is set to LoadBalancer, cloud services will allocate a hosted LB service, in which case the apilb.annotations should be provided, in order to pass configuration such as “internal” or “external” to the cloud service. Example: apilb: serviceType: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-internal: “true”
plateau
Plateau is a low-profile fixed-footprint log processor / event store for fast storage of inference results. The amount of disk space provisioned is adjustable. Smaller than “100Gi” is not recommended for performance reasons.
pythonAPIServer
Model conversion is an optional service that allows converting non-onnx models (keras, sklearn, and xgboost) to onnx and adding them to your pipeline, without extensive manual conversion or processing steps. This allows more rapid iteration over models or experiments.
4 - Installation Troubleshooting Guide
Troubleshooting
I’m Getting a Timeout Error
Depending on the connection and resources, the installation process may time out. If that occurs, use the --wait-duration flag to provide additional time. The time must be provided in Go duration format (for example: 60s, 1m, etc). The following example extends the wait duration to 10 minutes:
If your system does not meet all of the preflight requirements, the installation process may fail when performing an automated installation. It is highly recommended to install Wallaroo on a system that meets all requirements or else performance will be degraded.
To ignore preflight checks, use the --skip-preflights flag, as in the following example:
When Launching JupyterHb, I get a Server 500 error
If you shut down and restart a Wallaroo instance in a new environment or change the IP address, some settings may not be updated. Run the following command to restart the deployment process and update the settings to match the current environment. Note that the namespace wallaroo is used - modify this to match the environment where Wallaroo is installed.
The following guides demonstrate how to install Wallaroo with different options to best fit your organizations needs, and are meant to supplement the standard install guides.
5.1 - Install Wallaroo with Minimum Services
How to install Wallaroo with disabled services for lower core environments
Wallaroo requires at least 8 cores with 16 GB of RAM per node, and runs best in an environment with a total of 16 cores. Wallaroo recommends at least 16 cores total to enable all services. At less than 16 cores, services will have to be disabled to allow basic functionality as detailed in this table.
Note that even when disabling these services, Wallaroo performance may be impacted by the models, pipelines, and data used. The greater the size of the models and steps in a pipeline, the more resources will be required for Wallaroo to operate efficiently.
Cluster Size
< 8 core
8 core/48GB
16 core/48GB
32 core/48GB
Inference
✔
✔
✔
✔
Dashboard
✘
✔
✔
✔
Jupyter HUB/Lab
Single Lab
✘
✔
✔
✔
Multiple Labs
✘
✘
✔
✔
Prometheus
✘
✔
✔
✔
Alerting
✘
✘
✔
✔
Model Validation
✘
✘
✔
✔
Dashboard Graphs
✘
✔
✔
✔
Plateau
✘
✘
✔
✔
Model Insights
✘
✘
✔
✔
Python API
Model Conversion
✘
✔
✔
✔
To install Wallaroo with minimum services, a configuration file will be used as parts of the kots based installation. For full details on the Wallaroo installation process, see the Wallaroo Install Guides.
Organizations that share their Kubernetes environment with other applications may want to install Wallaroo to specific nodes in their cluster. The following guide demonstrates how to install a Wallaroo instance into specific nodes in the Kubernetes cluster.
Users who are familiar with Kubernetes clusters can skip ahead directly to the Install Steps.
Description
When installed into a Kubernetes cluster, Wallaroo will use available nodes to maximize its performance. Some organizations may use specific nodepools or nodes for specific applications.
One option is to use Kubernetes metadata to assign Node labels to nodes, then specify that Wallaroo can be installed into specific nodes that match that label. This can be done with specifying configuration options during the install process using the kots option --config-values={CONFIG YAML FILE}. For more information, see the kots set config documentation.
Install Steps
In this example, an instance of Wallaroo Community will be installed into a Kubernetes cluster that has four nodes assigned with the label wallaroo.ai/node=true;
kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-mainpool-18670167-vmss000000 Ready agent 84m v1.23.8
aks-wallarooai-12293194-vmss000000 Ready agent 75m v1.23.8
aks-wallarooai-12293194-vmss000001 Ready agent 75m v1.23.8
aks-wallarooai-12293194-vmss000002 Ready agent 75m v1.23.8
aks-wallarooai-12293194-vmss000003 Ready agent 75m v1.23.8
kubectl get nodes -l wallaroo.ai/node=trueNAME STATUS ROLES AGE VERSION
aks-wallarooai-12293194-vmss000000 Ready agent 75m v1.23.8
aks-wallarooai-12293194-vmss000001 Ready agent 75m v1.23.8
aks-wallarooai-12293194-vmss000002 Ready agent 75m v1.23.8
aks-wallarooai-12293194-vmss000003 Ready agent 75m v1.23.8
Create a kots configuration file and specify the label to use for installing nodes. For this example, we will use wallaroo.ai/node: "true" as the label. Any nodes with that label will be used by Wallaroo during the installation. For this example, this configuration is saved to the file test-node.yaml.
• Deploying Admin Console
• Creating namespace ✓
• Waiting for datastore to be ready ✓
• Waiting for Admin Console to be ready ✓
• Press Ctrl+C to exit• Go to http://localhost:8800 to access the Admin Console
Proceed with the installation as normal, including uploading the required license file, etc.
Once complete, verify Wallaroo was installed to specific nodes with the kubectl get pods command. The following shows the pods in the wallaroo namespace where the Wallaroo Community instance was installed, and the pods used for the deployed pipeline ccfraudpipeline.
kubectl get pods --all-namespaces -o=custom-columns=NAME:.metadata.name,Namespace:.metadata.namespace,Node:.spec.nodeName
For other instructions on how to deploy or configure a Wallaroo instance, see the Wallaroo Operations Guides.
5.3 - Taints and Tolerations Guide
Configure custom taints and toleration for a cluster for Wallaroo
Organizations can customize the taints and tolerances for their Kubernetes cluster running Wallaroo. Nodes in a Kubernetes cluster can have a taint applied to them. Any pod that does not have a toleration matching the taint can be rejected and will not be applied to that node.
This allows organizations to determine which pods can be accepted or rejected into specific nodes, reserving their Kubernetes resources for other services. Combined with the Install Wallaroo to Specific Nodes guide this ensures that Wallaroo pods are contained to specific cluster nodes, and prevents non-Wallaroo pods from being scheduled into the same nodes to reserve those resources for the Wallaroo instance.
In this example, the node Postgres has the taint wallaroo.ai/postgres=true:NoSchedule. The pod postgres has the tolerance wallaroo.ai/postgres:NoSchedule op=Exists, so it is scheduled into the node Postgres. The pod nginx has no tolerations, so it is not scheduled into the node Postgres.
The Wallaroo Enterprise Install Guides specify default taints applied to nodepools. These can be used to contain pod scheduling only to specific nodes where the pod tolerations match the nodes taints. By default, the following nodepools and their associated taints are created
After Wallaroo release September 2022 (Codename Cobra):
Nodepool
Taints
postgres
wallaroo.ai/postgres=true:NoSchedule
enginelb
wallaroo.ai/enginelb=true:NoSchedule
engine
wallaroo.ai/engine=true:NoSchedule
mainpool
N/A
Before Wallaroo release September 2022 (Code name Mustang and before)
Nodepool
Taints
postgres
wallaroo-postgres=true:NoSchedule
enginelb
wallaroo-enginelb=true:NoSchedule
engine
wallaroo-engine=true:NoSchedule
mainpool
N/A
The nodepool mainpool is not assigned any taints to allow other Kubernetes services to run as part of the cluster.
The taint wallaroo.ai/reserved=true:NoSchedule can be applied to other nodepools. This allows additional Wallaroo resources to be scheduled in those nodes while rejecting other pods that do not have a matching toleration.
Default Tolerations
By default, the following tolerations are applied for Wallaroo pods. Organizations can add a corresponding Any pod that does not contain a taint to match these tolerances will have the condition effect:NoSchedule for the specified node.
Toleration key for all Wallaroo pods
wallaroo.ai/reserved
Engine toleration key
wallaroo.ai/engine
Engine LB toleration key
wallaroo.ai/enginelb
Postgres toleration key
wallaroo.ai/postgres
Note that these taint values are applied to the nodepools as part of the Wallaroo Enterprise Setup guides. They are not typically set up or required for Wallaroo Community instances.
Custom Tolerations
To customize the tolerations applied to Wallaroo nodes, the following prerequisites must be met:
Access to the Kubernetes environment running the Wallaroo instances.
Have kubectl and kots installed and connected to the Kubernetes environment.
From a terminal with kubectl and kots installed and connected to the Kubernetes environment, run:
kubectl kots admin-console --namespace wallaroo
This will provide access to the Wallaroo Administrative Dashboard through http://localhost:8800:
• Press Ctrl+C to exit • Go to http://localhost:8800 to access the Admin Console
Launch a browser and connect to http://localhost:8800.
Enter the password created during the Wallaroo Install process. The Wallaroo Administrative Dashboard will now be available.
From the Wallaroo Administrative Dashboard, select Config -> Taints and Tolerations.
Set the custom tolerations as required by your organization. The following nodes and tolerations can be changed:
Toleration key for all Wallaroo pods
Default value: wallaroo.ai/reserved
Engine toleration key
Default value: wallaroo.ai/engine
Engine LB toleration key
Default value: wallaroo.ai/enginelb
Postgres toleration key
Default value: wallaroo.ai/postgres
6 - How to Uninstall Wallaroo
How to Uninstall Wallaroo
If the install procedure for Wallaroo goes awry, one option is to uninstall the incomplete Wallaroo installation and start again. The following procedure will remove Wallaroo from a Kubernetes cluster.
WARNING
This procedure will delete all Wallaroo data from the Kubernetes environment. Make sure that all data is backed up before proceeding with the uninstall process.
Remove all Kubernetes namespaces with the kubectl delete namespaces {list of namespaces}command except the following : default, kube* (any namespaces with kube before it).
For example, in the following environment wallaroo, model1 and model2 would be deleted with the following:
-> kubectl get namespaces
NAME STATUS AGE
default Active 7d4h
kube-node-lease Active 7d4h
kube-public Active 7d4h
model1 Active 4h23m
model2 Active 4h23m
wallaroo Active 3d6h
kubectl delete namespaces wallaroo model1 model2
Use the following bash script or run the commands individually:
