.

.

Wallaroo Install Guides

How to set up Wallaroo in the minimum number of steps

This guide is targeted towards system administrators and data scientists who want to work with the easiest, fastest, and free method of running your own machine learning models. Some knowledge of the following will be useful in working with this guide:

  • Working knowledge of Linux distributions, particularly Ubuntu.
  • Either Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure experience.
  • Working knowledge of Kubernetes, mainly kubectl and kots.
  • Desire to see your models working in the cloud.

Select either Wallaroo Community or Enterprise for the general steps on how to install Wallaroo. Organizations that already have a prepared environment can skip directly to the respective installation guide for their edition of Wallaroo.

Step   Description      Average Setup Time   
Setup Environment   Create an environment that meets the Wallaroo prerequisites   30 minutes
Install Wallaroo   Install Wallaroo into a prepared environment   15 minutes
Step   Description      Average Setup Time   
Setup Environment   Create an environment that meets the Wallaroo prerequisites   30 minutes
Install Wallaroo   Install Wallaroo into a prepared environment   15 minutes
Configure Wallaroo   Update Wallaroo post-install with DNS integration and user setup.   Variable

Note the differences between the Wallaroo Community and Wallaroo Enterprise. Wallaroo Community is limited to a maximum of 32 cores and 2 pipelines. For organizations that require more resources, the Wallaroo Enterprise Edition may be more appropriate.

For more information, Contact Us so we can help you find out which is better for your needs.

1 - Wallaroo Prerequisites Guide

Software and other local system requirements before installing Wallaroo

General Time to Completion: 30 minutes.

Before installing Wallaroo version, verify that the following hardware and software requirements are met.

Environment Requirements

Environment Hardware Requirements

The following system requirements are required for the minimum settings for running Wallaroo in a Kubernetes cloud cluster.

  • Minimum number of nodes: 4
  • Minimum Number of CPU Cores: 8
  • Minimum RAM per node: 16 GB
  • Minimum Storage: A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.

Wallaroo recommends at least 16 cores total to enable all services. At less than 16 cores, services will have to be disabled to allow basic functionality as detailed in this table.

Note that even when disabling these services, Wallaroo performance may be impacted by the models, pipelines, and data used. The greater the size of the models and steps in a pipeline, the more resources will be required for Wallaroo to operate efficiently. Pipeline resources are set by the pipeline configuration to control how many resources are allocated from the cluster to maintain peak effectiveness for other Wallaroo services. See the following guides for more details.

      
Cluster Size 8 core16 core32 coreDescription
Inference The Wallaroo inference engine that performs inference requests from deployed pipelines.
Dashboard The graphics user interface for configuring workspaces, deploying pipelines, tracking metrics, and other uses.
Jupyter HUB/LabThe JupyterHub service for running Python scripts, JupyterNotebooks, and other related tasks within the Wallaroo instance.
Single Lab
Multiple Labs
PrometheusUsed for collecting and reporting on metrics. Typical metrics are values such as CPU utilization and memory usage.
Alerting 
Model Validation 
Dashboard Graphs 
PlateauA Wallaroo developed service for storing inference logs at high speed. This is not a long term service; organizations are encouraged to store logs in long term solutions if required.
Model Insights 
Python API 
Model ConversionConverts models into a native runtime for use with the Wallaroo inference engine.

To install Wallaroo with minimum services, a configuration file will be used as parts of the kots based installation. For full details on the Wallaroo installation process, see the Wallaroo Install Guides.

Enterprise Network Requirements

The following network requirements are required for the minimum settings for running Wallaroo:

  • For Wallaroo Enterprise users: 200 IP addresses are required to be allocated per cloud environment.

  • For Wallaroo Community users: 98 IP addresses are required to be allocated per cloud environment.

  • DNS services integration is required for Wallaroo Enterprise edition. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.

    DNS services integration is required to provide access to the various supporting services that are part of the Wallaroo instance. These include:

    • Simplified user authentication and management.
    • Centralized services for accessing the Wallaroo Dashboard, Wallaroo SDK and Authentication.
    • Collaboration features allowing teams to work together.
    • Managed security, auditing and traceability.

Environment Software Requirements

The following software or runtimes are required for Wallaroo 2023.4.1. Most are automatically available through the supported cloud providers.

Software or RuntimeDescriptionMinimum Supported VersionPreferred Version(s)
KubernetesCluster deployment management1.231.26
containerdContainer Management1.7.01.7.0
kubectlKubernetes administrative console application1.261.26

Node Selectors

Wallaroo uses different nodes for various services, which can be assigned to a different node pool to contain resources separate from other nodes. The following nodes selectors can be configured:

  • ML Engine node selector
  • ML Engine Load Balance node selector
  • Database Node Selector
  • Grafana node selector
  • Prometheus node selector
  • Each Lab * Node Selector

  • For Kots based installs:

Cost Calculators

Organizations that intend to install Wallaroo into a Cloud environment can obtain an estimate of environment costs. The Wallaroo Install Guides list recommended virtual machine types and other settings that can be used to calculate costs for the environment.

For more information, see the pricing calculators for the following cloud services:

Kubernetes Admin Requirements

Before installing Wallaroo, the administrative node managing the Kubernetes cluster will require these tools.

The following are quick guides on how to install kubectl and kots to install and perform updates to Wallaroo. For a helm based installation, see the How to Install Wallaroo Enterprise via Helm guides.

kubectl Quick Install Guide

The following are quick guides for installing kubectl for different operating systems. For more details, see the instructions for your specific environment.

kubectl Install For Deb Package based Linux Systems

For users running a deb based package system such as Ubuntu Linux, the following commands will install kubectl and kots into the local system. They assume the user has sudo level access to the system.

  1. Update the apt-get repository:

    sudo apt-get update
    
  2. Install the prerequisite software apt-transport-https, ca-certificates, and curl.

    sudo apt-get install -y \
        apt-transport-https \
        ca-certificates curl
    
  3. Download the install the Google Cloud repository key:

    sudo curl -fsSLo \
        /usr/share/keyrings/kubernetes-archive-keyring.gpg \
        https://packages.cloud.google.com/apt/doc/apt-key.gpg
    
  4. Install the Google Cloud repository into the local repository configuration:

    echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" \
        | sudo tee /etc/apt/sources.list.d/kubernetes.list
    
  5. Update the apt-get repository, then install kubectl:

    sudo apt-get update
    
    sudo apt-get install -y kubectl
    
  6. Verify the kubectl installation:

    kubectl version --client
    

kubectl Install For macOS Using Homebrew

To install kubectl on a macOS system using Homebrew:

  1. Issue the brew install command:

    brew install kubectl
    
  2. Verify the installation:

    kubectl version --client
    

kots Quick Install Guide

The following are quick guides for installing kots for different operating systems. For more details, see the instructions for your specific environment.

  • IMPORTANT NOTE

    As of this time, Wallaroo requires kots version 1.91.3. Please verify that version is installed before starting the Wallaroo installation process.

  1. Install curl.

    1. For deb based Linux systems, update the apt-get repository and install curl:

      sudo apt-get update
      sudo apt-get install curl
      
    2. For macOS based systems curl is installed by default.

  2. Install kots by downloading the script and piping it into the bash shell:

    curl https://kots.io/install/1.103.3 | REPL_USE_SUDO=y bash
    

For instructions on updating the kots version for the Wallaroo Ops installation, see Updating KOTS.

Manual Kots Install

A manual method to install KOTS is:

  1. Download from https://github.com/replicatedhq/kots/releases/tag/v1.91.3. Linux and MacOS are supported.

  2. Unpack the release

  3. Rename the kots executable to kubectl-kots.

  4. Copy the renamed kubectl-kots to anywhere on the PATH.

  5. Next, verify successful installation.

    ~ kubectl kots version
    
    Replicated KOTS 1.91.3
    

2 - Wallaroo Enterprise Install Guides

2.1 - Wallaroo Enterprise Comprehensive Install Guides

How to set up Wallaroo Enterprise, environments, and other configurations.

This guides are targeted towards system administrators and data scientists who want to work with the easiest, fastest, and comprehensive method of running your own machine learning models.

A typical installation of Wallaroo follows this process:

StepDescription   Average Setup Time   
Setup EnvironmentCreate an environment that meets the Wallaroo prerequisites30 minutes
Install WallarooInstall Wallaroo into a prepared environment15 minutes
Configure WallarooUpdate Wallaroo with required post-install configurations.Variable

Some knowledge of the following will be useful in working with this guide:

  • Working knowledge of Linux distributions, particularly Ubuntu.
  • A cloud provider including Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure experience.
  • Working knowledge of Kubernetes, mainly kubectl and kots or helm.

For more information, Contact Us for additional details.

The following software or runtimes are required for Wallaroo 2023.4.1. Most are automatically available through the supported cloud providers.

Software or RuntimeDescriptionMinimum Supported VersionPreferred Version(s)
KubernetesCluster deployment management1.231.26
containerdContainer Management1.7.01.7.0
kubectlKubernetes administrative console application1.261.26

Custom Configurations

Wallaroo can be configured with custom installations depending on your organization’s needs. The following options are available:

2.1.1 - Wallaroo Enterprise Comprehensive Install Guide: Amazon Web Services EKS

How to set up Wallaroo Enterprise in AWS EKS via eksctl

Uninstall Guides

The following is a short version of the uninstall procedure to remove a previously installed version of Wallaroo. For full details, see the How to Uninstall Wallaroo. These instructions assume administrative use of the Kubernetes command kubectl.

To uninstall a previously installed Wallaroo instance:

  1. Delete any Wallaroo pipelines still deployed with the command kubectl delete namespace {namespace}. Typically these are the pipeline name with some numerical ID. For example, in the following list of namespaces the namespace ccfraud-pipeline-21 correspond to the Wallaroo pipeline ccfraud-pipeline. Verify these are Wallaroo pipelines before deleting.

      -> kubectl get namespaces
        NAME    STATUS    AGE
        default    Active    7d4h
        kube-node-lease    Active    7d4h
        kube-public    Active    7d4h
        ccfraud-pipeline-21    Active    4h23m
        wallaroo    Active    3d6h
    
      -> kubectl delete namespaces ccfraud-pipeline-21
    
  2. Use the following bash script or run the commands individually. Warning: If the selector is incorrect or missing from the kubectl command, the cluster could be damaged beyond repair. For a default installation, the selector and namespace will be wallaroo.

    #!/bin/bash
    kubectl delete ns wallaroo && \ 
    kubectl delete all,secret,configmap,clusterroles,clusterrolebindings,storageclass,crd \
    --selector app.kubernetes.io/part-of=wallaroo --selector kots.io/app-slug=wallaroo
    

Wallaroo can now be reinstalled into this environment.

AWS Cluster for Wallaroo Enterprise Instructions

The following steps are guidelines to assist new users in setting up their AWS environment for Wallaroo. Feel free to replace these with commands with ones that match your needs.

  • AWS Prerequisites

To install Wallaroo in your AWS environment based on these instructions, the following prerequisites must be met:

  • Register an AWS account: https://aws.amazon.com/ and assign the proper permissions according to your organization’s needs.
  • The Kubernetes cluster must include the following minimum settings:
    • Nodes must be OS type Linux with using the containerd driver.
    • Role-based access control (RBAC) must be enabled.
    • Minimum of 4 nodes, each node with a minimum of 8 CPU cores and 16 GB RAM. 50 GB will be allocated per node for a total of 625 GB for the entire cluster.
    • RBAC is enabled.
    • Recommended Aws Machine type: c5.4xlarge. For more information, see the AWS Instance Types.
  • Installed eksctl version 0.101.0 and above.
  • If the cluster will utilize autoscaling, install the Cluster Autoscaler on AWS.
  • IMPORTANT NOTE
    • Organizations that intend to stop and restart their Kubernetes environment on an intentional or regular basis are recommended to use a single availability zone for their nodes. This minimizes issues such as persistent volumes in different availability zones, etc.
    • Organizations that intend to use Wallaroo Enterprise in a high availability cluster are encouraged to follow best practices including using separate availability zones for redundancy, etc.

EKSCTL Based Instructions

These commands make use of the command line tool eksctl which streamlines the process in creating Amazon Elastic Kubernetes Service clusters for our Wallaroo environment.

The following are used for the example commands below. Replace them with your specific environment settings:

  • AWS Cluster Name: wallarooAWS

Create an AWS EKS Cluster

The following eksctl configuration file is an example of setting up the AWS environment for a Wallaroo cluster, including the static and adaptive nodepools. Adjust these names and settings based on your organizations requirements.

This sample YAML file can be downloaded from here:wallaroo_enterprise_aws_install.yaml

Or copied from here:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: wallarooAWS
  region: us-east-1
  version: "1.25"

addons:
  - name: aws-ebs-csi-driver

iam:
  withOIDC: true
  serviceAccounts:
  - metadata:
      name: cluster-autoscaler
      namespace: kube-system
      labels: {aws-usage: "cluster-ops"}
    wellKnownPolicies:
      autoScaler: true
    roleName: eksctl-cluster-autoscaler-role

nodeGroups:
  - name: mainpool
    instanceType: m5.2xlarge
    desiredCapacity: 3
    containerRuntime: containerd
    amiFamily: AmazonLinux2
    availabilityZones:
      - us-east-1a
  - name: postgres
    instanceType: m5.2xlarge
    desiredCapacity: 1
    taints:
      - key: wallaroo.ai/postgres
        value: "true"
        effect: NoSchedule
    containerRuntime: containerd
    amiFamily: AmazonLinux2
    availabilityZones:
      - us-east-1a
  - name: engine-lb
    instanceType: c5.4xlarge
    minSize: 1
    maxSize: 3
    taints:
      - key: wallaroo.ai/enginelb
        value: "true"
        effect: NoSchedule
    tags:
      k8s.io/cluster-autoscaler/node-template/label/k8s.dask.org/node-purpose: engine-lb
      k8s.io/cluster-autoscaler/node-template/taint/k8s.dask.org/dedicated: "true:NoSchedule"
    iam:
      withAddonPolicies:
        autoScaler: true
    containerRuntime: containerd
    amiFamily: AmazonLinux2
    availabilityZones:
      - us-east-1a
  - name: engine
    instanceType: c5.2xlarge
    minSize: 1
    maxSize: 3
    taints:
      - key: wallaroo.ai/engine
        value: "true"
        effect: NoSchedule
    tags:
      k8s.io/cluster-autoscaler/node-template/label/k8s.dask.org/node-purpose: engine
      k8s.io/cluster-autoscaler/node-template/taint/k8s.dask.org/dedicated: "true:NoSchedule"
    iam:
      withAddonPolicies:
        autoScaler: true
    containerRuntime: containerd
    amiFamily: AmazonLinux2
    availabilityZones:
      - us-east-1a

Create the Cluster

Create the cluster with the following command, which creates the environment and sets the correct Kubernetes version.

eksctl create cluster -f wallaroo_enterprise_aws_install.yaml

During the process the Kubernetes credentials will be copied into the local environment. To verify the setup is complete, use the kubectl get nodes command to display the available nodes as in the following example:

kubectl get nodes
NAME                                           STATUS   ROLES    AGE     VERSION
ip-192-168-21-253.us-east-2.compute.internal   Ready    <none>   13m     v1.23.8-eks-9017834
ip-192-168-30-36.us-east-2.compute.internal    Ready    <none>   13m     v1.23.8-eks-9017834
ip-192-168-38-31.us-east-2.compute.internal    Ready    <none>   9m46s   v1.23.8-eks-9017834
ip-192-168-55-123.us-east-2.compute.internal   Ready    <none>   12m     v1.23.8-eks-9017834
ip-192-168-79-70.us-east-2.compute.internal    Ready    <none>   13m     v1.23.8-eks-9017834
ip-192-168-37-222.us-east-2.compute.internal   Ready    <none>   13m     v1.23.8-eks-9017834

Install Wallaroo

Organizations that use cloud services such as Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure can install Wallaroo Enterprise through the following process. These instructions also work with Single Node Linux based installations.

Before installation, the following prerequisites must be met:

  • Have a Wallaroo Enterprise license file. For more information, you can request a demonstration.
  • Set up a cloud Kubernetes environment that meets the requirements. Clusters must meet the following minimum specifications:
    • Minimum number of nodes: 4
    • Minimum Number of CPU Cores: 8
    • Minimum RAM: 16 GB
    • A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.
    • Runtime: containerd is required.
  • DNS services for integrating your Wallaroo Enterprise instance. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.

Wallaroo Enterprise can be installed either interactively or automatically through the kubectl and kots applications.

Automated Install

To automatically install Wallaroo into the namespace wallaroo, specify the administrative password and the license file during the installation as in the following format with the following variables:

  • NAMESPACE: The namespace for the Wallaroo Enterprise install, typically wallaroo.
  • LICENSEFILE: The location of the Wallaroo Enterprise license file.
  • SHAREDPASSWORD: The password of for the Wallaroo Administrative Dashboard.
kubectl kots install wallaroo/ee -n $NAMESPACE --license-file $LICENSEFILE --shared-password $SHAREDPASSWORD

For example, the following settings translate to the following install command:

  • NAMESPACE: wallaroo.
  • LICENSEFILE: myWallaroolicense.yaml
  • SHAREDPASSWORD: snugglebunnies

kubectl kots install wallaroo/ee -n wallaroo --license-file myWallaroolicense.yaml --shared-password wallaroo

Interactive Install

The Interactive Install process allows users to adjust the configuration settings before Wallaroo is deployed. It requires users be able to access the Wallaroo Administrative Dashboard through a browser, typically on port 8080.

  • IMPORTANT NOTE: Users who install Wallaroo through another node such as in the single node installation can port use SSH tunneling to access the Wallaroo Administrative Dashboard. For example:

    ssh IP -L8800:localhost:8800
    
  1. Install the Wallaroo Enterprise Edition using kots install wallaroo/ee, specifying the namespace to install Wallaroo into. For example, if wallaroo is the namespace, then the command is:

    kubectl kots install wallaroo/ee --namespace wallaroo
    
  2. Wallaroo Enterprise Edition will be downloaded and installed into your Kubernetes environment in the namespace specified. When prompted, set the default password for the Wallaroo environment. When complete, Wallaroo Enterprise Edition will display the URL for the Admin Console, and how to end the Admin Console from running.

    • Deploying Admin Console
    • Creating namespace ✓
    • Waiting for datastore to be ready ✓
        Enter a new password to be used for the Admin Console: •••••••••••••
      • Waiting for Admin Console to be ready ✓
    
    • Press Ctrl+C to exit
    • Go to http://localhost:8800 to access the Admin Console
    

To relaunch the Wallaroo Administrative Dashboard and make changes or updates, use the following command:

kubectl-kots admin-console --namespace wallaroo

Configure Wallaroo

Once installed, Wallaroo will continue to run until terminated.

Change Wallaroo Administrative Dashboard Password

To change the password to the Wallaroo Administrative Dashboard:

  1. From the command line, use the command:

    kubectl kots reset-password -n {namespace}
    

    For example, for default installations where the Kubernetes namespace is wallaroo, the command would be:

    kubectl kots reset-password -n wallaroo
    

    From here, enter the new password.

  2. From the Wallaroo Administrative Dashboard:

    1. Login and authenticate with the current password.

    2. From the upper right hand corner, select to access the menu and select Change password.

      Select Change Password
    3. Enter the current password, then update and verify with the new password.

      Change Password

Setup DNS Services

Wallaroo Enterprise requires integration into your organizations DNS services.

The DNS Integration Guide details adding the Wallaroo instance to an organizations DNS services. The following is an abbreviated guide that assumes that certificates were already generated.

  1. From the Wallaroo Dashboard, select Config and set the following:

    1. Networking Configuration
      1. Ingress Mode for Wallaroo Endpoints:
        1. None: Port forwarding or other methods are used for access.
        2. Internal: For environments where only nodes within the same Kubernetes environment and no external connections are required.
        3. External: Connections from outside the Kubernetes environment is allowed.
          1. Enable external URL inference endpoints: Creates pipeline inference endpoints. For more information, see Model Endpoints Guide.
    2. DNS
      1. DNS Suffix (Mandatory): The domain name for your Wallaroo instance.
    3. TLS Certificates
      1. Use custom TLS Certs: Checked
      2. TLS Certificate: Enter your TLS Certificate (.crt file).
      3. TLS Private Key: Enter your TLS private key (.key file).
    4. Other settings as desired.
    Wallaroo DNS Records
  2. Once complete, scroll to the bottom of the Config page and select Save config.

  3. A pop-up window will display The config for Wallaroo Enterprise has been updated.. Select Go to updated version to continue.

  4. From the Version History page, select Deploy. Once the new deployment is finished, you will be able to access your Wallaroo services via their DNS addresses.

To verify the configuration is complete, access the Wallaroo Dashboard through the suffix domain. For example if the suffix domain is wallaroo.example.com then access https://wallaroo.example.com in a browser and verify the connection and certificates.

Setup Users

User management is handled through the Wallaroo instance Keycloak service. See the Wallaroo User Management for full guides on setting up users, identity providers, and other user configuration options. This step must be completed before using Wallaroo.

The following is an abbreviated guide on setting up new Wallaroo users.

Accessing The Wallaroo Keycloak Dashboard

Enterprise customers may access their Wallaroo Keycloak dashboard by navigating to https://keycloak.<suffix>, depending on their choice domain suffix supplied during installation.

Obtaining Administrator Credentials

The standard Wallaroo installation creates the user admin by default and assigns them a randomly generated password. The admin user credentials are obtained which may be obtained directly from Kubernetes with the following commands, assuming the Wallaroo instance namespace is wallaroo.

  • Retrieve Keycloak Admin Username

    kubectl -n wallaroo \
    get secret keycloak-admin-secret \
    -o go-template='{{.data.KEYCLOAK_ADMIN_USER | base64decode }}'
    
  • Retrieve Keycloak Admin Password

    kubectl -n wallaroo \
    get secret keycloak-admin-secret \
    -o go-template='{{.data.KEYCLOAK_ADMIN_PASSWORD | base64decode }}'
    

Accessing the User Management Panel

In the Keycloak Administration Console, click Manage -> Users in the left-hand side menu. Click the View all users button to see existing users. This will be under the host name keycloak.$WALLAROO_SUFFIX. For example, if the $WALLAROO_SUFFIX is wallaroo.example.com, the Keycloak Administration Console would be keycloak.wallaroo.example.com.

Adding Users

To add a user through the Keycloak interface:

  1. Click the Add user button in the top-right corner.

  2. Enter the following:

    Wallaroo Enterprise New User
    1. A unique username and email address.
    2. Ensure that the Email Verified checkbox is checked - Wallaroo does not perform email verification.
    3. Under Required User Actions, set Update Password so the user will update their password the next time they log in.
  3. Click Save.

  4. Once saved, select Credentials tab, then the Set Password section, enter the new user’s desired initial password in the Password and Password Confirmation fields.

    Wallaroo Enterprise New User
  5. Click Set Password. Confirm the action when prompted. This will force the user to set their own password when they log in to Wallaroo.

  6. To log into the Wallaroo dashboard, log out as the Admin user and login to the Wallaroo Dashboard as a preconfigured user or via SSO.

2.1.2 - Wallaroo Enterprise Comprehensive Install Guide: Microsoft Azure Kubernetes Services

How to set up Wallaroo Enterprise in Azure Kubernetes

Uninstall Guides

The following is a short version of the uninstall procedure to remove a previously installed version of Wallaroo. For full details, see the How to Uninstall Wallaroo. These instructions assume administrative use of the Kubernetes command kubectl.

To uninstall a previously installed Wallaroo instance:

  1. Delete any Wallaroo pipelines still deployed with the command kubectl delete namespace {namespace}. Typically these are the pipeline name with some numerical ID. For example, in the following list of namespaces the namespace ccfraud-pipeline-21 correspond to the Wallaroo pipeline ccfraud-pipeline. Verify these are Wallaroo pipelines before deleting.

      -> kubectl get namespaces
        NAME    STATUS    AGE
        default    Active    7d4h
        kube-node-lease    Active    7d4h
        kube-public    Active    7d4h
        ccfraud-pipeline-21    Active    4h23m
        wallaroo    Active    3d6h
    
      -> kubectl delete namespaces ccfraud-pipeline-21
    
  2. Use the following bash script or run the commands individually. Warning: If the selector is incorrect or missing from the kubectl command, the cluster could be damaged beyond repair. For a default installation, the selector and namespace will be wallaroo.

    #!/bin/bash
    kubectl delete ns wallaroo && \ 
    kubectl delete all,secret,configmap,clusterroles,clusterrolebindings,storageclass,crd \
    --selector app.kubernetes.io/part-of=wallaroo --selector kots.io/app-slug=wallaroo
    

Wallaroo can now be reinstalled into this environment.

Azure Cluster for Wallaroo Enterprise Instructions

The following instructions are made to assist users set up their Microsoft Azure Kubernetes environment for running Wallaroo Enterprise. These represent a recommended setup, but can be modified to fit your specific needs.

If your prepared to install the environment now, skip to Setup Environment Steps.

There are two methods we’ve detailed here on how to setup your Kubernetes cloud environment in Azure:

  • Quick Setup Script Download a bash script to automatically set up the Azure environment through the Microsoft Azure command line interface az.
  • Manual Setup Guide A list of the az commands used to create the environment through manual commands.

Azure Prerequisites

To install Wallaroo in your Microsoft Azure environment, the following prerequisites must be met:

  • Register a Microsoft Azure account: https://azure.microsoft.com/.
  • Install the Microsoft Azure CLI and complete the Azure CLI Get Started Guide to connect your az application to your Microsoft Azure account.
  • The Kubernetes cluster must include the following minimum settings:
    • Nodes must be OS type Linux the containerd driver as the default.
    • Role-based access control (RBAC) must be enabled.
    • Minimum of 4 nodes, each node with a minimum of 8 CPU cores and 16 GB RAM. 50 GB will be allocated per node for a total of 625 GB for the entire cluster.
    • RBAC is enabled.
    • Minimum machine type is set to to Standard_D8s_v4.
  • IMPORTANT NOTE
    • Organizations that intend to stop and restart their Kubernetes environment on an intentional or regular basis are recommended to use a single availability zone for their nodes. This minimizes issues such as persistent volumes in different availability zones, etc.
    • Organizations that intend to use Wallaroo Enterprise in a high availability cluster are encouraged to follow best practices including using separate availability zones for redundancy, etc.

Standard Setup Variables

The following variables are used in the Quick Setup Script and the Manual Setup Guide detailed below. Modify them as best fits your organization.

Variable NameDefault ValueDescription
WALLAROO_RESOURCE_GROUPwallaroogroupThe Azure Resource Group used for the KUbernetes environment.
WALLAROO_GROUP_LOCATIONeastusThe region that the Kubernetes environment will be installed to.
WALLAROO_CONTAINER_REGISTRYwallarooacrThe Azure Container Registry used for the Kubernetes environment.
WALLAROO_CLUSTERwallarooaksThe name of the Kubernetes cluster that Wallaroo is installed to.
WALLAROO_SKU_TYPEBaseThe Azure Kubernetes Service SKU type.
WALLAROO_VM_SIZEStandard_D8s_v4The VM type used for the standard Wallaroo cluster nodes.
POSTGRES_VM_SIZEStandard_D8s_v4The VM type used for the postgres nodepool.
ENGINELB_VM_SIZEStandard_D8s_v4The VM type used for the engine-lb nodepool.
ENGINE_VM_SIZEStandard_F8s_v2The VM type used for the engine nodepool.

Setup Environment Steps

Quick Setup Script

A sample script is available here, and creates an Azure Kubernetes environment ready for use with Wallaroo Enterprise. This script requires the following prerequisites listed above and uses the variables listed in Standard Setup Variables. Modify them as best fits your organization’s needs.

The following script is available for download: wallaroo_enterprise_azure_expandable.bash

The following steps are geared towards a standard Linux or macOS system that supports the prerequisites listed above. Modify these steps based on your local environment.

  1. Download the script above.
  2. In a terminal window set the script status as execute with the command chmod +x wallaroo_enterprise_install_azure_expandable.bash.
  3. Modify the script variables listed above based on your requirements.
  4. Run the script with either bash wallaroo_enterprise_install_azure_expandable.bash or ./wallaroo_enterprise_install_azure_expandable.bash from the same directory as the script.

Manual Setup Guide

The following steps are guidelines to assist new users in setting up their Azure environment for Wallaroo.

The process uses the variables listed in Standard Setup Variables. Modify them as best fits your organization’s needs.

See the Azure Command-Line Interface for full details on commands and settings.

Setting up an Azure AKS environment is based on the Azure Kubernetes Service tutorial, streamlined to show the minimum steps in setting up your own Wallaroo environment in Azure.

This follows these major steps:

Set Variables

The following are the variables used for the rest of the commands. Modify them as fits your organization’s needs.

WALLAROO_RESOURCE_GROUP=wallaroogroup
WALLAROO_GROUP_LOCATION=eastus
WALLAROO_CONTAINER_REGISTRY=wallarooacr
WALLAROO_CLUSTER=wallarooaks
WALLAROO_SKU_TYPE=Base
WALLAROO_VM_SIZE=Standard_D8s_v4
POSTGRES_VM_SIZE=Standard_D8s_v4
ENGINELB_VM_SIZE=Standard_D8s_v4
ENGINE_VM_SIZE=Standard_F8s_v2
  • Create an Azure Resource Group

To create an Azure Resource Group for Wallaroo in Microsoft Azure, use the following template:

az group create --name $WALLAROO_RESOURCE_GROUP --location $WALLAROO_GROUP_LOCATION

(Optional): Set the default Resource Group to the one recently created. This allows other Azure commands to automatically select this group for commands such as az aks list, etc.

az configure --defaults group={Resource Group Name}

For example:

az configure --defaults group=wallarooGroup
  • Create an Azure Container Registry

An Azure Container Registry(ACR) manages the container images for services includes Kubernetes. The template for setting up an Azure ACR that supports Wallaroo is the following:

az acr create -n $WALLAROO_CONTAINER_REGISTRY \
-g $WALLAROO_RESOURCE_GROUP \
--sku $WALLAROO_SKU_TYPE \
--location $WALLAROO_GROUP_LOCATION
  • Create an Azure Kubernetes Services

Now we can create our Kubernetes service in Azure that will host our Wallaroo with the az aks create command.

az aks create \
--resource-group $WALLAROO_RESOURCE_GROUP \
--name $WALLAROO_CLUSTER \
--node-count 3 \
--generate-ssh-keys \
--vm-set-type VirtualMachineScaleSets \
--load-balancer-sku standard \
--node-vm-size $WALLAROO_VM_SIZE \
--nodepool-name mainpool \
--attach-acr $WALLAROO_CONTAINER_REGISTRY \
--kubernetes-version=1.23.15 \
--zones 1 \
--location $WALLAROO_GROUP_LOCATION

Wallaroo Enterprise Nodepools

Wallaroo Enterprise supports autoscaling and static nodepools. The following commands are used to create both to support the Wallaroo Enterprise cluster.

The following static nodepools are set up to support the Wallaroo cluster for postgres. Update the VM_SIZE based on your requirements.

az aks nodepool add \
--resource-group $WALLAROO_RESOURCE_GROUP \
--cluster-name $WALLAROO_CLUSTER \
--name postgres \
--node-count 1 \
--node-vm-size $POSTGRES_VM_SIZE \
--no-wait \
--node-taints wallaroo.ai/postgres=true:NoSchedule \
--zones 1

The following autoscaling nodepools are used for the engineLB and the engine nodepools. Adjust the settings based on your organizations requirements.

az aks nodepool add \
--resource-group $WALLAROO_RESOURCE_GROUP \
--cluster-name $WALLAROO_CLUSTER \
--name enginelb \
--node-count 1 \
--node-vm-size $ENGINELB_VM_SIZE \
--no-wait \
--enable-cluster-autoscaler \
--max-count 3 \
--min-count 1 \
--node-taints wallaroo.ai/enginelb=true:NoSchedule \
--labels wallaroo-node-type=enginelb \
--zones 1
az aks nodepool add \
--resource-group $WALLAROO_RESOURCE_GROUP \
--cluster-name $WALLAROO_CLUSTER \
--name engine \
--node-count 1 \
--node-vm-size $ENGINE_VM_SIZE \
--no-wait \
--enable-cluster-autoscaler \
--max-count 3 \
--min-count 1 \
--node-taints wallaroo.ai/engine=true:NoSchedule \
--labels wallaroo-node-type=engine \
--zones 1

For additional settings such as customizing the node pools for your Wallaroo Kubernetes cluster to customize the type of virtual machines used and other settings, see the Microsoft Azure documentation on using system node pools.

  • Download Wallaroo Kubernetes Configuration

Once the Kubernetes environment is complete, associate it with the local Kubernetes configuration by importing the credentials through the following template command:

az aks get-credentials --resource-group $WALLAROO_RESOURCE_GROUP --name $WALLAROO_CLUSTER

Verify the cluster is available through the kubectl get nodes command.

kubectl get nodes

NAME                               STATUS   ROLES   AGE   VERSION
aks-engine-99896855-vmss000000     Ready    agent   40m   v1.23.8
aks-enginelb-54433467-vmss000000   Ready    agent   48m   v1.23.8
aks-mainpool-37402055-vmss000000   Ready    agent   81m   v1.23.8
aks-mainpool-37402055-vmss000001   Ready    agent   81m   v1.23.8
aks-mainpool-37402055-vmss000002   Ready    agent   81m   v1.23.8
aks-postgres-40215394-vmss000000   Ready    agent   52m   v1.23.8

Install Wallaroo

Organizations that use cloud services such as Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure can install Wallaroo Enterprise through the following process. These instructions also work with Single Node Linux based installations.

Before installation, the following prerequisites must be met:

  • Have a Wallaroo Enterprise license file. For more information, you can request a demonstration.
  • Set up a cloud Kubernetes environment that meets the requirements. Clusters must meet the following minimum specifications:
    • Minimum number of nodes: 4
    • Minimum Number of CPU Cores: 8
    • Minimum RAM: 16 GB
    • A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.
    • Runtime: containerd is required.
  • DNS services for integrating your Wallaroo Enterprise instance. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.

Wallaroo Enterprise can be installed either interactively or automatically through the kubectl and kots applications.

Automated Install

To automatically install Wallaroo into the namespace wallaroo, specify the administrative password and the license file during the installation as in the following format with the following variables:

  • NAMESPACE: The namespace for the Wallaroo Enterprise install, typically wallaroo.
  • LICENSEFILE: The location of the Wallaroo Enterprise license file.
  • SHAREDPASSWORD: The password of for the Wallaroo Administrative Dashboard.
kubectl kots install wallaroo/ee -n $NAMESPACE --license-file $LICENSEFILE --shared-password $SHAREDPASSWORD

For example, the following settings translate to the following install command:

  • NAMESPACE: wallaroo.
  • LICENSEFILE: myWallaroolicense.yaml
  • SHAREDPASSWORD: snugglebunnies

kubectl kots install wallaroo/ee -n wallaroo --license-file myWallaroolicense.yaml --shared-password wallaroo

Interactive Install

The Interactive Install process allows users to adjust the configuration settings before Wallaroo is deployed. It requires users be able to access the Wallaroo Administrative Dashboard through a browser, typically on port 8080.

  • IMPORTANT NOTE: Users who install Wallaroo through another node such as in the single node installation can port use SSH tunneling to access the Wallaroo Administrative Dashboard. For example:

    ssh IP -L8800:localhost:8800
    
  1. Install the Wallaroo Enterprise Edition using kots install wallaroo/ee, specifying the namespace to install Wallaroo into. For example, if wallaroo is the namespace, then the command is:

    kubectl kots install wallaroo/ee --namespace wallaroo
    
  2. Wallaroo Enterprise Edition will be downloaded and installed into your Kubernetes environment in the namespace specified. When prompted, set the default password for the Wallaroo environment. When complete, Wallaroo Enterprise Edition will display the URL for the Admin Console, and how to end the Admin Console from running.

    • Deploying Admin Console
    • Creating namespace ✓
    • Waiting for datastore to be ready ✓
        Enter a new password to be used for the Admin Console: •••••••••••••
      • Waiting for Admin Console to be ready ✓
    
    • Press Ctrl+C to exit
    • Go to http://localhost:8800 to access the Admin Console
    

To relaunch the Wallaroo Administrative Dashboard and make changes or updates, use the following command:

kubectl-kots admin-console --namespace wallaroo

Configure Wallaroo

Once installed, Wallaroo will continue to run until terminated.

Change Wallaroo Administrative Dashboard Password

To change the password to the Wallaroo Administrative Dashboard:

  1. From the command line, use the command:

    kubectl kots reset-password -n {namespace}
    

    For example, for default installations where the Kubernetes namespace is wallaroo, the command would be:

    kubectl kots reset-password -n wallaroo
    

    From here, enter the new password.

  2. From the Wallaroo Administrative Dashboard:

    1. Login and authenticate with the current password.

    2. From the upper right hand corner, select to access the menu and select Change password.

      Select Change Password
    3. Enter the current password, then update and verify with the new password.

      Change Password

Setup DNS Services

Wallaroo Enterprise requires integration into your organizations DNS services.

The DNS Integration Guide details adding the Wallaroo instance to an organizations DNS services. The following is an abbreviated guide that assumes that certificates were already generated.

  1. From the Wallaroo Dashboard, select Config and set the following:

    1. Networking Configuration
      1. Ingress Mode for Wallaroo Endpoints:
        1. None: Port forwarding or other methods are used for access.
        2. Internal: For environments where only nodes within the same Kubernetes environment and no external connections are required.
        3. External: Connections from outside the Kubernetes environment is allowed.
          1. Enable external URL inference endpoints: Creates pipeline inference endpoints. For more information, see Model Endpoints Guide.
    2. DNS
      1. DNS Suffix (Mandatory): The domain name for your Wallaroo instance.
    3. TLS Certificates
      1. Use custom TLS Certs: Checked
      2. TLS Certificate: Enter your TLS Certificate (.crt file).
      3. TLS Private Key: Enter your TLS private key (.key file).
    4. Other settings as desired.
    Wallaroo DNS Records
  2. Once complete, scroll to the bottom of the Config page and select Save config.

  3. A pop-up window will display The config for Wallaroo Enterprise has been updated.. Select Go to updated version to continue.

  4. From the Version History page, select Deploy. Once the new deployment is finished, you will be able to access your Wallaroo services via their DNS addresses.

To verify the configuration is complete, access the Wallaroo Dashboard through the suffix domain. For example if the suffix domain is wallaroo.example.com then access https://wallaroo.example.com in a browser and verify the connection and certificates.

Setup Users

User management is handled through the Wallaroo instance Keycloak service. See the Wallaroo User Management for full guides on setting up users, identity providers, and other user configuration options. This step must be completed before using Wallaroo.

The following is an abbreviated guide on setting up new Wallaroo users.

Accessing The Wallaroo Keycloak Dashboard

Enterprise customers may access their Wallaroo Keycloak dashboard by navigating to https://keycloak.<suffix>, depending on their choice domain suffix supplied during installation.

Obtaining Administrator Credentials

The standard Wallaroo installation creates the user admin by default and assigns them a randomly generated password. The admin user credentials are obtained which may be obtained directly from Kubernetes with the following commands, assuming the Wallaroo instance namespace is wallaroo.

  • Retrieve Keycloak Admin Username

    kubectl -n wallaroo \
    get secret keycloak-admin-secret \
    -o go-template='{{.data.KEYCLOAK_ADMIN_USER | base64decode }}'
    
  • Retrieve Keycloak Admin Password

    kubectl -n wallaroo \
    get secret keycloak-admin-secret \
    -o go-template='{{.data.KEYCLOAK_ADMIN_PASSWORD | base64decode }}'
    

Accessing the User Management Panel

In the Keycloak Administration Console, click Manage -> Users in the left-hand side menu. Click the View all users button to see existing users. This will be under the host name keycloak.$WALLAROO_SUFFIX. For example, if the $WALLAROO_SUFFIX is wallaroo.example.com, the Keycloak Administration Console would be keycloak.wallaroo.example.com.

Adding Users

To add a user through the Keycloak interface:

  1. Click the Add user button in the top-right corner.

  2. Enter the following:

    Wallaroo Enterprise New User
    1. A unique username and email address.
    2. Ensure that the Email Verified checkbox is checked - Wallaroo does not perform email verification.
    3. Under Required User Actions, set Update Password so the user will update their password the next time they log in.
  3. Click Save.

  4. Once saved, select Credentials tab, then the Set Password section, enter the new user’s desired initial password in the Password and Password Confirmation fields.

    Wallaroo Enterprise New User
  5. Click Set Password. Confirm the action when prompted. This will force the user to set their own password when they log in to Wallaroo.

  6. To log into the Wallaroo dashboard, log out as the Admin user and login to the Wallaroo Dashboard as a preconfigured user or via SSO.

2.1.3 - Wallaroo Enterprise Comprehensive Install Guide: Google Cloud Platform Kubernetes Engine

How to set up Wallaroo Enterprise in GCP Kubernetes Engine

Uninstall Guides

The following is a short version of the uninstall procedure to remove a previously installed version of Wallaroo. For full details, see the How to Uninstall Wallaroo. These instructions assume administrative use of the Kubernetes command kubectl.

To uninstall a previously installed Wallaroo instance:

  1. Delete any Wallaroo pipelines still deployed with the command kubectl delete namespace {namespace}. Typically these are the pipeline name with some numerical ID. For example, in the following list of namespaces the namespace ccfraud-pipeline-21 correspond to the Wallaroo pipeline ccfraud-pipeline. Verify these are Wallaroo pipelines before deleting.

      -> kubectl get namespaces
        NAME    STATUS    AGE
        default    Active    7d4h
        kube-node-lease    Active    7d4h
        kube-public    Active    7d4h
        ccfraud-pipeline-21    Active    4h23m
        wallaroo    Active    3d6h
    
      -> kubectl delete namespaces ccfraud-pipeline-21
    
  2. Use the following bash script or run the commands individually. Warning: If the selector is incorrect or missing from the kubectl command, the cluster could be damaged beyond repair. For a default installation, the selector and namespace will be wallaroo.

    #!/bin/bash
    kubectl delete ns wallaroo && \ 
    kubectl delete all,secret,configmap,clusterroles,clusterrolebindings,storageclass,crd \
    --selector app.kubernetes.io/part-of=wallaroo --selector kots.io/app-slug=wallaroo
    

Wallaroo can now be reinstalled into this environment.

GCP Kubernetes Engine Instructions

The following instructions are made to assist users set up their Google Cloud Platform (GCP) Kubernetes environment for running Wallaroo. These represent a recommended setup, but can be modified to fit your specific needs. In particular, these instructions will provision a GKE cluster with 56 CPUs in total. Please ensure that your project’s resource limits support that.

  • Quick Setup Script: Download a bash script to automatically set up the GCP environment through the Google Cloud Platform command line interface gcloud.

  • Manual Setup Guide: A list of the gcloud commands used to create the environment through manual commands.

    • GCP Prerequisites

    Organizations that wish to run Wallaroo in their Google Cloud Platform environment must complete the following prerequisites:

    • IMPORTANT NOTE
      • Organizations that intend to stop and restart their Kubernetes environment on an intentional or regular basis are recommended to use a single availability zone for their nodes. This minimizes issues such as persistent volumes in different availability zones, etc.
      • Organizations that intend to use Wallaroo Enterprise in a high availability cluster are encouraged to follow best practices including using separate availability zones for redundancy, etc.

    Standard Setup Variables

    The following variables are used in the Quick Setup Script and the Manual Setup Guide. Modify them as best fits your organization.

    Variable NameDefault ValueDescription
    WALLAROO_GCP_PROJECTwallarooThe name of the Google Project used for the Wallaroo instance.
    WALLAROO_CLUSTERwallarooThe name of the Kubernetes cluster for the Wallaroo instance.
    WALLAROO_GCP_REGIONus-central1The region the Kubernetes environment is installed to. Update this to your GCP Computer Engine region.
    WALLAROO_NODE_LOCATIONus-central1-fThe location the Kubernetes nodes are installed to. Update this to your GCP Compute Engine Zone.
    WALLAROO_GCP_NETWORK_NAMEwallaroo-networkThe Google network used with the Kubernetes environment.
    WALLAROO_GCP_SUBNETWORK_NAMEwallaroo-subnet-1The Google network subnet used with the Kubernets environment.
    DEFAULT_VM_SIZEe2-standard-8The VM type used for the default nodepool.
    POSTGRES_VM_SIZEn2-standard-8The VM type used for the postgres nodepool.
    ENGINELB_VM_SIZEc2-standard-8The VM type used for the engine-lb nodepool.
    ENGINE_VM_SIZEc2-standard-8The VM type used for the engine nodepool.

    Quick Setup Script

    A sample script is available here, and creates a Google Kubernetes Engine cluster ready for use with Wallaroo Enterprise. This script requires the prerequisites listed above and uses the variables as listed in Standard Setup Variables

    The following script is available for download: wallaroo_enterprise_gcp_expandable.bash

    The following steps are geared towards a standard Linux or macOS system that supports the prerequisites listed above. Modify these steps based on your local environment.

    1. Download the script above.
    2. In a terminal window set the script status as execute with the command chmod +x bash wallaroo_enterprise_gcp_expandable.bash.
    3. Modify the script variables listed above based on your requirements.
    4. Run the script with either bash wallaroo_enterprise_gcp_expandable.bash or ./wallaroo_enterprise_gcp_expandable.bash from the same directory as the script.

    Set Variables

    The following are the variables used in the environment setup process. Modify them as best fits your organization’s needs.

    WALLAROO_GCP_PROJECT=wallaroo
    WALLAROO_CLUSTER=wallaroo
    WALLAROO_GCP_REGION=us-central1
    WALLAROO_NODE_LOCATION=us-central1-f
    WALLAROO_GCP_NETWORK_NAME=wallaroo-network
    WALLAROO_GCP_SUBNETWORK_NAME=wallaroo-subnet-1
    DEFAULT_VM_SIZE=n2-standard-8
    POSTGRES_VM_SIZE=n2-standard-8
    ENGINELB_VM_SIZE=c2-standard-8
    ENGINE_VM_SIZE=c2-standard-8
    

    Manual Setup Guide

    The following steps are guidelines to assist new users in setting up their GCP environment for Wallaroo. The variables used in the commands are as listed in Standard Setup Variables listed above. Feel free to replace these with ones that match your needs.

    See the Google Cloud SDK for full details on commands and settings.

    Create a GCP Network

    First create a GCP network that is used to connect to the cluster with the gcloud compute networks create command. For more information, see the gcloud compute networks create page.

    gcloud compute networks \
    create $WALLAROO_GCP_NETWORK_NAME \
    --bgp-routing-mode regional \
    --subnet-mode custom
    

    Verify it’s creation by listing the GCP networks:

    gcloud compute networks list
    

    Create the GCP Wallaroo Cluster

    Once the network is created, the gcloud container clusters create command is used to create a cluster. For more information see the gcloud container clusters create page.

    The following is a recommended format, replacing the {} listed variables based on your setup. For Google GKE containerd is enabled by default.

    gcloud container clusters \
    create $WALLAROO_CLUSTER \
    --region $WALLAROO_GCP_REGION \
    --node-locations $WALLAROO_NODE_LOCATION \
    --machine-type $DEFAULT_VM_SIZE \
    --network $WALLAROO_GCP_NETWORK_NAME \
    --create-subnetwork name=$WALLAROO_GCP_SUBNETWORK_NAME \
    --enable-ip-alias \
    --cluster-version=1.23
    

    The command can take several minutes to complete based on the size and complexity of the clusters. Verify the process is complete with the clusters list command:

    gcloud container clusters list
    

    Wallaroo Enterprise Nodepools

    The following static nodepools can be set based on your organizations requirements. Adjust the settings or names based on your requirements.

    gcloud container node-pools create postgres \
    --cluster=$WALLAROO_CLUSTER \
    --machine-type=$POSTGRES_VM_SIZE \
    --num-nodes=1 \
    --region $WALLAROO_GCP_REGION \
    --node-taints wallaroo.ai/postgres=true:NoSchedule
    

    The following autoscaling nodepools are used for the engine load balancers and Wallaroo engine. Again, replace names and virtual machine types based on your organizations requirements.

    gcloud container node-pools create engine-lb \
    --cluster=$WALLAROO_CLUSTER \
    --machine-type=$ENGINELB_VM_SIZE \
    --enable-autoscaling \
    --num-nodes=1 \
    --min-nodes=0 \
    --max-nodes=3 \
    --region $WALLAROO_GCP_REGION \
    --node-taints wallaroo-engine-lb=true:NoSchedule,wallaroo.ai/enginelb=true:NoSchedule \
    --node-labels wallaroo-node-type=engine-lb
    
    gcloud container node-pools create engine \
    --cluster=$WALLAROO_CLUSTER \
    --machine-type=$ENGINE_VM_SIZE \
    --enable-autoscaling \
    --num-nodes=1 \
    --min-nodes=0 \
    --max-nodes=3 \
    --region $WALLAROO_GCP_REGION \
    --node-taints wallaroo.ai/engine=true:NoSchedule \
    --node-labels=wallaroo-node-type=engine
    

    Retrieving Kubernetes Credentials

    Once the GCP cluster is complete, the Kubernetes credentials can be installed into the local administrative system with the gcloud container clusters get-credentials command:

    gcloud container clusters \
    get-credentials $WALLAROO_CLUSTER \
    --region $WALLAROO_GCP_REGION
    

    To verify the Kubernetes credentials for your cluster have been installed locally, use the kubectl get nodes command. This will display the nodes in the cluster as demonstrated below:

    kubectl get nodes
    
    NAME                                         STATUS   ROLES    AGE   VERSION
    gke-wallaroo-default-pool-863f02db-7xd4   Ready    <none>   39m   v1.21.6-gke.1503
    gke-wallaroo-default-pool-863f02db-8j2d   Ready    <none>   39m   v1.21.6-gke.1503
    gke-wallaroo-default-pool-863f02db-hn06   Ready    <none>   39m   v1.21.6-gke.1503
    gke-wallaroo-engine-3946eaca-4l3s         Ready    <none>   89s   v1.21.6-gke.1503
    gke-wallaroo-engine-lb-2e33a27f-64wb      Ready    <none>   26m   v1.21.6-gke.1503
    gke-wallaroo-postgres-d22d73d3-5qp5       Ready    <none>   28m   v1.21.6-gke.1503
    

    Troubleshooting

    • What does the error Insufficient project quota to satisfy request: resource "CPUS_ALL_REGIONS" mean?
      • Make sure that the Compute Engine Zone and Region are properly set based on your organization’s requirements. The instructions above default to us-central1, so change that zone to install your Wallaroo instance in the correct location.

    Install Wallaroo

    Organizations that use cloud services such as Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure can install Wallaroo Enterprise through the following process. These instructions also work with Single Node Linux based installations.

    Before installation, the following prerequisites must be met:

    • Have a Wallaroo Enterprise license file. For more information, you can request a demonstration.
    • Set up a cloud Kubernetes environment that meets the requirements. Clusters must meet the following minimum specifications:
      • Minimum number of nodes: 4
      • Minimum Number of CPU Cores: 8
      • Minimum RAM: 16 GB
      • A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.
      • Runtime: containerd is required.
    • DNS services for integrating your Wallaroo Enterprise instance. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.

    Wallaroo Enterprise can be installed either interactively or automatically through the kubectl and kots applications.

    Automated Install

    To automatically install Wallaroo into the namespace wallaroo, specify the administrative password and the license file during the installation as in the following format with the following variables:

    • NAMESPACE: The namespace for the Wallaroo Enterprise install, typically wallaroo.
    • LICENSEFILE: The location of the Wallaroo Enterprise license file.
    • SHAREDPASSWORD: The password of for the Wallaroo Administrative Dashboard.
    kubectl kots install wallaroo/ee -n $NAMESPACE --license-file $LICENSEFILE --shared-password $SHAREDPASSWORD
    

    For example, the following settings translate to the following install command:

    • NAMESPACE: wallaroo.
    • LICENSEFILE: myWallaroolicense.yaml
    • SHAREDPASSWORD: snugglebunnies

    kubectl kots install wallaroo/ee -n wallaroo --license-file myWallaroolicense.yaml --shared-password wallaroo

    Interactive Install

    The Interactive Install process allows users to adjust the configuration settings before Wallaroo is deployed. It requires users be able to access the Wallaroo Administrative Dashboard through a browser, typically on port 8080.

    • IMPORTANT NOTE: Users who install Wallaroo through another node such as in the single node installation can port use SSH tunneling to access the Wallaroo Administrative Dashboard. For example:

      ssh IP -L8800:localhost:8800
      
    1. Install the Wallaroo Enterprise Edition using kots install wallaroo/ee, specifying the namespace to install Wallaroo into. For example, if wallaroo is the namespace, then the command is:

      kubectl kots install wallaroo/ee --namespace wallaroo
      
    2. Wallaroo Enterprise Edition will be downloaded and installed into your Kubernetes environment in the namespace specified. When prompted, set the default password for the Wallaroo environment. When complete, Wallaroo Enterprise Edition will display the URL for the Admin Console, and how to end the Admin Console from running.

      • Deploying Admin Console
      • Creating namespace ✓
      • Waiting for datastore to be ready ✓
          Enter a new password to be used for the Admin Console: •••••••••••••
        • Waiting for Admin Console to be ready ✓
      
      • Press Ctrl+C to exit
      • Go to http://localhost:8800 to access the Admin Console
      

    To relaunch the Wallaroo Administrative Dashboard and make changes or updates, use the following command:

    kubectl-kots admin-console --namespace wallaroo
    

    Configure Wallaroo

    Once installed, Wallaroo will continue to run until terminated.

    Change Wallaroo Administrative Dashboard Password

    To change the password to the Wallaroo Administrative Dashboard:

    1. From the command line, use the command:

      kubectl kots reset-password -n {namespace}
      

      For example, for default installations where the Kubernetes namespace is wallaroo, the command would be:

      kubectl kots reset-password -n wallaroo
      

      From here, enter the new password.

    2. From the Wallaroo Administrative Dashboard:

      1. Login and authenticate with the current password.

      2. From the upper right hand corner, select to access the menu and select Change password.

        Select Change Password
      3. Enter the current password, then update and verify with the new password.

        Change Password

    Setup DNS Services

    Wallaroo Enterprise requires integration into your organizations DNS services.

    The DNS Integration Guide details adding the Wallaroo instance to an organizations DNS services. The following is an abbreviated guide that assumes that certificates were already generated.

    1. From the Wallaroo Dashboard, select Config and set the following:

      1. Networking Configuration
        1. Ingress Mode for Wallaroo Endpoints:
          1. None: Port forwarding or other methods are used for access.
          2. Internal: For environments where only nodes within the same Kubernetes environment and no external connections are required.
          3. External: Connections from outside the Kubernetes environment is allowed.
            1. Enable external URL inference endpoints: Creates pipeline inference endpoints. For more information, see Model Endpoints Guide.
      2. DNS
        1. DNS Suffix (Mandatory): The domain name for your Wallaroo instance.
      3. TLS Certificates
        1. Use custom TLS Certs: Checked
        2. TLS Certificate: Enter your TLS Certificate (.crt file).
        3. TLS Private Key: Enter your TLS private key (.key file).
      4. Other settings as desired.
      Wallaroo DNS Records
    2. Once complete, scroll to the bottom of the Config page and select Save config.

    3. A pop-up window will display The config for Wallaroo Enterprise has been updated.. Select Go to updated version to continue.

    4. From the Version History page, select Deploy. Once the new deployment is finished, you will be able to access your Wallaroo services via their DNS addresses.

    To verify the configuration is complete, access the Wallaroo Dashboard through the suffix domain. For example if the suffix domain is wallaroo.example.com then access https://wallaroo.example.com in a browser and verify the connection and certificates.

    Setup Users

    User management is handled through the Wallaroo instance Keycloak service. See the Wallaroo User Management for full guides on setting up users, identity providers, and other user configuration options. This step must be completed before using Wallaroo.

    The following is an abbreviated guide on setting up new Wallaroo users.

    Accessing The Wallaroo Keycloak Dashboard

    Enterprise customers may access their Wallaroo Keycloak dashboard by navigating to https://keycloak.<suffix>, depending on their choice domain suffix supplied during installation.

    Obtaining Administrator Credentials

    The standard Wallaroo installation creates the user admin by default and assigns them a randomly generated password. The admin user credentials are obtained which may be obtained directly from Kubernetes with the following commands, assuming the Wallaroo instance namespace is wallaroo.

    • Retrieve Keycloak Admin Username

      kubectl -n wallaroo \
      get secret keycloak-admin-secret \
      -o go-template='{{.data.KEYCLOAK_ADMIN_USER | base64decode }}'
      
    • Retrieve Keycloak Admin Password

      kubectl -n wallaroo \
      get secret keycloak-admin-secret \
      -o go-template='{{.data.KEYCLOAK_ADMIN_PASSWORD | base64decode }}'
      

    Accessing the User Management Panel

    In the Keycloak Administration Console, click Manage -> Users in the left-hand side menu. Click the View all users button to see existing users. This will be under the host name keycloak.$WALLAROO_SUFFIX. For example, if the $WALLAROO_SUFFIX is wallaroo.example.com, the Keycloak Administration Console would be keycloak.wallaroo.example.com.

    Adding Users

    To add a user through the Keycloak interface:

    1. Click the Add user button in the top-right corner.

    2. Enter the following:

      Wallaroo Enterprise New User
      1. A unique username and email address.
      2. Ensure that the Email Verified checkbox is checked - Wallaroo does not perform email verification.
      3. Under Required User Actions, set Update Password so the user will update their password the next time they log in.
    3. Click Save.

    4. Once saved, select Credentials tab, then the Set Password section, enter the new user’s desired initial password in the Password and Password Confirmation fields.

      Wallaroo Enterprise New User
    5. Click Set Password. Confirm the action when prompted. This will force the user to set their own password when they log in to Wallaroo.

    6. To log into the Wallaroo dashboard, log out as the Admin user and login to the Wallaroo Dashboard as a preconfigured user or via SSO.

    2.1.4 - Wallaroo Enterprise Comprehensive Install Guide: Single Node Linux

    How to set up Wallaroo Enterprise on Single Node Linux

    Single Node Linux

    Organizations can run Wallaroo within a single node Linux environment that meet the prerequisites.

    The following guide is based on installing Wallaroo Enterprise into virtual machines based on Ubuntu 22.04.

    For other environments and configurations, consult your Wallaroo support representative.

    • Prerequisites

    Before starting the bare Linux installation, the following conditions must be met:

    • Have a Wallaroo Enterprise license file. For more information, you can request a demonstration.

    • A Linux bare-metal system or virtual machine with at least 32 cores and 64 GB RAM with Ubuntu 20.04 installed.

    • 650 GB allocated for the root partition, plus 50 GB allocated per node and another 50 GB for the JupyterHub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.

    • Ensure memory swapping is disabled by removing it from /etc/fstab if needed.

    • DNS services for integrating your Wallaroo Enterprise instance. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.

    • IMPORTANT NOTE

      • Wallaroo requires out-bound network connections to download the required container images and other tasks. For situations that require limiting out-bound access, refer to the air-gap installation instructions or contact your Wallaroo support representative. Also note that if Wallaroo is being installed into a cloud environment such as Google Cloud Platform, Microsoft Azure, Amazon Web Services, etc, then additional considerations such as networking, DNS, certificates, and other considerations must be accounted for. For IP address restricted environments, see the Air Gap Installation Guide.
      • The steps below are based on minimum requirements for install Wallaroo in a single node environment.
      • For situations that require limiting external IP access or other questions, refer to your Wallaroo support representative.
    • Template Single Node Scripts

    The following template scripts are provided as examples on how to create single node virtual machines that meet the requirements listed above in AWS, GCP, and Microsoft Azure environments.

    Download template script here: aws-single-node-vm.bash

    # Variables
    
    # The name of the virtual machine
    NAME=$USER-demo-vm                     # eg bob-demo-vm
    
    # The image used : ubuntu/images/2023.4.1/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20230208
    IMAGE_ID=ami-0557a15b87f6559cf
    
    # Instance type meeting the Wallaroo requirements.
    INSTANCE_TYPE=c6i.8xlarge # c6a.8xlarge is also acceptable
    
    # key name - generate keys using Amazon EC2 Key Pairs
    # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html
    # Wallaroo people: https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#KeyPairs:v=3 - 
    MYKEY=DocNode
    
    
    # We will whitelist the our source IP for maximum security -- just use 0.0.0.0/0 if you don't care.
    MY_IP=$(curl -s https://checkip.amazonaws.com)/32
    
    # Create security group in the Default VPC
    aws ec2 create-security-group --group-name $NAME --description "$USER demo" --no-cli-pager
    
    # Open port 22 and 443
    aws ec2 authorize-security-group-ingress --group-name $NAME --protocol tcp --port 22 --cidr $MY_IP --no-cli-pager
    aws ec2 authorize-security-group-ingress --group-name $NAME --protocol tcp --port 443 --cidr $MY_IP --no-cli-pager
    
    # increase Boot device size to 650 GB
    # Change the location from `/tmp/device.json` as required.
    # cat <<EOF > /tmp/device.json 
    # [{
    #   "DeviceName": "/dev/sda1",
    #   "Ebs": { 
    #     "VolumeSize": 650,
    #     "VolumeType": "gp2"
    #   }
    # }]
    # EOF
    
    # Launch instance with a 650 GB Boot device.
    aws ec2 run-instances --image-id $IMAGE_ID --count 1 --instance-type $INSTANCE_TYPE \
        --no-cli-pager \
        --key-name $MYKEY \
        --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":650,"VolumeType":"gp2"}}]'  \
        --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=$NAME}]" \
        --security-groups $NAME
    
    # Sample output:
    # {
    #     "Instances": [
    #         {
    #             ...
    #             "InstanceId": "i-0123456789abcdef",     # Keep this instance-id for later
    #             ...
    #         }
    #     ]
    # }
    
    #INSTANCEID=YOURINSTANCE
          
    # After several minutes, a public IP will be known. This command will retrieve it.
    # aws ec2 describe-instances  --output text --instance-id $INSTANCEID \
    #    --query 'Reservations[*].Instances[*].{ip:PublicIpAddress}'
    
    # Sample Output
    # 12.23.34.56
    
    # KEYFILE=KEYFILELOCATION       #usually ~/.ssh/key.pem - verify this is the same as the key above.
    # SSH to the VM - replace $INSTANCEIP
    #ssh -i $KEYFILE ubuntu@$INSTANCEIP
    
    # Stop the VM - replace the $INSTANCEID
    #aws ec2 stop-instances --instance-id $INSTANCEID
    
    # Restart the VM
    #aws ec2 start-instances --instance-id $INSTANCEID
    
    # Clean up - destroy VM
    #aws ec2 terminate-instances --instance-id $INSTANCEID
    
    • Azure VM Template Script

    • Dependencies

    Download template script here: azure-single-node-vm.bash

    #!/bin/bash
    
    # Variables list.  Update as per your organization's settings
    NAME=$USER-demo-vm                          # eg bob-demo-vm
    RESOURCEGROUP=YOURRESOURCEGROUP
    LOCATION=eastus
    IMAGE=Canonical:0001-com-ubuntu-server-jammy:22_04-lts:22.04.202301140
    
    # Pick a location
    az account list-locations  -o table |egrep 'US|----|Name'
    
    # Create resource group
    az group create -l $LOCATION --name $USER-demo-$(date +%y%m%d)
    
    # Create VM. This will create ~/.ssh/id_rsa and id_rsa.pub - store these for later use.
    az vm create --resource-group $RESOURCEGROUP --name $NAME --image $IMAGE  --generate-ssh-keys \
       --size Standard_D32s_v4 --os-disk-size-gb 500 --public-ip-sku Standard
    
    # Sample output
    # {
    #   "location": "eastus",
    #   "privateIpAddress": "10.0.0.4",
    #   "publicIpAddress": "20.127.249.196",    <-- Write this down as MYPUBIP
    #   "resourceGroup": "mnp-demo-230213",
    #   ...
    # }
    
    # SSH port is open by default. This adds an application port.
    az vm open-port --resource-group $RESOURCEGROUP --name $NAME --port 443
    
    # SSH to the VM - assumes that ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub from above are availble.
    # ssh $MYPUBIP
    
    # Use this Stop the VM ("deallocate" frees resources and billing; "stop" does not)
    # az vm deallocate --resource-group $RESOURCEGROUP --name $NAME
    
    # Restart the VM
    # az vm start --resource-group $RESOURCEGROUP --name $NAME
    • GCP VM Template Script

    Dependencies:

    Download template script here: gcp-single-node-vm.bash

    # Settings
    
    NAME=$USER-demo-$(date +%y%m%d)      # eg bob-demo-230210
    ZONE=us-west1-a                      # For a complete list, use `gcloud compute zones list | egrep ^us-`
    PROJECT=wallaroo-dev-253816          # Insert the GCP Project ID here.  This is the one for Wallaroo.
    
    # Create VM
    
    IMAGE=projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20231030
    
    # Port 22 and 443 open by default
    gcloud compute instances create $NAME \
        --project=$PROJECT \
        --zone=$ZONE \
        --machine-type=e2-standard-32 \
        --network-interface=network-tier=STANDARD,subnet=default \
        --maintenance-policy=MIGRATE \
        --provisioning-model=STANDARD \
        --no-service-account \
        --no-scopes \
        --tags=https-server \
        --create-disk=boot=yes,image=${IMAGE},size=500,type=pd-standard \
        --no-shielded-secure-boot \
        --no-shielded-vtpm \
        --no-shielded-integrity-monitoring \
        --reservation-affinity=any
    
    
    # Get the external IP address
    gcloud compute instances describe $NAME --zone $ZONE --format='get(networkInterfaces[0].accessConfigs[0].natIP)'
    
    # SSH to the VM
    #gcloud compute ssh $NAME --zone $ZONE
    
    # SCP file to the instance - replace $FILE with the file path.  Useful for copying up the license file up to the instance.
    
    #gcloud compute scp --zone $ZONE $FILE $NAME:~/
    
    # SSH port forward to the VM
    #gcloud compute ssh $NAME --zone $ZONE -- -NL 8800:localhost:8800
    
    # Suspend the VM
    #gcloud compute instances stop $NAME --zone $ZONE
    
    # Restart the VM
    #gcloud compute instances start $NAME --zone $ZONE
    
    • Kubernetes Installation Steps

    The following script and steps will install the Kubernetes version and requirements into the Linux node that supports a Wallaroo single node installation.

    The process includes these major steps:

    • Install Kubernetes

    • Install Kots Version

    • Install Kubernetes

    curl is installed in the default scripts provided above. Verify that it is installed if using some other platform.

    1. Verify that the Ubuntu distribution is up to date, and reboot if necessary after updating.

      sudo apt update
      sudo apt upgrade
      
    2. Start the Kubernetes installation with the following script, substituting the URL path as appropriate for your license.

      For Wallaroo versions 2022.4 and below:

      curl https://kurl.sh/9398a3a | sudo bash
      

      For Wallaroo versions 2023.1 and later, the install is based on the license channel. For example, if your license uses the EE channel, then the path is /wallaroo-ee; that is, /wallaroo- plus the lower-case channel name. Note that the Kubernetes install channel must match the License version. Check with your Wallaroo support representative with any questions about your version.

      curl https://kurl.sh/wallaroo-ee | sudo bash
      
      1. If prompted with This application is incompatible with memory swapping enabled. Disable swap to continue? (Y/n), reply Y.
    3. Set up the Kubernetes configuration with the following commands:

      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
      chmod u+w $HOME/.kube/config
      echo 'export KUBECONFIG=$HOME/.kube/config' >> ~/.bashrc
      
    4. Log out, and log back in as the same user. Verify the installation was successful with the following:

      kubectl get nodes
      

      It should return results similar to the following:

      NAME     STATUS   ROLES                  AGE     VERSION
      wallux   Ready    control-plane,master   6m26s   v1.23.6
      
    • Install Kots

    Install kots with the following process.

    1. Run the following script and provide your password for the sudo based commands when prompted.

      curl https://kots.io/install/1.103.3 | REPL_USE_SUDO=y bash
      
    2. Verify kots was installed with the following command:

      kubectl kots version
      

      It should return results similar to the following:

      Replicated KOTS 1.103.3
      

    For instructions on updating the kots version for the Wallaroo Ops installation, see Updating KOTS.

    • Connection Options

    Once Kubernetes has been set up on the Linux node, users can opt to copy the Kubernetes configuration to a local system, updating the IP address and other information as required. See the Configure Access to Multiple Clusters.

    The easiest method is to create a SSH tunnel to the Linux node. Usually this will be in the format:

    ssh $IP -L8800:localhost:8800
    

    For example, in an AWS instance that may be as follows, replaying $KEYFILE with the link to the keyfile and $IP with the IP address of the Linux node.

    ssh -i $KEYFILE ubuntu@$IP -L8800:localhost:8800
    

    In a GCP instance, gcloud can be used as follows, replacing $NAME with the name of the GCP instance, $ZONE with the zone it was installed into.

    gcloud compute ssh $NAME --zone $ZONE -- -NL 8800:localhost:8800
    

    Port forwarding port 8800 is used for kots based installation to access the Wallaroo Administrative Dashboard.

    • Network Configurations

    Note that the standard procedure of installing Wallaroo, the Model Endpoints Guide details how to enable external public communications with the Wallaroo instance.

    When Ingress Mode for Wallaroo interactive services are set to None, the user will have to use port forwarding services to access the Wallaroo instance.

    When Ingress Mode for Wallaroo interactive services are set to Internal or External, the IP address is set via NodePort, and requires the following ports be open to access from remote locations:

    • 80
    • 443
    • 8081
    • 8083

    Check the network settings for the single node linux hosting the Wallaroo instance for instructions on how to enable external or port forwarding access as required.

    Install Wallaroo

    Organizations that use cloud services such as Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure can install Wallaroo Enterprise through the following process. These instructions also work with Single Node Linux based installations.

    Before installation, the following prerequisites must be met:

    • Have a Wallaroo Enterprise license file. For more information, you can request a demonstration.
    • Set up a cloud Kubernetes environment that meets the requirements. Clusters must meet the following minimum specifications:
      • Minimum number of nodes: 4
      • Minimum Number of CPU Cores: 8
      • Minimum RAM: 16 GB
      • A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.
      • Runtime: containerd is required.
    • DNS services for integrating your Wallaroo Enterprise instance. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.

    Wallaroo Enterprise can be installed either interactively or automatically through the kubectl and kots applications.

    Automated Install

    To automatically install Wallaroo into the namespace wallaroo, specify the administrative password and the license file during the installation as in the following format with the following variables:

    • NAMESPACE: The namespace for the Wallaroo Enterprise install, typically wallaroo.
    • LICENSEFILE: The location of the Wallaroo Enterprise license file.
    • SHAREDPASSWORD: The password of for the Wallaroo Administrative Dashboard.
    kubectl kots install wallaroo/ee -n $NAMESPACE --license-file $LICENSEFILE --shared-password $SHAREDPASSWORD
    

    For example, the following settings translate to the following install command:

    • NAMESPACE: wallaroo.
    • LICENSEFILE: myWallaroolicense.yaml
    • SHAREDPASSWORD: snugglebunnies

    kubectl kots install wallaroo/ee -n wallaroo --license-file myWallaroolicense.yaml --shared-password wallaroo

    Interactive Install

    The Interactive Install process allows users to adjust the configuration settings before Wallaroo is deployed. It requires users be able to access the Wallaroo Administrative Dashboard through a browser, typically on port 8080.

    • IMPORTANT NOTE: Users who install Wallaroo through another node such as in the single node installation can port use SSH tunneling to access the Wallaroo Administrative Dashboard. For example:

      ssh IP -L8800:localhost:8800
      
    1. Install the Wallaroo Enterprise Edition using kots install wallaroo/ee, specifying the namespace to install Wallaroo into. For example, if wallaroo is the namespace, then the command is:

      kubectl kots install wallaroo/ee --namespace wallaroo
      
    2. Wallaroo Enterprise Edition will be downloaded and installed into your Kubernetes environment in the namespace specified. When prompted, set the default password for the Wallaroo environment. When complete, Wallaroo Enterprise Edition will display the URL for the Admin Console, and how to end the Admin Console from running.

      • Deploying Admin Console
      • Creating namespace ✓
      • Waiting for datastore to be ready ✓
          Enter a new password to be used for the Admin Console: •••••••••••••
        • Waiting for Admin Console to be ready ✓
      
      • Press Ctrl+C to exit
      • Go to http://localhost:8800 to access the Admin Console
      

    To relaunch the Wallaroo Administrative Dashboard and make changes or updates, use the following command:

    kubectl-kots admin-console --namespace wallaroo
    

    Configure Wallaroo

    Once installed, Wallaroo will continue to run until terminated.

    Change Wallaroo Administrative Dashboard Password

    To change the password to the Wallaroo Administrative Dashboard:

    1. From the command line, use the command:

      kubectl kots reset-password -n {namespace}
      

      For example, for default installations where the Kubernetes namespace is wallaroo, the command would be:

      kubectl kots reset-password -n wallaroo
      

      From here, enter the new password.

    2. From the Wallaroo Administrative Dashboard:

      1. Login and authenticate with the current password.

      2. From the upper right hand corner, select to access the menu and select Change password.

        Select Change Password
      3. Enter the current password, then update and verify with the new password.

        Change Password

    Setup DNS Services

    Wallaroo Enterprise requires integration into your organizations DNS services.

    The DNS Integration Guide details adding the Wallaroo instance to an organizations DNS services. The following is an abbreviated guide that assumes that certificates were already generated.

    1. From the Wallaroo Dashboard, select Config and set the following:

      1. Networking Configuration
        1. Ingress Mode for Wallaroo Endpoints:
          1. None: Port forwarding or other methods are used for access.
          2. Internal: For environments where only nodes within the same Kubernetes environment and no external connections are required.
          3. External: Connections from outside the Kubernetes environment is allowed.
            1. Enable external URL inference endpoints: Creates pipeline inference endpoints. For more information, see Model Endpoints Guide.
      2. DNS
        1. DNS Suffix (Mandatory): The domain name for your Wallaroo instance.
      3. TLS Certificates
        1. Use custom TLS Certs: Checked
        2. TLS Certificate: Enter your TLS Certificate (.crt file).
        3. TLS Private Key: Enter your TLS private key (.key file).
      4. Other settings as desired.
      Wallaroo DNS Records
    2. Once complete, scroll to the bottom of the Config page and select Save config.

    3. A pop-up window will display The config for Wallaroo Enterprise has been updated.. Select Go to updated version to continue.

    4. From the Version History page, select Deploy. Once the new deployment is finished, you will be able to access your Wallaroo services via their DNS addresses.

    To verify the configuration is complete, access the Wallaroo Dashboard through the suffix domain. For example if the suffix domain is wallaroo.example.com then access https://wallaroo.example.com in a browser and verify the connection and certificates.

    Setup Users

    User management is handled through the Wallaroo instance Keycloak service. See the Wallaroo User Management for full guides on setting up users, identity providers, and other user configuration options. This step must be completed before using Wallaroo.

    The following is an abbreviated guide on setting up new Wallaroo users.

    Accessing The Wallaroo Keycloak Dashboard

    Enterprise customers may access their Wallaroo Keycloak dashboard by navigating to https://keycloak.<suffix>, depending on their choice domain suffix supplied during installation.

    Obtaining Administrator Credentials

    The standard Wallaroo installation creates the user admin by default and assigns them a randomly generated password. The admin user credentials are obtained which may be obtained directly from Kubernetes with the following commands, assuming the Wallaroo instance namespace is wallaroo.

    • Retrieve Keycloak Admin Username

      kubectl -n wallaroo \
      get secret keycloak-admin-secret \
      -o go-template='{{.data.KEYCLOAK_ADMIN_USER | base64decode }}'
      
    • Retrieve Keycloak Admin Password

      kubectl -n wallaroo \
      get secret keycloak-admin-secret \
      -o go-template='{{.data.KEYCLOAK_ADMIN_PASSWORD | base64decode }}'
      

    Accessing the User Management Panel

    In the Keycloak Administration Console, click Manage -> Users in the left-hand side menu. Click the View all users button to see existing users. This will be under the host name keycloak.$WALLAROO_SUFFIX. For example, if the $WALLAROO_SUFFIX is wallaroo.example.com, the Keycloak Administration Console would be keycloak.wallaroo.example.com.

    Adding Users

    To add a user through the Keycloak interface:

    1. Click the Add user button in the top-right corner.

    2. Enter the following:

      Wallaroo Enterprise New User
      1. A unique username and email address.
      2. Ensure that the Email Verified checkbox is checked - Wallaroo does not perform email verification.
      3. Under Required User Actions, set Update Password so the user will update their password the next time they log in.
    3. Click Save.

    4. Once saved, select Credentials tab, then the Set Password section, enter the new user’s desired initial password in the Password and Password Confirmation fields.

      Wallaroo Enterprise New User
    5. Click Set Password. Confirm the action when prompted. This will force the user to set their own password when they log in to Wallaroo.

    6. To log into the Wallaroo dashboard, log out as the Admin user and login to the Wallaroo Dashboard as a preconfigured user or via SSO.

    2.2 - Wallaroo Enterprise Simple Install Guide

    How to set up Wallaroo Enterprise for prepared environments.

    The following guide is prepared for organizations that have an environment that meets the prerequisites for installing Wallaroo, and want to jump directly to the installation process.

    For a complete guide that includes environment setup for different cloud providers, select the Wallaroo Enterprise Comprehensive Install Guide.

    Some knowledge of the following will be useful in working with this guide:

    • Working knowledge of Linux distributions, particularly Ubuntu.

    • A cloud provider including Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure experience.

    • Working knowledge of Kubernetes, mainly kubectl and kots or helm.

      The following software or runtimes are required for Wallaroo 2023.4.1. Most are automatically available through the supported cloud providers.

    Software or RuntimeDescriptionMinimum Supported VersionPreferred Version(s)
    KubernetesCluster deployment management1.231.26
    containerdContainer Management1.7.01.7.0
    kubectlKubernetes administrative console application1.261.26

    Install Wallaroo

    1. Install the Wallaroo Enterprise Edition using kots install wallaroo/ee, specifying the namespace to install Wallaroo into. For example, if wallaroo is the namespace, then the command is:

      kubectl kots install wallaroo/ee --namespace wallaroo
      
    2. Wallaroo Enterprise Edition will be downloaded and installed into your Kubernetes environment in the namespace specified. When prompted, set the default password for the Wallaroo environment. When complete, Wallaroo Enterprise Edition will display the URL for the Admin Console, and how to end the Admin Console from running.

      • Deploying Admin Console
      • Creating namespace ✓
      • Waiting for datastore to be ready ✓
          Enter a new password to be used for the Admin Console: •••••••••••••
        • Waiting for Admin Console to be ready ✓
      
      • Press Ctrl+C to exit
      • Go to http://localhost:8800 to access the Admin Console
      

    Configure Wallaroo

    Once installed, Wallaroo will continue to run until terminated.

    To relaunch the Wallaroo Administrative Dashboard and make changes or updates, use the following command:

    kubectl-kots admin-console --namespace wallaroo
    

    DNS Services

    Wallaroo Enterprise requires integration into your organizations DNS services.

    The DNS Integration Guide details adding the Wallaroo instance to an organizations DNS services.

    User Management

    User management is handled through the Wallaroo instance Keycloak service. See the Wallaroo User Management for full guides on setting up users, identity providers, and other user configuration options.

    2.3 - Wallaroo Enterprise Air Gap Install Guide

    Organizations that require Wallaroo be installed into an “air gap” environment - where the Wallaroo instance does not connect to the public Internet - can use these instructions to install Wallaroo into an existing Kubernetes cluster.

    This guide assumes knowledge of how to use Kubernetes and work with internal clusters. The following conditions must be completed before starting an air gap installation of Wallaroo:

    If all prerequisites are met, skip directly to Install Instructions

    General Time to Completion: 30 minutes.

    Before installing Wallaroo version, verify that the following hardware and software requirements are met.

    Environment Requirements

    Environment Hardware Requirements

    The following system requirements are required for the minimum settings for running Wallaroo in a Kubernetes cloud cluster.

    • Minimum number of nodes: 4
    • Minimum Number of CPU Cores: 8
    • Minimum RAM per node: 16 GB
    • Minimum Storage: A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.

    Wallaroo recommends at least 16 cores total to enable all services. At less than 16 cores, services will have to be disabled to allow basic functionality as detailed in this table.

    Note that even when disabling these services, Wallaroo performance may be impacted by the models, pipelines, and data used. The greater the size of the models and steps in a pipeline, the more resources will be required for Wallaroo to operate efficiently. Pipeline resources are set by the pipeline configuration to control how many resources are allocated from the cluster to maintain peak effectiveness for other Wallaroo services. See the following guides for more details.

          
    Cluster Size 8 core16 core32 coreDescription
    Inference The Wallaroo inference engine that performs inference requests from deployed pipelines.
    Dashboard The graphics user interface for configuring workspaces, deploying pipelines, tracking metrics, and other uses.
    Jupyter HUB/LabThe JupyterHub service for running Python scripts, JupyterNotebooks, and other related tasks within the Wallaroo instance.
    Single Lab
    Multiple Labs
    PrometheusUsed for collecting and reporting on metrics. Typical metrics are values such as CPU utilization and memory usage.
    Alerting 
    Model Validation 
    Dashboard Graphs 
    PlateauA Wallaroo developed service for storing inference logs at high speed. This is not a long term service; organizations are encouraged to store logs in long term solutions if required.
    Model Insights 
    Python API 
    Model ConversionConverts models into a native runtime for use with the Wallaroo inference engine.

    To install Wallaroo with minimum services, a configuration file will be used as parts of the kots based installation. For full details on the Wallaroo installation process, see the Wallaroo Install Guides.

    Enterprise Network Requirements

    The following network requirements are required for the minimum settings for running Wallaroo:

    • For Wallaroo Enterprise users: 200 IP addresses are required to be allocated per cloud environment.

    • For Wallaroo Community users: 98 IP addresses are required to be allocated per cloud environment.

    • DNS services integration is required for Wallaroo Enterprise edition. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.

      DNS services integration is required to provide access to the various supporting services that are part of the Wallaroo instance. These include:

      • Simplified user authentication and management.
      • Centralized services for accessing the Wallaroo Dashboard, Wallaroo SDK and Authentication.
      • Collaboration features allowing teams to work together.
      • Managed security, auditing and traceability.

    Environment Software Requirements

    The following software or runtimes are required for Wallaroo 2023.4.1. Most are automatically available through the supported cloud providers.

    Software or RuntimeDescriptionMinimum Supported VersionPreferred Version(s)
    KubernetesCluster deployment management1.231.26
    containerdContainer Management1.7.01.7.0
    kubectlKubernetes administrative console application1.261.26

    Node Selectors

    Wallaroo uses different nodes for various services, which can be assigned to a different node pool to contain resources separate from other nodes. The following nodes selectors can be configured:

    • ML Engine node selector
    • ML Engine Load Balance node selector
    • Database Node Selector
    • Grafana node selector
    • Prometheus node selector
    • Each Lab * Node Selector

    Install Instructions

    The installation is broken into the following major processes:

    Download Assets

    The Wallaroo delivery team the URL and password to your organization’s License and Air Gap Download page. The following links are provided:

    Wallaroo Airgap Download Files
    • (A) Wallaroo Enterprise License File: The Wallaroo enterprise license file for this account. This is downloaded as a yaml file.

    • (B) Wallaroo Airgap Installation File: The air gap installation file that includes the necessary containers for the Wallaroo installation. This is typically about 6 GB in size. By selecting the link icon, the Wallaroo Airgap Installation File URL will be copied to the clipboard that can be used for curl or similar download commands. This file is typically downloaded as wallaroo.airgap.

    • (C) KOTS CLI: The installation files to install kots into the node that manages the Kubernetes cluster. This file is typically downloaded as kots_linux_amd64.tar.gz.

    • (D) KOTS Airgap Bundle: A set of files required by the Kubernetes environment to install Wallaroo via the air gap method. This file is typically downloaded as kotsadm.tar.gz.

    Download these files either through the provided License and Airgap Download page, or by copying the links from the page and using the following command line commands into node performing the air gap installation with curl as follows:

    1. Wallaroo Enterprise License File:

      curl -LO {Link to Wallaroo Enterprise License File}
      
    2. Airgap Installation File. Note the use of the -Lo option to download the Wallaroo air gap file as wallaroo.airgap, and the use of the single quotes around the Wallaroo Air Gap Installation File URL.

      curl -Lo wallaroo.airgap '{Wallaroo Airgap Installation File URL}'
      
    3. KOTS CLI

      curl -LO {Link to KOTS CLI}
      
    4. KOTS Airgap Bundle

      curl -LO {Link to KOTS Airgap Bundle}
      

    Place these files onto the air gap server or node that administrates the Kubernetes cluster. Once these files are on the node, the cluster can be air gapped and the required software installed through the next steps.

    Install Kots

    Install kots into the node managing the Kubernetes cluster with the following commands:

    1. Extract the archive:

      tar zxvf kots_linux_amd64.tar.gz kots
      
    2. Install kots to the /usr/local/bin directory. Adjust this directory to match the location of the kubectl command.

      sudo mv kots /usr/local/bin/kubectl-kots
      
    3. Verify the kots installation by checking the version. The result should be similar to the following:

      kubectl kots version
      Replicated KOTS 1.91.3
      

    Install the Kots Admin Console

    This step will Extract the KOTS Admin Console container images and push them into a private registry. Registry credentials provided in this step must have push access. These credentials will not be stored anywhere or reused later.

    This requires the following:

    • Private Registry Host: The URL of the private registry host used by the Kubernetes cluster.
    • Private Registry Port: The port of the private registry used by the Kubernetes cluster (5000 by default).
    • KOTS Airgap Bundle (default: kotsadm.tar.gz): Downloaded as part of Download Assets step.
    • Registry Push Username: The username with push access to the private registry.
    • Registry Push Password: The password of the registry user with push access to the private registry.

    This command takes the following format:

    kubectl kots admin-console push-images {KOTS Airgap Bundle} \
        {Private Registry Host}:{Private Registry Port} \
        --registry-username {Registry Push Username} \
        --registry-password {Registry Push Password}
    

    Adjust the command based on your organizations registry setup.

    Install Wallaroo Airgap

    This step will install the Wallaroo air gap file into the Kubernetes cluster through the Kots Admin images.

    Registry credentials provided in this step only need to have read access, and they will be stored in a Kubernetes secret in the same namespace where Admin Console will be installed. These credentials will be used to pull the images, and will be automatically created as an imagePullSecret on all of the Admin Console pods.

    This requires the following:

    • Private Registry Host: The URL of the private registry host used by the Kubernetes cluster.
    • Private Registry Port: The port of the private registry used by the Kubernetes cluster (5000 by default).
    • Wallaroo Namespace (default: wallaroo): The kubernetes namespace used to install the Wallaroo isntance.
    • Wallaroo Airgap Installation File (default: wallaroo.airgap): Downloaded as part of Download Assets step.
    • Wallaroo License File: Downloaded as part of Download Assets step.
    • Registry Read Username: The username with read access to the private registry.
    • Registry Read Password: The password of the registry user with read access to the private registry.

    The command will take the following format. Note that the option --license-file {Wallaroo License File} is required. This will point to the license REQUIRED for an air gap installation.

    kubectl kots install wallaroo/ea \
        --kotsadm-registry {Private Registry Host}:{Private Registry Port} \
        --registry-username {Registry Read Username} --registry-password {Registry Read Password} \
        --airgap-bundle {Wallaroo Airgap Installation File} \
        --namespace {Wallaroo Namespace} \
        --license-file {Wallaroo License File}
    

    The following flags can be added to speed up the configuration process:

    • --shared-password {Wallaroo Admin Dashboard Password}: The password used to access the Wallaroo Admin Dashboard.
    • --config-values config.yaml: Sets up the Wallaroo instance configuration based on the supplied yaml file.
    • --no-port-forward: Does not forward port 8800 for use.
    • --skip-preflights: Skip the standard preflight checks and launch the Wallaroo instance.

    For example, the following will install Wallaroo Enterprise into the namespace wallaroo using the provided license file, using the shared password wallaroo and skipping the preflight checks:

    kubectl kots install wallaroo/ea \
        --kotsadm-registry private.host:5000 \
        --registry-username xxx --registry-password yyy \
        --airgap-bundle wallaroo.airgap \
        --namespace wallaroo \
        --license-file license.yaml \
        --shared-password wallaroo \
        --skip-preflights
    

    When complete, a link to the Wallaroo Admin Console will be made available unless the option --no-port-forward is selected.

      • Press Ctrl+C to exit
      • Go to http://localhost:8800 to access the Admin Console
    

    Using Ctrl+C will disable the Wallaroo Admin Console, but the Wallaroo instance and services will continue to run in the cluster.

    To reenable the Wallaroo Admin Console, use the following command:

    kubectl-kots admin-console --namespace {Wallaroo Namespace}
    

    Preflight Checks

    Preflight checks will verify that the Wallaroo instance meets the prerequisites. If any fail, check your Kubernetes environment and verify they are in alignment.

    Preflight checks will be skipped if Wallaroo was installed with the --skip-preflights option.

    Wallaroo Admin Console

    If no license file was provided through the command line, it can be provided through the Wallaroo Admin Console on port 8800. To access the Wallaroo Admin Console, some method of port forwarding through the jump box will have to be configured to the air gapped cluster.

    Status Checks

    While the installer allocates resources and deploys workloads, the status page will show as Missing or Unavailable. If it stays in this state for more than twenty minutes, proceed to troubleshooting or contact Wallaroo technical support.

    Status showing unavailable

    Once the application has become ready, the status indication will turn green and ready Ready.

    Status Ready

    Troubleshooting

    At any time, the administration console can create troubleshooting bundles for Wallaroo technical support to assess product health and help with problems. Support bundles contain logs and configuration files which can be examined before downloading and transmitting to Wallaroo. The console also has a configurable redaction mechanism in cases where sensitive information such as passwords, tokens, or PII (Personally Identifiable Information) need to be removed from logs in the bundle.

    Status Ready

    To manage support bundles:

    1. Log into the administration console.
    2. Select the Troubleshoot tab.
    3. Select Analyze Wallaroo.
    4. Select Download bundle to save the bundle file as a compressed archive. Depending on your browser settings the file download location can be specified.
    5. Send the file to Wallaroo technical support.

    At any time, any existing bundle can be examined and downloaded from the Troubleshoot tab.

    Example Registry Service Install

    The following example demonstrates how to set up an unsecure local registry service that can be used for testing. This process is not advised for production systems, and it only provided as an example for testing the air gap install process. This example uses an Ubuntu 20.04 instance as the installation environment.

    This example assumes that the containerd service is installed and used by the Kubernetes cluster.

    Private Container Registry Service Install Process

    To install a demo container registry service on an Ubuntu 20.04 instance:

    1. Install the registry service:

      sudo apt update
      sudo apt install docker-registry jq
      
    2. Replace the file /etc/docker/registry/config.yml with the following. Note that this configures the service with no security:

      version: 0.1
      log:
      fields:
          service: registry
      storage:
      cache:
          blobdescriptor: inmemory
      filesystem:
          rootdirectory: /var/lib/docker-registry
      http:
      addr: :5000
      headers:
          X-Content-Type-Options: [nosniff]
      health:
      storagedriver:
          enabled: true
          interval: 10s
          threshold: 3
      
    3. Update the containerd service as follows, replacing YOUR-HOST-HERE with the hostname of the registry service configured above. Comment out any existing registry entries and replace with the new insecure registry service:

          [plugins."io.containerd.grpc.v1.cri".registry]
          [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
              [plugins."io.containerd.grpc.v1.cri".registry.mirrors."YOUR-HOST-HERE:5000"]
              endpoint = ["http://YOUR-HOST-HERE:5000"]
          [plugins."io.containerd.grpc.v1.cri".registry.configs]
              [plugins."io.containerd.grpc.v1.cri".registry.configs."YOUR-HOST-HERE:5000".tls]
              insecure_skip_verify = true
      
          # [plugins."io.containerd.grpc.v1.cri".registry]
          #   config_path = ""
          #   [plugins."io.containerd.grpc.v1.cri".registry.auths]
          #   [plugins."io.containerd.grpc.v1.cri".registry.configs]
          #   [plugins."io.containerd.grpc.v1.cri".registry.headers]
          #   [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      
    4. Restart the registry service and containerd service.

      sudo systemctl restart docker-registry
      sudo systemctl restart containerd
      

    2.4 - Wallaroo Enterprise Helm Setup and Install Guides

    Organizations that prefer to use the Helm package manager for Kubernetes can install Wallaroo versions 2022.4 and above via Helm.

    The following procedures demonstrates how to install Wallaroo using Helm. For more information on settings and options for a Helm based install, see the Wallaroo Helm Reference Guides.

    2.4.1 - Wallaroo Helm Standard Cloud Install Procedures

    General Time to Completion: 30 minutes.

    Before installing Wallaroo version, verify that the following hardware and software requirements are met.

    Environment Requirements

    Environment Hardware Requirements

    The following system requirements are required for the minimum settings for running Wallaroo in a Kubernetes cloud cluster.

    • Minimum number of nodes: 4
    • Minimum Number of CPU Cores: 8
    • Minimum RAM per node: 16 GB
    • Minimum Storage: A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.

    Wallaroo recommends at least 16 cores total to enable all services. At less than 16 cores, services will have to be disabled to allow basic functionality as detailed in this table.

    Note that even when disabling these services, Wallaroo performance may be impacted by the models, pipelines, and data used. The greater the size of the models and steps in a pipeline, the more resources will be required for Wallaroo to operate efficiently. Pipeline resources are set by the pipeline configuration to control how many resources are allocated from the cluster to maintain peak effectiveness for other Wallaroo services. See the following guides for more details.

          
    Cluster Size 8 core16 core32 coreDescription
    Inference The Wallaroo inference engine that performs inference requests from deployed pipelines.
    Dashboard The graphics user interface for configuring workspaces, deploying pipelines, tracking metrics, and other uses.
    Jupyter HUB/LabThe JupyterHub service for running Python scripts, JupyterNotebooks, and other related tasks within the Wallaroo instance.
    Single Lab
    Multiple Labs
    PrometheusUsed for collecting and reporting on metrics. Typical metrics are values such as CPU utilization and memory usage.
    Alerting 
    Model Validation 
    Dashboard Graphs 
    PlateauA Wallaroo developed service for storing inference logs at high speed. This is not a long term service; organizations are encouraged to store logs in long term solutions if required.
    Model Insights 
    Python API 
    Model ConversionConverts models into a native runtime for use with the Wallaroo inference engine.

    To install Wallaroo with minimum services, a configuration file will be used as parts of the kots based installation. For full details on the Wallaroo installation process, see the Wallaroo Install Guides.

    Enterprise Network Requirements

    The following network requirements are required for the minimum settings for running Wallaroo:

    • For Wallaroo Enterprise users: 200 IP addresses are required to be allocated per cloud environment.

    • For Wallaroo Community users: 98 IP addresses are required to be allocated per cloud environment.

    • DNS services integration is required for Wallaroo Enterprise edition. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.

      DNS services integration is required to provide access to the various supporting services that are part of the Wallaroo instance. These include:

      • Simplified user authentication and management.
      • Centralized services for accessing the Wallaroo Dashboard, Wallaroo SDK and Authentication.
      • Collaboration features allowing teams to work together.
      • Managed security, auditing and traceability.

    Environment Software Requirements

    The following software or runtimes are required for Wallaroo 2023.4.1. Most are automatically available through the supported cloud providers.

    Software or RuntimeDescriptionMinimum Supported VersionPreferred Version(s)
    KubernetesCluster deployment management1.231.26
    containerdContainer Management1.7.01.7.0
    kubectlKubernetes administrative console application1.261.26

    Node Selectors

    Wallaroo uses different nodes for various services, which can be assigned to a different node pool to contain resources separate from other nodes. The following nodes selectors can be configured:

    • ML Engine node selector
    • ML Engine Load Balance node selector
    • Database Node Selector
    • Grafana node selector
    • Prometheus node selector
    • Each Lab * Node Selector

    Kubernetes Installation Instructions

    This sample Helm installation procedure has the following steps:

    Install Kubernetes

    This example requires the user use a Cloud Kubernetes installation.

    Setup the Kubernetes Cloud cluster as defined in the Wallaroo Enterprise Environment Setup Guides.

    Install Helm

    The follow the instructions from the Installing Helm guide for your environment.

    Install Krew

    The following instructions were taken from the Install Krew guide.

    To install the kubectl plugin krew:

    1. Verify that git is installed in the local system.

    2. Run the following to install krew:

      (
      set -x; cd "$(mktemp -d)" &&
      OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
      ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
      KREW="krew-${OS}_${ARCH}" &&
      curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
      tar zxvf "${KREW}.tar.gz" &&
      ./"${KREW}" install krew
      )
      
    3. Once complete, add the following to the .bashrc file:

      export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
      

    Install Support Tools

    Install the preflight and support-bundle Krew tools via the following commands:

    kubectl krew install preflight
    
    kubectl krew install support-bundle
    

    Install Wallaroo via Helm

    Wallaroo Provided Data

    Members of the Wallaroo support staff will provide the following information:

    • Wallaroo Container Registration Login: Commands to login to the Wallaroo container registry.
    • Preflight and Support Bundle configuration files: The files preflight.yaml and support-bundle.yaml are used in the commands below to complete the preflight process and generate the support bundle package as needed for troubleshooting needs.
    • Preflight verification command: The commands to verify that the Kubernetes environment meets the requirements for the Wallaroo install.
    • Install Wallaroo Command: Instructions on installations into the Kubernetes environment using Helm through the Wallaroo container registry.

    The following steps are used with these command and configuration files to install Wallaroo Enterprise via Helm.

    Registration Login

    The first step in the Wallaroo installation process via Helm is to connect to the Kubernetes environment that will host the Wallaroo Enterprise instance and login into the Wallaroo container registry through the command provided by the Wallaroo support staff. The command will take the following format, replacing $YOURUSERNAME and $YOURPASSWORD with the respective username and password provided.

    helm registry login registry.replicated.com --username $YOURUSERNAME --password $YOURPASSWORD
    

    Preflight Verification

    Preflight verification is performed with the following command, using the preflight.yaml configuration file provided by the Wallaroo support representative as listed above.

    kubectl preflight --interactive=false preflight.yaml
    

    If successful, the tests will show PASS for each preflight requirement as in the following example:

    name: cluster-resources    status: running         completed: 0    total: 2
    name: cluster-resources    status: completed       completed: 1    total: 2
    name: cluster-info         status: running         completed: 1    total: 2
    name: cluster-info         status: completed       completed: 2    total: 2
    
       --- PASS Required Kubernetes Version
          --- Your cluster meets the recommended and required versions of Kubernetes.
       --- PASS Container Runtime
          --- Containerd container runtime was found.
       --- PASS Check Kubernetes environment.
          --- KURL is a supported distribution
       --- PASS Cluster Resources
          --- Cluster resources are satisfactory
       --- PASS Every node in the cluster must have at least 12Gi of memory
          --- All nodes have at least 12 GB of memory capacity
       --- PASS Every node in the cluster must have at least 8 cpus allocatable.
          --- All nodes have at least 8 CPU capacity
    --- PASS   wallaroo
    PASS
    

    The following instructions detail how to install Wallaroo Enterprise via Helm for Kubernetes cloud environments such as Microsoft Azure, Amazon Web Service, and Google Cloud Platform.

    Install Wallaroo

    With the preflight checks and prerequisites met, Wallaroo can be installed via Helm through the following process:

    1. Create namespace. By default, the namespace wallaroo is used:

      kubectl create namespace wallaroo
      
    2. Set the new namespace as the current namespace:

      kubectl config set-context --current --namespace wallaroo
      
    3. Set the TLS certificate secret in the Kubernetes environment:

      1. Create the certificate and private key. It is recommended to name it after the domain name of your Wallaroo instance. For example: wallaroo.example.com. For production environments, organizations are recommended to use certificates from their certificate authority. Note that the Wallaroo SDK will not connect from an external connection without valid certificates. For more information on using DNS settings and certificates, see the Wallaroo DNS Integration Guide.

      2. Create the Kubernetes secret from the certificates created in the previous step, replacing $TLSCONFIG with the name of the Kubernetes secret. Store the secret name for a the step Configure local values file.

        kubectl create secret tls $TLSCONFIG --cert=$TLSSECRETS --key=$TLSSECRETS
        

        For example, if $TLSCONFIG is my-tls-secrets with example.com.crt and key example.com.key, then the command would be translated as

        kubectl create secret tls my-tls-secrets --cert=example.com.crt --key=example.com.key
        
    4. Configure local values file: The default Helm install of Wallaroo contains various default settings. The local values file overwrites values based on the organization needs. The following represents the minimum mandatory values for a Wallaroo installation using certificates and the default LoadBalancer for a cloud Kubernetes cluster. The configuration details below is saved as local-values.yaml for these examples.

      For information on taints and tolerations settings, see the Taints and Tolerations Guide.

      Note the following required settings:

      • domainPrefix and domainSuffix: Used to set the DNS settings for the Wallaroo instance. For more information, see the Wallaroo DNS Integration Guide.
      • deploymentStage and custTlsSecretName: These are set for use with the Kubernetes secret created in the previous step. External connections through the Wallaroo SDK require valid certificates.
      • replImagePrefix: proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs: Sets the Replicated installation containe proxy. Set to proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs unless using a private container registry. Contact a Wallaroo Support representative for details.
      • generate_secrets: Secrets for administrative and other users can be generated by the Helm install process, or set manually. This setting scrambles the passwords during installation.
      • apilb: Sets the apilb service options including the following:
        • serviceType: LoadBalancer: Uses the default LoadBalancer setting for the Kubernetes cloud service the Wallaroo instance is installed into. Replace with the specific service connection settings as required.
        • external_inference_endpoints_enabled: true: This setting is required for performing external SDK inferences to a Wallaroo instance. For more information, see the Wallaroo Model Endpoints Guide
    domainPrefix: "" # optional if using a DNS Prefix
    domainSuffix: {Your Wallaroo DNS Suffix}
    
    deploymentStage: cust
    custTlsSecretName: cust-cert-secret
    
    generate_secrets: true
    
    apilb:
      serviceType: LoadBalancer
      external_inference_endpoints_enabled: true
    
    dashboard:
      clientName: "xx" # Insert the name displayed in the Wallaroo Dashboard
    
    arbEx:
      enabled: true
    
    nats:
      enabled: true
    
    orchestration:
      enabled: true
    
    pipelines:
      enabled: false
    
    imageRegistry: proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs
    replImagePrefix: proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs
    
    minio:
      persistence:
        size: 25Gi     # Minio model storage disk size. Smaller than 10Gi is not recommended.
    
    models:
      enabled: true
    
    pythonAPIServer:
      enabled: true
    1. Install Wallaroo: The Wallaroo support representative will provide the installation command for the Helm install that will use the Wallaroo container registry. This assumes that the preflight checks were successful. This command uses the following format:

      helm install $RELEASE $REGISTRYURL --version $VERSION--values $LOCALVALUES.yaml
      

      Where:

      1. $RELEASE: The name of the Helm release. By default, wallaroo.
      2. $REGISTRYURL: The URl for the Wallaroo container registry service.
      3. $VERSION: The version of Wallaroo to install. For this example, 2022.4.0-main-2297.
      4. $LOCALVALUES: The .yaml file containing the local values overrides. For this example, local-values.yaml.

      For example, for the registration wallaroo the command would be:

      helm install wallaroo oci://registry.replicated.com/wallaroo/EE/wallaroo --version 2022.4.0-main-2297 --values local-values.yaml
      
    2. Verify the Installation: Once the installation is complete, verify the installation with the helm test $RELEASE command. With the settings above, this would be:

      helm test wallaroo
      

      A successful installation will resemble the following:

      NAME: wallaroo
      LAST DEPLOYED: Wed Dec 21 09:15:23 2022
      NAMESPACE: wallaroo
      STATUS: deployed
      REVISION: 1
      TEST SUITE:     wallaroo-fluent-bit-test-connection
      Last Started:   Wed Dec 21 11:58:34 2022
      Last Completed: Wed Dec 21 11:58:37 2022
      Phase:          Succeeded
      TEST SUITE:     wallaroo-test-connections-hook
      Last Started:   Wed Dec 21 11:58:37 2022
      Last Completed: Wed Dec 21 11:58:41 2022
      Phase:          Succeeded
      TEST SUITE:     wallaroo-test-objects-hook
      Last Started:   Wed Dec 21 11:58:41 2022
      Last Completed: Wed Dec 21 11:58:53 2022
      Phase:          Succeeded
      

    At this point, the installation is complete and can be accessed through the fully qualified domain names set in the installation process above. Verify that the DNS settings are accurate before attempting to connect to the Wallaroo instance. For more information, see the Wallaroo DNS Integration Guide.

    To add the initial users if they were not set up through Helm values, see the Wallaroo Enterprise User Management guide.

    Network Configuration

    apilb.serviceType and edgelb.serviceType settings have the following effects depending on whether they are installed on single node Linux installations, or part of a cloud Kubernetes installation.

    SettingSingle Node LinuxCloud Kubernetes
    Internal Only ConnectionsClusterIPClusterIP
    External ConnectionsNodePortLoadBalancer

    Refer to the instructions for environment host for details on IP address allocation and support.

    Troubleshoot Wallaroo

    If issues are detected in the Wallaroo instance, a support bundle file is generated using the support-bundle.yaml file provided by the Wallaroo support representative.

    This creates a collection of log files, configuration files and other details into a .tar.gz file in the same directory as the command is run from in the format support-bundle-YYYY-MM-DDTHH-MM-SS.tar.gz. This file is submitted to the Wallaroo support team for review.

    This support bundle is generated through the following command:

    kubectl support-bundle support-bundle.yaml --interactive=false
    

    Uninstall

    To uninstall Wallaroo via Helm, use the following command replacing the $RELEASE with the name of the release used to install Wallaroo. By default, this is wallaroo:

    helm uninstall wallaroo
    

    It is also recommended to remove the wallaroo namespace after the helm uninstall is complete.

    kubectl delete namespace wallaroo
    

    2.4.2 - Wallaroo Helm Reference Guides

    The following guides include reference details related to installing Wallaroo via Helm.

    2.4.2.1 - Wallaroo Helm Reference Table

    Wallaroo

    A Helm chart for the control plane for Wallaroo

    Configuration

    The following table lists the configurable parameters of the Wallaroo chart and their default values.

    ParameterDescriptionDefault
    kubernetes_distributionOne of: aks, eks, gke, or kurl. May be safe to leave defaulted.""
    imageRegistryimageRegistry where images are pulled from"ghcr.io/wallaroolabs"
    replImagePrefiximageRegistry where images are pulled from, as overridden by Kots"ghcr.io/wallaroolabs"
    assays.enabledControls the display of Assay data in the Dashboardtrue
    custTlsSecretNameName of existing Kubernetes TLS type secret""
    deploymentStageDeployment stage, must be set to “cust” when deployed"dev"
    custTlsCertCustomer provided certificate chain when deploymentStage is “cust”.""
    custTlsKeyCustomer provided private key when deploymentStage is “cust”.""
    nodeSelectorGlobal node selector{}
    tolerationsGlobal tolerations[{"key": "wallaroo", "operator": "Exists", "effect": "NoSchedule"}]
    domainPrefixDNS prefix of Wallaroo endpoints, can be empty for none"xxx"
    domainSuffixDNS suffix of Wallaroo endpoints, MUST be provided"yyy"
    externalIpOverrideUsed in cases where we can’t accurately determine our external, inbound IP address. Normally “”.""
    imagePullPolicyGlobal policy saying when K8s pulls images: Always, Never, or IfNotPresent."Always"
    wallarooSecretNameSecret name for pulling Wallaroo images"regcred"
    privateModelRegistry.enabledIf true, external containerized models can be accessedfalse
    privateModelRegistry.registryRegistry URL, eg “reg.big.corp:3579”""
    privateModelRegistry.emailOptional, for bookkeeping""
    privateModelRegistry.usernameUsername access credential""
    privateModelRegistry.passwordPassword access credential""
    ociRegistry.enabledIf true, pipelines can be published to this OCI registry for use in edge deploymentsfalse
    ociRegistry.registryRegistry URL, eg “reg.big.corp:3579”""
    ociRegistry.repositoryRepository within the registry. May contain cloud account, eg “account123/wallaroothings”""
    ociRegistry.emailOptional, for bookkeeping""
    ociRegistry.usernameUsername access credential""
    ociRegistry.passwordPassword access credential""
    ociRegistry.noTlsSet to true if the registry does not support TLS - for development onlyfalse
    apilb.nodeSelectorstandard node selector for API-LB{}
    apilb.annotationsAnnotations for api-lb service{}
    apilb.serviceTypeService type of api-lb service"ClusterIP"
    apilb.external_inference_endpoints_enabledEnable external URL inference endpoints: pipeline inference endpoints that are accessible outside of the Wallaroo cluster.true
    jupyter.enabledIf true, a jupyer hub was deployed which components can point to.false
    keycloak.useradministrative username"admin"
    keycloak.passworddefault admin password: overridden if generate_secrets is true"admin"
    keycloak.provider.clientIdupstream client id""
    keycloak.provider.clientSecretupstream client secret""
    keycloak.provider.namehuman name for provider""
    keycloak.provider.idType of provider, one of: “github”, “google”, or “OIDC”""
    keycloak.provider.authorizationUrlURL to contact the upstream client for auth requestsnull
    keycloak.provider.clientAuthMethodclient auth method - Must be client_secret_post for OIDC provider type, leave blank otherwise.null
    keycloak.provider.displayNamehuman name for provider, displayed to end user in login dialogsnull
    keycloak.provider.tokenUrlUsed only for ODIC, see token endpoint under Azure endpoints.null
    dbcleaner.schedulewhen the cleaner runs, default is every eight hours"* */8 * * *"
    dbcleaner.maxAgeDaysdelete older than this many days"30"
    plateau.enabledEnable Plateau deploymenttrue
    plateau.diskSizeDisk space to allocate. Smaller than 100Gi is not recommended."100Gi"
    telemetry.enabledUsed only for our CE product. Leave disabled for EE/Helm installs.false
    dashboard.enabledEnable dashboard servicetrue
    dashboard.clientNameCustomer display name which appears at the top of the dashboard window."Fitzroy Macropods, LLC"
    minio.imagePullSecretsMust override for helm + private registry; eg -name: "some-secret"[]
    minio.image.repositoryMust override for helm + private registry"quay.io/minio/minio"
    minio.mcImage.repositoryMust override for helm + private registry"quay.io/minio/mc"
    minio.persistence.sizeMinio model storage disk size. Smaller than 10Gi is not recommended."10Gi"
    fluent-bit.imagePullSecretsMust override for helm + private registry; eg -name: "some-secret"[]
    fluent-bit.image.repositoryMust override for helm + private registry"cr.fluentbit.io/fluent/fluent-bit"
    helmTests.enabledWhen enabled, create “helm test” resources.true
    helmTests.nodeSelectorWhen helm test is run, this selector places the test pods.{}
    explainabilityServer.enabledEnable the model explainability servicefalse
    replImagePrefixSets the replicated image prefix for installation containers. Set to replImagePrefix: proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs unless otherwise instructed.

    2.4.2.2 - Wallaroo Helm Reference Details

    post_delete_hook

    This hook runs when you do helm uninstall unless:

    • you give –no-hooks to helm
    • you set the enable flag to False at INSTALL time.

    imageRegistry

    Registry and Tag portion of Wallaroo images. Third party images are not included. Tag is
    computed at runtime and overridden. In online Helm installs, these should not be touched; in
    airgap Helm installs imageRegistry must be overridden to local registry.

    generate_secrets

    If true, generate random secrets for several services at install time.
    If false, use the generic defaults listed here, which can also be overridden by caller.

    assays

    This is a (currently) Dashboard-specific feature flag to control the display of Assays.

    custTlsSecretName

    To provide TLS certificates, (1) set deploymentStage to “cust”, then (2) provide EITHER the
    name of an existing Kubernetes TLS secret in custTlsSecret OR provide base64 encoded secrets
    in custTlsCert and custTlsKey.

    domainPrefix

    DNS specification for our named external service endpoints.

    To form URLs, we concatenate the optional domainPrefix, the service name in question, and then
    the domainSuffix. Their values are based on license, type, and customer config inputs. They
    MUST be overriden per install via helm values, or by Replicated.

    Community – prefix/suffix in license

    domainPrefixdomainSuffixdashboard_fqdnthing_fqdn (thing = jup, kc, etc)
    ""wallaroo.community(never)(never)
    cust123wallaroo.communitycust123.wallaroo.communitycust123.thing.wallaroo.community

    Enterprise et al – prefix/suffix from config

    domainPrefixdomainSuffixdashboard_fqdnthing_fqdn (thing = jup, kc, etc)
    ""wl.bigcowl.bigcothing.wl.bigco
    cust123wl.bigcocust123.wl.bigcocust123.thing.wl.bigco

    wallarooSecretName

    In online Helm installs, an image pull secret is created and this is its name. The secret allows
    the Kubernetes node to pull images from proxy.replicated.com. In airgap Helm installs, a local
    Secret of type docker-registry must be created and this value set to its name.

    privateModelRegistry

    If the customer has specified a private model container registry, the enable flag will reflect
    and the secret will be populated. registry, username, and password are mandatory. email
    is optional. registry is of the form “hostname:port”. See the Wallaroo Private Model Registry
    Guide for registry specific details.

    ociRegistry

    In order to support edge deployments, a customer-supplied OCI registry is required. The enable
    flag turns on the feature, which causes the secret to be populated. registry, repository,
    username, and password are mandatory. email is optional. registry is of the form
    “hostname:port”. Important: some cloud OCI registries require creation of the repository before
    it can be published to. See the Wallaroo Private Model Registry Guide for registry specific
    details.

    apilb

    Main ingress LB for Wallaroo services.

    The Kubernetes Ingress object is not used, instead we deploy a single Envoy load balancer with a
    single IP in all cases, which serves: TLS termination, authentication (JWT) checking, and both
    host based and path based application routing. Customer should be aware of two values in particular.

    api.serviceType defaults to ClusterIP. If api.serviceType is set to LoadBalancer, cloud
    services will allocate a hosted LB service, in which case the apilb.annotations should be
    provided, in order to pass configuration such as “internal” or “external” to the cloud service.

    Example:

        apilb:
            serviceType: LoadBalancer
            annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    

    keycloak

    Wallaroo can connect to a variety of identity providers, broker OpenID Connect authentication
    requests, and then limit access to endpoints. This section configures a https://www.keycloak.org
    installation. If a provider is specified here, Keycloak will configure itself to use that on
    install. If no providers are specified here, the administrator must login to the Keycloak
    service as the administrative user and either add users by hand or create an auth provider. In
    general, a client must be created upstream and a URL, client ID, and secret (token) for that
    client is entered here.

    dbcleaner

    Manage retention for fluentbit table. This contains log message outputs from orchestration tasks.

    plateau

    Plateau is a low-profile fixed-footprint log processor / event store for fast storage of
    inference results. The amount of disk space provisioned is adjustable. Smaller than “100Gi” is
    not recommended for performance reasons.

    wsProxy

    This controls the wsProxy, and should only be enabled if nats (ArbEx) is also enabled.
    wsProxy is required for the Dashboard to subscribe to events and show notifications.

    arbEx

    Arbitrary Execution

    orchestration

    Pipeline orchestration is general task execution service that allows users to upload arbitrary
    code and have it executed on their behalf by the system. nats and arbex must be enabled.

    pipelines

    Pipelines is service that supports packaging and publishing pipelines suitable for edge deployments.
    It requires ociRegistry to be configured.

    wallsvc

    Wallsvc runs arbex, models, pipelines and orchestration.

    3 - Wallaroo Community Install Guides

    3.1 - Wallaroo Community Simple Install Guide

    How to set up Wallaroo Community in a prepared environment.

    The following guide details how to set up Wallaroo Community in a prepared environment. For a comprehensive guide for the entire process including sample scripts for setting up a cloud environment, see the Wallaroo Community Comprehensive Install Guide

    Install Wallaroo Community

    Wallaroo Community can be installed into a Kubernetes cloud environment, or into a Kubernetes environment that meets the Wallaroo Prerequisites Guide. Organizations that use the Wallaroo Community AWS EC2 Setup procedure do not have to set up a Kubernetes environment, as it is already configured for them.

    This video demonstrates that procedure:

    The procedure assumes at least a basic knowledge of Kubernetes and how to use the kubectl and kots version 1.91.3 applications.

    The procedure involves the following major steps:

    Prerequisites

    Local Software Requirements

    Before starting, verify that all local system requirements are complete as detailed in the Wallaroo Community Local System Prerequisites guide:

    • kubectl: This interfaces with the Kubernetes server created in the Wallaroo environment.

      • For Kots based installs:
      • Cloud Kubernetes environment has been prepared.
      • You have downloaded your Wallaroo Community License file.

      Install Wallaroo

      The environment is ready, the tools are installed - let’s install Wallaroo! The following will use kubectl and kots through the following procedure:

      1. Install the Wallaroo Community Edition using kots install wallaroo/ce, specifying the namespace to install. For example, if wallaroo is the namespace, then the command is:

        kubectl kots install wallaroo/ce --namespace wallaroo
        
      2. Wallaroo Community Edition will be downloaded and installed into your Kubernetes environment in the namespace specified. When prompted, set the default password for the Wallaroo environment. When complete, Wallaroo Community Edition will display the URL for the Admin Console, and how to end the Admin Console from running.

        • Deploying Admin Console
        • Creating namespace ✓
        • Waiting for datastore to be ready ✓
            Enter a new password to be used for the Admin Console: •••••••••••••
          • Waiting for Admin Console to be ready ✓
        
        • Press Ctrl+C to exit
        • Go to http://localhost:8800 to access the Admin Console
        

      Wallaroo Community edition will continue to run until terminated. To relaunch in the future, use the following command:

      kubectl-kots admin-console --namespace wallaroo
      

      Initial Configuration and License Upload Procedure

      Once Wallaroo Community edition has been installed for the first time, we can perform initial configuration and load our Wallaroo Community license file through the following process:

      1. If Wallaroo Community Edition has not started, launch it with the following command:

        ❯ kubectl-kots admin-console --namespace wallaroo
          • Press Ctrl+C to exit
          • Go to http://localhost:8800 to access the Admin Console
        
      2. Enter the Wallaroo Community Admin Console address into a browser. You will be prompted for the default password as set in the step above. Enter it and select Log in.

        Wallaroo Admin Console Initial Login
      3. Upload your license file.

        Wallaroo Admin Upload License
      4. The Configure Wallaroo Community page will be displayed which allows you to customize your Wallaroo environment. For now, scroll to the bottom and select Continue. These settings can be customized at a later date.

      5. The Wallaroo Community Admin Console will run the preflight checks to verify that all of the minimum requirements are not met. This may take a few minutes. If there are any issues, Wallaroo can still be launched but may not function properly. When ready, select Continue.

        Wallaroo Admin Preflight Successful
      6. The Wallaroo Community Dashboard will be displayed. There may be additional background processes that are completing their setup procedures, so there may be a few minute wait until those are complete. If everything is ready, then the Wallaroo Dashboard will show a green Ready.

        Wallaroo Admin Admin Ready
      7. Under the license information is the DNS entry for your Wallaroo instance. This is where you and other users of your Wallaroo instance can log in. In this example, the URL will be https://beautiful-horse-9537.wallaroo.community. Note that it may take a few minutes for the DNS entries to propagate and this URL to be available.

        Wallaroo Instance URL
      8. You will receive an email invitation for the email address connected to this URL with a temporary password and a link to this Wallaroo instance’s URL. Either enter the URL for your Wallaroo instance or use the link in the email.

      9. To login to your new Wallaroo instance, enter the email address and temporary password associated with the license.

        Wallaroo Initial Login

      With that, Wallaroo Community edition is launched and ready for use! You can end the Admin Console from your terminal session above. From this point on you can just use the Wallaroo instance URL.

      3.2 - Wallaroo Community GitHub Codespaces Install Guide

      How to set up Wallaroo Community using GitHub Codespaces

      The following details how to set up Wallaroo Community using GitHub Codespaces. This provides a quick method of setting up Wallaroo Community with just a few commands. You get the free license from the Wallaroo Community Portal site, and the rest of the process is handled by GitHub and Wallaroo.

      Prerequisites

      Installation Steps

      1. From the Wallaroo Community Portal page, follow the registration instructions. If a Wallaroo Community license already exists for this account, enter the same name and email address to retrieve the registration link.

      2. Download the Wallaroo Community license. This provides both the Wallaroo Community license in YAML format and the base64 encoded version. Store both in a secure location. The base64 ended version will be used in this process.

      3. From the License Download page:

        1. Select Download license file. This provides two files: The YAML version of the license file, and an encoded base 64 version of the license file.
        2. Select Github Codespaces, then select Click to install Wallaroo in your GitHub Codespaces.
      4. From the Create codespace GitHub page for Wallaroo Community, set the following:

        Wallaroo GitHub Install Options
        1. Dev container configuration: Set to Wallaroo Codespaces Data Science.
        2. Region: Select the region where the codespace will run. This should be the region geographically closest to the end user.
          1. Europe West
          2. Southeast Asia
          3. US West
          4. US East
        3. Machine Type: Set the number of CPUs to use. Leave as 16-cores - anything less the containers will not run.
        4. Wallaroo Community License: Provide the base64 version of the Wallaroo license here.
      5. When complete, select Create Codespace. Wait until the process is complete - this usually takes about 15 minutes.

      To see the installation process in action, use one of the following options:

      • Enter either Control-Shift-P or Command-Shift-P (for MacOS systems) and select Codespaces: View Creation Log.
      • Select from the lower left corner Codespaces and then View Creation Log.

      Once the entry “Finished configuring codespace” is displayed, the Wallaroo Community installation is complete.

      Wallaroo Community First Login

      If this is your first login, you will receive an email invitation for the email address connected to this URL with a temporary password. Follow the link provided in the email. If not already logged into GitHub, a prompt will appear for you to login to GitHub to manage the port forwarding required to connect to the Wallaroo Community instance through your browser.

      To login to your new Wallaroo instance, enter the email address and temporary password associated with the license. Enter a new password twice to set a new permanent password, and you will be logged into the Wallaroo Community instance.

      Subsequent logins will are performed through the same URL.

      Stopping and Starting Wallaroo Community

      To start and stop the Wallaroo Community codespace:

      • To stop the current Wallaroo Community codespace:
        • From the Wallaroo Community Codespace page, from the bottom left select Codespaces, then select the action Stop Current Codespace.
        • The other option is to press F1 and select Stop Current Codespace.
      • To start a Wallaroo Community codespace:
        • Access your GitHub Codespaces page from GitHub by selecting your user profile icon in the upper right corner, then selecting My Codespaces.
        • From your GitHub Codespaces page, select your Wallaroo Community code space.
        • From the menu list on the right side, select Open In…, then Open in Browser to launch the Wallaroo Community Codespace page. This will launch the Wallaroo Community containers and make them available once they are done starting up.

      See Setting your timeout period for GitHub Codespaces for information on timeout settings.

      Additional GitHub Codespaces Options

      Update the Wallaroo Community License

      When Wallaroo Community is installed through GitHub Codespaces, the base64 hash of the Wallaroo license used in the installation process is stored in GitHub.

      To update this license:

      1. Select your GitHub user profile, then Settings.

      2. Select Codespaces, then Codespaces secrets. The Wallaroo Community license is listed as WALLAROO_LICENSE. Use Update to update the license or Delete to remove if it is no longer needed.

        Wallaroo GitHub Codespaces Community License

      Open Ports for Multiple Users

      To provide for multiple users, the following ports must be set from private to public. This step is not required for users who will be the only one accessing Wallaroo Community though their codespace.

      • keycloak: Port 8080, used for authentication to the Wallaroo service.
      • hub: Port 8081, used for the JupyterHub service.
      • dashboard: Port 8443, used for the Wallaroo Dashboard.

      To update the network settings:

      1. Access the Wallaroo Community codespace settings by entering either Control-Shift-P or Command-Shift-P (for MacOS systems) and select View: Toggle Terminal.
      2. Right click keycloak, hub, and dashboard and set Port Visibility to Public.

      3.3 - Wallaroo Community Comprehensive Install Guide

      How to set up Wallaroo Community in various environments.

      This guide is targeted towards system administrators and data scientists who want to work with the easiest, fastest, and free method of running your own machine learning models.

      A typical installation of Wallaroo Community follows this process:

      StepDescription   Average Setup Time   
      Download LicenseCreate an environment that meets the Wallaroo prerequisites5 minutes
      Set Up EnvironmentSet up the cloud environment hosting the Wallaroo instance.30 minutes
      Install WallarooInstall Wallaroo into a prepared environment15 minutes

      Register Your Wallaroo Community Account

      The first step to installing Wallaroo CE is to set up your Wallaroo Community Account at the web site https://portal.wallaroo.community. This process typically takes about 5 minutes.

      Registration Portal

      Once you’ve submitted your credentials, you’ll be sent an email with a link to your license file.

      Invitation Email

      Follow the link and download your license file. Store it in a secure location.

      Download license

      Redownload License

      If your license is misplaced or otherwise lost, it can be downloaded again later from the same link, or by following the registration steps again to be provided with a link to your license file.


      Setup Environments

      The following setup guides are used to set up the environment that will host the Wallaroo instance. Verify that the environment is prepared and meets the Wallaroo Prerequisites Guide.

      • Wallaroo Community AWS EC2 Setup Instructions

      The following instructions are made to assist users set up their Amazon Web Services (AWS) environment for running Wallaroo using AWS virtual servers with EC2. This allows organizations to stand a single virtual machine and used a pre-made Amazon Machine Images (AMIs) to quickly stand up an environment that can be used to install Wallaroo.

      • AWS Prerequisites

      To install Wallaroo in your AWS environment based on these instructions, the following prerequisites must be met:

      • Register an AWS account: https://aws.amazon.com/ and assign the proper permissions according to your organization’s needs. This must be a paid AWS account - Wallaroo will not operate on the free tier level of virtual machines.

      • Steps

        • Create the EC2 VM

      To create your Wallaroo instance using a pre-made AMI:

      1. Log into AWS cloud console.

      2. Set the region to N. Virginia. Other regions will be added over time.

        Set the region
      3. Select Services -> EC2.

        Select EC2
      4. Select Instances, then from the upper right hand section Launch Instances->Launch Instances.

        Select Launch Instances
      5. Set the Name and any additional tags.

      6. In Application and OS Images, enter Wallaroo Install and press Enter.

      7. From the search results, select Community AMIs and select Wallaroo Installer 3a.

        Select AMI
      8. Set the Instance Type as c6i.8xlarge or c6a.8xlarge as the minimum machine type. This provides 32 cores with 60 GB memory.

        Select Instance Type
      9. For Key pair (login) select one of the following:

        Select or Create Key Pair
        1. Select an existing Key pair name
        2. Select Create new key pair and set the following:
          1. Name: The name of the new key pair.
          2. Key pair type: Select either RSA or ED25519.
          3. Private key file format: Select either .pem or .ppk. These instructions are based on the .pem file.
          4. Select Create key pair when complete.
      10. Set the following for Network settings:

        Set Network
        1. Firewall: Select Create security group or select from an existing one that best fits your organization.
        2. Allow SSH traffic from: Set to Enabled and Anywhere 0.0.0.0/0.
        3. Allow HTTPs traffic from the internet: Set to Enabled.
      11. Set the following for Configure Storage:

        1. Set Root volume to at least 400 GiB, type standard.
      12. Review the Summary and verify the following:

        1. Number of instances: 1
        2. Virtual server type: Matches the minimum requirement listed above.
        3. Verify the other settings are accurate.
      13. Select Launch Instance.

      It is recommended to give the instance time to complete its setup process. This typically takes 20 minutes.

      • Verify the Setup

      To verify the environment is setup for Wallaroo:

      1. From the EC2 Dashboard, select the virtual machine created for your Wallaroo instance.

      2. Note the Public IPv4 DNS address.

        Instance Public DNS
      3. From a terminal, run ssh to connect to your virtual machine. The installation requires access to port 8800 and the private key selected or created in the instructions above.

        The ssh command format for connecting to your virtual machine uses the following format, replacing the $keyfile, $VM_DNS with your private key file and the DNS address to your Amazon VM:

        ssh -i "$keyfile" ubuntu@$VM_DNS -L8800:localhost:8800
        

        For example, a $keyfile of Doc Sample Key.pem and $VM_DNS of ec2-54-160-227-100.compute-1.amazonaws.com would be as follows:

        ssh -i "Doc Sample Key.pem" ubuntu@ec2-54-160-227-100.compute-1.amazonaws.com -L8800:localhost:8800
        
      4. If the Kubernetes setup is still installing, wait until complete and when prompted select EXIT to complete the process. This process may take up to 20 to 30 minutes.

        Complete Kubernetes Install
      • Cost Saving Tips The following tips can be used to save costs on your AWS EC2 instance.
        • Stop Instances When Not In Use

      One cost saving measure is to stop instances when not in use. If you intend to stop an instance, register it with static IP address so when it is turned back on your services will continue to function without interruption.

      Stop instance.

      Reference: How do I associate a static public IP address with my EC2 Windows or Linux instance?.

      • Troubleshooting
        • I keep seeing the errors such as connect failed. Is this a problem?
          • Sometimes you may see an error such as channel 3: open failed: connect failed: Connection refused. This is the ssh port forwarding attempting to connect to port 8800 during the installation, and can be ignored.
        • When Launching JupyterHub, I get a Server 500 error.
          • If you shut down and restart a Wallaroo instance in a new environment or change the IP address, some settings may not be updated. Run the following command to restart the deployment process and update the settings to match the current environment:

            kubectl rollout restart deployment hub
            
      • Setup AWS EKS Environment for Wallaroo

      The following instructions are made to assist users set up their Amazon Web Services (AWS) environment for running Wallaroo using AWS Elastic Kubernetes Service (EKS).

      These represent a recommended setup, but can be modified to fit your specific needs.

      If the prerequisites are already met, skip ahead to Install Wallaroo.

      The following video demonstrates this process:

      • AWS Prerequisites

      To install Wallaroo in your AWS environment based on these instructions, the following prerequisites must be met:

      • Register an AWS account: https://aws.amazon.com/ and assign the proper permissions according to your organization’s needs.
      • The Kubernetes cluster must include the following minimum settings:
        • Nodes must be OS type Linux with using the containerd driver.
        • Role-based access control (RBAC) must be enabled.
        • Minimum of 4 nodes, each node with a minimum of 8 CPU cores and 16 GB RAM. 50 GB will be allocated per node for a total of 625 GB for the entire cluster.
        • RBAC is enabled.
        • Recommended Aws Machine type: c5.4xlarge. For more information, see the AWS Instance Types.
      • Installed eksctl version 0.101.0 and above.
      • If the cluster will utilize autoscaling, install the Cluster Autoscaler on AWS.
      • AWS Cluster Recommendations

      The following recommendations will assist in reducing the cost of a cloud based Kubernetes Wallaroo cluster.

      • Turn off the cluster when not in use. An AWS EKS (Elastic Kubernetes Services) cluster can be turn off when not in use, then turned back on again when needed. If organizations adopt this process, be aware of the following issues:

        • IP Address Reassignment: The load balancer public IP address may be reassigned when the cluster is restarted by the cloud service unless a static IP address is assigned. For more information in Amazon Web Services see the Associate Elastic IP addresses with resources in your VPC user guide.
      • Assign to a Single Availability Zone: Clusters that span multiple availability zones may have issues accessing persistent volumes that were provisioned in another availability zone from the node when the node is restarted. The simple solution is to assign the entire cluster into a single availability zone. For more information in Amazon Web Services see the Regions and Zones guide.

        The scripts and configuration files are set up to create the AWS environment for a Wallaroo instance are based on a single availability zone. Modify the script as required for your organization.

      • Community Cluster Setup Instructions

      The following is based on the requirements for Wallaroo Community. Note that Wallaroo Community does not use adaptive nodepools. Adapt the settings as required for your organization’s needs, as long as they meet the prerequisites listed above.

      This sample YAML file can be downloaded from here:

      wallaroo_community_aws_install.yaml

      apiVersion: eksctl.io/v1alpha5
      kind: ClusterConfig
      
      metadata:
        # replace with the name of your server
        name: wallarooAWS
        # replace with your location
        region: us-east-1
        version: "1.25"
      
      addons:
        - name: aws-ebs-csi-driver
      
      iam:
        withOIDC: true
      
      nodeGroups:
        - name: mainpool
          instanceType: m5.2xlarge
          desiredCapacity: 4
          containerRuntime: containerd
          amiFamily: AmazonLinux2
          availabilityZones:
            - us-east-1a
      
      
      • Install AWS Command Line Tools

      The following steps require the installation of the following Amazon Web Services command line tools:

      Create the cluster with the following command, which creates the environment and sets the correct Kubernetes version.

      eksctl create cluster -f aws.yaml
      

      During the process the Kuberntes credentials will be copied into the local environment. To verify the setup is complete, use the kubectl get nodes command to display the available nodes as in the following example:

      kubectl get nodes
      
      NAME                                           STATUS   ROLES    AGE     VERSION
      ip-192-168-21-253.us-east-2.compute.internal   Ready    <none>   13m     v1.23.8-eks-9017834
      ip-192-168-30-36.us-east-2.compute.internal    Ready    <none>   13m     v1.23.8-eks-9017834
      ip-192-168-55-123.us-east-2.compute.internal   Ready    <none>   12m     v1.23.8-eks-9017834
      ip-192-168-79-70.us-east-2.compute.internal    Ready    <none>   13m     v1.23.8-eks-9017834
      
      • Setup Azure Environment for Wallaroo

      The following instructions are made to assist users set up their Microsoft Azure Kubernetes environment for running Wallaroo Community. These represent a recommended setup, but can be modified to fit your specific needs.

      If your prepared to install the environment now, skip to Setup Environment Steps.

      There are two methods we’ve detailed here on how to setup your Kubernetes cloud environment in Azure:

      • Quick Setup Script: Download a bash script to automatically set up the Azure environment through the Microsoft Azure command line interface az.
      • Manual Setup Guide: A list of the az commands used to create the environment through manual commands.

      The following video demonstrates the manual guide:

      • Azure Prerequisites

      To install Wallaroo in your Microsoft Azure environment, the following prerequisites must be met:

      • Register a Microsoft Azure account: https://azure.microsoft.com/.
      • Install the Microsoft Azure CLI and complete the Azure CLI Get Started Guide to connect your az application to your Microsoft Azure account.
      • The Kubernetes cluster must include the following minimum settings:
        • Nodes must be OS type Linux the containerd driver as the default.
        • Role-based access control (RBAC) must be enabled.
        • Minimum of 4 nodes, each node with a minimum of 8 CPU cores and 16 GB RAM. 50 GB will be allocated per node for a total of 625 GB for the entire cluster.
        • RBAC is enabled.
        • Minimum machine type is set to to Standard_D8s_v4.
      • Azure Cluster Recommendations

      The following recommendations will assist in reducing the cost of a cloud based Kubernetes Wallaroo cluster.

      • Turn off the cluster when not in use. An Azure Kubernetes Service (AKS) cluster can be turn off when not in use, then turned back on again when needed to save on costs. For more information on starting and stopping an AKS cluster, see the Stop and Start an Azure Kubernetes Service (AKS) cluster guide.

        If organizations adopt this process, be aware of the following issues:

      • Assign to a Single Availability Zone: Clusters that span multiple availability zones may have issues accessing persistent volumes that were provisioned in another availability zone from the node when the cluster is restarted. The simple solution is to assign the entire cluster into a single availability zone. For more information in Microsoft Azure see the Create an Azure Kubernetes Service (AKS) cluster that uses availability zones guide.

        The scripts and configuration files are set up to create the Azure environment for a Wallaroo instance are based on a single availability zone. Modify the script as required for your organization.

      Variable NameDefault ValueDescription
      WALLAROO_RESOURCE_GROUPwallaroocegroupThe Azure Resource Group used for the Kubernetes environment.
      WALLAROO_GROUP_LOCATIONeastusThe region that the Kubernetes environment will be installed to.
      WALLAROO_CONTAINER_REGISTRYwallarooceacrThe Azure Container Registry used for the Kubernetes environment.
      WALLAROO_CLUSTERwallarooceaksThe name of the Kubernetes cluster that Wallaroo is installed to.
      WALLAROO_SKU_TYPEBaseThe Azure Kubernetes Service SKU type.
      WALLAROO_NODEPOOLwallaroocepoolThe main nodepool for the Kubernetes cluster.
      WALLAROO_VM_SIZEStandard_D8s_v4The VM type used for the standard Wallaroo cluster nodes.
      WALLAROO_CLUSTER_SIZE4The number of nodes in the cluster.
      • Quick Setup Script

      The following sample script creates an Azure Kubernetes environment ready for use with Wallaroo Community. This script requires the following prerequisites listed above.

      Modify the installation file to fit for your organization. The only parts that require modification are the variables listed in the beginning as follows:

      The following script is available for download: wallaroo_community_azure_install.bash

      The following steps are geared towards a standard Linux or macOS system that supports the prerequisites listed above. Modify these steps based on your local environment.

      1. Download the script above.

      2. In a terminal window set the script status as execute with the command chmod +x wallaroo_community_azure_install.bash.

      3. Modify the script variables listed above based on your requirements.

      4. Run the script with either bash wallaroo_community_azure_install.bash or ./wallaroo_community_azure_install.bash from the same directory as the script.

        Azure Quick Setup Script
      • Manual Setup Guide

      The following steps are guidelines to assist new users in setting up their Azure environment for Wallaroo. Feel free to replace these with commands with ones that match your needs.

      See the Azure Command-Line Interface for full details on commands and settings.

      The following are used for the example commands below. Replace them with your specific environment settings:

      • Azure Resource Group: wallarooCEGroup
      • Azure Resource Group Location: eastus
      • Azure Container Registry: wallarooCEAcr
      • Azure Kubernetes Cluster: wallarooCEAKS
      • Azure Container SKU type: Base
      • Azure Nodepool Name: wallarooCEPool

      Setting up an Azure AKS environment is based on the Azure Kubernetes Service tutorial, streamlined to show the minimum steps in setting up your own Wallaroo environment in Azure.

      This follows these major steps:

      • Create an Azure Resource Group

      • Create an Azure Container Registry

      • Create the Azure Kubernetes Environment

      • Set Variables

      The following are the variables used in the environment setup process. Modify them as best fits your organization’s needs.

      WALLAROO_RESOURCE_GROUP=wallaroocegroupdocs
      WALLAROO_GROUP_LOCATION=eastus
      WALLAROO_CONTAINER_REGISTRY=wallarooceacrdocs
      WALLAROO_CLUSTER=wallarooceaksdocs
      WALLAROO_SKU_TYPE=Base
      WALLAROO_NODEPOOL=wallaroocepool
      WALLAROO_VM_SIZE=Standard_D8s_v4
      WALLAROO_CLUSTER_SIZE=4
      
      • Create an Azure Resource Group

      To create an Azure Resource Group for Wallaroo in Microsoft Azure, use the following template:

      az group create --name $WALLAROO_RESOURCE_GROUP --location $WALLAROO_GROUP_LOCATION
      

      (Optional): Set the default Resource Group to the one recently created. This allows other Azure commands to automatically select this group for commands such as az aks list, etc.

      az configure --defaults group=$WALLAROO_RESOURCE_GROUP
      
      • Create an Azure Container Registry

      An Azure Container Registry(ACR) manages the container images for services includes Kubernetes. The template for setting up an Azure ACR that supports Wallaroo is the following:

      az acr create -n $WALLAROO_CONTAINER_REGISTRY -g $WALLAROO_RESOURCE_GROUP --sku $WALLAROO_SKU_TYPE --location $WALLAROO_GROUP_LOCATION
      
      • Create an Azure Kubernetes Services

      And now we can create our Kubernetes service in Azure that will host our Wallaroo that meet the prerequisites. Modify the settings to meet your organization’s needs. This creates a 4 node cluster with a total of 32 cores.

      az aks create \
      --resource-group $WALLAROO_RESOURCE_GROUP \
      --name $WALLAROO_CLUSTER \
      --node-count $WALLAROO_CLUSTER_SIZE \
      --generate-ssh-keys \
      --vm-set-type VirtualMachineScaleSets \
      --load-balancer-sku standard \
      --node-vm-size $WALLAROO_VM_SIZE \
      --nodepool-name $WALLAROO_NODEPOOL \
      --nodepool-name mainpool \
      --attach-acr $WALLAROO_CONTAINER_REGISTRY \
      --kubernetes-version=1.23.8 \
      --zones 1 \
      --location $WALLAROO_GROUP_LOCATION
      
      • Download Wallaroo Kubernetes Configuration

      Once the Kubernetes environment is complete, associate it with the local Kubernetes configuration by importing the credentials through the following template command:

      az aks get-credentials --resource-group $WALLAROO_RESOURCE_GROUP --name $WALLAROO_CLUSTER
      

      Verify the cluster is available through the kubectl get nodes command.

      kubectl get nodes
      
      NAME                               STATUS   ROLES   AGE   VERSION
      aks-mainpool-37402055-vmss000000   Ready    agent   81m   v1.23.8
      aks-mainpool-37402055-vmss000001   Ready    agent   81m   v1.23.8
      aks-mainpool-37402055-vmss000002   Ready    agent   81m   v1.23.8
      aks-mainpool-37402055-vmss000003   Ready    agent   81m   v1.23.8
      
      • Setup GCP Environment for Wallaroo

      The following instructions are made to assist users set up their Google Cloud Platform (GCP) Kubernetes environment for running Wallaroo. These represent a recommended setup, but can be modified to fit your specific needs. In particular, these instructions will provision a GKE cluster with 32 CPUs in total. Please ensure that your project’s resource limits support that.

      • Quick Setup Guide: Download a bash script to automatically set up the GCP environment through the Google Cloud Platform command line interface gcloud.
      • Manual Setup Guide: A list of the gcloud commands used to create the environment through manual commands.

      The following video demonstrates the manual guide:

      • GCP Prerequisites

      Organizations that wish to run Wallaroo in their Google Cloud Platform environment must complete the following prerequisites:

      • GCP Cluster Recommendations

      The following recommendations will assist in reducing the cost of a cloud based Kubernetes Wallaroo cluster.

      • Turn off the cluster when not in use. A GCP Google Kubernetes Engine (GKE) cluster can be turn off when not in use, then turned back on again when needed. If organizations adopt this process, be aware of the following issues:

        • IP Address Reassignment: The load balancer public IP address may be reassigned when the cluster is restarted by the cloud service unless a static IP address is assigned. For more information in Google Cloud Platform see the Configuring domain names with static IP addresses user guide.
      • Assign to a Single Availability Zone: Clusters that span multiple availability zones may have issues accessing persistent volumes that were provisioned in another availability zone from the node when the cluster is restarted. The simple solution is to assign the entire cluster into a single availability zone. For more information in Google Cloud Platform see the Regions and zones guide.

        The scripts and configuration files are set up to create the GCP environment for a Wallaroo instance are based on a single availability zone. Modify the script as required for your organization.

      • Standard Setup Variables

      The following variables are used in the Quick Setup Script and the Manual Setup Guide. Modify them as best fits your organization.

      Variable NameDefault ValueDescription
      WALLAROO_GCP_PROJECTwallaroo-ceThe name of the Google Project used for the Wallaroo instance.
      WALLAROO_CLUSTERwallaroo-ceThe name of the Kubernetes cluster for the Wallaroo instance.
      WALLAROO_GCP_REGIONus-central1The region the Kubernetes environment is installed to. Update this to your GCP Computer Engine region.
      WALLAROO_NODE_LOCATIONus-central1-fThe location the Kubernetes nodes are installed to. Update this to your GCP Compute Engine Zone.
      WALLAROO_GCP_NETWORK_NAMEwallaroo-networkThe Google network used with the Kubernetes environment.
      WALLAROO_GCP_SUBNETWORK_NAMEwallaroo-subnet-1The Google network subnet used with the Kubernetes environment.
      WALLAROO_GCP_MACHINE_TYPEe2-standard-8Recommended VM size per GCP node.
      WALLAROO_CLUSTER_SIZE4Number of nodes installed into the cluster. 4 nodes will create a 32 core cluster.
      • Quick Setup Script

      A sample script is available here, and creates a Google Kubernetes Engine cluster ready for use with Wallaroo Community. This script requires the prerequisites listed above, and uses the variables as listed in Standard Setup Variables.

      The following script is available for download: wallaroo_community_gcp_install.bash

      The following steps are geared towards a standard Linux or macOS system that supports the prerequisites listed above. Modify these steps based on your local environment.

      1. Download the script above.

      2. In a terminal window set the script status as execute with the command chmod +x bash wallaroo_community_gcp_install.bash.

      3. Modify the script variables listed above based on your requirements.

      4. Run the script with either bash wallaroo_community_gcp_install.bash or ./wallaroo_community_gcp_install.bash from the same directory as the script.

        GCP Quick Setup Script
      • Manual Setup Guide

      The following steps are guidelines to assist new users in setting up their GCP environment for Wallaroo. Feel free to replace these with commands with ones that match your needs.

      See the Google Cloud SDK for full details on commands and settings.

      The commands below are set to meet the prerequisites listed above, and uses the variables as listed in Standard Setup Variables. Modify them as best fits your organization’s needs.

      • Set Variables

      The following are the variables used in the environment setup process. Modify them as best fits your organization’s needs.

      WALLAROO_GCP_PROJECT=wallaroo-ce
      WALLAROO_CLUSTER=wallaroo-ce
      WALLAROO_GCP_REGION=us-central1
      WALLAROO_NODE_LOCATION=us-central1-f
      WALLAROO_GCP_NETWORK_NAME=wallaroo-network
      WALLAROO_GCP_SUBNETWORK_NAME=wallaroo-subnet-1
      WALLAROO_GCP_MACHINE_TYPE=e2-standard-8
      WALLAROO_CLUSTER_SIZE=4
      
      • Create a GCP Network

      First create a GCP network that is used to connect to the cluster with the gcloud compute networks create command. For more information, see the gcloud compute networks create page.

      gcloud compute networks \
      create $WALLAROO_GCP_NETWORK_NAME \
      --bgp-routing-mode regional \
      --subnet-mode custom
      

      Verify it’s creation by listing the GCP networks:

      gcloud compute networks list
      
      • Create the GCP Wallaroo Cluster

      Once the network is created, the gcloud container clusters create command is used to create a cluster. For more information see the gcloud container clusters create page.

      Note that three nodes are created by default, so one more is added with the --num-nodes setting to meet the Wallaroo prerequisites. For Google GKE, containerd is enabled by default and so does not need to be specified during the setup procedure: (https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd)[https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd].

      gcloud container clusters \
      create $WALLAROO_CLUSTER \
      --region $WALLAROO_GCP_REGION \
      --node-locations $WALLAROO_NODE_LOCATION \
      --machine-type $WALLAROO_GCP_MACHINE_TYPE \
      --num-nodes $WALLAROO_CLUSTER_SIZE \
      --network $WALLAROO_GCP_NETWORK_NAME \
      --create-subnetwork name=$WALLAROO_GCP_SUBNETWORK_NAME \
      --enable-ip-alias \
      --cluster-version=1.23
      

      The command can take several minutes to complete based on the size and complexity of the clusters. Verify the process is complete with the clusters list command:

      gcloud container clusters list
      
      • Retrieving Kubernetes Credentials

      Once the GCP cluster is complete, the Kubernetes credentials can be installed into the local administrative system with the the gcloud container clusters get-credentials (https://cloud.google.com/sdk/gcloud/reference/container/clusters/get-credentials) command:

      gcloud container clusters \
      get-credentials $WALLAROO_CLUSTER \
      --region $WALLAROO_GCP_REGION
      

      To verify the Kubernetes credentials for your cluster have been installed locally, use the kubectl get nodes command. This will display the nodes in the cluster as demonstrated below:

      kubectl get nodes
      
      NAME                                         STATUS   ROLES    AGE   VERSION
      gke-wallaroo-ce-default-pool-863f02db-7xd4   Ready    <none>   39m   v1.21.6-gke.1503
      gke-wallaroo-ce-default-pool-863f02db-8j2d   Ready    <none>   39m   v1.21.6-gke.1503
      gke-wallaroo-ce-default-pool-863f02db-hn06   Ready    <none>   39m   v1.21.6-gke.1503
      gke-wallaroo-ce-default-pool-3946eaca-4l3s   Ready    <none>   39m   v1.21.6-gke.1503
      
      • Troubleshooting
        • What does the error ‘Insufficient project quota to satisfy request: resource “CPUS_ALL_REGIONS”’ mean?

      Make sure that the Compute Engine Zone and Region are properly set based on your organization’s requirements. The instructions above default to us-central1, so change that zone to install your Wallaroo instance in the correct location.

      In the case of the script, this would mean changing the region and location from:

      WALLAROO_GCP_REGION=us-central1
      WALLAROO_NODE_LOCATION=us-central1-f
      
      WALLAROO_GCP_REGION={Your Region}
      WALLAROO_NODE_LOCATION={Your Location}
      

      Install Wallaroo Community

      Wallaroo Community can be installed into a Kubernetes cloud environment, or into a Kubernetes environment that meets the Wallaroo Prerequisites Guide. Organizations that use the Wallaroo Community AWS EC2 Setup procedure do not have to set up a Kubernetes environment, as it is already configured for them.

      If the prerequisites are already configured, jump to Install Wallaroo to start installing.

      This video demonstrates that procedure:

      The procedure assumes at least a basic knowledge of Kubernetes and how to use the kubectl and kots version 1.91.3 applications.

      The procedure involves the following major steps:

      Prerequisites

      Local Software Requirements

      Before starting, verify that all local system requirements are complete as detailed in the Wallaroo Community Local System Prerequisites guide:

      • kubectl: This interfaces with the Kubernetes server created in the Wallaroo environment.

        • For Kots based installs:
        • Cloud Kubernetes environment has been prepared.
        • You have downloaded your Wallaroo Community License file.

        Install Wallaroo

        The environment is ready, the tools are installed - let’s install Wallaroo! The following will use kubectl and kots through the following procedure:

        1. Install the Wallaroo Community Edition using kots install wallaroo/ce, specifying the namespace to install. For example, if wallaroo is the namespace, then the command is:

          kubectl kots install wallaroo/ce --namespace wallaroo
          
        2. Wallaroo Community Edition will be downloaded and installed into your Kubernetes environment in the namespace specified. When prompted, set the default password for the Wallaroo environment. When complete, Wallaroo Community Edition will display the URL for the Admin Console, and how to end the Admin Console from running.

          • Deploying Admin Console
          • Creating namespace ✓
          • Waiting for datastore to be ready ✓
              Enter a new password to be used for the Admin Console: •••••••••••••
            • Waiting for Admin Console to be ready ✓
          
          • Press Ctrl+C to exit
          • Go to http://localhost:8800 to access the Admin Console
          

        Wallaroo Community edition will continue to run until terminated. To relaunch in the future, use the following command:

        kubectl-kots admin-console --namespace wallaroo
        

        Initial Configuration and License Upload Procedure

        Once Wallaroo Community edition has been installed for the first time, we can perform initial configuration and load our Wallaroo Community license file through the following process:

        1. If Wallaroo Community Edition has not started, launch it with the following command:

          ❯ kubectl-kots admin-console --namespace wallaroo
            • Press Ctrl+C to exit
            • Go to http://localhost:8800 to access the Admin Console
          
        2. Enter the Wallaroo Community Admin Console address into a browser. You will be prompted for the default password as set in the step above. Enter it and select Log in.

          Wallaroo Admin Console Initial Login
        3. Upload your license file.

          Wallaroo Admin Upload License
        4. The Configure Wallaroo Community page will be displayed which allows you to customize your Wallaroo environment. For now, scroll to the bottom and select Continue. These settings can be customized at a later date.

        5. The Wallaroo Community Admin Console will run the preflight checks to verify that all of the minimum requirements are not met. This may take a few minutes. If there are any issues, Wallaroo can still be launched but may not function properly. When ready, select Continue.

          Wallaroo Admin Preflight Successful
        6. The Wallaroo Community Dashboard will be displayed. There may be additional background processes that are completing their setup procedures, so there may be a few minute wait until those are complete. If everything is ready, then the Wallaroo Dashboard will show a green Ready.

          Wallaroo Admin Admin Ready
        7. Under the license information is the DNS entry for your Wallaroo instance. This is where you and other users of your Wallaroo instance can log in. In this example, the URL will be https://beautiful-horse-9537.wallaroo.community. Note that it may take a few minutes for the DNS entries to propagate and this URL to be available.

          Wallaroo Instance URL
        8. You will receive an email invitation for the email address connected to this URL with a temporary password and a link to this Wallaroo instance’s URL. Either enter the URL for your Wallaroo instance or use the link in the email.

        9. To login to your new Wallaroo instance, enter the email address and temporary password associated with the license.

          Wallaroo Initial Login

        With that, Wallaroo Community edition is launched and ready for use! You can end the Admin Console from your terminal session above. From this point on you can just use the Wallaroo instance URL.

        Now that your Wallaroo Community edition has been installed, let’s work with some sample models to show off what you can do. Check out either the Wallaroo 101 if this is your first time using Wallaroo, or for more examples of how to use Wallaroo see the Wallaroo Tutorials.

        Troubleshooting

        4 - Installation Configurations

        Guides for different install options for Wallaroo

        The following guides demonstrate how to install Wallaroo with different options to best fit your organizations needs, and are meant to supplement the standard install guides.

        4.1 - Create ARM Nodepools for Kubernetes Clusters

        How to create ARM nodepools for Kubernetes clusters.

        The following guide demonstrates how to create nodepools with ARM architecture.

        Wallaroo supports for ARM architecture CPUs. For example, Azure supports Ampere® Altra® Arm-based processor included with the following virtual machines:

        The following templates demonstrate how to create an ARM nodepool, then assign that nodepool to an existing cluster. These steps can be used in conjunction with Wallaroo Enterprise Install Guides.

        • ARM nodes are required both for the Wallaroo inference engine, and for converting Wallaroo non-native runtimes. For standard installs of Wallaroo, the options are either:
          • Create two nodepools:
            • One nodepool with the wallaroo.ai/engine=true:NoSchedule taint for the Wallaroo Engine
            • One nodepool without taints for Wallaroo non-native runtime conversions.
          • Create one nodepool without taints used for both auto-conversion and engine deployments.

        For custom tolerations, see Taints and Tolerations Guide.

        The following scripts show examples of setting up an ARM nodepool for use with the Wallaroo instance.

        The following sample script for Microsoft Azure command line tool creates a nodepool into an existing cluster with the Standard_D4ps_v5 virtual machine, providing 4 cpus and 16 GB RAM under the the Ampere® Altra® Arm-based processors. For more details, see Azure Virtual Machines Dpsv5 and Dpdsv5-series.

        This provides as 0-3 nodes in the nodepool, allowing it to spin up or down VMs as required.

        RESOURCE_GROUP="YOUR RESOURCE GROUP"
        CLUSTER_NAME="YOUR CLUSTER NAME"
        ARM_NODEPOOL_NAME="YOUR ARM NODEPOOL NAME"
        
        az aks nodepool add \                
            --resource-group $RESOURCE_GROUP \
            --cluster-name $CLUSTER_NAME \
            --name $ARM_NODEPOOL_NAME \
            --node-count 0 \
            --node-vm-size Standard_D4ps_v5 \
            --node-taints wallaroo.ai/engine=true:NoSchedule \
            --enable-cluster-autoscaler \
            --min-count 0 \
            --max-count 3
        

        For example, to create an ARM nodepool arm_node_01 to the existing cluster wallaroo-cluster in the resource group sample-group, the following would be used:

        RESOURCE_GROUP="sample-group"
        CLUSTER_NAME="wallaroo-cluster"
        ARM_NODEPOOL_NAME="arm_node_01"
        
        az aks nodepool add \                
            --resource-group $RESOURCE_GROUP \
            --cluster-name $CLUSTER_NAME \
            --name $ARM_NODEPOOL_NAME \
            --node-count 0 \
            --node-vm-size Standard_D4ps_v5 \
            --node-taints wallaroo.ai/engine=true:NoSchedule \
            --enable-cluster-autoscaler \
            --min-count 0 \
            --max-count 3
        

        The following sample script for Amazon Web Services tool eksctl creates a nodepool with the m6g.xlarge virtual machine, providing 4 cpus and 16 GB RAM under the the Arm-based AWS Graviton2 processors processors. For more details, see Amazon EC2 M6g Instances.

        The following example template shows adding the nodepool to an existing cluster.

        apiVersion: eksctl.io/v1alpha5
        kind: ClusterConfig
        
        metadata:
          name: YOUR CLUSTER NAME HERE # This must match the name of the existing cluster
          region: YOUR REGION HERE 
        
        managedNodeGroups:
        - name: YOUR NODEPOOL NAME HERE
          instanceType: m6g.medium # the ARM based virtual machine to use
          minSize: 1
          maxSize: 1
          taints:
            - key: wallaroo.ai/engine # allows the Wallaroo engine to run in this nodepool 
              value: "true"
              effect: NoSchedule
          tags:
            k8s.io/cluster-autoscaler/node-template/label/k8s.dask.org/node-purpose: engine
            k8s.io/cluster-autoscaler/node-template/taint/k8s.dask.org/dedicated: "true:NoSchedule"
          iam:
            withAddonPolicies:
              autoScaler: true # used to autoscale between the minimum and maximum nodes
          containerRuntime: containerd
          amiFamily: AmazonLinux2
          availabilityZones:
            - INSERT YOUR ZONE HERE # this should match the region for optimal performance
          volumeSize: 100

        The following sample script for Google Cloud Platform command line tool creates a nodepool with the t2a-standard-4 virtual machine, providing 4 cpus and 16 GB RAM under the the Ampere® Altra® Arm-based processors. For more details, see Tau T2A machine series.

        The following must be specified:

        • Cluster Name: The cluster the nodepool is attached to.
        • Zone and node location: The location the nodepool is hosted in. For optimal performance, this should match the zone the cluster is in.
        • Machine type: The virtual machine to use.
        • Number of nodes: The number of nodes used for the nodepool. This example uses 1.
        • Service account: The IAM service account. If none is specified, then the default is used.
        CLUSTER_NAME="YOUR CLUSTER NAME"
        ARM_NODEPOOL_NAME="YOUR NODEPOOL NAME"
        ZONE="YOUR ZONE"
        NODE_LOCATIONS="YOUR LOCATIONS"
        NUM_NODES=YOUR NUMBER OF NODES
        
        
        gcloud container node-pools create $ARM_NODEPOOL_NAME \
            --cluster $CLUSTER_NAME \
            --zone $ZONE \
            --node-locations NODE_LOCATIONS \
            --node-taints=wallaroo.ai/engine=true:NoSchedule \
            --machine-type T2A_MACHINE_TYPE \
            --num-nodes NUM_NODES
        

        For example, to create an ARM nodepool arm_node_01 to the existing cluster wallaroo-cluster in the resource group sample-group, the following would be used:

        CLUSTER_NAME="wallaroo-cluster"
        ZONE="us-west1-a"
        NODE_LOCATIONS="us-west1-a"
        ARM_NODEPOOL_NAME="arm_node_01"
        NUM_NODES=1
        
        gcloud container node-pools create $ARM_NODEPOOL_NAME \
            --cluster $CLUSTER_NAME \
            --zone $ZONE \
            --node-locations NODE_LOCATIONS \
            --node-taints=wallaroo.ai/engine=true:NoSchedule \
            --machine-type t2a-standard-4 \
            --num-nodes NUM_NODES
        

        Deployment Tutorials

        The following tutorials demonstrate deploying a pipeline with the specified architecture.

        4.2 - Wallaroo Enterprise Azure integration Overview

        An overview of the Wallaroo Enterprise for Azure Cloud

        Wallaroo is proud to announce Wallaroo Enterprise for the Microsoft Azure Marketplace. This brings Wallaroo to even more organizations who want to use Wallaroo with their other Microsoft Azure services.

        The following diagram displays the architecture for this service.

        Wallaroo Azure Marketplace Architecture
        1. Users and application integrations connect to Jupyter Lab and the Wallaroo ML Ops APIs hosted in an AKS cluster.
        2. Wallaroo cluster services are hosted in a Kubernetes namespace and manage deployments of ML models.
        3. ML models are deployed in AKS to scale across as many VMs as needed to handle the load.
        4. ML inference services are provided via a web API to allow integration with data storage systems or other services.

        4.3 - Create GPU Nodepools for Kubernetes Clusters

        How to create GPU nodepools for Kubernetes clusters.

        Wallaroo provides support for ML models that use GPUs. The following templates demonstrate how to create a nodepool in different cloud providers, then assign that nodepool to an existing cluster. These steps can be used in conjunction with Wallaroo Enterprise Install Guides.

        Note that deploying pipelines with GPU support is only available for Wallaroo Enterprise.

        For standard Wallaroo installations, GPU nodepools must include the following taints:

        • sku=gpu:NoSchedule
        • wallaroo.ai/engine=true:NoSchedule

        For custom tolerations, see Taints and Tolerations Guide.

        The following script creates a nodepool with NVidia Tesla K80 gpu using the Standard_NC6 machine type and autoscales from 0-3 nodes. Each node has one GPU in this example so the max .gpu() that can be requested by a pipeline step is 1.

        For detailed steps on adding GPU to a cluster, see Microsoft Azure Use GPUs for compute-intensive workloads on Azure Kubernetes Service (AKS) guide.

        Note that the labels are required as part of the Wallaroo pipeline deployment with GPU support. The label below is an example, but a label must be provided.

        RESOURCE_GROUP="YOUR RESOURCE GROUP"
        CLUSTER_NAME="YOUR CLUSTER NAME"
        GPU_NODEPOOL_NAME="YOUR GPU NODEPOOL NAME"
        
        az extension add --name aks-preview
        
        az extension update --name aks-preview
        
        az feature register --namespace "Microsoft.ContainerService" --name "GPUDedicatedVHDPreview"
        
        az provider register -n Microsoft.ContainerService
        
        az aks nodepool add \                
            --resource-group $RESOURCE_GROUP \
            --cluster-name $CLUSTER_NAME \
            --name $GPU_NODEPOOL_NAME \
            --node-count 0 \
            --node-vm-size Standard_NC6 \
            --node-taints "sku=gpu:NoSchedule,wallaroo.ai/engine=true:NoSchedule" \
            --aks-custom-headers UseGPUDedicatedVHD=true \
            --enable-cluster-autoscaler \
            --min-count 0 \
            --max-count 3 \
            --labels doc-gpu-label=true
        

        The following script creates a nodepool uses NVidia T4 GPUs and autoscales from 0-3 nodes. Each node has one GPU in this example so the max .gpu() that can be requested by a pipeline step is 1.

        Google GKE automatically adds the following taint to the created nodepool.

        NO_SCHEDULE nvidia.com/gpu present

        Note that the labels are required as part of the Wallaroo pipeline deployment with GPU support. The label below is an example, but a label must be provided.

        GCP_PROJECT="YOUR GCP PROJECT"
        GCP_CLUSTER="YOUR CLUSTER NAME"
        GPU_NODEPOOL_NAME="YOUR GPU NODEPOOL NAME"
        REGION="YOUR REGION"
        
        gcloud container \
            --project $GCP_PROJECT \
            node-pools create $GPU_NODEPOOL_NAME \
            --cluster $GCP_CLUSTER \
            --region $REGION \
            --node-version "1.25.8-gke.500" \
            --machine-type "n1-standard-1" \
            --accelerator "type=nvidia-tesla-t4,count=1" \
            --image-type "COS_CONTAINERD" \
            --disk-type "pd-balanced" \
            --disk-size "100" \
            --node-labels doc-gpu-label=true \
            --metadata disable-legacy-endpoints=true \
            --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
            --num-nodes "3" \
            --enable-autoscaling \
            --min-nodes "0" \
            --max-nodes "3" \
            --location-policy "BALANCED" \
            --enable-autoupgrade \
            --enable-autorepair \
            --max-surge-upgrade 1 \
            --node-taints=sku=gpu:NoSchedule,wallaroo.ai/engine=true:NoSchedule \
            --max-unavailable-upgrade 0
        

        The following steps are used to create a AWS EKS Nodepool with GPU nodes.

        • Prerequisites: An existing AWS (Amazon Web Service) EKS (Elastic Kubernetes Service) cluster. See Wallaroo Enterprise Comprehensive Install Guide: Environment Setup Guides for a sample creation of an AWS EKS cluster for hosting a Wallaroo Enterprise instance.
        • eksctl: Command line tool for installating and updating EKS clusters.
        • Administrator access to the EKS cluster and capabilty of running kubectl commands.
        1. Create the nodepool with the following configuration file. Note that the labels are required as part of the Wallaroo pipeline deployment with GPU support. The label below is an example, but a label must be provided.
          1. The sample configuration file below uses the AWS instance type g5.2xlarge. Modify as required.
        eksctl create nodegroup --config-file=<path>
        

        Sample config file:

        # aws-gpu-nodepool.yaml
        apiVersion: eksctl.io/v1alpha5
        kind: ClusterConfig
        
        metadata:
          name: YOUR CLUSTER NAME HERE # This must match the name of the existing cluster
          region: YOUR REGION HERE
        
        managedNodeGroups:
        - name: YOUR NODEPOOL NAME HERE
          instanceType: g5.2xlarge
          minSize: 1
          maxSize: 3
          labels:
            wallaroo.ai/gpu: "true"
            doc-gpu-label: "true"
          taints:
            - key: wallaroo.ai/engine
              value: "true"
              effect: NoSchedule
            - key: sku
              value: gpu
              effect: NoSchedule
          tags:
            k8s.io/cluster-autoscaler/node-template/label/k8s.dask.org/node-purpose: engine
            k8s.io/cluster-autoscaler/node-template/taint/k8s.dask.org/dedicated: "true:NoSchedule"
          iam:
            withAddonPolicies:
              autoScaler: true
          containerRuntime: containerd
          amiFamily: AmazonLinux2
          availabilityZones:
            - INSERT YOUR ZONE HERE
          volumeSize: 100

        4.4 - Install Wallaroo with Minimum Services

        How to install Wallaroo with disabled services for lower core environments

        The following system requirements are required for the minimum settings for running Wallaroo in a Kubernetes cloud cluster.

        The following system requirements are required for the minimum settings for running Wallaroo in a Kubernetes cloud cluster.

        • Minimum number of nodes: 4
        • Minimum Number of CPU Cores: 8
        • Minimum RAM per node: 16 GB
        • Minimum Storage: A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.

        Wallaroo recommends at least 16 cores total to enable all services. At less than 16 cores, services will have to be disabled to allow basic functionality as detailed in this table.

        Note that even when disabling these services, Wallaroo performance may be impacted by the models, pipelines, and data used. The greater the size of the models and steps in a pipeline, the more resources will be required for Wallaroo to operate efficiently. Pipeline resources are set by the pipeline configuration to control how many resources are allocated from the cluster to maintain peak effectiveness for other Wallaroo services. See the following guides for more details.

              
        Cluster Size 8 core16 core32 coreDescription
        Inference The Wallaroo inference engine that performs inference requests from deployed pipelines.
        Dashboard The graphics user interface for configuring workspaces, deploying pipelines, tracking metrics, and other uses.
        Jupyter HUB/LabThe JupyterHub service for running Python scripts, JupyterNotebooks, and other related tasks within the Wallaroo instance.
        Single Lab
        Multiple Labs
        PrometheusUsed for collecting and reporting on metrics. Typical metrics are values such as CPU utilization and memory usage.
        Alerting 
        Model Validation 
        Dashboard Graphs 
        PlateauA Wallaroo developed service for storing inference logs at high speed. This is not a long term service; organizations are encouraged to store logs in long term solutions if required.
        Model Insights 
        Python API 
        Model ConversionConverts models into a native runtime for use with the Wallaroo inference engine.

        To install Wallaroo with minimum services, a configuration file will be used as parts of the kots based installation. For full details on the Wallaroo installation process, see the Wallaroo Install Guides.

        Wallaroo Installation with less than 16 Cores

        To install Wallaroo with less than 16 cores and 8 cores or greater, the following services must be disabled:

        • Model Conversion
        • Model Insights
        • Plateau

        The following configuration settings can be used at the installation procedure to disable these services.

        apiVersion: kots.io/v1beta1
        kind: ConfigValues
        metadata:
          name: wallaroo
        spec:
          values:
            dashboard_enabled:
              value: "1"
            enable_model_insights:
              value: "0"
            model_conversion_enabled:
              value: "1"
            plateau_enabled:
              value: "0"

        The configuration file can be applied via the --config-values={CONFIG YAML FILE} option. For example:

        kubectl kots install "wallaroo/ce" \
        -n wallaroo \ 
        --config-values=wallaroo-install-8-cores.yaml
        

        4.5 - Install Wallaroo to Specific Nodes

        How to install Wallaroo to specific nodes

        Organizations that share their Kubernetes environment with other applications may want to install Wallaroo to specific nodes in their cluster. The following guide demonstrates how to install a Wallaroo instance into specific nodes in the Kubernetes cluster.

        This example uses Wallaroo Community as the example. For other guides for installing a Wallaroo instance, see the Wallaroo Community Setup Guides and the Wallaroo Enterprise Setup Guides.

        Users who are familiar with Kubernetes clusters can skip ahead directly to the Install Steps.

        Description

        When installed into a Kubernetes cluster, Wallaroo will use available nodes to maximize its performance. Some organizations may use specific nodepools or nodes for specific applications.

        One option is to use Kubernetes metadata to assign Node labels to nodes, then specify that Wallaroo can be installed into specific nodes that match that label. This can be done with specifying configuration options during the install process using the kots option --config-values={CONFIG YAML FILE}. For more information, see the kots set config documentation.

        Install Steps

        In this example, an instance of Wallaroo Community will be installed into a Kubernetes cluster that has four nodes assigned with the label wallaroo.ai/node=true;

        kubectl get nodes
        NAME                                 STATUS   ROLES   AGE   VERSION
        aks-mainpool-18670167-vmss000000     Ready    agent   84m   v1.23.8
        aks-wallarooai-12293194-vmss000000   Ready    agent   75m   v1.23.8
        aks-wallarooai-12293194-vmss000001   Ready    agent   75m   v1.23.8
        aks-wallarooai-12293194-vmss000002   Ready    agent   75m   v1.23.8
        aks-wallarooai-12293194-vmss000003   Ready    agent   75m   v1.23.8
        
        kubectl get nodes -l wallaroo.ai/node=true
        NAME                                 STATUS   ROLES   AGE   VERSION
        aks-wallarooai-12293194-vmss000000   Ready    agent   75m   v1.23.8
        aks-wallarooai-12293194-vmss000001   Ready    agent   75m   v1.23.8
        aks-wallarooai-12293194-vmss000002   Ready    agent   75m   v1.23.8
        aks-wallarooai-12293194-vmss000003   Ready    agent   75m   v1.23.8
        
        1. Create a kots configuration file and specify the label to use for installing nodes. For this example, we will use wallaroo.ai/node: "true" as the label. Any nodes with that label will be used by Wallaroo during the installation. For this example, this configuration is saved to the file test-node.yaml.

          apiVersion: kots.io/v1beta1
          kind: ConfigValues
          metadata:
              creationTimestamp: null
              name: wallaroo
          spec:
              values:
                  wallaroo_node_selector: 
                      value: 'wallaroo.ai/node: "true"'
          status: {}
          
        2. During installation, specify the configuration file to be used with the --config-values option.

          kubectl kots install "wallaroo/ce" \
          -n wallaroo \ 
          --config-values=test-node.yaml
          
          • Deploying Admin Console
          • Creating namespace ✓
          • Waiting for datastore to be ready ✓
            • Waiting for Admin Console to be ready ✓
          • Press Ctrl+C to exit
          • Go to http://localhost:8800 to access the Admin Console
          
        3. Proceed with the installation as normal, including uploading the required license file, etc.

        4. Once complete, verify Wallaroo was installed to specific nodes with the kubectl get pods command. The following shows the pods in the wallaroo namespace where the Wallaroo Community instance was installed, and the pods used for the deployed pipeline ccfraudpipeline.

          kubectl get pods --all-namespaces -o=custom-columns=NAME:.metadata.name,Namespace:.metadata.namespace,Node:.spec.nodeName
          
          NAME                                       Namespace           Node
          engine-6469d85b5c-5pz75                    ccfraudpipeline-1   aks-wallarooai-12293194-vmss000003
          engine-lb-db4f647fb-m9bkl                  ccfraudpipeline-1   aks-wallarooai-12293194-vmss000001
          helm-runner-xz4vn                          ccfraudpipeline-1   aks-wallarooai-12293194-vmss000003
          azure-ip-masq-agent-26cnd                  kube-system         aks-wallarooai-12293194-vmss000002
          azure-ip-masq-agent-745hs                  kube-system         aks-wallarooai-12293194-vmss000000
          azure-ip-masq-agent-f2nl2                  kube-system         aks-mainpool-18670167-vmss000000
          azure-ip-masq-agent-hjxbr                  kube-system         aks-wallarooai-12293194-vmss000003
          azure-ip-masq-agent-nktlq                  kube-system         aks-wallarooai-12293194-vmss000001
          cloud-node-manager-6twk7                   kube-system         aks-mainpool-18670167-vmss000000
          cloud-node-manager-g2bql                   kube-system         aks-wallarooai-12293194-vmss000003
          cloud-node-manager-j4xdq                   kube-system         aks-wallarooai-12293194-vmss000001
          cloud-node-manager-q6b2k                   kube-system         aks-wallarooai-12293194-vmss000000
          cloud-node-manager-rsrsg                   kube-system         aks-wallarooai-12293194-vmss000002
          coredns-autoscaler-7d56cd888-t28v5         kube-system         aks-mainpool-18670167-vmss000000
          coredns-dc97c5f55-8v7lh                    kube-system         aks-mainpool-18670167-vmss000000
          coredns-dc97c5f55-p2dc2                    kube-system         aks-mainpool-18670167-vmss000000
          csi-azuredisk-node-5hlxc                   kube-system         aks-mainpool-18670167-vmss000000
          csi-azuredisk-node-6bp8l                   kube-system         aks-wallarooai-12293194-vmss000003
          csi-azuredisk-node-mthtd                   kube-system         aks-wallarooai-12293194-vmss000000
          csi-azuredisk-node-p6w8w                   kube-system         aks-wallarooai-12293194-vmss000002
          csi-azuredisk-node-sqznw                   kube-system         aks-wallarooai-12293194-vmss000001
          csi-azurefile-node-7kw5p                   kube-system         aks-wallarooai-12293194-vmss000002
          csi-azurefile-node-9zb6l                   kube-system         aks-wallarooai-12293194-vmss000001
          csi-azurefile-node-grs6g                   kube-system         aks-wallarooai-12293194-vmss000000
          csi-azurefile-node-z84nz                   kube-system         aks-mainpool-18670167-vmss000000
          csi-azurefile-node-zzqdf                   kube-system         aks-wallarooai-12293194-vmss000003
          konnectivity-agent-6c57d77bcd-5tvbh        kube-system         aks-mainpool-18670167-vmss000000
          konnectivity-agent-6c57d77bcd-z5q48        kube-system         aks-mainpool-18670167-vmss000000
          kube-proxy-4nz25                           kube-system         aks-wallarooai-12293194-vmss000000
          kube-proxy-8fv76                           kube-system         aks-wallarooai-12293194-vmss000002
          kube-proxy-c5nvs                           kube-system         aks-wallarooai-12293194-vmss000001
          kube-proxy-lvlwc                           kube-system         aks-wallarooai-12293194-vmss000003
          kube-proxy-vbvfr                           kube-system         aks-mainpool-18670167-vmss000000
          metrics-server-64b66fbbc8-tvxpj            kube-system         aks-mainpool-18670167-vmss000000
          api-lb-bbc98488d-24qxb                     wallaroo           aks-wallarooai-12293194-vmss000002
          continuous-image-puller-8sfw9              wallaroo           aks-wallarooai-12293194-vmss000003
          continuous-image-puller-bbt7c              wallaroo           aks-wallarooai-12293194-vmss000000
          continuous-image-puller-ngr75              wallaroo           aks-wallarooai-12293194-vmss000002
          continuous-image-puller-stxpq              wallaroo           aks-wallarooai-12293194-vmss000001
          dashboard-677df986d9-8c5mz                 wallaroo           aks-wallarooai-12293194-vmss000000
          deploymentmanager-69b4c6d449-j8jct         wallaroo           aks-wallarooai-12293194-vmss000002
          graphql-api-9c664ddf-t7cnr                 wallaroo           aks-wallarooai-12293194-vmss000000
          hub-668d49b7b4-jspqj                       wallaroo           aks-wallarooai-12293194-vmss000001
          jupyter-john-2ehansarick-40wallaroo-2eai   wallaroo           aks-wallarooai-12293194-vmss000002
          keycloak-85cf99c7bf-8vvb5                  wallaroo           aks-wallarooai-12293194-vmss000002
          kotsadm-cbf8d8ccb-qgx2w                    wallaroo           aks-wallarooai-12293194-vmss000002
          kotsadm-minio-0                            wallaroo           aks-wallarooai-12293194-vmss000002
          kotsadm-postgres-0                         wallaroo           aks-wallarooai-12293194-vmss000000
          minio-68bc498d6d-xm5ht                     wallaroo           aks-wallarooai-12293194-vmss000000
          model-insights-7dcccb976-ttz64             wallaroo           aks-wallarooai-12293194-vmss000000
          plateau-5b777686dd-8c69s                   wallaroo           aks-wallarooai-12293194-vmss000000
          postgres-6c5fff5c57-9hr4n                  wallaroo           aks-wallarooai-12293194-vmss000002
          prometheus-deployment-7dcb484c56-7jwq4     wallaroo           aks-wallarooai-12293194-vmss000000
          proxy-755778dccd-mwhwz                     wallaroo           aks-wallarooai-12293194-vmss000002
          python-api-787bcb7764-nvdb4                wallaroo           aks-wallarooai-12293194-vmss000002
          rest-api-677dc6bdcf-q9b62                  wallaroo           aks-wallarooai-12293194-vmss000002
          wallaroo-fluent-bit-standard-h7mrl         wallaroo           aks-wallarooai-12293194-vmss000002
          wallaroo-fluent-bit-standard-jss2d         wallaroo           aks-wallarooai-12293194-vmss000000
          wallaroo-fluent-bit-standard-l75cj         wallaroo           aks-wallarooai-12293194-vmss000001
          wallaroo-fluent-bit-standard-m55tk         wallaroo           aks-wallarooai-12293194-vmss000003
          wallaroo-telemetry-27687782-g6mhm          wallaroo           aks-wallarooai-12293194-vmss000001
          wallaroo-telemetry-27687783-xgqpm          wallaroo           aks-wallarooai-12293194-vmss000001
          wallaroo-telemetry-27687784-9b85g          wallaroo           aks-wallarooai-12293194-vmss000001
          

        For other instructions on how to deploy or configure a Wallaroo instance, see the Wallaroo Operations Guides.

        4.6 - Taints and Tolerations Guide

        Configure custom taints and toleration for a cluster for Wallaroo

        Organizations can customize the taints and tolerances for their Kubernetes cluster running Wallaroo. Nodes in a Kubernetes cluster can have a taint applied to them. Any pod that does not have a toleration matching the taint can be rejected and will not be applied to that node.

        This allows organizations to determine which pods can be accepted or rejected into specific nodes, reserving their Kubernetes resources for other services. Combined with the Install Wallaroo to Specific Nodes guide this ensures that Wallaroo pods are contained to specific cluster nodes, and prevents non-Wallaroo pods from being scheduled into the same nodes to reserve those resources for the Wallaroo instance.

        In this example, the node Postgres has the taint wallaroo.ai/postgres=true:NoSchedule. The pod postgres has the tolerance wallaroo.ai/postgres:NoSchedule op=Exists, so it is scheduled into the node Postgres. The pod nginx has no tolerations, so it is not scheduled into the node Postgres.

        Tolerations Example
        Node: Postgres
        Taints:wallaroo.ai/postgres=true:NoSchedule
        Scheduled
        Postgres
        Tolerations: wallaroo.ai/postgres:NoSchedule op=Exists

        nginx
        Tolerations: None

        🚫

        See the Kubernetes Taints and Tolerations documentation for more information.

        Setting Tolerations and Taints

        The Wallaroo Enterprise Install Guides specify default taints applied to nodepools. These can be used to contain pod scheduling only to specific nodes where the pod tolerations match the nodes taints. By default, the following nodepools and their associated taints are created

        After Wallaroo release September 2022 (Codename Cobra):

        NodepoolTaints
        postgreswallaroo.ai/postgres=true:NoSchedule
        enginelbwallaroo.ai/enginelb=true:NoSchedule
        enginewallaroo.ai/engine=true:NoSchedule
        mainpoolN/A

        Before Wallaroo release September 2022 (Code name Mustang and before)

        NodepoolTaints
        postgreswallaroo-postgres=true:NoSchedule
        enginelbwallaroo-enginelb=true:NoSchedule
        enginewallaroo-engine=true:NoSchedule
        mainpoolN/A

        The nodepool mainpool is not assigned any taints to allow other Kubernetes services to run as part of the cluster.

        The taint wallaroo.ai/reserved=true:NoSchedule can be applied to other nodepools. This allows additional Wallaroo resources to be scheduled in those nodes while rejecting other pods that do not have a matching toleration.

        Default Tolerations

        By default, the following tolerations are applied for Wallaroo pods. Organizations can add a corresponding Any pod that does not contain a taint to match these tolerances will have the condition effect:NoSchedule for the specified node.

        • Toleration key for all Wallaroo pods
          • wallaroo.ai/reserved
        • Engine toleration key
          • wallaroo.ai/engine
        • Engine LB toleration key
          • wallaroo.ai/enginelb
        • Postgres toleration key
          • wallaroo.ai/postgres

        Note that these taint values are applied to the nodepools as part of the Wallaroo Enterprise Setup guides. They are not typically set up or required for Wallaroo Community instances.

        Custom Tolerations

        To customize the tolerations applied to Wallaroo nodes, the following prerequisites must be met:

        • Access to the Kubernetes environment running the Wallaroo instances.
        • Have kubectl and kots installed and connected to the Kubernetes environment.

        For full details on installing Wallaroo and the prerequisite software, see the Wallaroo Prerequisites Guide.

        1. Access the Wallaroo Administrative Dashboard.

          1. From a terminal with kubectl and kots installed and connected to the Kubernetes environment, run:

            kubectl kots admin-console --namespace wallaroo
            

            This will provide access to the Wallaroo Administrative Dashboard through http://localhost:8800:

              • Press Ctrl+C to exit
              • Go to http://localhost:8800 to access the Admin Console
            
          2. Launch a browser and connect to http://localhost:8800.

          3. Enter the password created during the Wallaroo Install process. The Wallaroo Administrative Dashboard will now be available.

        2. From the Wallaroo Administrative Dashboard, select Config -> Taints and Tolerations.

        3. Set the custom tolerations as required by your organization. The following nodes and tolerations can be changed:

          Wallaroo Taints and Tolerances
        • Toleration key for all Wallaroo pods
          • Default value: wallaroo.ai/reserved
        • Engine toleration key
          • Default value: wallaroo.ai/engine
        • Engine LB toleration key
          • Default value: wallaroo.ai/enginelb
        • Postgres toleration key
        • Default value: wallaroo.ai/postgres

        5 - Installation Troubleshooting Guide

        Troubleshooting

        I’m Getting a Timeout Error

        Depending on the connection and resources, the installation process may time out. If that occurs, use the --wait-duration flag to provide additional time. The time must be provided in Go duration format (for example: 60s, 1m, etc). The following example extends the wait duration to 10 minutes:

        kubectl kots install wallaroo/ea -n wallaroo --license-file myfile.yaml --shared-password wallaroo --wait-duration 600s

        Preflight Checks are Failing at the Command Line

        If your system does not meet all of the preflight requirements, the installation process may fail when performing an automated installation. It is highly recommended to install Wallaroo on a system that meets all requirements or else performance will be degraded.

        Before continuing, use the following command and note down any and all pre-flight checks that are listed as a failure. The license will be installed in later steps through the browser.

        install wallaroo/ea -n wallaroo
        

        To ignore preflight checks, use the --skip-preflights flag, as in the following example (Note: This is not recommended, only provided as an example.):

        kubectl kots install wallaroo/ea -n wallaroo --license-file myfile.yaml --shared-password wallaroo --skip-preflights

        When Launching JupyterHb, I get a Server 500 error

        If you shut down and restart a Wallaroo instance in a new environment or change the IP address, some settings may not be updated. Run the following command to restart the deployment process and update the settings to match the current environment. Note that the namespace wallaroo is used - modify this to match the environment where Wallaroo is installed.

        kubectl rollout restart deployment hub -n wallaroo
        

        The resource wallaroo/daemonset/spire-agent shows as not loading - how do I fix it?

        If the resource wallaroo/daemonset/spire-agent does not load, verify the kots version is 1.103.3. To upgrade the kots version in your Kubernetes cluster, see see Updating KOTS.

        Usually this can be done by:

        1. Updating the version of kots with the command:

          curl https://kots.io/install/1.103.3 | REPL_USE_SUDO=y bash
          
        2. Upgrade kots in the Kubernetes cluster with the following, replacing the namespace wallaroo with the namespace you have installed the Wallaroo Ops instance in:

          kubectl kots admin-console upgrade -namespace wallaroo
          

        How do I Send Logs and Configurations to Wallaroo?

        See the Wallaroo Support Bundle Generation Guide for instructions on how to create a support bundle used to troubleshoot installation and configuration issues.

        6 - How to Uninstall Wallaroo

        How to Uninstall Wallaroo

        If the install procedure for Wallaroo goes awry, one option is to uninstall the incomplete Wallaroo installation and start again. The following procedure will remove Wallaroo from a Kubernetes cluster.

        1. Remove all Kubernetes namespaces that correlate to a Wallaroo pipeline with the kubectl delete namespaces {list of namespaces}command except the following : default, kube* (any namespaces with kube before it), and wallaroo. wallaroo will be removed in the next step.

          For example, in the following environment model1 and model2 would be deleted with the following:

            -> kubectl get namespaces
              NAME			    STATUS        AGE
              default		        Active        7d4h
              kube-node-lease	    Active		    7d4h
              kube-public		    Active		    7d4h
              model1               Active         4h23m
              model2               Active         4h23m
              wallaroo             Active         3d6h
          
              kubectl delete namespaces model1 model2
          
        2. Use the following bash script or run the commands individually. Warning: If the selector is incorrect or missing from the kubectl command, the cluster could be damaged beyond repair.

          #!/bin/bash
          kubectl delete ns wallaroo && kubectl delete all,secret,configmap,clusterroles,clusterrolebindings,storageclass,crd --selector app.kubernetes.io/part-of=wallaroo --selector kots.io/app-slug=wallaroo
          
        3. Once complete, the kubectl get namespaces will return only the default namespaces:

          ❯ kubectl get namespaces
          NAME              STATUS   AGE
          default           Active   3h47m
          kube-node-lease   Active   3h47m
          kube-public       Active   3h47m
          kube-system       Active   3h47m
          

        Wallaroo can now be reinstalled into this environment.