Uninstall Guides
The following is a short version of the uninstall procedure to remove a previously installed version of Wallaroo. For full details, see the How to Uninstall Wallaroo. These instructions assume administrative use of the Kubernetes command kubectl
.
To uninstall a previously installed Wallaroo instance:
Delete any Wallaroo pipelines still deployed with the command
kubectl delete namespace {namespace}
. Typically these are the pipeline name with some numerical ID. For example, in the following list of namespaces the namespaceccfraud-pipeline-21
correspond to the Wallaroo pipelineccfraud-pipeline
. Verify these are Wallaroo pipelines before deleting.-> kubectl get namespaces NAME STATUS AGE default Active 7d4h kube-node-lease Active 7d4h kube-public Active 7d4h ccfraud-pipeline-21 Active 4h23m wallaroo Active 3d6h -> kubectl delete namespaces ccfraud-pipeline-21
Use the following bash script or run the commands individually. Warning: If the selector is incorrect or missing from the kubectl command, the cluster could be damaged beyond repair. For a default installation, the selector and namespace will be
wallaroo
.#!/bin/bash kubectl delete ns wallaroo && \ kubectl delete all,secret,configmap,clusterroles,clusterrolebindings,storageclass,crd \ --selector app.kubernetes.io/part-of=wallaroo --selector kots.io/app-slug=wallaroo
Wallaroo can now be reinstalled into this environment.
GCP Kubernetes Engine Instructions
The following instructions are made to assist users set up their Google Cloud Platform (GCP) Kubernetes environment for running Wallaroo. These represent a recommended setup, but can be modified to fit your specific needs. In particular, these instructions will provision a GKE cluster with 56 CPUs in total. Please ensure that your project’s resource limits support that.
Quick Setup Script: Download a bash script to automatically set up the GCP environment through the Google Cloud Platform command line interface
gcloud
.Manual Setup Guide: A list of the
gcloud
commands used to create the environment through manual commands.
Organizations that wish to run Wallaroo in their Google Cloud Platform environment must complete the following prerequisites:
- Register a Google Cloud Account: https://cloud.google.com/
- Create a Google Cloud project: https://cloud.google.com/resource-manager/docs/creating-managing-projects
- Install
gcloud
and rungcloud init
orgcloud init
–console on the local system used to set up your environment: https://cloud.google.com/sdk/docs/install - Enable the Google Compute Engine(GCE): https://cloud.google.com/endpoints/docs/openapi/enable-api
- Enable the Google Kubernetes Engine(GKE) on your project: https://console.cloud.google.com/apis/enableflow?apiid=container.googleapis.com
- Select a default Computer Engine region and zone: https://cloud.google.com/compute/docs/regions-zones.
- IMPORTANT NOTE
- Organizations that intend to stop and restart their Kubernetes environment on an intentional or regular basis are recommended to use a single availability zone for their nodes. This minimizes issues such as persistent volumes in different availability zones, etc.
- Organizations that intend to use Wallaroo Enterprise in a high availability cluster are encouraged to follow best practices including using separate availability zones for redundancy, etc.
Standard Setup Variables
The following variables are used in the Quick Setup Script and the Manual Setup Guide. Modify them as best fits your organization.
Variable Name | Default Value | Description |
---|---|---|
WALLAROO_GCP_PROJECT | wallaroo | The name of the Google Project used for the Wallaroo instance. |
WALLAROO_CLUSTER | wallaroo | The name of the Kubernetes cluster for the Wallaroo instance. |
WALLAROO_GCP_REGION | us-central1 | The region the Kubernetes environment is installed to. Update this to your GCP Computer Engine region. |
WALLAROO_NODE_LOCATION | us-central1-f | The location the Kubernetes nodes are installed to. Update this to your GCP Compute Engine Zone. |
WALLAROO_GCP_NETWORK_NAME | wallaroo-network | The Google network used with the Kubernetes environment. |
WALLAROO_GCP_SUBNETWORK_NAME | wallaroo-subnet-1 | The Google network subnet used with the Kubernets environment. |
DEFAULT_VM_SIZE | e2-standard-8 | The VM type used for the default nodepool. |
POSTGRES_VM_SIZE | n2-standard-8 | The VM type used for the postgres nodepool. |
ENGINELB_VM_SIZE | c2-standard-8 | The VM type used for the engine-lb nodepool. |
ENGINE_VM_SIZE | c2-standard-8 | The VM type used for the engine nodepool. |
Quick Setup Script
A sample script is available here, and creates a Google Kubernetes Engine cluster ready for use with Wallaroo Enterprise. This script requires the prerequisites listed above and uses the variables as listed in Standard Setup Variables
The following script is available for download: wallaroo_enterprise_gcp_expandable.bash
The following steps are geared towards a standard Linux or macOS system that supports the prerequisites listed above. Modify these steps based on your local environment.
- Download the script above.
- In a terminal window set the script status as
execute
with the commandchmod +x bash wallaroo_enterprise_gcp_expandable.bash
. - Modify the script variables listed above based on your requirements.
- Run the script with either
bash wallaroo_enterprise_gcp_expandable.bash
or./wallaroo_enterprise_gcp_expandable.bash
from the same directory as the script.
Set Variables
The following are the variables used in the environment setup process. Modify them as best fits your organization’s needs.
WALLAROO_GCP_PROJECT=wallaroo
WALLAROO_CLUSTER=wallaroo
WALLAROO_GCP_REGION=us-central1
WALLAROO_NODE_LOCATION=us-central1-f
WALLAROO_GCP_NETWORK_NAME=wallaroo-network
WALLAROO_GCP_SUBNETWORK_NAME=wallaroo-subnet-1
DEFAULT_VM_SIZE=n2-standard-8
POSTGRES_VM_SIZE=n2-standard-8
ENGINELB_VM_SIZE=c2-standard-8
ENGINE_VM_SIZE=c2-standard-8
Manual Setup Guide
The following steps are guidelines to assist new users in setting up their GCP environment for Wallaroo. The variables used in the commands are as listed in Standard Setup Variables listed above. Feel free to replace these with ones that match your needs.
See the Google Cloud SDK for full details on commands and settings.
Create a GCP Network
First create a GCP network that is used to connect to the cluster with the gcloud compute networks create
command. For more information, see the gcloud compute networks create page.
gcloud compute networks \
create $WALLAROO_GCP_NETWORK_NAME \
--bgp-routing-mode regional \
--subnet-mode custom
Verify it’s creation by listing the GCP networks:
gcloud compute networks list
Create the GCP Wallaroo Cluster
Once the network is created, the gcloud container clusters create
command is used to create a cluster. For more information see the gcloud container clusters create page.
The following is a recommended format, replacing the {} listed variables based on your setup. For Google GKE containerd
is enabled by default.
gcloud container clusters \
create $WALLAROO_CLUSTER \
--region $WALLAROO_GCP_REGION \
--node-locations $WALLAROO_NODE_LOCATION \
--machine-type $DEFAULT_VM_SIZE \
--network $WALLAROO_GCP_NETWORK_NAME \
--create-subnetwork name=$WALLAROO_GCP_SUBNETWORK_NAME \
--enable-ip-alias \
--cluster-version=1.23
The command can take several minutes to complete based on the size and complexity of the clusters. Verify the process is complete with the clusters list
command:
gcloud container clusters list
Wallaroo Enterprise Nodepools
The following static nodepools can be set based on your organizations requirements. Adjust the settings or names based on your requirements.
gcloud container node-pools create postgres \
--cluster=$WALLAROO_CLUSTER \
--machine-type=$POSTGRES_VM_SIZE \
--num-nodes=1 \
--region $WALLAROO_GCP_REGION \
--node-taints wallaroo.ai/postgres=true:NoSchedule
The following autoscaling nodepools are used for the engine load balancers and Wallaroo engine. Again, replace names and virtual machine types based on your organizations requirements.
gcloud container node-pools create engine-lb \
--cluster=$WALLAROO_CLUSTER \
--machine-type=$ENGINELB_VM_SIZE \
--enable-autoscaling \
--num-nodes=1 \
--min-nodes=0 \
--max-nodes=3 \
--region $WALLAROO_GCP_REGION \
--node-taints wallaroo-engine-lb=true:NoSchedule,wallaroo.ai/enginelb=true:NoSchedule \
--node-labels wallaroo-node-type=engine-lb
gcloud container node-pools create engine \
--cluster=$WALLAROO_CLUSTER \
--machine-type=$ENGINE_VM_SIZE \
--enable-autoscaling \
--num-nodes=1 \
--min-nodes=0 \
--max-nodes=3 \
--region $WALLAROO_GCP_REGION \
--node-taints wallaroo.ai/engine=true:NoSchedule \
--node-labels=wallaroo-node-type=engine
Retrieving Kubernetes Credentials
Once the GCP cluster is complete, the Kubernetes credentials can be installed into the local administrative system with the gcloud container clusters get-credentials
command:
gcloud container clusters \
get-credentials $WALLAROO_CLUSTER \
--region $WALLAROO_GCP_REGION
To verify the Kubernetes credentials for your cluster have been installed locally, use the kubectl get nodes
command. This will display the nodes in the cluster as demonstrated below:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-wallaroo-default-pool-863f02db-7xd4 Ready <none> 39m v1.21.6-gke.1503
gke-wallaroo-default-pool-863f02db-8j2d Ready <none> 39m v1.21.6-gke.1503
gke-wallaroo-default-pool-863f02db-hn06 Ready <none> 39m v1.21.6-gke.1503
gke-wallaroo-engine-3946eaca-4l3s Ready <none> 89s v1.21.6-gke.1503
gke-wallaroo-engine-lb-2e33a27f-64wb Ready <none> 26m v1.21.6-gke.1503
gke-wallaroo-postgres-d22d73d3-5qp5 Ready <none> 28m v1.21.6-gke.1503
Troubleshooting
- What does the error
Insufficient project quota to satisfy request: resource "CPUS_ALL_REGIONS"
mean?- Make sure that the Compute Engine Zone and Region are properly set based on your organization’s requirements. The instructions above default to
us-central1
, so change that zone to install your Wallaroo instance in the correct location.
- Make sure that the Compute Engine Zone and Region are properly set based on your organization’s requirements. The instructions above default to
Install Wallaroo
Organizations that use cloud services such as Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure can install Wallaroo Enterprise through the following process. These instructions also work with Single Node Linux based installations.
Before installation, the following prerequisites must be met:
- Have a Wallaroo Enterprise license file. For more information, you can request a demonstration.
- Set up a cloud Kubernetes environment that meets the requirements. Clusters must meet the following minimum specifications:
- Minimum number of nodes: 4
- Minimum Number of CPU Cores: 8
- Minimum RAM: 16 GB
- A total of 625 GB of storage will be allocated for the entire cluster based on 5 users with up to four pipelines with five steps per pipeline, with 50 GB allocated per node, including 50 GB specifically for the Jupyter Hub service. Enterprise users who deploy additional pipelines will require an additional 50 GB of storage per lab node deployed.
- Runtime: containerd is required.
- DNS services for integrating your Wallaroo Enterprise instance. See the DNS Integration Guide for the instructions on configuring Wallaroo Enterprise with your DNS services.
IMPORTANT NOTE
Wallaroo requires out-bound network connections to download the required container images and other tasks. For situations that require limiting out-bound access, refer to the air-gap installation instructions or contact your Wallaroo support representative.Wallaroo Enterprise can be installed either interactively or automatically through the kubectl
and kots
applications.
Automated Install
To automatically install Wallaroo into the namespace wallaroo
, specify the administrative password and the license file during the installation as in the following format with the following variables:
NAMESPACE
: The namespace for the Wallaroo Enterprise install, typicallywallaroo
.LICENSEFILE
: The location of the Wallaroo Enterprise license file.SHAREDPASSWORD
: The password of for the Wallaroo Administrative Dashboard.
kubectl kots install wallaroo/ee -n $NAMESPACE --license-file $LICENSEFILE --shared-password $SHAREDPASSWORD
For example, the following settings translate to the following install command:
NAMESPACE
:wallaroo
.LICENSEFILE
:myWallaroolicense.yaml
SHAREDPASSWORD
:snugglebunnies
kubectl kots install wallaroo/ee -n wallaroo --license-file myWallaroolicense.yaml --shared-password wallaroo
Interactive Install
The Interactive Install process allows users to adjust the configuration settings before Wallaroo is deployed. It requires users be able to access the Wallaroo Administrative Dashboard through a browser, typically on port 8080
.
IMPORTANT NOTE: Users who install Wallaroo through another node such as in the single node installation can port use SSH tunneling to access the Wallaroo Administrative Dashboard. For example:
ssh IP -L8800:localhost:8800
Install the Wallaroo Enterprise Edition using
kots install wallaroo/ee
, specifying the namespace to install Wallaroo into. For example, ifwallaroo
is the namespace, then the command is:kubectl kots install wallaroo/ee --namespace wallaroo
Wallaroo Enterprise Edition will be downloaded and installed into your Kubernetes environment in the namespace specified. When prompted, set the default password for the Wallaroo environment. When complete, Wallaroo Enterprise Edition will display the URL for the Admin Console, and how to end the Admin Console from running.
• Deploying Admin Console • Creating namespace ✓ • Waiting for datastore to be ready ✓ Enter a new password to be used for the Admin Console: ••••••••••••• • Waiting for Admin Console to be ready ✓ • Press Ctrl+C to exit • Go to http://localhost:8800 to access the Admin Console
To relaunch the Wallaroo Administrative Dashboard and make changes or updates, use the following command:
kubectl-kots admin-console --namespace wallaroo
Configure Wallaroo
Once installed, Wallaroo will continue to run until terminated.
Change Wallaroo Administrative Dashboard Password
To change the password to the Wallaroo Administrative Dashboard:
From the command line, use the command:
kubectl kots reset-password -n {namespace}
For example, for default installations where the Kubernetes namespace is
wallaroo
, the command would be:kubectl kots reset-password -n wallaroo
From here, enter the new password.
From the Wallaroo Administrative Dashboard:
Login and authenticate with the current password.
From the upper right hand corner, select … to access the menu and select Change password.
Enter the current password, then update and verify with the new password.
Setup DNS Services
Wallaroo Enterprise requires integration into your organizations DNS services.
The DNS Integration Guide details adding the Wallaroo instance to an organizations DNS services. The following is an abbreviated guide that assumes that certificates were already generated.
From the Wallaroo Dashboard, select Config and set the following:
- Networking Configuration
- Ingress Mode for Wallaroo Endpoints:
- None: Port forwarding or other methods are used for access.
- Internal: For environments where only nodes within the same Kubernetes environment and no external connections are required.
- External: Connections from outside the Kubernetes environment is allowed.
- Enable external URL inference endpoints: Creates pipeline inference endpoints. For more information, see Model Endpoints Guide.
- Ingress Mode for Wallaroo Endpoints:
- DNS
- DNS Suffix (Mandatory): The domain name for your Wallaroo instance.
- TLS Certificates
- Use custom TLS Certs: Checked
- TLS Certificate: Enter your TLS Certificate (.crt file).
- TLS Private Key: Enter your TLS private key (.key file).
- Other settings as desired.
- Networking Configuration
Once complete, scroll to the bottom of the Config page and select Save config.
A pop-up window will display The config for Wallaroo Enterprise has been updated.. Select Go to updated version to continue.
From the Version History page, select Deploy. Once the new deployment is finished, you will be able to access your Wallaroo services via their DNS addresses.
To verify the configuration is complete, access the Wallaroo Dashboard through the suffix domain
. For example if the suffix domain
is wallaroo.example.com
then access https://wallaroo.example.com
in a browser and verify the connection and certificates.
Setup Users
User management is handled through the Wallaroo instance Keycloak service. See the Wallaroo User Management for full guides on setting up users, identity providers, and other user configuration options. This step must be completed before using Wallaroo.
The following is an abbreviated guide on setting up new Wallaroo users.
IMPORTANT NOTE
At least one user must be created before using Wallaroo.
Accessing The Wallaroo Keycloak Dashboard
Enterprise customers may access their Wallaroo Keycloak dashboard by navigating to https://keycloak.<suffix>
, depending on their choice domain suffix supplied during installation.
Obtaining Administrator Credentials
The standard Wallaroo installation creates the user admin
by default and assigns them a randomly generated password. The admin
user credentials are obtained which may be obtained directly from Kubernetes with the following commands, assuming the Wallaroo instance namespace is wallaroo
.
Retrieve Keycloak Admin Username
kubectl -n wallaroo \ get secret keycloak-admin-secret \ -o go-template='{{.data.KEYCLOAK_ADMIN_USER | base64decode }}'
Retrieve Keycloak Admin Password
kubectl -n wallaroo \ get secret keycloak-admin-secret \ -o go-template='{{.data.KEYCLOAK_ADMIN_PASSWORD | base64decode }}'
Accessing the User Management Panel
In the Keycloak Administration Console, click Manage -> Users in the left-hand side menu. Click the View all users button to see existing users. This will be under the host name keycloak.$WALLAROO_SUFFIX
. For example, if the $WALLAROO_SUFFIX is wallaroo.example.com
, the Keycloak Administration Console would be keycloak.wallaroo.example.com
.
Adding Users
To add a user through the Keycloak interface:
Click the Add user button in the top-right corner.
Enter the following:
- A unique username and email address.
- Ensure that the Email Verified checkbox is checked - Wallaroo does not perform email verification.
- Under Required User Actions, set Update Password so the user will update their password the next time they log in.
Click Save.
Once saved, select Credentials tab, then the Set Password section, enter the new user’s desired initial password in the Password and Password Confirmation fields.
Click Set Password. Confirm the action when prompted. This will force the user to set their own password when they log in to Wallaroo.
To log into the Wallaroo dashboard, log out as the Admin user and login to the Wallaroo Dashboard as a preconfigured user or via SSO.