Upgrade Standard Cluster Installations of Wallaroo via Helm

Organizations have the ability to upgrade Wallaroo to newer versions while keeping their workspaces, model uploads, ML workload orchestrations, and other artifacts intact. This process is fast and easy, allowing users to seamlessly resume their work with Wallaroo Ops without any changes.

The duration of a standard upgrade may vary from 30 to 60 minutes, depending on the size and quantity of workspaces and artifacts. Please note that services to the Wallaroo Ops instance will be temporarily interrupted during the upgrade process.

The upgrade Wallaroo follows these general steps.

Pre-upgrade Checklist: Actions that should be performed before the upgrade process is initiated. See Upgrade Prerequisites for more details.
Upgrade Procedure: Steps for upgrading Wallaroo to a specified version.

Upgrade Procedure

Depending on the size and number of workspaces and artifacts, a typical upgrade can take 30-60 minutes.

Verify allowVolumeExpansion is True

IMPORTANT NOTE

The Kubernetes setting allowVolumeExpansion must be true before starting the upgrade procedure. If volume expansion is required during the upgrade process and allowVolumeExpansion=false, the upgrade will fail.

Verify that the Kubernetes setting allowVolumeExpansion is set to true through the following procedure.

From the command line of the Kubernetes administrative node, run kubectl get sc. For example:

kubectl get sc

The result should resemble the following:

NAME                          PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
premium-rwo                   pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   453d
standard                      kubernetes.io/gce-pd    Delete          Immediate              true                   453d
standard-rwo (default)        pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   453d
wallaroo-standard (default)   pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   6d1h

For any nodes marked (default) that are not named wallaroo-standard, verify the AllowVolumeExpansion value through the following command:

kubectl describe sc {name}

Based on the following example, this node is standard-rwo, making the describe command the following:

kubectl describe sc standard-rwo

The following is a sample result from this command.

Name:                  standard-rwo
IsDefaultClass:        Yes
Annotations:           components.gke.io/component-name=pdcsi,components.gke.io/component-version=0.16.18,components.gke.io/layer=addon,storageclass.kubernetes.io/is-default-class=true
Provisioner:           pd.csi.storage.gke.io
Parameters:            type=pd-balanced
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

If AllowVolumeExpansion is False, see the Kubernetes documentation on Volume expansion and follow the steps for your host environment on setting the AllowVolumeExpansion value to True before proceeding.

Upgrade via Helm

The following procedure is used to upgrade a Wallaroo Ops instance via helm.

Helm Client Software Prerequisites

For Helm installs:
- helm: Install Helm
  - Minimum supported version: Helm 3.11.2
- krew: Install Krew
- krew preflight and krew support-bundle. Install with the following commands:
  - kubectl krew install support-bundle
  - kubectl krew install preflight

Upgrade via Helm Procedure

To upgrade a helm based installation of Wallaroo:

From a Wallaroo Support representative, retrieve the following:
1. The license channel. This will be in the form of oci://registry.replicated.com/wallaroo/$CHANNEL/wallaroo, where $CHANNEL represents the channel type. For example, oci://registry.replicated.com/wallaroo/2025-1/wallaroo.
2. The version to be upgrade to. For example: 2025.1.0-6158.
3. OCI Registry login. This will be in the format: helm registry login registry.replicated.com --username $YOURUSERNAME --password $YOURPASSWORD
4. Helm Release Name: This was determined during the Wallaroo Install process.
Prepare the local-values.yaml file that will store the essential configurations options. It is highly recommended to use the same local-values.yaml file used during the Wallaroo installation for minimum changes. The following is an example of the local-values.yaml file settings. See Wallaroo Helm Reference Guides for additional settings.

wallarooDomain: "wallaroo.example.com" # change to match the domain name

custTlsSecretName: cust-cert-secret

ingress_mode: internal # internal (Default), external,or none

dashboard:
  clientName: "Wallaroo Helm Example" # Insert the name displayed in the Wallaroo Dashboard

kubernetes_distribution: ""   # Required. One of: aks, eks, gke, oke, or kurl.

From a terminal with helm and administrative access to the Kubernetes cluster Wallaroo Ops is installed to, perform the following:

Login to the OCI registry, replacing $YOURUSERNAME and $YOURPASSWORD with the ones provided by Wallaroo:
```
helm registry login registry.replicated.com --username $YOURUSERNAME --password $YOURPASSWORD
```
Set the default Kubernetes namespace to the one used for the Wallaroo installation. By default, the namespace wallaroo is used. For example:
```
kubectl config set-context --current --namespace wallaroo
```
Perform the preflight check. Preflight verification is performed with the following command format. The variables $LICENSE_CHANNEL and $VERSION is supplied by your Wallaroo support representative.
```
helm template --is-upgrade \
oci://registry.replicated.com/wallaroo/$LICENSE_CHANNEL/wallaroo --version $VERSION \
| kubectl preflight -
```
For example, the $LICENSE_CHANNEL=2025-1 and the $VERSION=2025.1.0-6158
```
helm template --is-upgrade \
oci://registry.replicated.com/wallaroo/2025-1/wallaroo --version 2025.1.0-6158 \
| kubectl preflight -
```
This displays the Preflight Checks report. Verify that all checks are completed successfully before proceeding.

Perform the upgrade with the following command, replacing the following:

$LICENSE_CHANNEL: The channel for the installation upgrade.
$VERSION: The version to upgrade to.
$RELEASE: The helm name of the installation release. This does not need to match the name of the release already used. By default, wallaroo.

helm upgrade --install --version $VERSION \
--wait --timeout 10m \
--values local-values.yaml \
--debug \
$RELEASE \
oci://registry.replicated.com/wallaroo/$LICENSE_CHANNEL/wallaroo \
--wait \
--wait-for-jobs

For example, the $LICENSE_CHANNEL=2025-1, $VERSION=2025.1.0-6158, and $RELEASE=wallaroo, the command would be:

helm upgrade --install --version 2025.1.0-6158 \
--wait --timeout 10m \
--values local-values.yaml \
--debug \
wallaroo \
oci://registry.replicated.com/wallaroo/2025-1/wallaroo \
--wait \
--wait-for-jobs

Once the installation is complete, verify the installation with the helm test $RELEASE command. With the settings above, this would be:

helm test wallaroo

A successful upgrade will resemble the following:

NAME: wallaroo
LAST DEPLOYED: Thu Apr 11 09:56:17 2024
NAMESPACE: default
STATUS: pending-upgrade
REVISION: 2
TEST SUITE:     wallaroo-fluent-bit-test-connection
Last Started:   Thu Apr 11 10:03:52 2024
Last Completed: Thu Apr 11 10:03:56 2024
Phase:          Succeeded
TEST SUITE:     nats-test-request-reply
Last Started:   Thu Apr 11 10:03:44 2024
Last Completed: Thu Apr 11 10:03:52 2024
Phase:          Succeeded
TEST SUITE:     wallaroo-test-connections-hook
Last Started:   Thu Apr 11 10:03:56 2024
Last Completed: Thu Apr 11 10:06:11 2024
Phase:          Succeeded
TEST SUITE:     wallaroo-test-objects-hook
Last Started:   Thu Apr 11 10:06:12 2024
Last Completed: Thu Apr 11 10:06:21 2024
Phase:          Succeeded