Upgrade Air-Gapped Cluster Installations of Wallaroo via Helm

Table of Contents

Organizations have the ability to upgrade Wallaroo to newer versions while keeping their workspaces, model uploads, ML workload orchestrations, and other artifacts intact. This process is fast and easy, allowing users to seamlessly resume their work with Wallaroo Ops without any changes.

The duration of a standard upgrade may vary from 30 to 60 minutes, depending on the size and quantity of workspaces and artifacts. Please note that services to the Wallaroo Ops instance will be temporarily interrupted during the upgrade process.

The upgrade Wallaroo follows these general steps.

  • Pre-upgrade Checklist: Actions that should be performed before the upgrade process is initiated. See Upgrade Prerequisites for more details.
  • Upgrade Procedure: Steps for upgrading Wallaroo to a specified version.

Upgrade Procedure

Depending on the size and number of workspaces and artifacts, a typical upgrade can take 30-60 minutes.

Verify allowVolumeExpansion is True

Verify that the Kubernetes setting allowVolumeExpansion is set to true through the following procedure.

  1. From the command line of the Kubernetes administrative node, run kubectl get sc. For example:

    kubectl get sc

    The result should resemble the following:

    NAME                          PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
premium-rwo                   pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   453d
standard                      kubernetes.io/gce-pd    Delete          Immediate              true                   453d
standard-rwo (default)        pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   453d
wallaroo-standard (default)   pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   6d1h

  2. For any nodes marked (default) that are not named wallaroo-standard, verify the AllowVolumeExpansion value through the following command:

    kubectl describe sc {name}

    Based on the following example, this node is standard-rwo, making the describe command the following:

    kubectl describe sc standard-rwo

    The following is a sample result from this command.

    Name:                  standard-rwo
IsDefaultClass:        Yes
Annotations:           components.gke.io/component-name=pdcsi,components.gke.io/component-version=0.16.18,components.gke.io/layer=addon,storageclass.kubernetes.io/is-default-class=true
Provisioner:           pd.csi.storage.gke.io
Parameters:            type=pd-balanced
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

  3. If AllowVolumeExpansion is False, see the Kubernetes documentation on Volume expansion and follow the steps for your host environment on setting the AllowVolumeExpansion value to True before proceeding.

Air-Gapped Helm Prerequisites

For installations of Wallaroo in air-gapped environments, the following steps are required before performing the upgrade. Select the installation method used to install Wallaroo and follow the steps provided.

Amazon Web Services EKS Air-Gapped Installations

Using the same Elastic Container Registry used to Install Wallaroo in Amazon Web Services EKS, execute the following commands.

  1. Download the following bash scripts:

    1. generate-airgap-image-values.bash
    2. load-images-ecr.bash.

  2. Open a terminal and set the following terminal convenience variables. These should be the same as used to store Wallaroo installation files except Version and LICENSE_CHANNEL - these must be updated to reflect the version of Wallaroo being upgraded to.

    REGISTRY_ID=YOUR AWS ECR REGISTRY ID
REGION=YOUR AWS ECR REGION
REGISTRY_URI=${REGISTRY_ID}.dkr.ecr.${REGION}.amazonaws.com
LICENSE_CHANNEL=YOUR LICENSE CHANNEL
VERSION=YOUR VERSION
WALLAROO_LICENSE_USERNAME=YOUR WALLAROO INSTALL USER NAME
WALLAROO_LICENSE_PASSWORD=YOUR WALLAROO INSTALL USER PASSWORD

    For example:

    REGISTRY_ID=9876543210
REGION=us-east-2
REGISTRY_URI=${REGISTRY_ID}.dkr.ecr.${REGION}.amazonaws.com
LICENSE_CHANNEL=2025-1
VERSION=2025.1.0-5924
WALLAROO_LICENSE_USERNAME=abcdefg
WALLAROO_LICENSE_PASSWORD=12345679

  3. Load the updated Wallaroo images to the ECR via the load-images-ecr.bash using the following command:

    bash load-images-ecr --region $REGION --wallaroo-version $VERSION --wallaroo-license $WALLAROO_LICENSE_PASSWORD --wallaroo-username $WALLAROO_LICENSE_USERNAME

  4. Generate the updated image-values.yaml file via the following command:

    bash generate-airgap-image-values.bash --registry $REGISTRY_URI > image-values.yaml

Follow the helm steps as follows, noting where Air-Gapped commands are required.

Upgrade via Helm Procedure

To upgrade a helm based installation of Wallaroo:

  1. From a Wallaroo Support representative, retrieve the following:
    1. The license channel. This will be in the form of oci://registry.replicated.com/wallaroo/$CHANNEL/wallaroo, where $CHANNEL represents the channel type. For example, oci://registry.replicated.com/wallaroo/2025-1/wallaroo.
    2. The version to be upgrade to. For example: 2025.1.0-6158.
    3. OCI Registry login. This will be in the format: helm registry login registry.replicated.com --username $WALLAROO_LICENSE_USERNAME --password $WALLAROO_LICENSE_PASSWORD
    4. Helm Release Name: This was determined during the Wallaroo Install process.
  2. Prepare the local-values.yaml file that will store the essential configurations options. It is highly recommended to use the same local-values.yaml file used during the Wallaroo installation for minimum changes. The following is an example of the local-values.yaml file settings with the minimum air-gapped install settings. See Wallaroo Helm Reference Guides for additional settings.
wallarooDomain: "wallaroo.example.com" # change to match the domain name

custTlsSecretName: cust-cert-secret

ingress_mode: internal # internal (Default), external,or none

dashboard:
  clientName: "Wallaroo Helm Example" # Insert the name displayed in the Wallaroo Dashboard

kubernetes_distribution: ""   # Required. One of: aks, eks, gke, oke, or kurl.

# enable private container repository for installation.  REQUIRED for air-gapped installs
imageRegistry: example.registry.com

From a terminal with helm and administrative access to the Kubernetes cluster Wallaroo Ops is installed to, perform the following:

  1. Login to the Wallaroo OCI registry, replacing $WALLAROO_LICENSE_USERNAME and $WALLAROO_LICENSE_PASSWORD with the ones provided by Wallaroo:

    helm registry login registry.replicated.com --username $WALLAROO_LICENSE_USERNAME --password $WALLAROO_LICENSE_PASSWORD

  2. Set the default Kubernetes namespace to the one used for the Wallaroo installation. By default, the namespace wallaroo is used. For example:

    kubectl config set-context --current --namespace wallaroo

  3. Perform the preflight check. Preflight verification is performed with the following command format. The variables $LICENSE_CHANNEL and $VERSION is supplied by your Wallaroo support representative.

    helm template --is-upgrade \
oci://registry.replicated.com/wallaroo/$LICENSE_CHANNEL/wallaroo --version $VERSION \
| kubectl preflight -

    For example, the $LICENSE_CHANNEL=2025-1 and the $VERSION=2025.1.0-6158

    helm template --is-upgrade \
oci://registry.replicated.com/wallaroo/2025-1/wallaroo --version 2025.1.0-6158 \
| kubectl preflight -

    This displays the Preflight Checks report. Verify that all checks are completed successfully before proceeding.

  4. Perform the upgrade with the following command, replacing the following:

    • $LICENSE_CHANNEL: The channel for the installation upgrade.
    • $VERSION: The version to upgrade to.
    • $RELEASE: The helm name of the installation release. This does not need to match the name of the release already used. By default, wallaroo.

    For Air-Gapped installations, the Helm upgrade command is as follows, using the variables from the step Air-Gapped Helm Prerequisites.

    helm upgrade --install --version $VERSION \
--wait --timeout 10m \
--values local-values.yaml \
--values image-values.yaml \
--debug \
$RELEASE \
oci://registry.replicated.com/wallaroo/$LICENSE_CHANNEL/wallaroo \
--wait \
--wait-for-jobs

    Both $LICENSE_CHANNEL and $VERSION are the same values used during the Wallaroo installation process, while $RELEASE is up to the organization. For example, if the $LICENSE_CHANNEL is 2025-1, the $VERSION is 2025.1.0-6158, and $RELEASE is wallaroo, then the command becomes:

    helm upgrade --install --version 2025.1.0-6158 \
--wait --timeout 10m \
--values local-values.yaml \
--values image-values.yaml \
--debug \
wallaroo \
oci://registry.replicated.com/wallaroo/2025-1/wallaroo \
--wait \
--wait-for-jobs

  5. Once the installation is complete, verify the installation with the helm test $RELEASE command. With the settings above, this would be:

    helm test wallaroo

    A successful upgrade will resemble the following:

    NAME: wallaroo
LAST DEPLOYED: Thu Apr 11 09:56:17 2024
NAMESPACE: default
STATUS: pending-upgrade
REVISION: 2
TEST SUITE:     wallaroo-fluent-bit-test-connection
Last Started:   Thu Apr 11 10:03:52 2024
Last Completed: Thu Apr 11 10:03:56 2024
Phase:          Succeeded
TEST SUITE:     nats-test-request-reply
Last Started:   Thu Apr 11 10:03:44 2024
Last Completed: Thu Apr 11 10:03:52 2024
Phase:          Succeeded
TEST SUITE:     wallaroo-test-connections-hook
Last Started:   Thu Apr 11 10:03:56 2024
Last Completed: Thu Apr 11 10:06:11 2024
Phase:          Succeeded
TEST SUITE:     wallaroo-test-objects-hook
Last Started:   Thu Apr 11 10:06:12 2024
Last Completed: Thu Apr 11 10:06:21 2024
Phase:          Succeeded