Organizations can manage their ML Model storage in their Wallaroo instances through the MinIO interface and its related PersistentVolumeClaim (PVC) included in the standard Wallaroo installation.

The following details how to access and the Wallaroo MinIO service. For full details on using the MinIO service, see the MinIO Documentation site.

All of the steps below require administrative access to the Kubernetes service hosting the Wallaroo instance.

Wallaroo MinIO Model Storage

Wallaroo ML Models are stored in the MinIO bucket model-bucket.

Retrieving the Wallaroo MinIO Password

Access to the Wallaroo MinIO service is password protected. DevOps with administrative access to the Kubernetes cluster hosting the Wallaroo instance can retrieve this password with the following:

• The kubectl command.
• The namespace the Wallaroo instance is installed to.

This command takes the following format:

kubectl -n {Wallaroo Namespace} get secret minio -o 'jsonpath={.data.rootPassword}' | base64 -d

For example, if the Wallaroo instance is installed into the namespace wallaroo this command would be:

kubectl -n wallaroo get secret minio -o 'jsonpath={.data.rootPassword}' | base64 -d

Accessing the MinIO Service

Access to the MinIO service in a Wallaroo instance is performed either with the Command Line Interface (CLI), or through a browser based User Interface (UI).

Accessing the MinIO Service Through CLI

Access to the MinIO service included with the Wallaroo instance can be performed with the command line tool mc. For more details, see the MinIO Client documentation.

Installing Minio CLI

The following demonstrates installing the mc command for MacOS and Linux.

Installing for MacOS with Brew

MacOS users who have installed Homebrew can install mc with the following:

brew install minio/stable/mc

Installing for Linux

Linux users can install the MinIO CLI tool mc with the following:

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin/mc

Port Forward for MinIO CLI Access

To access the Wallaroo MinIO service, use Kubernetes port-forward to connect. By default this is on port 9000. This command requires the following:

• The kubectl command.
• The namespace the Wallaroo instance is installed to.

This command is in the following format:

kubectl port-forward services/minio 9000:9000 -n {Wallaroo Namespace}

For example, if the Wallaroo instance is installed to the default namespace wallaroo, this command is:

kubectl port-forward services/minio 9000:9000 -n wallaroo

Show MinIO Disk Usage through CLI

To view the Wallaroo MinIO service through the CLI, the following is required:

• The kubectl command.
• The namespace the Wallaroo instance is installed to.
• The MinIO CLI tool mc.
• The MinIO password.

Here’s an example showing MinIO disk usage via the CLI for the namespace wallaroo. When prompted for the Secret Key, press Enter for none.

The following script displays the space used for a default installation of Wallaroo in the namespace wallaroo.

#!/bin/bash
if kubectl -n wallaroo get secret minio >& /dev/null; then
    u="$(kubectl -n wallaroo get secret minio -o jsonpath='{ .data.rootUser }' | base64 -d)"
    p="$(kubectl -n wallaroo get secret minio -o jsonpath='{ .data.rootPassword }' | base64 -d)"
    creds="$u $p"
fi

mc alias set --insecure wallaroo http://localhost:9000 $creds; mc du --recursive wallaroo

The output:

Added `wallaroo` successfully.
2.2GiB	22 objects	model-bucket
2.2GiB	22 objects

Accessing the MinIO Service Through the UI

The MinIO service included with the Wallaroo instance can be access through the MinIO user interface. By default this is port 9001.

Port Forward for MinIO UI Access

The MinIO UI port 9001 can be access through the kubectl port-forward command.

This command requires the following:

• The kubectl command.
• The namespace the Wallaroo instance is installed to.

This command is in the following format:

kubectl port-forward services/minio-console 9001:9001 -n {Wallaroo Namespace}

For example, to port forward through the default installation namespace wallaroo:

kubectl port-forward services/minio-console 9001:9001 -n wallaroo

Accessing the Wallaroo MinIO UI

Once the port forward command is running, the MinIO UI is access through a browser on port 9001 with the user minio and the MinIO administrative password retrieved through the step Retrieving the Minio Administrative Password.

Viewing General Storage

General disk usage is displayed through Monitoring->Metrics.

Viewing ML Model Storage

ML Models stored for Wallaroo are accessed through the bucket model-bucket.

Select Browse to view the contents of the model-bucket. To determine the specific file name, access the Name of the object and view the Tags. The file name is access via the Tag file-name.

Objects can be deleted from this bucket with the Delete option.

Increase MinIO Storage

Storage capacity increases for MinIO requires that allowVolumeExpansion: true is set in your cluster. Check with your cloud service provider to verify this setting can be enabled.

The following procedures are based on the method for installing Wallaroo:

• kots
• helm

Increase MinIO Storage for Kots

For installation of Wallaroo via kots:

1. Access the Kots Administrative Dashboard with the kubectl kots command, replacing $NAMESPACE with the Kubernetes namespace used to install Wallaroo. By default this is wallaroo.

   kubectl kots admin-console --namespace $NAMESPACE
   

2. A URL will be provided to access the dashboard in a browser. By default, this is http://localhost:8800.

3. Select Config, then the field Wallaroo ML Engine->Size of model data store. In units of Gi, which must be provided.

4. Update the storage capacity in terms of Gi. By default, this is 10Gi.

5. Once updated, scroll to the bottom and select Save config, then deploy the new configuration.

Increase MinIO Storage for Helm

To increase the model storage size for helm based installations of Wallaroo:

1. Edit your local values.yaml with the following fragment:

   minio:
     persistence:
       size: 25Gi # Minio model storage disk size. Smaller than 10Gi is not recommended.
   

2. Apply the update with helm upgrade. For example, to upgrade the registration wallaroo from the EE channel the command would be:

   helm upgrade wallaroo oci://registry.replicated.com/wallaroo/2024-1/wallaroo --version 2024.1.0-5097 --values local-values.yaml
   

Troubleshooting

If the PersistentVolumeClaim (PVC) status still shows Waiting, check with the PersistentVolume (PV) settings and verify that the storage settings match the PVC.