Inference Log Storage

How to manage Inference Log Storage in Wallaroo

Targeted Role
Dev Ops

Organizations can manage the amount of Inference Log Storage is allocated to Wallaroo, either to save on costs or expand the amount of inference logs stored before old logs are pruned. The amount of disk space provisioned is adjustable, and set to 100Gi storage by default. Storage smaller than 100Gi is not recommended for performance reasons.

These values are adjusted depending on the Wallaroo installation method used: kots or helm.

Update Inference Log Storage for Kots

For installation of Wallaroo via kots:

Access the Kots Administrative Dashboard with the kubectl kots command, replacing $NAMESPACE with the Kubernetes namespace used to install Wallaroo. By default this is wallaroo.
```
kubectl kots admin-console --namespace $NAMESPACE
```
A URL will be provided to access the dashboard in a browser. By default, this is http://localhost:8800.
Select Config, then the field Plateau Inference Logging Server->Plateau disk space in units which must include Gi suffix
Update the storage capacity in terms of Gi. By default, this is 100Gi. Storage smaller than 100Gi is not recommended for performance reasons.
Once updated, scroll to the bottom and select Save config, then Deploy the new configuration.

Update Inference Log Storage for Helm

To increase the model storage size for helm based installations of Wallaroo:

Edit your local values.yaml with the following fragment. Note that the tolerations and nodeSelector must match any custom Kubernetes taints or tolerations. The values below show the default settings.

# For each service below, adjust the disk size and resources as required.
# If the nodeSelector or tolerations are changed for one service, 
# the other services nodeSelector and tolerations **must** be changed to match
#
#
plateau:
  diskSize: 100Gi
  resources:
    limits:
      memory: 4Gi
      cpu: 1000m
    requests:
      memory: 128Mi
      cpu: 100m
  nodeSelector:
    wallaroo.ai/node-purpose: persistent
  tolerations:
    - key: "wallaroo.ai/persistent"
      operator: "Exists"
      effect: "NoSchedule"

Apply the update with helm upgrade. For example, to upgrade the registration wallaroo from the EE channel the command would be:

helm upgrade wallaroo oci://registry.replicated.com/wallaroo/2024-2/wallaroo --version 2024.2.1 --values local-values.yaml --timeout 10m