Metrics Storage

How to modify metrics aka prometheus storage.

Adjusting the storage space for metrics in Wallaroo depends on whether the installation used kots or helm. For more details on metrics in Wallaroo, see the following.

Metric Storage Conditions

Metric data is stored until either the retention data period is passed or the disk space is full. At that point, data is removed from the oldest records first for the new records.

Set Metrics Storage via Kots

For kots based installations of Wallaroo, the following procedure updates the storage capacity used for metrics logging.

Kots Prerequisites

kubectl
For Kots based installs:
- kots Version 1.112.2

Kots Procedure

Access the Kots Administrative Dashboard for the Wallaroo installation via the following command. Replace the namespace wallaroo with the one used for the Wallaroo installation.
```
  kubectl kots admin-console --namespace wallaroo
    • Press Ctrl+C to exit
    • Go to http://localhost:8800 to access the Admin Console
```
Select Config, then scroll to Prometheus Data and Metrics Logging.
Update the following settings.
1. Prometheus retention time in days (Default: 90): Set the number of days to keep metric data.
2. Prometheus disk space in GB (Default: 100)
When finished, scroll to the bottom of the Config page and select Save Config, then Go to updated version.
For the latest configuration, select Deploy. When complete, the metric storage capacity is updated.

Set Metrics Storage via Helm

The following instructions are for installations of Wallaroo in a helm based installation.

JupyterHub Helm Settings

The following metrics storage settings are determined via helm:

Parameter	Description	Default
`prometheus.storageRetentionSizeGb`	Sets the storage capacity for metric data.	`100`
`prometheus.storageRetentionTimeDays`	Sets the retention period for metric data in days.	`90`

Helm Procedure

Update a helm yaml file to indicate the modified settings. The following is a minimum local-values.yaml file with the Wallaroo Domain set to wallaroo.example.com and the optional metric storage capacity commented out.

wallarooDomain: "wallaroo.example.com" # change to match the domain name

custTlsSecretName: cust-cert-secret

apilb:
  serviceType: LoadBalancer
  external_inference_endpoints_enabled: true
  ingress_mode: internal # internal (Default), external,or none

dashboard:
  clientName: "Wallaroo Helm Example" # Insert the name displayed in the Wallaroo Dashboard

kubernetes_distribution: ""   # Required. One of: aks, eks, gke, oke, or kurl.

prometheus:
  storageRetentionSizeGb: "100"        # Retain this much data, in GB.
  storageRetentionTimeDays: "90"

Update the helm based installation with the helm upgrade command in the following format:
```
helm upgrade $RELEASE $REGISTRYURL --version $VERSION --values $LOCALVALUES.yaml --timeout 10m
```
Where:
1. $RELEASE: The name of the Helm release. By default, wallaroo.
2. $REGISTRYURL: The URl for the Wallaroo container registry service.
3. $VERSION: The version of Wallaroo to install. For this example, 2024.4.0-5879.
4. $LOCALVALUES: The .yaml file containing the local values overrides. For this example, local-values.yaml.
For example, for the release wallaroo the command would be:
```
helm upgrade wallaroo oci://registry.replicated.com/wallaroo/2024-4/wallaroo --version 2024.4.0-5879 --values local-values.yaml --timeout 10m
```

Once the upgrade is complete, the metric storage and retention period is adjusted to the new values.