Monitor Wallaroo Pipeline Logs through Kubernetes

How to retrieve pipeline inference logs through Kubernetes.

Table of Contents

Wallaroo provides interactive error messages and pipeline inference logs available through the Wallaroo SDK and Wallaroo MLOps API.

The following provides additional methods for tracing logs through the Kubernetes (K8s) logs interface. The instructions below focus on using the kubectl command line interface. Other Kubernetes monitoring tools, such as Lens are also useful for monitoring Kubernetes based logs through a friendlier user interface.

These instructions are valid for nearly any Kubernetes deployment in cloud or stand-alone environments. Check with the specific provider for additional details.

Note that Kubernetes logs are short term and are not persistent; once a Wallaroo pipeline is undeployed or the Kubernetes cluster is halted, these logs are no longer available. This troubleshooting process is best in gathering logs from Kubernetes to debug the ML models within the Wallaroo inference engine while it is deployed.

Prerequisites

Kubectl access.

Kubernetes kubectl Steps

  1. Retrieve the pipeline name. Remember this name.

  2. Use kubectl get ns to list
    namespaces.

  3. Choose the namespace best matching the pipeline name (there are some extra digits
    on the end indicating the version number).

  4. In that namespace will be the following Kubernetes pods, where xxx is the unique identifier for the pod:

    • engine-xxx: This is the Wallaroo Inference Engine for Native Runtimes (onnx, tensorflow, etc). See the complete list of Models and Runtimes for full details.
    • engine-sidekick-PIPELINE-xxx: This is the Wallaroo Inference Engine for Containerized Runtimes (hugging-face, BYOP, etc). See the complete list of Models and Runtimes for full details.
    • engine-lb-xxx - this is the engine load balancer and is not used for retrieving inference logs.
  5. For Native Runtime deployments, list the Inference Engine logs with the following command:

    kubectl logs -n NAMESPACE engine-xxxxxx
    
  6. For Containerized Runtime deployments, list the Inference Engine logs with the following command:

    kubectl -n NAMESPACE engine-sidekick-xxxxx
    

Kubernetes Log Example

The following is an example transcript of retrieving logs for a ML Model of framework wallaroo.framework.Framework.PYTHON.

kubectl get namespaces
NAMESTATUSAGE
forecast-14Active86m
defaultActive21d
kube-flannelActive21d
kube-node-leaseActive21d
kube-publicActive21d
kube-systemActive21d
wallarooActive21d
pov.wallaroo.io ~ kubectl get pods -n forecast-14
NAMEREADYSTATUSAGE
engine-f4b858d65-k7fnr1/1RUNNING87m
engine-lb-74b4969486-fhdpd1/1RUNNING87m
engine-sidekick-forecast-14-6999d7644f-2tb6g1/1RUNNING87m
forecast-14 engine-f4b858d65-k7fnr | less

2024-01-01T19:15:06.249058Z  INFO fitzroy::model::manager: Loaded model SHA 3ed5cd199e0e6e419bd3d474cf74f2e378aacbf586e40f24d1f8c89c2c476a08
2024-01-01T19:15:06.249083Z  INFO fitzroy::model::manager: Adding model with SHA 3ed5cd199e0e6e419bd3d474cf74f2e378aacbf586e40f24d1f8c89c2c476a08
2024-01-01T19:15:06.413770Z DEBUG hyper::proto::h1::io: parsed 4 headers
2024-01-01T19:15:06.413796Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-01-01T19:15:06.413891Z DEBUG hyper::proto::h1::io: flushed 117 bytes
2024-01-01T19:15:06.750506Z  INFO fitzroy::model::manager: Loaded model SHA 3ed5cd199e0e6e419bd3d474cf74f2e378aacbf586e40f24d1f8c89c2c476a08
2024-01-01T19:15:06.750531Z  INFO fitzroy::model::manager: Adding model with SHA 3ed5cd199e0e6e419bd3d474cf74f2e378aacbf586e40f24d1f8c89c2c476a08
2024-01-01T19:15:07.251029Z  INFO fitzroy::model::manager: Loaded model SHA 3ed5cd199e0e6e419bd3d474cf74f2e378aacbf586e40f24d1f8c89c2c476a08
2024-01-01T19:15:07.251087Z  INFO fitzroy::model::manager: Adding model with SHA 3ed5cd199e0e6e419bd3d474cf74f2e378aacbf586e40f24d1f8c89c2c476a08
...(additional entries follow)

Using less

less is a Linux util for paging through large amounts of text. Putting it at the end of a command, such as “kubectl logs” will let you page through the logs. When viewing the log data you can navigate through with a variety of single-character commands. The most important are:

KeyFunction
qQuit
GGo to the end of the file / log
gGo to the top of the log
Page up / downPage through the file
?Search backwards through the file
/Search forwards through the file