The following steps are used to view the Kubernetes pod log during Pipeline deployment and execution. DevOps engineers modify the deployed Pipeline in Kubernetes to allow for trace level logs and display them as needed.

Set Log Level to Trace

Wallaroo pipelines are deployed through its own Kubernetes namespace in the format , we can retrieve additional log data directly from the Kubernetes deployment.

1. From the terminal prompt, determine the namespace for the deployed pipeline with the command kubectl get namespaces. This is in the format {pipeline-name}-{id}. For example, the pipeline python-step-demo-pipeline is python-step-demo-pipeline-48:

   kubectl get namespaces
   NAME                           STATUS        AGE
   default                        Active        153d
   edge-pipeline-6                Terminating   22m
   kube-node-lease                Active        153d
   kube-public                    Active        153d
   kube-system                    Active        153d
   python-step-demo-pipeline-48   Active        53s
   velero                         Active        140d
   wallaroo                       Active        4d2h
   

2. Edit the engine for the pipeline with the command kubectl edit deploy engine -n {pipeline-namespace}. For example, using the pipeline deployed as python-step-demo-pipeline-48, that command is:

   kubectl edit deploy engine -n python-step-demo-pipeline-48
   

3. Within the deploy engine config, scroll to spec.template.spec.containers.env where name=RUST_LOG. The easiest way is to search by pressing the / and typing in RUST_LOG. This is how it looks by default:

   env:
   - name: RUST_LOG
       value: debug,fitzroy::sink=info
   - name: RUST_BACKTRACE
       value: full
   - name: MINIO_NAMESPACE
       value: wallaroo
   

4. Update the RUST_LOG value from debug to trace. This is done by placing the cursor over at debug, then typing i (for insert), and deleting debug to trace. Look up instructions on using the vi editor. It will look like this.

   env:
   - name: RUST_LOG
       value: trace,fitzroy::sink=info
   - name: RUST_BACKTRACE
       value: full
   - name: MINIO_NAMESPACE
       value: wallaroo
   

5. When finished, hit ESC to exit the editor, then enter :wq. Wait for the engine pod to reset. Use the command kubectl get pods -n {pipeline_namespace} to view the pods. The engine pod(s) will reset. For example:

   kubectl get pods -n python-step-demo-pipeline-48
   NAME                         READY   STATUS        RESTARTS   AGE
   engine-85cb7b6bd8-bqf6v      1/1     Running       0          17s
   engine-d4cfb84c8-fs89h       1/1     Terminating   0          2m55s
   engine-lb-584f54c899-qj55z   1/1     Running       0          2m55s
   helm-runner-s5t5k            0/1     Completed     0          2m56s
   

View Kubernetes Pipeline Logs

1. Now we can view the logs with kubectl logs --follow -n {pipeline_namespace} {engine_pod}. With the pipeline and pods above:

   kubectl logs --follow  -n python-step-demo-pipeline-48 engine-85cb7b6bd8-bqf6v
   

   Here’s the standard output:

   {"msg":"deregistering event source from poller","level":"TRCE","ts":"2023-09-12T16:42:52.660761783Z","file_line":"/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.8/src/poll.rs:662"}
   {"msg":"registering event source with poller: token=Token(419430405), interests=READABLE | WRITABLE","level":"TRCE","ts":"2023-09-12T16:43:02.659772451Z","file_line":"/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.8/src/poll.rs:531"}
   {"msg":"deregistering event source from poller","level":"TRCE","ts":"2023-09-12T16:43:02.661156669Z","file_line":"/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.8/src/poll.rs:662"}
   

   Here’s the output when an inference is run for the following Python step model. The outputs are from the lines {"msg":"Python: stdout:

   import pandas as pd
   
   # take a dataframe output of the house price model, and reformat the `dense_2`
   # column as `output`
   def wallaroo_json(data: pd.DataFrame):
       print(data)
       return [{"output": [data["dense_2"].to_list()[0][0]]}]
   

   {"msg":"beginning inference for python-step","level":"TRCE","ts":"2023-09-12T16:43:25.176094863Z","file_line":"src/model/manager.rs:416"}
   {"msg":"input: {\"_wallaroo_pandas\":[{\"dense_2\":[12.886651]}]}","level":"TRCE","ts":"2023-09-12T16:43:25.176198964Z","file_line":"src/runtimes/python/pipeline_session.rs:92"}
   {"msg":"Python: stdout:        dense_2","level":"INFO","ts":"2023-09-12T16:43:25.265281832Z","file_line":"src/runtimes/python/process.rs:196"}
   {"msg":"Python: stdout: 0  [12.886651]","level":"INFO","ts":"2023-09-12T16:43:25.265315833Z","file_line":"src/runtimes/python/process.rs:196"}
   {"msg":"Python result receiver: Got result from python","level":"INFO","ts":"2023-09-12T16:43:25.265321733Z","file_line":"src/runtimes/python/session_wrapper.rs:79"}
   {"msg":"output: [{\"output\":[12.886651]}]","level":"TRCE","ts":"2023-09-12T16:43:25.265327433Z","file_line":"src/runtimes/python/pipeline_session.rs:103"}
   

The list of tasks in the Wallaroo instance is retrieves through the Wallaroo Client list_tasks() method. The task list returned is based on the workspaces the user is a member of and the filtered parameters, in reverse chronological order.

Admin users have unrestricted access to all workspaces. For more details, see Wallaroo Enterprise User Management.

The following demonstrates retrieving the task list, and how to assign a task to a variable.

wl.list_tasks()

task = wl.list_tasks()[0]

Orchestration tasks logs are retrieved via the Wallaroo SDK. For full details, see Task Run Logs.

The output of a task is displayed with the Task Run logs() method that takes the following parameters.

The Task Run logs() returns the log entries as a string list, with each entry as an item in the list. For example, if the task has run 5 times, then the last_runs() command shows the first task as last_runs()[0], last_runs()[1], etc.

• IMPORTANT NOTE: It may take around a minute for task run logs to be integrated into the Wallaroo log database.

# give time for the task to complete and the log files entered
time.sleep(60)
recent_run = task.last_runs()[0]
display(recent_run.logs())

2023-22-May 19:59:29 Getting the pipeline orchestrationpipelinetgiq
2023-22-May 19:59:29 Getting arrow table file
2023-22-May 19:59:29 Inference time.  Displaying results after.
2023-22-May 19:59:29 pyarrow.Table
2023-22-May 19:59:29 time: timestamp[ms]
2023-22-May 19:59:29 in.tensor: list<item: float> not null
2023-22-May 19:59:29   child 0, item: float
2023-22-May 19:59:29 out.variable: list<inner: float not null> not null
2023-22-May 19:59:29 anomaly.count: int8
2023-22-May 19:59:29   child 0, inner: float not null
2023-22-May 19:59:29 ----
2023-22-May 19:59:29 time: [[2023-05-22 19:58:49.767,2023-05-22 19:58:49.767,2023-05-22 19:58:49.767,2023-05-22 19:58:49.767,2023-05-22 19:58:49.767,...,2023-05-22 19:58:49.767,2023-05-22 19:58:49.767,2023-05-22 19:58:49.767,2023-05-22 19:58:49.767,2023-05-22 19:58:49.767]]
2023-22-May 19:59:29 in.tensor: [[[4,2.5,2900,5505,2,...,2970,5251,12,0,0],[2,2.5,2170,6361,1,...,2310,7419,6,0,0],...,[3,1.75,2910,37461,1,...,2520,18295,47,0,0],[3,2,2005,7000,1,...,1750,4500,34,0,0]]]
2023-22-May 19:59:29 check_failures: [[0,0,0,0,0,...,0,0,0,0,0]]
2023-22-May 19:59:29 out.variable: [[[718013.75],[615094.56],...,[706823.56],[581003]]]</code></pre>

During the Orchestration upload process, the packaging and conversion process is viewable through the following procedures using kubectl.

1. During the orchestration upload process, use kubectl get namespaces to identify that namespace used. It will be in the format tasks-{identifier}. For example, in the following the tasks-1f26cbf9-ccc9-4c-20 1 second old, indicating it is the task we want for a recent upload orchestration command.

   kubectl get namespaces
   NAME                                    STATUS   AGE
   api-pipeline-with-models-11             Active   23d
   cellai-pipeline-sdk-10                  Active   28d
   default                                 Active   588d
   gke-managed-system                      Active   588d
   gke-managed-volumepopulator             Active   159d
   gmp-public                              Active   588d
   gmp-system                              Active   588d
   kube-node-lease                         Active   588d
   kube-public                             Active   588d
   kube-system                             Active   588d
   metrics-retrieval-tutorial-pipeline-9   Active   30d
   simpleorchestrationtutorial-12          Active   3m22s
   tasks-1f26cbf9-ccc9-4c-20               Active   1s <--- OUR TASK>
   velero                                  Active   512d
   wallaroo                                Active   140d
   

2. Use the get pods -n {namespace} command to retrieve a list of the pods used. For example:

   kubectl get pods -n tasks-1f26cbf9-ccc9-4c-20
   NAME                                        READY   STATUS    RESTARTS   AGE
   1f2-exec-orch-packaging-arb-one-exe-76r4f   1/1     Running   0          25s
   

3. Use the command kubectl -f {POD NAME} -n {NAMESPACE} to retrieve and continue to follow the process. For example:

   kubectl logs -f 1f2-exec-orch-packaging-arb-one-exe-76r4f -n tasks-1f26cbf9-ccc9-4c-20
   +++ dirname /build/mkenv.sh
   ++ realpath /build
   + BUILD_PATH=/build
   + MINIO_NAME=minio
   + MINIO_HOST=http://minio.wallaroo.svc.cluster.local:9000
   + MINIO_BASE=model-bucket
   + MINIO_USER=minio
   + MINIO_PASS=AYf47eeewd67ELd0kyzisr2Zcm35zhRg
   + SCRIPT_PATH=/tmp
   + NATS_SUBJECT_SUB=orch_packaging.start
   + NATS_SUBJECT_PUB=orch_packaging.update
   + READY_NATS_SUBJECT_PUB=orch_packaging.ready
   + VENV_PATH=/home/jovyan/venv
   + PYPI_INDEX_ARGS=
   + '[' -n '' ']'
   + echo 'No private PyPI configuration found, using default public PyPI'
   No private PyPI configuration found, using default public PyPI
   + trap 'cat /build/start.json | jq -cM '\''.data.status="failure"'\'' | nats -s nats.wallaroo.svc.cluster.local publish --force-stdin orch_packaging.update' ERR
   + sleep 5
   + nats -s nats.wallaroo.svc.cluster.local sub orch_packaging.start -r --count=1
   + nats -s nats.wallaroo.svc.cluster.local publish --force-stdin orch_packaging.ready
   ++ date -u --iso-8601=ns
   ++ sed s/+00:00/Z/
   ++ sed s/,/./