Pipeline Deployment Troubleshooting

How to detect issues with pipeline deployments

The following methods assist in determining issues during pipeline deployments.

Typical Deployment Issues

When checking for deployment issues:

Verify the resources specified in the Pipeline Deployment Configuration are available.
Verify any model steps assigned to the pipeline meet the requirements for the specific model framework.
Examine the Inference Logging options to examine outputs and performance.
For models deployed in the Wallaroo Containerized Runtime, use the sidekick pipeline logs to retrieve more data on the processes and messages occurring within the model operations.

Pipeline Sidekick Logs

Pipelines with models deployed in the Wallaroo Containerized Runtime store log outputs of their operations as sidekick pod logs. These represent the outputs of Kubernetes or Podman pods, Python script outputs of operations for Python and Wallaroo Custom Models, etc.

These logs are useful for determining where errors may have occurred during deployment or inference steps. For issues involving errors during model upload, see Model Auto-Packaging Troubleshooting.

Models deployed in the Wallaroo Containerized Runtime store the outputs of the Kubernetes or Podman pod logs, referred to as pipeline sidekick pod logs. These logs have the following qualities:

Each pipeline sidekick pod log is a specific output for the deployed model in its replica. For example, if the pipeline deployment configuration uses multiple replicas, the same model is deployed multiple times, each with its own pod logs.
Pipeline sidekick pod logs are tied to the specific model version deployed in the pipeline. For example, if the pipeline has the model step named sample-model, and the pipeline is deployed first with sample-model version A, and later sample-model version B, each deployed model version has its own sidekick pod logs.
Logs are deleted from oldest to newest with data retention clean up run regularly enough to prevent reaching the DB storage limits.
Returned log results limit is 1 million log lines or 10 MB, whichever limit is reached first. For large log amounts, restrict the date and time requests to collect smaller amounts.

Get Pipeline Sidekick Logs

The pod logs for these sidekick models are retrievable via the Wallaroo SDK and the Wallaroo MLOps API.

Get Pipeline Sidekick Logs via the Wallaroo SDK

Wallaroo.Pipeline.pipeline.get_sidekick_pod_logs(sidekick_name, start_datetime, end_datetime, limit) and returns the most recent log lines per specified pod name. sidekick_name is retrieved via the Wallaroo.Pipeline.pipeline.status() method.

Get Pipeline Sidekick Logs via the Wallaroo SDK Parameters

pipeline.get_sidekick_pod_logs takes the following parameters.

Parameter	Type	Description
`sidekick_name`	String (Required)	The name of the specific sidekick pod logs to retrieve.
`start_datetime`	Datetime (Optional)	The start date and time to begin retrieving logs.
`end_datetime`	Datetime (Optional)	The end date and time to begin retrieving logs.
`limit`	Integer (Optional) (Default: 100)	The number of records to retrieve; default is `100`.

Get Pipeline Sidekick Logs via the Wallaroo SDK Returns

pipeline.get_sidekick_pod_logs returns the following values as a typing.List[String], where each List consists of a single line of log outputs.

Get Pipeline Sidekick Logs via the Wallaroo SDK Example

The following is an example of retrieving pipeline sidekick pod logs for a currently deployed pipeline.

Retrieve the sidekick name from pipeline.status. The sidekick model name is in the sidekicks[{replica}][name] field.

pipeline.status()

{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.4.1.5',
   'name': 'engine-97c8b5466-mr2s8',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'sidekick-logs-tutorial-demo',
      'status': 'Running',
      'version': '94be98b0-e1b8-40f8-b6bf-2001fb9a80f6'}]},
   'model_statuses': {'models': [{'model_version_id': 1515,
      'name': 'vgg16-clustering',
      'sha': '79472d4b9652937c15e1887478c23effe1160363b985920e083fec72868810f4',
      'status': 'Running',
      'version': '93770042-147b-4d07-97f4-2f91e41f6dcf'}]}}],
 'engine_lbs': [{'ip': '10.4.1.3',
   'name': 'engine-lb-8576cc4974-z4f7d',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': [{'ip': '10.4.1.4',
   'name': 'engine-sidekick-vgg16-clustering-1515-5ddd999747-6fjbd',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'}]}

For this example, the sidekick model name is engine-sidekick-vgg16-clustering-1515-5ddd999747-6fjbd.

Execute pipeline.get_sidekick_pod_logs(sidekick_name) for the specific sidekick model. For example:

pipeline.get_sidekick_pod_logs(sidekick_name="engine-sidekick-vgg16-clustering-1515-5ddd999747-6fjbd")

'2026-02-23T12:30:55.449982309Z stdout F          successful.                                                            ',
'2026-02-23T12:30:55.451582846Z stdout F INFO     mac.service.arrow_flight.arrow_flight_servic arrow_flight_service.py:58',
'2026-02-23T12:30:55.451638904Z stdout F          e - INFO: Creating synchronous Arrow Flight                            ',
'2026-02-23T12:30:55.451647026Z stdout F          RPC service...                                                         ',
'2026-02-23T12:30:55.697992122Z stdout F INFO     mac.service.arrow_flight.arrow_flight_servic arrow_flight_service.py:68',
'2026-02-23T12:30:55.698050612Z stdout F          e - INFO: Successfully created Arrow Flight                            ',
'2026-02-23T12:30:55.698057233Z stdout F          RPC service.                                                           ',
'2026-02-23T12:30:55.6995903Z stdout F INFO     root - INFO: [📡] Starting server on         arrow_flight_service.py:30',

Get Sidekick Logs History

Sidekick logs for previous model versions deployed as pipeline steps are retrieved with the Wallaroo SDK via the pipeline.get_sidekick_pod_logs(sidekick_name, start_datetime, end_datetime, limit). The Sidekick Logs History command retrieves a list of previous models used as pipeline steps and the sidekick_name for each model.

Get Sidekick Logs History via the Wallaroo SDK

The sidekick pod logs history is retrieved via the Wallaroo SDK using the Pipeline.pipeline.get_sidekick_pods_history. This retrieves a list of current and previous models used in the pipeline’s Wallaroo Containerized Runtime, thier associated models names, and other useful details.

Get Sidekick Logs History via the Wallaroo SDK Parameters

The method get_sidekick_pods_history accepts the following parameters.

Parameter	Type	Description
`model_name`	String (Optional)	Optional parameter to filter pipeline steps by model name.

Get Sidekick Logs History via the Wallaroo SDK Returns

The method get_sidekick_pods_history returns a List of historical sidekick pods used in the pipelines, excluding current pods. These results are sorted in reverse chronological order by the field started_at.

Each List item contains the following elements.

Fields	Type	Description
`sidekick_name`	String	The name of the sidekick pod associated with the pipeline for this model version.
`started_at`	DateTime	The date and time the model version was deployed as a pipeline model step.
`pipeline_model_step`	String	The model name as added as a pipeline step.
`model_sha`	String	The SHA hash of the model version file.
`model_version`	The UUID identifier for the model version.

Get Sidekick Logs History via the Wallaroo SDK Example

The following demonstrates retrieving the model history for a pipeline.

pipeline.get_sidekick_pods_history()

sidekick_name	started_at	pipeline_model_step	model_sha	model_version
engine-sidekick-noop-py-pre-b-5-0	2026-03-31T20:30:51.797Z	noop-py-pre-b	9b634f014101dc88c2a31ba9018dc5d0f364f5e44f4d4923f49078c9bd56cbed	1d9f841a-5a2e-496c-98e3-603b57b050fc
engine-sidekick-noop-py-post-7-0	2026-03-31T20:27:51.798Z	noop-py-post	167b50597f55e1fc774aefb7622d75f03ef542c6479971b8eb2de6210ab01813	4f3b5c97-566c-4ae0-8f7f-aa3697c5d530

Error Messages

The following error messages are displayed based on various input errors.

"Sidekick '{sidekick name}' not found in pipeline": The requested sidekick name does not exist in the requested pipeline.
No logs: No logs exist for the requested sidekick or were not found the for requested start and end time.
Invalid datetime format: Valid formats are RFC3339, ISO8601, or date-only (YYYY-MM-DD).
MLOps API Only Workspace name not found: XXXXX: The request with the specified workspace does not match any existing workspace.
MLOps API Only Pipeline not found: YYYYYYYYY: The request with the specified workspace does not match any existing workspace.