Inference Troubleshooting

How to detect issues with inferencing requests.

The following methods assist in determining issues during inferences.

Typical Inference Issues

When checking for deployment issues:

Verify the resources specified in the Pipeline Deployment Configuration are available.
Verify any model steps assigned to the pipeline meet the requirements for the specific model framework.
Examine the Inference Logging options to examine outputs and performance.
For models deployed in the Wallaroo Containerized Runtime, use the sidekick pipeline logs to retrieve more data on the processes and messages occurring within the model operations.

Pipeline Sidekick Logs

Pipelines with models deployed in the Wallaroo Containerized Runtime store log outputs of their operations as sidekick pod logs. These represent the outputs of Kubernetes or Podman pods, Python script outputs of operations for Python and Wallaroo Custom Models, etc.

These logs are useful for determining where errors may have occurred during deployment or inference steps. For issues involving errors during model upload, see Model Auto-Packaging Troubleshooting.

Models deployed in the Wallaroo Containerized Runtime store the outputs of the Kubernetes or Podman pod logs, referred to as pipeline sidekick pod logs. These logs have the following qualities:

Each pipeline sidekick pod log is a specific output for the deployed model in its replica. For example, if the pipeline deployment configuration uses multiple replicas, the same model is deployed multiple times, each with its own pod logs.
Pipeline sidekick pod logs are tied to the specific model version deployed in the pipeline. For example, if the pipeline has the model step named sample-model, and the pipeline is deployed first with sample-model version A, and later sample-model version B, each deployed model version has its own sidekick pod logs.
Logs are deleted from oldest to newest with data retention clean up run regularly enough to prevent reaching the DB storage limits.
Returned log results limit is 1 million log lines or 10 MB, whichever limit is reached first. For large log amounts, restrict the date and time requests to collect smaller amounts.

Get Pipeline Sidekick Logs

The pod logs for these sidekick models are retrievable via the Wallaroo SDK and the Wallaroo MLOps API.

Get Pipeline Sidekick Logs via the Wallaroo SDK

Wallaroo.Pipeline.pipeline.get_sidekick_pod_logs(sidekick_name, start_datetime, end_datetime, limit) and returns the most recent log lines per specified pod name. sidekick_name is retrieved via the Wallaroo.Pipeline.pipeline.status() method.

Get Pipeline Sidekick Logs via the Wallaroo SDK Parameters

pipeline.get_sidekick_pod_logs takes the following parameters.

Parameter	Type	Description
`sidekick_name`	String (Required)	The name of the specific sidekick pod logs to retrieve.
`start_datetime`	Datetime (Optional)	The start date and time to begin retrieving logs.
`end_datetime`	Datetime (Optional)	The end date and time to begin retrieving logs.
`limit`	Integer (Optional) (Default: 100)	The number of records to retrieve; default is `100`.

Get Pipeline Sidekick Logs via the Wallaroo SDK Returns

pipeline.get_sidekick_pod_logs returns the following values as a typing.List[String], where each List consists of a single line of log outputs.

Get Pipeline Sidekick Logs via the Wallaroo SDK Example

The following is an example of retrieving pipeline sidekick pod logs for a currently deployed pipeline.

Retrieve the sidekick name from pipeline.status. The sidekick model name is in the sidekicks[{replica}][name] field.

pipeline.status()

{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.4.1.5',
   'name': 'engine-97c8b5466-mr2s8',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'sidekick-logs-tutorial-demo',
      'status': 'Running',
      'version': '94be98b0-e1b8-40f8-b6bf-2001fb9a80f6'}]},
   'model_statuses': {'models': [{'model_version_id': 1515,
      'name': 'vgg16-clustering',
      'sha': '79472d4b9652937c15e1887478c23effe1160363b985920e083fec72868810f4',
      'status': 'Running',
      'version': '93770042-147b-4d07-97f4-2f91e41f6dcf'}]}}],
 'engine_lbs': [{'ip': '10.4.1.3',
   'name': 'engine-lb-8576cc4974-z4f7d',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': [{'ip': '10.4.1.4',
   'name': 'engine-sidekick-vgg16-clustering-1515-5ddd999747-6fjbd',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'}]}

For this example, the sidekick model name is engine-sidekick-vgg16-clustering-1515-5ddd999747-6fjbd.

Execute pipeline.get_sidekick_pod_logs(sidekick_name) for the specific sidekick model. For example:

pipeline.get_sidekick_pod_logs(sidekick_name="engine-sidekick-vgg16-clustering-1515-5ddd999747-6fjbd")

'2026-02-23T12:30:55.449982309Z stdout F          successful.                                                            ',
'2026-02-23T12:30:55.451582846Z stdout F INFO     mac.service.arrow_flight.arrow_flight_servic arrow_flight_service.py:58',
'2026-02-23T12:30:55.451638904Z stdout F          e - INFO: Creating synchronous Arrow Flight                            ',
'2026-02-23T12:30:55.451647026Z stdout F          RPC service...                                                         ',
'2026-02-23T12:30:55.697992122Z stdout F INFO     mac.service.arrow_flight.arrow_flight_servic arrow_flight_service.py:68',
'2026-02-23T12:30:55.698050612Z stdout F          e - INFO: Successfully created Arrow Flight                            ',
'2026-02-23T12:30:55.698057233Z stdout F          RPC service.                                                           ',
'2026-02-23T12:30:55.6995903Z stdout F INFO     root - INFO: [📡] Starting server on         arrow_flight_service.py:30',

Get Pipeline Sidekick Logs via the Wallaroo MLOps API

Sidekick logs are retrieved via following Wallaroo MLOps API endpoint.

Endpoint: /v1/api/pipelines/get_sidekick_pod_logs (POST)
Content-Type: application/json

Get Pipeline Sidekick Logs via the Wallaroo MLOps API Parameters

The API endpoint /v1/api/pipelines/get_sidekick_pod_logs accepts the following parameters.

Parameter	Type	Description
`sidekick_name`	String (Required)	The name of the specific sidekick pod logs to retrieve.
`workspace_name`	String (Required)	The name of the workspace the pipeline deploying the model is assigned to.
`pipeline_name`	String (Required)	The pipeline the model is deployed through.
`limit`	Integer (Optional) (Default: 100)	The number of records to retrieve; default is `100`.
`start_datetime`	String in ISO Datetime Format (Optional)	The date and time to start log collection. For example: `start_datetime = datetime.datetime.now().isoformat()`.
`end_datetime`	String in ISO Datetime Format (Optional)	The date and time to start log collection. For example: `end_datetime = datetime.datetime.now().isoformat()`.

Get Pipeline Sidekick Logs via the Wallaroo MLOps API Returns

/v1/api/pipelines/get_sidekick_pod_logs returns a List of logs, each row a line of the log output starting with the ISO datetime.

Get Pipeline Sidekick Logs via the Wallaroo MLOps API Example

The following examples demonstrates retrieving the sidekick pod logs from a sample sidekick model.

# Normal case

headers = wl.auth.auth_header()
data = { "sidekick_name": sample_sidekick_name,   # should match sidekicks in pipeline.status
        "workspace_name": workspace_name,
        "pipeline_name": pipeline_name,
         "limit": 1000
       }
response = requests.post(url, headers=headers, json=data)
response.json()

{'logs': ['2026-03-31T20:26:59.883566102Z stdout F \x1b[2m2026-03-31T20:26:59.883463Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv\x1b[0m\x1b[2m:\x1b[0m Starting model deployment',
  '2026-03-31T20:26:59.883611137Z stdout F \x1b[2m2026-03-31T20:26:59.883525Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv\x1b[0m\x1b[2m:\x1b[0m \x1b[3mconfig\x1b[0m\x1b[2m=\x1b[0mConfig { minio_url: "http://minio.wallaroo.svc.cluster.local:9000", minio_base: "/model-bucket", minio_user: "minio", minio_pass: "dSE1NUWyhqxQmUhjFUrLVD8FGRDdLFqu", model_sha: "167b50597f55e1fc774aefb7622d75f03ef542c6479971b8eb2de6210ab01813", venv_path: "/venvs/167b50597f55e1fc774aefb7622d75f03ef542c6479971b8eb2de6210ab01813", nats_host: "nats.wallaroo.svc.cluster.local", pipeline_config_subject: "stream.3.depman.deployment.1.deploy", install_qaic_vllm: false, install_sglang_rocm: false, install_vllm: false, skip_downloads: false, prestaged_venv_tar: None, prestaged_model: None, prestaged_qpcs: None }',
  '2026-03-31T20:26:59.883623988Z stdout F \x1b[2m2026-03-31T20:26:59.883540Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv\x1b[0m\x1b[2m:\x1b[0m Fetching pipeline deployment event from NATS',
  '2026-03-31T20:26:59.883627443Z stdout F \x1b[2m2026-03-31T20:26:59.883566Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv::nats_client\x1b[0m\x1b[2m:\x1b[0m Fetching pipeline deploy event \x1b[3msubject\x1b[0m\x1b[2m=\x1b[0mstream.3.depman.deployment.1.deploy \x1b[3mskip_downloads\x1b[0m\x1b[2m=\x1b[0mfalse',
  '2026-03-31T20:26:59.901215303Z stdout F \x1b[2m2026-03-31T20:26:59.900965Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2masync_nats\x1b[0m\x1b[2m:\x1b[0m event: connected',
  '2026-03-31T20:26:59.904556108Z stdout F \x1b[2m2026-03-31T20:26:59.904379Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv\x1b[0m\x1b[2m:\x1b[0m Extracted GPU count \x1b[3mgpu_count\x1b[0m\x1b[2m=\x1b[0m0',
  '2026-03-31T20:26:59.904581085Z stdout F \x1b[2m2026-03-31T20:26:59.904401Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv\x1b[0m\x1b[2m:\x1b[0m Model configuration extracted \x1b[3mmodel_name\x1b[0m\x1b[2m=\x1b[0mpassthrough_2.zip \x1b[3mframework\x1b[0m\x1b[2m=\x1b[0mPython \x1b[3mopenai_enabled\x1b[0m\x1b[2m=\x1b[0mfalse \x1b[3mruntime\x1b[0m\x1b[2m=\x1b[0mFlight',
  '2026-03-31T20:26:59.904583328Z stdout F \x1b[2m2026-03-31T20:26:59.904431Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2mstorage::s3compat\x1b[0m\x1b[2m:\x1b[0m Memory limit set to 1073741824',
  '2026-03-31T20:26:59.904585701Z stdout F \x1b[2m2026-03-31T20:26:59.904443Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv\x1b[0m\x1b[2m:\x1b[0m Downloading virtual environment',
  '2026-03-31T20:26:59.904587371Z stdout F \x1b[2m2026-03-31T20:26:59.904448Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv::storage_client\x1b[0m\x1b[2m:\x1b[0m Downloading and extracting tar from S3 \x1b[3ms3_path\x1b[0m\x1b[2m=\x1b[0m167b50597f55e1fc774aefb7622d75f03ef542c6479971b8eb2de6210ab01813.venv.tar \x1b[3mextract_to\x1b[0m\x1b[2m=\x1b[0m/',
  '2026-03-31T20:26:59.904600045Z stdout F \x1b[2m2026-03-31T20:26:59.904515Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv::storage_client\x1b[0m\x1b[2m:\x1b[0m Streaming download to temp file \x1b[3mtemp_path\x1b[0m\x1b[2m=\x1b[0m/tmp/useenv-29e0249b-b31a-4815-84a2-788b2aef922a.tar',
  '2026-03-31T20:27:38.44930103Z stdout F \x1b[2m2026-03-31T20:27:38.449061Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv\x1b[0m\x1b[2m:\x1b[0m Merging model configurations',
  '2026-03-31T20:27:38.449343329Z stdout F \x1b[2m2026-03-31T20:27:38.449090Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv::model\x1b[0m\x1b[2m:\x1b[0m Merging model configurations \x1b[3mmodel_json\x1b[0m\x1b[2m=\x1b[0m/venvs/167b50597f55e1fc774aefb7622d75f03ef542c6479971b8eb2de6210ab01813/model.json',
  '2026-03-31T20:27:38.449647158Z stdout F \x1b[2m2026-03-31T20:27:38.449489Z\x1b[0m \x1b[32m INFO\x1b[0m \x1b[2museenv\x1b[0m\x1b[2m:\x1b[0m Merged model configuration:',
  '2026-03-31T20:27:38.449676677Z stdout F {',
  '2026-03-31T20:27:38.449683366Z stdout F   "data": {',
  '2026-03-31T20:27:38.449689757Z stdout F     "id": "827c7906-4579-424e-850b-1a4deb323758",',
  '2026-03-31T20:27:38.449692432Z stdout F     "model": {',
  '2026-03-31T20:27:38.449695069Z stdout F       "config": {',
  '2026-03-31T20:27:38.449698274Z stdout F         "batch_config": null,',
  '2026-03-31T20:27:38.449702038Z stdout F         "continuous_batching_config": null,',
  '2026-03-31T20:27:38.449704719Z stdout F         "dynamic_batching_config": null,',
  '2026-03-31T20:27:38.449707161Z stdout F         "filter_threshold": null,',
  '2026-03-31T20:27:38.449709322Z stdout F         "id": 10,',
  '2026-03-31T20:27:38.449713266Z stdout F         "input_schema": "/////6gAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAEAAAAEAAAA0P///wAAAQwUAAAAIAAAAAQAAAABAAAAKAAAAAcAAABvdXRwdXRzAAQABAAEAAAAEAAUAAgABgAHAAwAAAAQABAAAAAAAAEDEAAAABwAAAAEAAAAAAAAAAQAAABpdGVtAAAGAAgABgAGAAAAAAABAAAAAAA=",',
  '2026-03-31T20:27:38.449725298Z stdout F         "model_version_id": 7,',
  '2026-03-31T20:27:38.449727187Z stdout F         "openai": null,',
  '2026-03-31T20:27:38.449728821Z stdout F         "output_schema": "/////6gAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAEAAAAEAAAA0P///wAAAQwUAAAAIAAAAAQAAAABAAAAKAAAAAYAAABvdXRwdXQAAAQABAAEAAAAEAAUAAgABgAHAAwAAAAQABAAAAAAAAEDEAAAABwAAAAEAAAAAAAAAAQAAABpdGVtAAAGAAgABgAGAAAAAAABAAAAAAA=",',
  ...,
  '2026-04-01T16:21:50.52001593Z stdout F          `grpc://0.0.0.0:8080`...                                               ']}

Get Sidekick Logs History

Sidekick logs for previous model versions deployed as pipeline steps are retrieved with the Wallaroo SDK via the pipeline.get_sidekick_pod_logs(sidekick_name, start_datetime, end_datetime, limit). The Sidekick Logs History command retrieves a list of previous models used as pipeline steps and the sidekick_name for each model.

Get Sidekick Logs History via the Wallaroo SDK

The sidekick pod logs history is retrieved via the Wallaroo SDK using the Pipeline.pipeline.get_sidekick_pods_history. This retrieves a list of current and previous models used in the pipeline’s Wallaroo Containerized Runtime, thier associated models names, and other useful details.

Get Sidekick Logs History via the Wallaroo SDK Parameters

The method get_sidekick_pods_history accepts the following parameters.

Parameter	Type	Description
`model_name`	String (Optional)	Optional parameter to filter pipeline steps by model name.

Get Sidekick Logs History via the Wallaroo SDK Returns

The method get_sidekick_pods_history returns a List of historical sidekick pods used in the pipelines, excluding current pods. These results are sorted in reverse chronological order by the field started_at.

Each List item contains the following elements.

Fields	Type	Description
`sidekick_name`	String	The name of the sidekick pod associated with the pipeline for this model version.
`started_at`	DateTime	The date and time the model version was deployed as a pipeline model step.
`pipeline_model_step`	String	The model name as added as a pipeline step.
`model_sha`	String	The SHA hash of the model version file.
`model_version`	The UUID identifier for the model version.

Get Sidekick Logs History via the Wallaroo SDK Example

The following demonstrates retrieving the model history for a pipeline.

pipeline.get_sidekick_pods_history()

sidekick_name	started_at	pipeline_model_step	model_sha	model_version
engine-sidekick-noop-py-pre-b-5-0	2026-03-31T20:30:51.797Z	noop-py-pre-b	9b634f014101dc88c2a31ba9018dc5d0f364f5e44f4d4923f49078c9bd56cbed	1d9f841a-5a2e-496c-98e3-603b57b050fc
engine-sidekick-noop-py-post-7-0	2026-03-31T20:27:51.798Z	noop-py-post	167b50597f55e1fc774aefb7622d75f03ef542c6479971b8eb2de6210ab01813	4f3b5c97-566c-4ae0-8f7f-aa3697c5d530

Get Sidekick Logs History via the Wallaroo MLOps API

Sidekick logs for previous model versions deployed as pipeline steps are retrieved with the Wallaroo MLOps API via the following endpoint:

Endpoint: /v1/api/pipelines/get_sidekick_pods_history (POST)
Content-Type: application/json

Get Sidekick Logs History via the Wallaroo MLOps API Parameters

The endpoint /v1/api/pipelines/get_sidekick_pods_history accepts the following parameters

Parameter	Type	Description
`workspace_name`	String (Required)	The name of the workspace containing the pipeline.
`pipeline_name`	String (Required)	The pipeline the model is deployed through.
`model_name`	String (Optional)	The model name used when assigned as a pipeline step. This is used to filter for model versions that match the `model_name` field.

Get Sidekick Logs History via the Wallaroo MLOps API Returns

The endpoint returns a List of pods with each having the following fields.

Fields	Type	Description
`sidekick_name`	String	The name of the sidekick pod associated with the pipeline for this model version.
`started_at`	DateTime	The date and time the model version was deployed as a pipeline model step.
`pipeline_model_step`	String	The model name as added as a pipeline step.
`model_sha`	String	The SHA hash of the model version file.
`model_version`	The UUID identifier for the model version.

Get Sidekick Logs History via the Wallaroo MLOps API Example

The following example demonstrates retrieving the sidekick pod logs history from a sample pipeline.

headers = wl.auth.auth_header()
data = { "pipeline_name": "double-noop-x",  
         "workspace_id": workspace_id}

response = requests.post(f"{wl.api_endpoint}/v1/api/pipelines/get_sidekick_pods_history", headers=headers, json=data)
response.json()

{'pods': [{'sidekick_name': 'engine-sidekick-noop-py-pre-b-5-0',
   'started_at': '2026-03-31T20:30:51.797Z',
   'pipeline_model_step': 'noop-py-pre-b',
   'model_sha': '9b634f014101dc88c2a31ba9018dc5d0f364f5e44f4d4923f49078c9bd56cbed',
   'model_version': '1d9f841a-5a2e-496c-98e3-603b57b050fc'},
  {'sidekick_name': 'engine-sidekick-noop-py-post-7-0',
   'started_at': '2026-03-31T20:27:51.798Z',
   'pipeline_model_step': 'noop-py-post',
   'model_sha': '167b50597f55e1fc774aefb7622d75f03ef542c6479971b8eb2de6210ab01813',
   'model_version': '4f3b5c97-566c-4ae0-8f7f-aa3697c5d530'}]}

Error Messages

The following error messages are displayed based on various input errors.

"Sidekick '{sidekick name}' not found in pipeline": The requested sidekick name does not exist in the requested pipeline.
No logs: No logs exist for the requested sidekick or were not found the for requested start and end time.
Invalid datetime format: Valid formats are RFC3339, ISO8601, or date-only (YYYY-MM-DD).
MLOps API Only Workspace name not found: XXXXX: The request with the specified workspace does not match any existing workspace.
MLOps API Only Pipeline not found: YYYYYYYYY: The request with the specified workspace does not match any existing workspace.