Retrieve Inference Metrics and Logs Via the Wallaroo Dashboard

Inference logs from Wallaroo deployments are available through the Wallaroo Dashboard through the Pipeline Metrics page.

How to Access the Pipeline Metrics Page

To access the metrics page for a specific pipeline:

1. Login to the Wallaroo Dashboard.
2. Select the workspace the pipeline is associated with.
3. Select View Pipelines.
4. From the Workspace Pipeline List page, select the pipeline.
5. From the Pipeline Details page, select Metrics.

Pipeline Metrics Overview

The Pipeline Metrics page contains the following elements.

• Pipeline Name and Identifier (A): The pipeline’s assigned name and unique identifier in UUID format.

• Filter Edges (B): By default, all locations are displayed. Filter Edges provides a list of available edge deployments are displayed. Selecting one or more filters from the list limits the available metrics and logs displayed to only the selected locations.

  

• Status (C): The status of the pipeline. The status only applies to the pipeline’s status in the Wallaroo Ops instance.. Options are:

  • Active: The pipeline is deployed.
  • Inactive: The pipeline is not deployed.

• Tags (D): Any tags applied to the pipeline.

• Inference Urls (E): The internal and external inference URLs for the deployed pipeline in the Wallaroo Ops instance.

• Date Filter (F): Filter date and time to specify the period of time for inference requests to collect for the metrics.

• Deploy/Undeploy the Pipeline (G): Deploy an inactive pipeline, or deploy an active pipeline in the Wallaroo Ops instance.

• Requests per Second (H): The number of inference requests per second in the filtered date period. This chart data can be downloaded and shared with other users.

• Cluster inference rate (I): The rate of inference requests completed in the filtered date period. This chart data can be downloaded and shared with other users.

• Inference Latency (J): The latency between when an inference request is received versus when it is completed in the filtered date period. This chart data can be downloaded and shared with other users.

• Engine Replicas (K): Tracks the number of replicas deployed for the Wallaroo Native Runtime. In the displayed example, the replica count started at 0, then went to 3 replicas when the pipeline was deployed. For details on deployment configurations, see Deployment Configuration with the Wallaroo SDK.

• EngineAux Replicas (L): Tracks the number of replicas deployed for the Wallaroo Containerized Runtime. In the displayed example, the replica count started at 0, then went to 3 replicas when the pipeline was deployed. For details on deployment configurations, see Deployment Configuration with the Wallaroo SDK.

• Activity (M): Comments left by users.

• Audit Log (N): The inference audit logs. The logs are filtered by the Filter Edges settings and Date Filter.

  • If an inference result output is greater than 100k in for any field, the field results show NULL in the Wallaroo Dashboard:

    

    Full inference results are always returned with either the Wallaroo SDK or the MLOps API. Large inference results are filtered only in the display.

• Anomaly Log (O): The anomaly audit logs. These are included when anomaly detection validation rules are triggered. The logs are filtered by the Filter Edges settings and Date Filter.

How to Filter by Edge Locations

By default, all location inference data is used for the Pipeline Metrics page.

To filter by Edge Locations:

1. Select Filter Edges.
2. Select the edges to filter inference results from.
3. Select Filter when to set the filters.
4. To clear the filters, select Clear filter next to the Filter Edges button.

How to Share Metric Charts and Data

To save metric data from the Requests per Second, Cluster inference rate and Inference Latency charts:

• Select the … menu option for the specific chart.
• Select one of the following:
  • Download CSV: Downloads a CSV file with the Date and Time stamp and data for the specified chart.
  • Copy shareable URL: Provides a publicly available URL for users to view the chart.
  • View Enlarged: Displays a larger version of the chart.

Retrieve Inference Request Logs Via the Wallaroo SDK

Pipeline have their own set of log files that are retrieved and analyzed as needed with the either through:

• The Pipeline logs method (returns either a DataFrame or Apache Arrow).
• The Pipeline export_logs method (saves either a DataFrame file in JSON format, or an Apache Arrow file).

Get Pipeline Logs

Pipeline logs are retrieved through the Pipeline logs method. By default, logs are returned as a DataFrame in reverse chronological order of insertion, with the most recent files displayed first.

Pipeline logs are segmented by pipeline versions. For example, if a new model step is added to a pipeline, a model swapped out of a pipeline step, etc - this generated a new pipeline version. log method requests will return logs based on the parameter that match the pipeline version. To request logs of a specific pipeline version, specify the start_datetime and end_datetime parameters based on the pipeline version logs requested.

This command takes the following parameters.

All of the parameters can be used together, but start_datetime and end_datetime must be combined; if one is used, then so must the other. If start_datetime and end_datetime are used with any other parameter, then the log results are in chronological order of record insertion.

Log requests are limited to around 100k in size. For requests greater than 100k in size, use the Pipeline export_logs() method.

Logs include the following standard datasets:

In this example, the last 50 logs to the pipeline mainpipeline between two sample dates. In this case, all of the time column fields are the same since the inference request was sent as a batch.

logs = mainpipeline.logs(start_datetime=date_start, end_datetime=date_end)

display(len(logs))
display(logs)

538

538 rows × 4 columns

Metadata Requests Restrictions

The following restrictions are in place when requesting the following datasets:

• metadata
• metadata.elasped
• metadata.last_model
• metadata.pipeline_version

Standard Pipeline Steps Log Requests

Effected pipeline steps:

• add_model_step
• replace_with_model_step

For log file requests, the following metadata dataset requests for standard pipeline steps are available:

• metadata

These must be paired with specific columns. * is not available when paired with metadata.

• in: All input fields.
• out: All output fields.
• time: The DateTime the inference request was made.
• in.{input_fields}: Any input fields (tensor, etc.)
• out.{output_fields}: Any output fields (out.house_price, out.variable, etc.)
• anomaly.count: Any anomalies detected from validations.
• anomaly.{validation}: The validation that triggered the anomaly detection and whether it is True (indicating an anomaly was detected) or False. For more details, see Wallaroo SDK Essentials Guide: Anomaly Detection

For example, the following requests the metadata plus any output fields.

metadatalogs = mainpipeline.logs(dataset=["out","metadata"])
display(metadatalogs.loc[:, ['out.variable', 'metadata.last_model']])

Shadow Deploy Testing Pipeline Steps

Effected pipeline steps:

• add_shadow_deploy
• replace_with_shadow_deploy

For log file requests, the following metadata dataset requests for shadow deploy testing pipeline steps are available:

• metadata

These must be paired with specific columns. * is not available when paired with metadata. time must be included if dataset is used.

• in: All input fields.
• out: All output fields.
• time: The DateTime the inference request was made.
• in.{input_fields}: Any input fields (tensor, etc.).
• out.{output_fields}: Any output fields matching the specific output_field (out.house_price, out.variable, etc.).
• out_: All shadow deployed challenger steps Any output fields matching the specific output_field (out.house_price, out.variable, etc.).
• anomaly.count: Any anomalies detected from validations.
• anomaly.{validation}: The validation that triggered the anomaly detection and whether it is True (indicating an anomaly was detected) or False. For more details, see Wallaroo SDK Essentials Guide: Anomaly Detection

The following example retrieves the logs from a pipeline with shadow deployed models, and displays the specific shadow deployed model outputs and the metadata.elasped field.

# Display metadata

metadatalogs = mainpipeline.logs(dataset=["out_logcontrolchallenger01.variable", 
                                          "out_logcontrolchallenger02.variable", 
                                          "metadata"
                                          ]
                                )

display(metadatalogs.loc[:, ['out_logcontrolchallenger01.variable',	
                             'out_logcontrolchallenger02.variable', 
                             'metadata.elapsed'
                             ]
                        ])

A/B Deploy Testing Pipeline Steps

Effected pipeline steps:

• add_random_split
• replace_with_random_split

For log file requests, the following metadata dataset requests for A/B testing pipeline steps are available:

• metadata

These must be paired with specific columns. * is not available when paired with metadata. time must be included if dataset is used.

• in: All input fields.
• out: All output fields.
• time: The DateTime the inference request was made. Must be requested in all dataset requests.
• in.{input_fields}: Any input fields (tensor, etc.).
• out.{output_fields}: Any output fields matching the specific output_field (out.house_price, out.variable, etc.).
• anomaly.count: Any anomalies detected from validations.
• anomaly.{validation}: The validation that triggered the anomaly detection and whether it is True (indicating an anomaly was detected) or False. For more details, see Wallaroo SDK Essentials Guide: Anomaly Detection

The following example retrieves the logs from a pipeline with A/B deployed models, and displays the output and the specific metadata.last_model field.

metadatalogs = mainpipeline.logs(dataset=["time",
                                          "out", 
                                          "metadata"
                                          ]
                                )

display(metadatalogs.loc[:, ['out.variable', 'metadata.last_model']])

Export Pipeline Logs as File

The Pipeline method export_logs returns the Pipeline records as either by default pandas records in Newline Delimited JSON (NDJSON) format, or an Apache Arrow table files.

The output files are by default stores in the current working directory ./logs with the default prefix as the {pipeline name}-1, {pipeline name}-2, etc.

The suffix by default will be json for pandas records in Newline Delimited JSON (NDJSON) format files. Logs are segmented by pipeline version across the limit, data_size_limit, or start_datetime and end_datetime parameters.

By default, logs are returned as a pandas record in NDJSON in reverse chronological order of insertion, with the most recent log insertions displayed first.

Pipeline logs are segmented by pipeline versions. For example, if a new model step is added to a pipeline, a model swapped out of a pipeline step, etc - this generated a new pipeline version.

This command takes the following parameters.

All of the parameters can be used together, but start_datetime and end_datetime must be combined; if one is used, then so must the other. If start_datetime and end_datetime are used with any other parameter, then the log results are in chronological order of record insertion.

File sizes are limited to around 10 MB in size. If the requested log file is greater than 10 MB, a Warning will be displayed indicating the end date of the log file downloaded so the request can be adjusted to capture the requested log files.

data_df = pd.read_json("mainpipeline_logs.df.json", lines=True)
display(data_df)

In this example, the log files are saved as both Pandas DataFrame and Apache Arrow.

# Save the DataFrame version of the log file

mainpipeline.export_logs()
display(os.listdir('./logs'))

mainpipeline.export_logs(arrow=True)
display(os.listdir('./logs'))

    Warning: There are more logs available. Please set a larger limit to export more data.
    

    ['pipeline-logs-1.json']

    Warning: There are more logs available. Please set a larger limit to export more data.
    

    ['pipeline-logs-1.arrow', 'pipeline-logs-1.json']

Pipeline Log Storage

Pipeline logs have a set allocation of storage space and data requirements.

Pipeline Log Storage Warnings

To prevent storage and performance issues, inference result data may be dropped from pipeline logs by the following standards:

• Columns are progressively removed from the row starting with the largest input data size and working to the smallest, then the same for outputs.

For example, Computer Vision ML Models typically have large inputs and output values - a single pandas DataFrame inference request may be over 13 MB in size, and the inference results nearly as large. To prevent pipeline log storage issues, the input may be dropped from the pipeline logs, and if additional space is needed, the inference outputs would follow. The time column is preserved.

If a pipeline has dropped columns for space purposes, this will be displayed when a log request is made with the following warning, with {columns} replaced with the dropped columns.

The inference log is above the allowable limit and the following columns may have been suppressed for various rows in the logs: {columns}. To review the dropped columns for an individual inference’s suppressed data, include dataset=["metadata"] in the log request.

Review Dropped Columns

To review what columns are dropped from pipeline logs for storage reasons, include the dataset metadata in the request to view the column metadata.dropped. This metadata field displays a List of any columns dropped from the pipeline logs.

For example:

metadatalogs = mainpipeline.logs(dataset=["time", "metadata"])

Suppressed Data Elements

Data elements that do not fit the supported data types below, such as None or Null values, are not supported in pipeline logs. When present, undefined data will be written in the place of the null value, typically zeroes. Any null list values will present an empty list.

Retrieve Inference Request Logs Via the Wallaroo MLOps API

Inference Logs Endpoints

Pipeline logs are retrieved through the Wallaroo MLOps API with the following request.

• REQUEST URL
  • v1/api/pipelines/get_logs
• Headers
  • Accept:
    • application/json; format=pandas-records: For the logs returned as pandas DataFrame
    • application/vnd.apache.arrow.file: for the logs returned as Apache Arrow
• PARAMETERS
  • pipeline_id (String Required): The name of the pipeline.
  • workspace_id (Integer Required): The numerical identifier of the workspace.
  • cursor (String Optional): Cursor returned with a previous page of results from a pipeline log request, used to retrieve the next page of information.
  • order (String Optional Default: Desc): The order for log inserts returned. Valid values are:
    • Asc: In chronological order of inserts.
    • Desc: In reverse chronological order of inserts.
  • page_size (Integer Optional Default: 1000.): Max records per page.
  • start_time (String Optional): The start time of the period to retrieve logs for in RFC 3339 format for DateTime. Must be combined with end_time.
  • end_time (String Optional): The end time of the period to retrieve logs for in RFC 3339 format for DateTime. Must be combined with start_time.
• RETURNS
  • The logs are returned by default as 'application/json; format=pandas-records' format. To request the logs as Apache Arrow tables, set the submission header Accept to application/vnd.apache.arrow.file.
  • Headers:
    • x-iteration-cursor: Used to retrieve the next page of results. This is not included if x-iteration-status is All.
    • x-iteration-status: Informs whether there are more records available outside of this log request parameters.
      • All: This page includes all logs available from this request. If x-iteration-status is All, then x-iteration-cursor is not provided.
      • SchemaChange: A change in the log schema caused by actions such as pipeline version, etc.
      • RecordLimited: The number of records exceeded from the page size, more records can be requested as the next page. There may be more records available to retrieve OR the record limit was reached for this request even if no more records are available in next cursor request.
      • ByteLimited: The number of records exceeded the pipeline log limit which is around 100K.

Inference Logs Example: Standard Model Deployment

The following demonstrates retrieving inference logs from a standard model deployment.

# retrieve the authorization token
headers = wl.auth.auth_header()

url = f"{APIURL}/v1/api/pipelines/get_logs"

# Standard log retrieval

data = {
    'pipeline_id': main_pipeline_name,
    'workspace_id': workspace_id
}

response = requests.post(url, headers=headers, json=data)
standard_logs = pd.DataFrame.from_records(response.json())

display(standard_logs.head(5).loc[:, ["time", "in", "out"]])

Inference Logs Example: Shadow Deploy Deployment

The following demonstrates retrieving inference logs from a model deployment with shadow deployed steps.

# Retrieve logs from specific date/time to only get the two DataFrame input inferences in ascending format

# retrieve the authorization token
headers = wl.auth.auth_header()

url = f"{APIURL}/v1/api/pipelines/get_logs"

# Standard log retrieval

data = {
    'pipeline_id': main_pipeline_name,
    'workspace_id': workspace_id,
    'order': 'Asc',
    'start_time': f'{shadow_date_start.isoformat()}',
    'end_time': f'{shadow_date_end.isoformat()}'
}

response = requests.post(url, headers=headers, json=data)
standard_logs = pd.DataFrame.from_records(response.json())

display(standard_logs.head(5).loc[:, ["time", "out", "out_logcontrolchallenger01", "out_logcontrolchallenger02"]])

Inference Logs Example: A/B Test Deployment

The following demonstrates retrieving inference logs from a model deployment with A/B Testing deployed steps.

# Retrieve logs from specific date/time to only get the two DataFrame input inferences in ascending format

# retrieve the authorization token
headers = wl.auth.auth_header()

url = f"{APIURL}/v1/api/pipelines/get_logs"

# Standard log retrieval

data = {
    'pipeline_id': main_pipeline_name,
    'workspace_id': workspace_id,
    'order': 'Asc',
    'start_time': f'{ab_date_start.isoformat()}',
    'end_time': f'{ab_date_end.isoformat()}'
}

response = requests.post(url, headers=headers, json=data)
standard_logs = pd.DataFrame.from_records(response.json())

display(standard_logs.head(5).loc[:, ["time", "out"]])

Pipeline Log Storage

Pipeline logs have a set allocation of storage space and data requirements.

Pipeline Log Storage Warnings

To prevent storage and performance issues, inference result data may be dropped from pipeline logs by the following standards:

• Columns are progressively removed from the row starting with the largest input data size and working to the smallest, then the same for outputs.

For example, Computer Vision ML Models typically have large inputs and output values - a single pandas DataFrame inference request may be over 13 MB in size, and the inference results nearly as large. To prevent pipeline log storage issues, the input may be dropped from the pipeline logs, and if additional space is needed, the inference outputs would follow. The time column is preserved.

If a pipeline has dropped columns for space purposes, this will be displayed when a log request is made with the following warning, with {columns} replaced with the dropped columns.

The inference log is above the allowable limit and the following columns may have been suppressed for various rows in the logs: {columns}. To review the dropped columns for an individual inference’s suppressed data, include dataset=["metadata"] in the log request.

Review Dropped Columns

To review what columns are dropped from pipeline logs for storage reasons, include the dataset metadata in the request to view the column metadata.dropped. This metadata field displays a List of any columns dropped from the pipeline logs.

For example:

# retrieve the authorization token
headers = wl.auth.auth_header()

url = f"{APIURL}/v1/api/pipelines/get_logs"

# Standard log retrieval

data = {
    'pipeline_id': main_pipeline_name,
    'workspace_id': workspace_id
}

response = requests.post(url, headers=headers, json=data)
standard_logs = pd.DataFrame.from_records(response.json())

display(len(standard_logs))
display(standard_logs.head(5).loc[:, ["time", "metadata"]])
cursor = response.headers['x-iteration-cursor']

Suppressed Data Elements

Data elements that do not fit the supported data types below, such as None or Null values, are not supported in pipeline logs. When present, undefined data will be written in the place of the null value, typically zeroes. Any null list values will present an empty list.