Inference requests sent to Wallaroo are submitted under the inference data schema definitions. The following details those requirements for:

• Submitting single inference requests versus batch inference requests
• Submitting inference requests with multidimensional array data.

Inferencing Data Types

Wallaroo pipelines accept either Apache Arrow tables natively or pandas DataFrames.

• If using the Wallaroo SDK with wallaroo.pipeline.infer or wallaroo.pipeline.infer_from_file, it is a “What you give is what you get” situation: If an Apache Arrow table is submitted for an inference request with the , then an Apache Arrow table is returned. If a pandas DataFrame is submitted, then a pandas DataFrame is returned.
• If using the Wallaroo pipeline’s inference URL as an API request, the API headers Content-Type and Accept determine what data type is submitted and what data type is received:
  • For DataFrame formatted JSON: application/json; format=pandas-records
  • For Arrow binary files: application/vnd.apache.arrow.file

Data Types

Data types for inputs and outputs to inference requests through the Wallaroo is based on the Apache Arrow Data Types, with the following exceptions:

• null: Not allowed. All fields must have submitted values that match their data type. For example, if the schema expects a float value, then some value of type float such as 0.0 must be submitted and cannot be None or Null. If a schema expects a string value, then some value of type string must be submitted, etc.
• time32 and time64: Datetime data types must be converted to string.

Data Constraints

Data submitted to Wallaroo for inference requests have the following data constraints.

• Equal rows constraint: The number of input rows and output rows must match.
• Data Type Consistency: Data types within each tensor are of the same type.

Equal Rows

Each input row for an inference is related directly to the inference row output.

For example, the INPUT and OUTPUT rows match, with each input row directly corresponding to an output row.

Equal Rows Input Example

Equal Rows Output Example

Data Type Consistency

Each element must have the same internal data type. For example, the following is valid because all of the data types within each element are float32.

t = [
    [2.35, 5.75],
    [3.72, 8.55],
    [5.55, 97.2]
]

The following is invalid, as it mixes floats and strings in each element:

t = [
    [2.35, "Bob"],
    [3.72, "Nancy"],
    [5.55, "Wani"]
]

The following inputs are valid, as each data type is consistent within the elements.

df = pd.DataFrame({
    "t": [
        [2.35, 5.75, 19.2],
        [5.55, 7.2, 15.7],
    ],
    "s": [
        ["Bob", "Nancy", "Wani"],
        ["Jason", "Rita", "Phoebe"]
    ]
})
df

Data Requirements for ONNX Models

ONNX models used for inference requests in Wallaroo have an additional requirement.

• The shape of each element within a field is the same.

For ONNX models, all inputs must be tensors. This requires that the shape of each element is the same. For example, the following is a proper input:

t =  [
    [2.35, 5.75],
    [3.72, 8.55],
    [5.55, 97.2]
]

Another example is a 2,2,3 tensor, where the shape of each element is (3,), and each element has 2 rows, and the outer row is bound by 2 rows.

t = [
        [2.35, 5.75, 19.2],
        [3.72, 8.55, 10.5]
    ],
    [
        [5.55, 7.2, 15.7],
        [9.6, 8.2, 2.3]
    ]

In this example each element has a shape of (2,). Tensors with elements of different shapes, known as ragged tensors, are not supported for batch inputs. For example:

t = [
    [2.35, 5.75],
    [3.72, 8.55, 10.5],
    [5.55, 97.2]
])

**INVALID SHAPE**

For this situation, the rows must be broken up to avoid the ragged tensors, or submitted as single row inputs. For example:

t1 = [[2.35, 5.75]]
t2 = [[3.72, 8.55, 10.5]]
t3 = [[5.55, 97.2]]

Batch vs Single Row Data Requirements for ONNX models

Data submitted to ONNX models deployed in Wallaroo is submitted as either batch data or single row input data based on the model configuration, typically at model upload.

ONNX models with the following data input conditions must be set as single row inputs:

• The inputs do not have an outer dimension.
• The output has an outer dimension of 1.

The simple rule of thumb is that for batch inputs, the data meets the following requirements.

• Equal rows constraint: The number of input rows and output rows must match.
• Data Type Consistency: Data types within each tensor are of the same type.
• Data Shape Consistency: The shape of each element within a field is the same.

For single row inputs, the data meets the following requirements.

• Equal rows constraint: The number of input rows and output rows must match.
• Data Type Consistency: Data types within each tensor are of the same type.

For single row inputs, the following image does not have an outer dimension, and depending on the size of the image, the shape of the data changes.

For example, the following images each have data type consistency when converted from an image to tensors. But the shape of each image differs from their size.

# a 640x480 image converted to tensors
df_image, resizedImage = utils.loadImageAndConvertToDataframe('./data/images/input/example/dairy_bottles.png', width, height)
# flatten the multi-dimensional array for inferencing through a Wallaroo deployed model
df_for_inference = pd.DataFrame(wallaroo.utils.flatten_np_array_columns(df_image, 'tensor'))
# get the shape - from a 640x480 image with 3 color values per pixel this is 921600 elements
print(df_for_inference['tensor'][0].shape)
(921600,)

# a 1920x1080 image converted to tensors
df_image, resizedImage = utils.loadImageAndConvertToDataframe('./data/images/input/example/crowd_of_people.png', width, height)
# flatten the multi-dimensional array for inferencing through a Wallaroo deployed model
df_for_inference = pd.DataFrame(wallaroo.utils.flatten_np_array_columns(df_image, 'tensor'))
# get the shape - from a 1920x1080 image with 3 color values per pixel this is 6220800 elements
df_for_inference['tensor'][0].shape
(6220800,)

display(df_for_inference)

For each input, the shape of the input field tensor depends on the size of the image, and there is no outer dimension. In these instances where single batch inputs are required because of the data schema, the ONNX model’s configuration must be set to batch_config=single.

mobilenet_model = wl.upload_model("mobilenet-demo", 
                './models/mobilenet.pt.onnx', 
                framework=Framework.ONNX).configure(batch_config="single")

Each inference request submitted must be a single row pandas DataFrame or Apache Arrow table when the model is configured as batch_config="single".

Inference Requests Examples

The following examples show different inference requests with data sets.

Inference Request Batch Inputs

For standard batch inputs, each column and row corresponds to one data input. For example, a set of house data fed into a house price model has the following inputs.

As the data types meet all three data requirements, the model is set with the configuration batch_config=None, allowing for batch inference submissions. The following represents a inference results from the data above for a house price model, with each output row corresponding to the input row.

Inference Request Batch Inputs for a Range of Data

For batch inputs, each inference request is contained in a single row of data. For example, a bike rental model predicts rentals based on previous dates for bikes sold, whether it was a week day, etc. These are submitted as a group of dates that are then used by the model to predict bike rentals on figure dates. For example, this 5 day period is submitted for the inference request.

To submit to the Wallaroo deployed pipeline as a batch, each range of dates and values is set into a single row of data. This is reflected in the input and output schemas defined for the model. The pyarrow schema pa.list_ is used to define the data being received as a List per field.

import pyarrow as pa
import wallaroo

wl = wallaroo.Client()

input_schema = pa.schema([
   pa.field('dteday', pa.list_(pa.string()) ),
   pa.field('site_id', pa.list_(pa.string()) ), 
   pa.field('cnt', pa.list_(pa.int64()) ),
   pa.field('season', pa.list_(pa.int64()) ),
   pa.field('holiday', pa.list_(pa.int64()) ),
   pa.field('weekday', pa.list_(pa.int64()) ),
   pa.field('workingday', pa.list_(pa.int64()) ),           
])

output_schema = pa.schema([
    pa.field('dteday', pa.list_(pa.string()) ),
    pa.field('site_id', pa.list_(pa.string()) ), 
    pa.field('forecast', pa.list_(pa.int64()) ),
])

arima_model = (wl.upload_model('bikeforecast-arima', 
                               './models/forecast.py', 
                               framework=Framework.PYTHON)
                               .configure(runtime="python", 
                               input_schema = input_schema,
                               output_schema = output_schema
                               )
              )

Here, a month of data for single site is prepared for inference through a Wallaroo deployed model. Each inference request contains in a single row per site. Additional sites would be in their own row, with their month’s worth of data set as lists within the cells.

display(input_frame)

The inference results are then converted to whatever format the data scientist and ML engineers require.

Using Multi-Dimensional Numpy Arrays for Inference Inputs

Wallaroo inference requests support either pandas DataFrame or Apache Arrows as inputs. When using a numpy array with multi-dimensional shapes, the Wallaroo SDK provides helper functions for data conversion to support the Wallaroo data requirements.

• Wallaroo SDK:
  • DataFrame: Numpy multi-dimensional array native support.
  • Apache Arrow: Numpy multi-dimensional array must be flattened.
• Wallaroo MLOps API:
  • DataFrame: Numpy multi-dimensional array must be flattened.
  • Apache Arrow: Numpy multi-dimensional array must be flattened.

Note that when using the Wallaroo MLOps with multi-dimensional numpy arrays, the helper functions are contained in the Wallaroo SDK to assist with data conversion.

Using Multi-Dimensional Numpy Arrays with the Wallaroo SDK

Inference requests made through the Wallaroo SDK accept Numpy arrays as input values within a pandas DataFrame. For Apache Arrow tables, the numpy array must be flattened.

The Wallaroo SDK method wallaroo.utils.flatten_np_array_columns(df, col) converts a pandas DataFrame column specified with the name col. The pandas DataFrame is then converted to an Apache Arrow table for inferencing.

Wallaroo SDK Numpy Arrays with DataFrames Example

The following example deploys a Computer Vision Mobilenet model. A multi-dimensional numpy array is generated from an image, then used it directly for inference request in a DataFrame.

import wallaroo
import torch
import cv2

wl = wallaroo.Client()

# upload the model
mobilenet_model = wl.upload_model("mobilenet-demo", 
                './models/mobilenet.pt.onnx', 
                framework=Framework.ONNX).configure(batch_config="single")

# build the pipeline with the model and deploy
pipeline = wl.build_pipeline("cv-sample-pipeline")
pipeline.add_model_step(mobilenet_model)
# deployed the pipeline
pipeline.deploy()

# convert the image to a tensor
image = cv2.imread("./data/images/input/example/dairy_bottles.png")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
tensor = torch.FloatTensor(image)

# get the numpy value and use in a dataframe
npArray = tensor.cpu().numpy()
df_with_numpy = pd.DataFrame({"tensor":[npArray]})

The following is an example of the the first row and the first 4 columns from the image converted into a multi-dimensional numpy array.

The DataFrame with the multi-dimensional numpy array is submitted to the Wallaroo pipeline in an inference request. The following isolates the first 4 objects detected and overall confidence value of all detected objects.

infResults = pipeline.infer(df_with_numpy, timeout=300)
df = pd.DataFrame([
    {"time": infResults.loc[0, ['time']][0],
    'out.avg_conf': [infResults.loc[0, ['out.avg_conf']][0]],
    'boxes': [infResults.loc[0, ['out.boxes']][0][0:4][0:4]],
    'classes': [infResults.loc[0, ['out.classes']][0][0:4]],
    'confidences': [infResults.loc[0, ['out.confidences']][0][0:4]]
    }
])
display(df)

Wallaroo SDK Numpy Arrays with Apache Arrow Example

Apache Arrow tables submitted for inference request must have multi-dimensional arrays flattened before they are submitted. The following example shows flattening the numpy array with the wallaroo.utils.flatten_np_array_columns(df, col) method, which returns the column’s values as flattened arrays.

import wallaroo
import torch
import cv2

wl = wallaroo.Client()

# upload the model
mobilenet_model = wl.upload_model("mobilenet-demo", 
                './models/mobilenet.pt.onnx', 
                framework=Framework.ONNX).configure(batch_config="single")

# build the pipeline with the model and deploy
pipeline = wl.build_pipeline("cv-sample-pipeline")
pipeline.add_model_step(mobilenet_model)
# deployed the pipeline
pipeline.deploy()

# convert the image to a tensor
image = cv2.imread("./data/images/input/example/dairy_bottles.png")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
tensor = torch.FloatTensor(image)

# get the numpy value and use in a dataframe
npArray = tensor.cpu().numpy()
df_with_numpy = pd.DataFrame({"tensor":[npArray]})

The following shows the the first 3 columns of the first 4 rows of the image file converted to tensors as a multi-dimensional numpy array in a pandas DataFrame.

Before submitting as an Apache Arrow table, the multi-dimensional arrays are flattened.

The following demonstrates using the wallaroo.utils.flatten_np_array_columns(df, col) takes a pandas DataFrame, and the column name of the arrays to flatten method. For this example, we will flatten the column and retrieve a new array from it.

wallaroo.utils.flatten_np_array_columns(df_with_numpy, 'tensor')

0    [0.9372549, 0.9529412, 0.9490196, 0.94509804, 0.94509804, 0.94509804 ...]
Name: tensor, dtype: object

We create an Apache Arrow table from the flattened numpy array and submit it for the inference request.

table = pa.table([
    pa.array(wallaroo.utils.flatten_np_array_columns(df_with_numpy, 'tensor'))
], names=["tensor"])

display(table)

pyarrow.Table
tensor: list<item: float>
  child 0, item: float
----
tensor: [[[0.9372549,0.9529412,0.9490196,0.94509804,...]]]

result = pipeline.infer(table)

display(result['out.avg_confidence'])

<pyarrow.lib.ChunkedArray object at 0x2a69b4b30>
[
  [
    [
      0.3588039
    ]
  ]
]

Using Multi-Dimensional Numpy Arrays with the Wallaroo MLOps API

The Wallaroo MLOps API supports the following Content-Type and Accept headers values:

• application/json; format=pandas-records: JSON in pandas Record format.
• application/vnd.apache.arrow.file: A binary in Apache Arrow format.

In both cases, multi-dimensional numpy array must be flattened before submitting it as part of an API Inference Request.

The following examples demonstrates using the Wallaroo SDK wallaroo.utils.flatten_np_array_columns(df, col) method to flatten a column in a pandas DataFrame that contains multi-dimensional numpy arrays.

Wallaroo MLOps API Numpy Arrays with DataFrames Example

The following is an example of a DataFrame with an image converted to a multi-dimensional numpy array of tensors.

Before submitting as an MLOps API request as a pandas DataFrame, the multi-dimensional arrays are flattened. The following demonstrates using the wallaroo.utils.flatten_np_array_columns(df, col) method. The flattened array is inserted as the tensor column for a new pandas DataFrame.

df_with_flattened_array = pd.DataFrame({'tensor': wallaroo.utils.flatten_np_array_columns(df_with_numpy, 'tensor')})

# show only the first 5 tensors
df = df_with_flattened_array.apply(lambda x: [i[0:5] for i in x])
display(df)

With the multi-dimensional arrays flattened, it is submitted as an inference request.

headers = wl.auth.auth_header()

headers['Content-Type'] = 'application/json; format=pandas-records'

deploy_url = pipeline._deployment._url()

response = requests.post(
                    deploy_url, 
                    headers=headers, 
                    data=df_with_flattened_array.to_json(orient="records")
                )

display(pd.DataFrame(response.json()).loc[:, ['time', 'out.avg_confidence']])

Wallaroo MLOps API Numpy Arrays with Apache Arrow Example

The following is an example of a DataFrame with an image converted to a multi-dimensional numpy array of tensors.

Before submitting as an MLOps API request as an Apache Arrow table, the multi-dimensional arrays are flattened. The following demonstrates using the wallaroo.utils.flatten_np_array_columns(df, col) method.

wallaroo.utils.flatten_np_array_columns(df_with_numpy, 'tensor')

0    [0.9372549, 0.9529412, 0.9490196, 0.94509804, 0.94509804, 0.94509804 ...]
Name: tensor, dtype: object

With the multi-dimensional arrays flattened, it is inserted into an Apache Arrow table and submitted in an inference request.

table = pa.table([
    pa.array(wallaroo.utils.flatten_np_array_columns(df_with_numpy, 'tensor'))
], names=["tensor"])

display(table)

pyarrow.Table
tensor: list<item: float>
  child 0, item: float
----
tensor: [[[0.9372549,0.9529412,0.9490196,0.94509804,...]]]

headers = wl.auth.auth_header()

# set Content-Type type
headers['Content-Type']='application/vnd.apache.arrow.file'

# set accept as apache arrow table
headers['Accept']="application/vnd.apache.arrow.file"

deploy_url = pipeline._deployment._url()

response = requests.post(
                    deploy_url, 
                    headers=headers, 
                    data=table, 
                    verify=True
                )

# Arrow table is retrieved 
with pa.ipc.open_file(response.content) as reader:
    arrow_table = reader.read_all()

display(result['out.avg_confidence'])

<pyarrow.lib.ChunkedArray object at 0x2a69b4b30>
[
  [
    [
      0.3588039
    ]
  ]
]

Tutorials

Other examples come from the Wallaroo Tutorials as listed below.

• Computer Vision: Yolov8n Demonstration
• Computer Vision Tutorials
• Wallaroo Edge Computer Vision Yolov8n Demonstration
• ARM Computer Vision Retail Yolov8 Demonstration
• Wallaroo Workshop: Edge Deployment: Computer Vision Yolov8n