Models uploaded and deployed to Wallaroo pipelines require that the input data matches the model’s data schema. Pipeline steps chained in Wallaroo pipelines accept the outputs of the previous pipeline step, which may or may not be a model. Any sequence of steps requires the previous step’s outputs to match the next step’s inputs.

This is where tools like BYOP (Bring Your Own Predict) aka Arbitrary Python, or Python Models are useful to reshape data submitted to a Wallaroo pipeline or between models deployed in a Wallaroo pipeline become essential.

This guide aims to demonstrate how Wallaroo handles data types in regards to models, and how to shape data for different Wallaroo deployed models.

Data Types

Data types for inputs and outputs to inference requests through the Wallaroo is based on the Apache Arrow Data Types, with the following exceptions:

• null: Not allowed. All fields must have submitted values that match their data type. For example, if the schema expects a float value, then some value of type float such as 0.0 must be submitted and cannot be None or Null. If a schema expects a string value, then some value of type string must be submitted, etc.
• time32 and time64: Datetime data types must be converted to string.

Data Constraints

Data submitted to Wallaroo for inference requests have the following data constraints.

• Equal rows constraint: The number of input rows and output rows must match.
• Data Type Consistency: Data types within each tensor are of the same type.

Equal Rows

Each input row for an inference is related directly to the inference row output.

For example, the INPUT and OUTPUT rows match, with each input row directly corresponding to an output row.

Equal Rows Input Example

Equal Rows Output Example

Data Type Consistency

Each element must have the same internal data type. For example, the following is valid because all of the data types within each element are float32.

t = [
    [2.35, 5.75],
    [3.72, 8.55],
    [5.55, 97.2]
]

The following is invalid, as it mixes floats and strings in each element:

t = [
    [2.35, "Bob"],
    [3.72, "Nancy"],
    [5.55, "Wani"]
]

The following inputs are valid, as each data type is consistent within the elements.

df = pd.DataFrame({
    "t": [
        [2.35, 5.75, 19.2],
        [5.55, 7.2, 15.7],
    ],
    "s": [
        ["Bob", "Nancy", "Wani"],
        ["Jason", "Rita", "Phoebe"]
    ]
})
df

Data Requirements for ONNX Models

ONNX models used for inference requests in Wallaroo have an additional requirement.

• The shape of each element within a field is the same.

For ONNX models, all inputs must be tensors. This requires that the shape of each element is the same. For example, the following is a proper input:

t =  [
    [2.35, 5.75],
    [3.72, 8.55],
    [5.55, 97.2]
]

Another example is a 2,2,3 tensor, where the shape of each element is (3,), and each element has 2 rows, and the outer row is bound by 2 rows.

t = [
        [2.35, 5.75, 19.2],
        [3.72, 8.55, 10.5]
    ],
    [
        [5.55, 7.2, 15.7],
        [9.6, 8.2, 2.3]
    ]

In this example each element has a shape of (2,). Tensors with elements of different shapes, known as ragged tensors, are not supported for batch inputs. For example:

t = [
    [2.35, 5.75],
    [3.72, 8.55, 10.5],
    [5.55, 97.2]
])

**INVALID SHAPE**

For this situation, the rows must be broken up to avoid the ragged tensors, or submitted as single row inputs. For example:

t1 = [[2.35, 5.75]]
t2 = [[3.72, 8.55, 10.5]]
t3 = [[5.55, 97.2]]

Batch vs Single Row Data Requirements for ONNX models

Data submitted to ONNX models deployed in Wallaroo is submitted as either batch data or single row input data based on the model configuration, typically at model upload.

ONNX models with the following data input conditions must be set as single row inputs:

• The inputs do not have an outer dimension.
• The output has an outer dimension of 1.

The simple rule of thumb is that for batch inputs, the data meets the following requirements.

• Equal rows constraint: The number of input rows and output rows must match.
• Data Type Consistency: Data types within each tensor are of the same type.
• Data Shape Consistency: The shape of each element within a field is the same.

For single row inputs, the data meets the following requirements.

• Equal rows constraint: The number of input rows and output rows must match.
• Data Type Consistency: Data types within each tensor are of the same type.

For single row inputs, the following image does not have an outer dimension, and depending on the size of the image, the shape of the data changes.

For example, the following images each have data type consistency when converted from an image to tensors. But the shape of each image differs from their size.

# a 640x480 image converted to tensors
df_image, resizedImage = utils.loadImageAndConvertToDataframe('./data/images/input/example/dairy_bottles.png', width, height)
# flatten the multi-dimensional array for inferencing through a Wallaroo deployed model
df_for_inference = pd.DataFrame(wallaroo.utils.flatten_np_array_columns(df_image, 'tensor'))
# get the shape - from a 640x480 image with 3 color values per pixel this is 921600 elements
print(df_for_inference['tensor'][0].shape)
(921600,)

# a 1920x1080 image converted to tensors
df_image, resizedImage = utils.loadImageAndConvertToDataframe('./data/images/input/example/crowd_of_people.png', width, height)
# flatten the multi-dimensional array for inferencing through a Wallaroo deployed model
df_for_inference = pd.DataFrame(wallaroo.utils.flatten_np_array_columns(df_image, 'tensor'))
# get the shape - from a 1920x1080 image with 3 color values per pixel this is 6220800 elements
df_for_inference['tensor'][0].shape
(6220800,)

display(df_for_inference)

For each input, the shape of the input field tensor depends on the size of the image, and there is no outer dimension. In these instances where single batch inputs are required because of the data schema, the ONNX model’s configuration must be set to batch_config=single.

mobilenet_model = wl.upload_model("mobilenet-demo", 
                './models/mobilenet.pt.onnx', 
                framework=Framework.ONNX).configure(batch_config="single")

Each inference request submitted must be a single row pandas DataFrame or Apache Arrow table when the model is configured as batch_config="single".

Setting Model Data Schemas

ML Models uploaded to Wallaroo except Wallaroo Native Runtimes require their fields and data types specified in Apache Arrow pyarrow.Schema. For full details on specific ML models and their data requirements, see Wallaroo SDK Essentials Guide: Model Uploads and Registrations.

Each field is defined by its data type. For example, the following input and output schema are used for a Pytorch single input and output model, which receives a list of float values and returns a list of float values.

input_schema = pa.schema([
    pa.field('input', pa.list_(pa.float32(), list_size=10))
])
output_schema = pa.schema([
    pa.field('output', pa.list_(pa.float32(), list_size=1))
])

Additional Examples

Other examples come from the Wallaroo Tutorials as listed below.

• Computer Vision: Yolov8n Demonstration
• Computer Vision Tutorials
• Wallaroo Edge Computer Vision Yolov8n Demonstration
• ARM Computer Vision Retail Yolov8 Demonstration
• Wallaroo Workshop: Edge Deployment: Computer Vision Yolov8n

Multiple Step Pipelines

Pipelines with multiple models as pipeline steps still have the same input and output data requirements. The inference input is fed to the pipeline, which submits it to the first model. The first model’s output becomes the input for the next model, until with the last model the pipeline submits the final inference results.

There are different ways of handling data in this scenario.

1. Set each model’s inputs and outputs to match. This will require each model’s outputs to match the next model’s inputs. This will require tailoring each model to match the previous model’s data types.

   

2. Use Python models, where a Python script a model to transform the data outputs from one model to match the data inputs of the next one. See Wallaroo SDK Essentials Guide: Model Uploads and Registrations: Python Models for full details and requirements.

   

3. Use BYOP (Bring Your Own Predict) aka Arbitrary Python models, which combines models and Python scripts as one cohesive model. See Wallaroo SDK Essentials Guide: Model Uploads and Registrations: Arbitrary Python for full details and requirements.