Models uploaded and deployed to Wallaroo pipelines require that the input data matches the model’s data schema. Pipeline steps chained in Wallaroo pipelines accept the outputs of the previous pipeline step, which may or may not be a model. Any sequence of steps requires the previous step’s outputs to match the next step’s inputs.

This is where tools like BYOP (Bring Your Own Predict) aka Arbitrary Python, or Python Models are useful to reshape data submitted to a Wallaroo pipeline or between models deployed in a Wallaroo pipeline become essential.

This guide aims to demonstrate how Wallaroo handles data types in regards to models, and how to shape data for different Wallaroo deployed models.

Data Types

Data types for inputs and outputs to inference requests through the Wallaroo is based on the Apache Arrow Data Types, with the following exceptions:

• null:
• time32 and time64: Datetime data types must be converted to string.

Model Uploads

ML Models uploaded to Wallaroo except ONNX require their fields and data types specified in Apache Arrow pyarrow.Schema. For full details on specific ML models and their data requirements, see Wallaroo SDK Essentials Guide: Model Uploads and Registrations.

Each field is defined by it’s data type. For example, the following input and output schema are used for a Pytorch single input and output model, which receives a list of float values and returns a list of float values.

input_schema = pa.schema([
    pa.field('input', pa.list_(pa.float32(), list_size=10))
])
output_schema = pa.schema([
    pa.field('output', pa.list_(pa.float32(), list_size=1))
])

Model Uploads Examples - Hugging Face Summarization

The following example demonstrates the data schema for the Hugging Face LLM Summarizer Model which takes the following inputs and outputs.

• Inputs

• Outputs

When uploading to Wallaroo using the Wallaroo SDK wallaroo.client.upload_model(name, path, framework, input_schema, output_schema, convert_wait, arch), the input_schema and output_schema define the fields the model uses as follows:

input_schema = pa.schema([
    pa.field('inputs', pa.string()),
    pa.field('return_text', pa.bool_()),
    pa.field('return_tensors', pa.bool_()),
    pa.field('clean_up_tokenization_spaces', pa.bool_())
])

output_schema = pa.schema([
    pa.field('summary_text', pa.string()),
])

model = wl.upload_model(name="HF summerizer model", 
                        path="./models/model-auto-conversion_hugging-face_complex-pipelines_hf-summarisation-bart-large-samsun.zip", 
                        framework=wallaroo.framework.Framework.HUGGING_FACE_SUMMARIZATION, 
                        input_schema=input_schema, 
                        output_schema=output_schema
                        )

Model Uploads Examples - Computer Vision Resnet

The following example demonstrates the data schema for the Computer Vision Resnet model, which has the following inputs and outputs.

• Inputs

• Outputs

This is translated into the following input and output schemas.

import pyarrow as py

input_schema = pa.schema([
    pa.field('tensor', pa.list_(pa.float64()))
])

output_schema = pa.schema([
    pa.field('forecast', pa.list_(pa.int64())),
    pa.field('average_forecast', pa.list_(pa.float64()))
])

Inferencing Data

Wallaroo pipelines accept either Apache Arrow tables natively or pandas DataFrames and transmit inference results data as the same.

• If using the Wallaroo SDK with wallaroo.pipeline.infer(https://docs.wallaroo.ai/20230201/wallaroo-developer-guides/wallaroo-sdk-guides/wallaroo-sdk-essentials-guide/wallaroo-sdk-essentials-inferences/) or wallaroo.pipeline.infer_from_file(https://docs.wallaroo.ai/20230201/wallaroo-developer-guides/wallaroo-sdk-guides/wallaroo-sdk-essentials-guide/wallaroo-sdk-essentials-inferences/), it is a “What you give is what you get” situation: If an Apache Arrow table is submitted for an inference request with the , then an Apache Arrow table is returned. If a pandas DataFrame is submitted, then a pandas DataFrame is returned.
• If using the Wallaroo pipeline’s inference URL as an API request, the API headers Content-Type and Accept determine what data type is submitted and what data type is received:
  • For DataFrame formatted JSON: application/json; format=pandas-records
  • For Arrow binary files: application/vnd.apache.arrow.file

Once the data is received by Wallaroo, it is converted to the Apache Arrow table format and submitted to each model deployed in the pipeline in the Apache Arrow format.

Batch Inputs

For batch inputs, each inference request is contained in a single row of data. For example, a bike rental model predicts rentals based on previous dates for bikes sold, whether it was a week day, etc. These are submitted as a group of dates that are then used by the model to predict bike rentals on figure dates. For example, this 5 day period is submitted for the inference request.

To submit to the Wallaroo deployed pipeline as a batch, each ranges of dates and values is set into a single row of data. This is reflected in the input and output schemas defined for the model. The pyarrow schema pa.list_ is used to define the data being received as a list per field.

import pyarrow as pa
import wallaroo

wl = wallaroo.Client()

input_schema = pa.schema([
   pa.field('dteday', pa.list_(pa.string()) ),
   pa.field('site_id', pa.list_(pa.string()) ), 
   pa.field('cnt', pa.list_(pa.int64()) ),
   pa.field('season', pa.list_(pa.int64()) ),
   pa.field('holiday', pa.list_(pa.int64()) ),
   pa.field('weekday', pa.list_(pa.int64()) ),
   pa.field('workingday', pa.list_(pa.int64()) ),           
])

output_schema = pa.schema([
    pa.field('dteday', pa.list_(pa.string()) ),
    pa.field('site_id', pa.list_(pa.string()) ), 
    pa.field('forecast', pa.list_(pa.int64()) ),
])

arima_model = (wl.upload_model('bikeforecast-arima', 
                               './models/forecast.py', 
                               framework=Framework.PYTHON)
                               .configure(runtime="python", 
                               input_schema = input_schema,
                               output_schema = output_schema
                               )
              )

Here, a month of data for single site is prepared for for a Wallaroo pipeline. Each inference request contained in a single row per site. Additional sites would be in their own row, with their month’s worth of data set as lists within the cells.

display(input_frame)

Then the inference request is submitted and returned. The inference results are then converted to whatever format the data scientist and ML engineers require.

results = pipeline.infer(input_frame)
display(results)

Multiple Step Pipelines

Pipelines with multiple models as pipeline steps still have the same input and output parameters. The inference input is fed to the pipeline, which submits it to the first model. The first model’s output becomes the input for the next model, until with the last model the pipeline submits the final inference results.

There are different ways of handling data in this scenario.

1. Set each model’s inputs and outputs to match. This will require each model’s outputs to match the next model’s inputs. This will require tailoring each model to match the previous model’s data types.

   

2. Use Python models, where a Python script a model to transform the data outputs from one model to match the data inputs of the next one. See Wallaroo SDK Essentials Guide: Model Uploads and Registrations: Python Models for full details and requirements.

   

3. Use BYOP (Bring Your Own Predict) aka Arbitary Python models, which combines models and Python scripts as one cohesive model. See Wallaroo SDK Essentials Guide: Model Uploads and Registrations: Arbitrary Python for full details and requirements.