Deploy the Model in Wallaroo

The following tutorials are available from the Wallaroo Tutorials Repository.

Stage 3: Deploy the Model in Wallaroo

In this stage, we upload the trained model and the processing steps to Wallaroo, then set up and deploy the inference pipeline.

Once deployed we can feed the newest batch of data to the pipeline, do the inferences and write the results to a results table.

For clarity in this demo, we have split the training/upload task into two notebooks:

  • 02_automated_training_process.ipynb: Train and pickle ML model.
  • 03_deploy_model.ipynb: Upload the model to Wallaroo and deploy into a pipeline.

Assuming no changes are made to the structure of the model, these two notebooks, or a script based on them, can then be scheduled to run on a regular basis, to refresh the model with more recent training data and update the inference pipeline.

This notebook is expected to run within the Wallaroo instance’s Jupyter Hub service to provide access to all required Wallaroo libraries and functionality.

Resources

The following resources are used as part of this tutorial:

  • data
    • data/seattle_housing_col_description.txt: Describes the columns used as part data analysis.
    • data/seattle_housing.csv: Sample data of the Seattle, Washington housing market between 2014 and 2015.
  • code
    • simdb.py: A simulated database to demonstrate sending and receiving queries.
  • models
    • housing_model_xgb.onnx: Model created in Stage 2: Training Process Automation Setup.
    • ./models/preprocess_byop.zip.: Formats the incoming data for the model.
    • ./models/postprocess_byop.zip: Formats the outgoing data for the model.

Steps

The process of uploading the model to Wallaroo follows these steps:

Connect to Wallaroo

First we import the required libraries to connect to the Wallaroo instance, then connect to the Wallaroo instance.

import json
import pickle
import pandas as pd
import numpy as np
import pyarrow as pa

import simdb # module for the purpose of this demo to simulate pulling data from a database

# from wallaroo.ModelConversion import ConvertXGBoostArgs, ModelConversionSource, ModelConversionInputType
import wallaroo
from wallaroo.object import EntityNotFoundError

# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

import datetime

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

def get_pipeline(name):
    try:
        pipeline = wl.pipelines_by_name(name)[0]
    except EntityNotFoundError:
        pipeline = wl.build_pipeline(name)
    return pipeline
workspace_name = 'housepricing'
model_name = "housepricemodel"
model_file = "./housing_model_xgb.onnx"
pipeline_name = "housing-pipe"

The workspace housepricing will either be created or, if already existing, used and set to the current workspace.

new_workspace = get_workspace(workspace_name)
new_workspace
{'name': 'housepricing', 'id': 5, 'archived': False, 'created_by': '4a24979e-63e9-4f04-bab4-0cf828f15ae7', 'created_at': '2024-03-13T17:58:08.872971+00:00', 'models': [], 'pipelines': []}
_ = wl.set_current_workspace(new_workspace)

Upload The Model

With the connection set and workspace prepared, upload the model created in 02_automated_training_process.ipynb into the current workspace.

To ensure the model input contract matches the provided input, the configuration tensor_fields=["tensor"] is used so regardless of what the model input type is, Wallaroo will ensure inputs of type tensor are accepted.

hpmodel = (wl.upload_model(model_name, 
                           model_file, 
                           framework=wallaroo.framework.Framework.ONNX)
                           .configure(tensor_fields=["tensor"]
                                    )
            )

Upload the Processing Modules

Preprocess model:

input_schema = pa.schema([
    pa.field('id', pa.int64()),
    pa.field('date', pa.string()),
    pa.field('list_price', pa.float64()),
    pa.field('bedrooms', pa.int64()),
    pa.field('bathrooms', pa.float64()),
    pa.field('sqft_living', pa.int64()),
    pa.field('sqft_lot', pa.int64()),
    pa.field('floors', pa.float64()),
    pa.field('waterfront', pa.int64()),
    pa.field('view', pa.int64()),
    pa.field('condition', pa.int64()),
    pa.field('grade', pa.int64()),
    pa.field('sqft_above', pa.int64()),
    pa.field('sqft_basement', pa.int64()),
    pa.field('yr_built', pa.int64()),
    pa.field('yr_renovated', pa.int64()),
    pa.field('zipcode', pa.int64()),
    pa.field('lat', pa.float64()),
    pa.field('long', pa.float64()),
    pa.field('sqft_living15', pa.int64()),
    pa.field('sqft_lot15', pa.int64()),
    pa.field('sale_price', pa.float64())
])

output_schema = pa.schema([
    pa.field('tensor', pa.list_(pa.float32(), list_size=18))
])

preprocess_model = wl.upload_model("preprocess-byop", "./models/preprocess_byop.zip", \
                                   framework=wallaroo.framework.Framework.CUSTOM, \
                                   input_schema=input_schema, output_schema=output_schema)
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime........successful

Ready

Postprocess model:

input_schema = pa.schema([
    pa.field('variable', pa.list_(pa.float32()))
])

output_schema = pa.schema([
    pa.field('variable', pa.list_(pa.float32()))
])

postprocess_model = wl.upload_model("postprocess-byop", "./models/postprocess_byop.zip", \
                                   framework=wallaroo.framework.Framework.CUSTOM, \
                                   input_schema=input_schema, output_schema=output_schema)
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime........successful

Ready

Create and Deploy the Pipeline

Create the pipeline with the preprocess module, housing model, and postprocess module as pipeline steps, then deploy the newpipeline.

pipeline = get_pipeline(pipeline_name)
# clear if the tutorial was run before
pipeline.clear()

pipeline.add_model_step(preprocess_model)
pipeline.add_model_step(hpmodel)
pipeline.add_model_step(postprocess_model)

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()
pipeline.deploy(deployment_config=deploy_config)
pipeline.status()
{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.28.0.131',
   'name': 'engine-686c9db4fd-jhxf2',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'housing-pipe',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'postprocess-byop',
      'version': 'a4f60a10-2097-4b89-ae9b-b97fc3343f12',
      'sha': '7b95b800dd05b11b664ae269820da05de0ba6c99fc7ef270f3d645f330608a05',
      'status': 'Running'},
     {'name': 'preprocess-byop',
      'version': '28e338eb-3dd6-40b8-a57e-f7405840d8c9',
      'sha': '4e5a026b792ec84baf54b04ebf3a5d5cb24079629921efede64872cf6ccde5ca',
      'status': 'Running'},
     {'name': 'housepricemodel',
      'version': 'a5198e1e-5a0e-42b3-86b5-4bcdfd330f2d',
      'sha': 'd8b79e526eed180d39d4653b39bebd9d06e6ae7f68293b5745775a9093a3ae7d',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.28.0.130',
   'name': 'engine-lb-d7cc8fc9c-r45rc',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': [{'ip': '10.28.0.129',
   'name': 'engine-sidekick-postprocess-byop-4-c987744bf-82kxx',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'},
  {'ip': '10.28.2.104',
   'name': 'engine-sidekick-preprocess-byop-3-ff88cfd96-gfmzr',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'}]}

Test the Pipeline

We will use a single query from the simulated housing_price table and infer. When successful, we will undeploy the pipeline to restore the resources back to the Kubernetes environment.

conn = simdb.simulate_db_connection()

# create the query
query = f"select * from {simdb.tablename} limit 1"
print(query)

# read in the data
singleton = pd.read_sql_query(query, conn)
conn.close()

display(singleton.loc[:, ["id", "date", "list_price", "bedrooms", "bathrooms", "sqft_living", "sqft_lot"]])
select * from house_listings limit 1
iddatelist_pricebedroomsbathroomssqft_livingsqft_lot
071293005202023-07-31221900.031.011805650
result = pipeline.infer(singleton)
display(result.loc[:, ['time', 'out.variable']])
timeout.variable
02024-03-13 18:28:37.873[224852.0]

When finished, we undeploy the pipeline to return the resources back to the environment.

pipeline.undeploy()
Waiting for undeployment - this will take up to 45s ..................................... ok
namehousing-pipe
created2024-03-13 18:19:37.008014+00:00
last_updated2024-03-13 18:19:37.077324+00:00
deployedFalse
archNone
tags
versionsbc52c554-6b82-4e3c-a3d1-f6402ee388b0, 6348f7aa-89ba-4051-ad91-1398e5cdd519
stepspreprocess-byop
publishedFalse

With this stage complete, we can proceed to Stage 4: Regular Batch Inference.