This tutorial can be downloaded as part of the Wallaroo Tutorials repository.

Wallaroo Model Upload via the Wallaroo SDK: TensorFlow keras Sequential Single IO

The following tutorial demonstrates how to upload a TensorFlow keras Sequential Single IO model to a Wallaroo instance.

Tutorial Goals

Demonstrate the following:

• Upload a TensorFlow keras Sequential Single IO to a Wallaroo instance.
• Create a pipeline and add the model as a pipeline step.
• Perform a sample inference.

Prerequisites

• Wallaroo Version 2023.2.1 or above instance.

References

• Wallaroo MLOps API Essentials Guide: Model Upload and Registrations
• Wallaroo API Connection Guide
• DNS Integration Guide

Tutorial Steps

Import Libraries

The first step is to import the libraries we’ll be using. These are included by default in the Wallaroo instance’s JupyterHub service.

import json
import os
import pickle

import wallaroo
from wallaroo.pipeline   import Pipeline
from wallaroo.deployment_config import DeploymentConfigBuilder
from wallaroo.framework import Framework
from wallaroo.object import EntityNotFoundError

import pyarrow as pa
import numpy as np
import pandas as pd

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

import datetime

Open a Connection to Wallaroo

The next step is connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). If logging in externally, update the wallarooPrefix and wallarooSuffix variables with the proper DNS information. For more information on Wallaroo DNS settings, see the Wallaroo DNS Integration Guide.

wl = wallaroo.Client()

Set Variables

We’ll set the name of our workspace, pipeline, models and files. Workspace names must be unique across the Wallaroo workspace. For this, we’ll add in a randomly generated 4 characters to the workspace name to prevent collisions with other users’ workspaces. If running this tutorial, we recommend hard coding the workspace name so it will function in the same workspace each time it’s run.

workspace_name = f'keras-sequential-single-io'
pipeline_name = f'keras-sequential-single-io'

model_name = 'keras-sequential-single-io'
model_file_name = 'models/model-auto-conversion_keras_single_io_keras_sequential_model.h5'

Create Workspace and Pipeline

We will now create the Wallaroo workspace to store our model and set it as the current workspace. Future commands will default to this workspace for pipeline creation, model uploads, etc. We’ll create our Wallaroo pipeline to deploy our model.

workspace = wl.get_workspace(name=workspace_name, create_if_not_exist=True)

wl.set_current_workspace(workspace)

pipeline = wl.build_pipeline(pipeline_name)

Configure Data Schemas

The following parameters are required for TensorFlow keras models. Note that while some fields are considered as optional for the upload_model method, they are required for proper uploading of a TensorFlow Keras model to Wallaroo.

Once the upload process starts, the model is containerized by the Wallaroo instance. This process may take up to 10 minutes.

input_schema = pa.schema([
    pa.field('input', pa.list_(pa.float64(), list_size=10))
])
output_schema = pa.schema([
    pa.field('output', pa.list_(pa.float64(), list_size=32))
])

Upload Model

The model will be uploaded with the framework set as Framework.KERAS.

framework=Framework.KERAS

model = wl.upload_model(model_name, 
                        model_file_name, 
                        framework=framework, 
                        input_schema=input_schema, 
                        output_schema=output_schema)
model

Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a native runtime.
Model is attempting loading to a native runtime..incompatible
Model is pending loading to a container runtime.

Model is attempting loading to a container runtime……………..successful
Ready

model.config().runtime()

'flight'

Deploy Pipeline

The model is uploaded and ready for use. We’ll add it as a step in our pipeline, then deploy the pipeline. For this example we’re allocated 0.25 cpu and 4 Gi RAM to the pipeline through the pipeline’s deployment configuration.

deployment_config = DeploymentConfigBuilder() \
    .cpus(0.25).memory('1Gi') \
    .build()

# clear the pipeline if used in a previous tutorial
pipeline.undeploy()
pipeline.clear()
pipeline.add_model_step(model)

pipeline.deploy(deployment_config=deployment_config)
pipeline.status()

{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.28.0.179',
   'name': 'engine-6498c8b7df-8p92g',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'keras-sequential-single-io',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'keras-sequential-single-io',
      'sha': 'f7e49538e38bebe066ce8df97bac8be239ae8c7d2733e500c8cd633706ae95a8',
      'status': 'Running',
      'version': '9ab17dc8-50e0-43b4-b0b9-76084f593d0c'}]}}],
 'engine_lbs': [{'ip': '10.28.3.156',
   'name': 'engine-lb-d7cc8fc9c-2vjds',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': [{'ip': '10.28.2.39',
   'name': 'engine-sidekick-keras-sequential-single-io-130-6bb87f86cf-rppkx',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'}]}

Run Inference

A sample inference will be run. First the pandas DataFrame used for the inference is created, then the inference run through the pipeline’s infer method.

input_data = np.random.rand(10, 10)
mock_dataframe = pd.DataFrame({
    "input": input_data.tolist()
})
mock_dataframe

pipeline.infer(mock_dataframe)

Undeploy Pipelines

With the tutorial complete, the pipeline is undeployed to return the resources back to the cluster.

pipeline.undeploy()