Wallaroo SDK Upload Tutorials: Arbitrary Python
How to upload different Arbitrary Python models to Wallaroo.
The following tutorials cover how to upload sample arbitrary python models into a Wallaroo instance.
Parameter | Description |
---|
Web Site | https://www.python.org/ |
Supported Libraries | python==3.8 |
Framework | Framework.CUSTOM aka custom |
Runtime | Containerized aka flight |
Arbitrary Python models, also known as Bring Your Own Predict (BYOP) allow for custom model deployments with supporting scripts and artifacts. These are used with pre-trained models (PyTorch, Tensorflow, etc) along with whatever supporting artifacts they require. Supporting artifacts can include other Python modules, model files, etc. These are zipped with all scripts, artifacts, and a requirements.txt
file that indicates what other Python models need to be imported that are outside of the typical Wallaroo platform.
Contrast this with Wallaroo Python models - aka “Python steps”. These are standalone python scripts that use the python libraries natively supported by the Wallaroo platform. These are used for either simple model deployment (such as ARIMA Statsmodels), or data formatting such as the postprocessing steps. A Wallaroo Python model will be composed of one Python script that matches the Wallaroo requirements.
Arbitrary Python File Requirements
Arbitrary Python (BYOP) models are uploaded to Wallaroo via a ZIP file with the following components:
Artifact | Type | Description |
---|
Python scripts aka .py files with classes that extend mac.inference.Inference and mac.inference.creation.InferenceBuilder | Python Script | Extend the classes mac.inference.Inference and mac.inference.creation.InferenceBuilder . These are included with the Wallaroo SDK. Further details are in Arbitrary Python Script Requirements. Note that there is no specified naming requirements for the classes that extend mac.inference.Inference and mac.inference.creation.InferenceBuilder - any qualified class name is sufficient as long as these two classes are extended as defined below. |
requirements.txt | Python requirements file | This sets the Python libraries used for the arbitrary python model. These libraries should be targeted for Python 3.8 compliance. These requirements and the versions of libraries should be exactly the same between creating the model and deploying it in Wallaroo. This insures that the script and methods will function exactly the same as during the model creation process. |
Other artifacts | Files | Other models, files, and other artifacts used in support of this model. |
For example, the if the arbitrary python model will be known as vgg_clustering
, the contents may be in the following structure, with vgg_clustering
as the storage directory:
vgg_clustering\
feature_extractor.h5
kmeans.pkl
custom_inference.py
requirements.txt
Note the inclusion of the custom_inference.py
file. This file name is not required - any Python script or scripts that extend the classes listed above are sufficient. This Python script could have been named vgg_custom_model.py
or any other name as long as it includes the extension of the classes listed above.
The sample arbitrary python model file is created with the command zip -r vgg_clustering.zip vgg_clustering/
.
Wallaroo Arbitrary Python uses the Wallaroo SDK mac
module, included in the Wallaroo SDK 2023.2.1 and above. See the Wallaroo SDK Install Guides for instructions on installing the Wallaroo SDK.
Arbitrary Python Script Requirements
The entry point of the arbitrary python model is any python script that extends the following classes. These are included with the Wallaroo SDK. The required methods that must be overridden are specified in each section below.
mac.inference.Inference
interface serves model inferences based on submitted input some input. Its purpose is to serve inferences for any supported arbitrary model framework (e.g. scikit
, keras
etc.).
classDiagram
class Inference {
<<Abstract>>
+model Optional[Any]
+expected_model_types()* Set
+predict(input_data: InferenceData)* InferenceData
-raise_error_if_model_is_not_assigned() None
-raise_error_if_model_is_wrong_type() None
}
mac.inference.creation.InferenceBuilder
builds a concrete Inference
, i.e. instantiates an Inference
object, loads the appropriate model and assigns the model to to the Inference object.
classDiagram
class InferenceBuilder {
+create(config InferenceConfig) * Inference
-inference()* Any
}
mac.inference.Inference
mac.inference.Inference Objects
Object | Type | Description |
---|
model (Required) | [Any] | One or more objects that match the expected_model_types . This can be a ML Model (for inference use), a string (for data conversion), etc. See Arbitrary Python Examples for examples. |
mac.inference.Inference Methods
Method | Returns | Description |
---|
expected_model_types (Required) | Set | Returns a Set of models expected for the inference as defined by the developer. Typically this is a set of one. Wallaroo checks the expected model types to verify that the model submitted through the InferenceBuilder method matches what this Inference class expects. |
_predict (input_data: mac.types.InferenceData) (Required) | mac.types.InferenceData | The entry point for the Wallaroo inference with the following input and output parameters that are defined when the model is updated.mac.types.InferenceData : The input InferenceData is a Dictionary of numpy arrays derived from the input_schema detailed when the model is uploaded, defined in PyArrow.Schema format.mac.types.InferenceData : The output is a Dictionary of numpy arrays as defined by the output parameters defined in PyArrow.Schema format. The InferenceDataValidationError exception is raised when the input data does not match mac.types.InferenceData . |
raise_error_if_model_is_not_assigned | N/A | Error when a model is not set to Inference . |
raise_error_if_model_is_wrong_type | N/A | Error when the model does not match the expected_model_types . |
IMPORTANT NOTE
Verify that the inputs and outputs match the InferenceData
input and output types: a Dictionary of numpy arrays defined by the input_schema
and output_schema
parameters when uploading the model to the Wallaroo instance. The following code is an example of a Dictionary of numpy arrays.
preds = self.model.predict(data)
preds = preds.numpy()
rows, _ = preds.shape
preds = preds.reshape((rows,))
return {"prediction": preds} # a Dictionary of numpy arrays.
The example, the expected_model_types
can be defined for the KMeans
model.
from sklearn.cluster import KMeans
class SampleClass(mac.inference.Inference):
@property
def expected_model_types(self) -> Set[Any]:
return {KMeans}
mac.inference.creation.InferenceBuilder
InferenceBuilder
builds a concrete Inference
, i.e. instantiates an Inference
object, loads the appropriate model and assigns the model to the Inference.
classDiagram
class InferenceBuilder {
+create(config InferenceConfig) * Inference
-inference()* Any
}
Each model that is included requires its own InferenceBuilder
. InferenceBuilder
loads one model, then submits it to the Inference
class when created. The Inference
class checks this class against its expected_model_types()
Set.
mac.inference.creation.InferenceBuilder Methods
Method | Returns | Description |
---|
create(config mac.config.inference.CustomInferenceConfig) (Required) | The custom Inference instance. | Creates an Inference subclass, then assigns a model and attributes. The CustomInferenceConfig is used to retrieve the config.model_path , which is a pathlib.Path object pointing to the folder where the model artifacts are saved. Every artifact loaded must be relative to config.model_path . This is set when the arbitrary python .zip file is uploaded and the environment for running it in Wallaroo is set. For example: loading the artifact vgg_clustering\feature_extractor.h5 would be set with config.model_path \ feature_extractor.h5 . The model loaded must match an existing module. For our example, this is from sklearn.cluster import KMeans , and this must match the Inference expected_model_types . |
inference | custom Inference instance. | Returns the instantiated custom Inference object created from the create method. |
Arbitrary Python Runtime
Arbitrary Python always run in the containerized model runtime.
1 - Wallaroo SDK Upload Arbitrary Python Tutorial: Generate VGG16 Model
How to generate a VGG166 model for arbitrary python model deployment in Wallaroo.
This tutorial can be downloaded as part of the Wallaroo Tutorials repository.
Wallaroo SDK Upload Arbitrary Python Tutorial: Generate Model
This tutorial demonstrates how to use arbitrary python as a ML Model in Wallaroo. Arbitrary Python allows organizations to use Python scripts that require specific libraries and artifacts as models in the Wallaroo engine. This allows for highly flexible use of ML models with supporting scripts.
Tutorial Goals
This tutorial is split into two parts:
- Wallaroo SDK Upload Arbitrary Python Tutorial: Generate Model: Train a dummy
KMeans
model for clustering images using a pre-trained VGG16
model on cifar10
as a feature extractor. The Python entry points used for Wallaroo deployment will be added and described.- A copy of the arbitrary Python model
models/model-auto-conversion-BYOP-vgg16-clustering.zip
is included in this tutorial, so this step can be skipped.
- Arbitrary Python Tutorial Deploy Model in Wallaroo Upload and Deploy: Deploys the
KMeans
model in an arbitrary Python package in Wallaroo, and perform sample inferences. The file models/model-auto-conversion-BYOP-vgg16-clustering.zip
is provided so users can go right to testing deployment.
Arbitrary Python Script Requirements
The entry point of the arbitrary python model is any python script that must include the following.
class ImageClustering(Inference)
: The default inference class. This is used to perform the actual inferences. Wallaroo uses the _predict
method to receive the inference data and call the appropriate functions for the inference.def __init__
: Used to initialize this class and load in any other classes or other required settings.def expected_model_types
: Used by Wallaroo to anticipate what model types are used by the script.def model(self, model)
: Defines the model used for the inference. Accepts the model instance used in the inference.self._raise_error_if_model_is_wrong_type(model)
: Returns the error if the wrong model type is used. This verifies that only the anticipated model type is used for the inference.self._model = model
: Sets the submitted model as the model for this class, provided _raise_error_if_model_is_wrong_type
is not raised.
def _predict(self, input_data: InferenceData)
: This is the entry point for Wallaroo to perform the inference. This will receive the inference data, then perform whatever steps and return a dictionary of numpy arrays.
class ImageClusteringBuilder(InferenceBuilder)
: Loads the model and prepares it for inferencing.def inference(self) -> ImageClustering
: Sets the inference class being used for the inferences.def create(self, config: CustomInferenceConfig) -> ImageClustering
: Creates an inference subclass, assigning the model and any attributes required for it to function.
All other methods used for the functioning of these classes are optional, as long as they meet the requirements listed above.
Tutorial Prerequisites
- A Wallaroo version 2023.2.1 or above instance.
References
VGG16 Model Training Steps
This process will train a dummy KMeans
model for clustering images using a pre-trained VGG16
model on cifar10
as a feature extractor. This model consists of the following elements:
- All elements are stored in the folder
models/vgg16_clustering
. This will be converted to the zip file model-auto-conversion-BYOP-vgg16-clustering.zip
. models/vgg16_clustering
will contain the following:- All necessary model artifacts
- One or multiple Python files implementing the classes
Inference
and InferenceBuilder
. The implemented classes can have any naming they desire as long as they inherit from the appropriate base classes. - a
requirements.txt
file with all necessary pip requirements to successfully run the inference
Import Libraries
The first step is to import the libraries we’ll be using. These are included by default in the Wallaroo instance’s JupyterHub service.
import numpy as np
import pandas as pd
import json
import os
import pickle
import pyarrow as pa
import tensorflow as tf
import warnings
warnings.filterwarnings('ignore')
from sklearn.cluster import KMeans
from tensorflow.keras.datasets import cifar10
from tensorflow.keras import Model
from tensorflow.keras.layers import Flatten
2023-07-07 16:16:26.511340: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-07-07 16:16:26.511369: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Variables
We’ll use these variables in later steps rather than hard code them in. In this case, the directory where we’ll store our artifacts.
model_directory = './models/vgg16_clustering'
Load Data Set
In this section, we will load our sample data and shape it.
# Load and preprocess the CIFAR-10 dataset
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
# Normalize the pixel values to be between 0 and 1
X_train = X_train / 255.0
X_test = X_test / 255.0
(50000, 32, 32, 3)
Now we will train our model.
pretrained_model = tf.keras.applications.VGG16(include_top=False,
weights='imagenet',
input_shape=(32, 32, 3)
)
embedding_model = Model(inputs=pretrained_model.input,
outputs=Flatten()(pretrained_model.output))
2023-07-07 16:16:30.207936: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2023-07-07 16:16:30.207966: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2023-07-07 16:16:30.207987: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (jupyter-john-2ehummel-40wallaroo-2eai): /proc/driver/nvidia/version does not exist
2023-07-07 16:16:30.208169: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
X_train_embeddings = embedding_model.predict(X_train[:100])
X_test_embeddings = embedding_model.predict(X_test[:100])
kmeans = KMeans(n_clusters=2, random_state=0).fit(X_train_embeddings)
Save Models
Let’s first create the directory where the model artifacts will be saved:
os.makedirs(model_directory, exist_ok=True)
And now save the two models:
with open(f'{model_directory}/kmeans.pkl', 'wb') as fp:
pickle.dump(kmeans, fp)
embedding_model.save(f'{model_directory}/feature_extractor.h5')
WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.
All needed model artifacts have been now saved under our model directory.
Sample Arbitrary Python Script
The following shows an example of extending the Inference and InferenceBuilder classes for our specific model. This script is located in our model directory under ./models/vgg16_clustering
.
"""This module features an example implementation of a custom Inference and its
corresponding InferenceBuilder."""
import pathlib
import pickle
from typing import Any, Set
import tensorflow as tf
from mac.config.inference import CustomInferenceConfig
from mac.inference import Inference
from mac.inference.creation import InferenceBuilder
from mac.types import InferenceData
from sklearn.cluster import KMeans
class ImageClustering(Inference):
"""Inference class for image clustering, that uses
a pre-trained VGG16 model on cifar10 as a feature extractor
and performs clustering on a trained KMeans model.
Attributes:
- feature_extractor: The embedding model we will use
as a feature extractor (i.e. a trained VGG16).
- expected_model_types: A set of model instance types that are expected by this inference.
- model: The model on which the inference is calculated.
"""
def __init__(self, feature_extractor: tf.keras.Model):
self.feature_extractor = feature_extractor
super().__init__()
@property
def expected_model_types(self) -> Set[Any]:
return {KMeans}
@Inference.model.setter # type: ignore
def model(self, model) -> None:
"""Sets the model on which the inference is calculated.
:param model: A model instance on which the inference is calculated.
:raises TypeError: If the model is not an instance of expected_model_types
(i.e. KMeans).
"""
self._raise_error_if_model_is_wrong_type(model) # this will make sure an error will be raised if the model is of wrong type
self._model = model
def _predict(self, input_data: InferenceData) -> InferenceData:
"""Calculates the inference on the given input data.
This is the core function that each subclass needs to implement
in order to calculate the inference.
:param input_data: The input data on which the inference is calculated.
It is of type InferenceData, meaning it comes as a dictionary of numpy
arrays.
:raises InferenceDataValidationError: If the input data is not valid.
Ideally, every subclass should raise this error if the input data is not valid.
:return: The output of the model, that is a dictionary of numpy arrays.
"""
# input_data maps to the input_schema we have defined
# with PyArrow, coming as a dictionary of numpy arrays
inputs = input_data["images"]
# Forward inputs to the models
embeddings = self.feature_extractor(inputs)
predictions = self.model.predict(embeddings.numpy())
# Return predictions as dictionary of numpy arrays
return {"predictions": predictions}
class ImageClusteringBuilder(InferenceBuilder):
"""InferenceBuilder subclass for ImageClustering, that loads
a pre-trained VGG16 model on cifar10 as a feature extractor
and a trained KMeans model, and creates an ImageClustering object."""
@property
def inference(self) -> ImageClustering:
return ImageClustering
def create(self, config: CustomInferenceConfig) -> ImageClustering:
"""Creates an Inference subclass and assigns a model and additionally
needed attributes to it.
:param config: Custom inference configuration. In particular, we're
interested in `config.model_path` that is a pathlib.Path object
pointing to the folder where the model artifacts are saved.
Every artifact we need to load from this folder has to be
relative to `config.model_path`.
:return: A custom Inference instance.
"""
feature_extractor = self._load_feature_extractor(
config.model_path / "feature_extractor.h5"
)
inference = self.inference(feature_extractor)
model = self._load_model(config.model_path / "kmeans.pkl")
inference.model = model
return inference
def _load_feature_extractor(
self, file_path: pathlib.Path
) -> tf.keras.Model:
return tf.keras.models.load_model(file_path)
def _load_model(self, file_path: pathlib.Path) -> KMeans:
with open(file_path.as_posix(), "rb") as fp:
model = pickle.load(fp)
return model
Create Requirements File
As a last step we need to create a requirements.txt
file and save it under our vgg_clustering/
. The file should contain all the necessary pip requirements needed to run the inference. It will have this data inside.
tensorflow==2.8.0
scikit-learn==1.2.2
Zip model folder
Assuming we have stored the following files inside out model directory models/vgg_clustering/
:
feature_extractor.h5
kmeans.pkl
custom_inference.py
requirements.txt
Now we will zip the file. This is performed with the zip
command and the -r
option to zip the contents of the entire directory.
zip -r model-auto-conversion-BYOP-vgg16-clustering.zip vgg16_clustering/
The arbitrary Python custom model can now be uploaded to the Wallaroo instance.
2 - Wallaroo SDK Upload Arbitrary Python Tutorial: Deploy VGG16 Model
How to deploy a VGG166 model as a arbitrary python model in Wallaroo.
This tutorial can be downloaded as part of the Wallaroo Tutorials repository.
Arbitrary Python Tutorial Deploy Model in Wallaroo Upload and Deploy
This tutorial demonstrates how to use arbitrary python as a ML Model in Wallaroo. Arbitrary Python allows organizations to use Python scripts that require specific libraries and artifacts as models in the Wallaroo engine. This allows for highly flexible use of ML models with supporting scripts.
Tutorial Goals
This tutorial is split into two parts:
- Wallaroo SDK Upload Arbitrary Python Tutorial: Generate Model: Train a dummy
KMeans
model for clustering images using a pre-trained VGG16
model on cifar10
as a feature extractor. The Python entry points used for Wallaroo deployment will be added and described.- A copy of the arbitrary Python model
models/model-auto-conversion-BYOP-vgg16-clustering.zip
is included in this tutorial, so this step can be skipped.
- Arbitrary Python Tutorial Deploy Model in Wallaroo Upload and Deploy: Deploys the
KMeans
model in an arbitrary Python package in Wallaroo, and perform sample inferences. The file models/model-auto-conversion-BYOP-vgg16-clustering.zip
is provided so users can go right to testing deployment.
Arbitrary Python Script Requirements
The entry point of the arbitrary python model is any python script that must include the following.
class ImageClustering(Inference)
: The default inference class. This is used to perform the actual inferences. Wallaroo uses the _predict
method to receive the inference data and call the appropriate functions for the inference.def __init__
: Used to initialize this class and load in any other classes or other required settings.def expected_model_types
: Used by Wallaroo to anticipate what model types are used by the script.def model(self, model)
: Defines the model used for the inference. Accepts the model instance used in the inference.self._raise_error_if_model_is_wrong_type(model)
: Returns the error if the wrong model type is used. This verifies that only the anticipated model type is used for the inference.self._model = model
: Sets the submitted model as the model for this class, provided _raise_error_if_model_is_wrong_type
is not raised.
def _predict(self, input_data: InferenceData)
: This is the entry point for Wallaroo to perform the inference. This will receive the inference data, then perform whatever steps and return a dictionary of numpy arrays.
class ImageClusteringBuilder(InferenceBuilder)
: Loads the model and prepares it for inferencing.def inference(self) -> ImageClustering
: Sets the inference class being used for the inferences.def create(self, config: CustomInferenceConfig) -> ImageClustering
: Creates an inference subclass, assigning the model and any attributes required for it to function.
All other methods used for the functioning of these classes are optional, as long as they meet the requirements listed above.
Tutorial Prerequisites
- A Wallaroo version 2023.2.1 or above instance.
References
Tutorial Steps
Import Libraries
The first step is to import the libraries we’ll be using. These are included by default in the Wallaroo instance’s JupyterHub service.
import numpy as np
import pandas as pd
import json
import os
import pickle
import pyarrow as pa
import tensorflow as tf
import wallaroo
from sklearn.cluster import KMeans
from tensorflow.keras.datasets import cifar10
from tensorflow.keras import Model
from tensorflow.keras.layers import Flatten
from wallaroo.pipeline import Pipeline
from wallaroo.deployment_config import DeploymentConfigBuilder
from wallaroo.framework import Framework
from wallaroo.object import EntityNotFoundError
Open a Connection to Wallaroo
The next step is connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. If logging in externally, update the wallarooPrefix
and wallarooSuffix
variables with the proper DNS information. For more information on Wallaroo DNS settings, see the Wallaroo DNS Integration Guide.
Set Variables and Helper Functions
We’ll set the name of our workspace, pipeline, models and files. Workspace names must be unique across the Wallaroo workspace. For this, we’ll add in a randomly generated 4 characters to the workspace name to prevent collisions with other users’ workspaces. If running this tutorial, we recommend hard coding the workspace name so it will function in the same workspace each time it’s run.
We’ll set up some helper functions that will either use existing workspaces and pipelines, or create them if they do not already exist.
# import string
# import random
# # make a random 4 character suffix to prevent overwriting other user's workspaces
# suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
suffix=''
workspace_name = f'vgg16-clustering-workspace{suffix}'
pipeline_name = f'vgg16-clustering-pipeline'
model_name = 'vgg16-clustering'
model_file_name = './models/model-auto-conversion-BYOP-vgg16-clustering.zip'
def get_workspace(name):
workspace = None
for ws in wl.list_workspaces():
if ws.name() == name:
workspace= ws
if(workspace == None):
workspace = wl.create_workspace(name)
return workspace
def get_pipeline(name):
try:
pipeline = wl.pipelines_by_name(name)[0]
except EntityNotFoundError:
pipeline = wl.build_pipeline(name)
return pipeline
Create Workspace and Pipeline
We will now create the Wallaroo workspace to store our model and set it as the current workspace. Future commands will default to this workspace for pipeline creation, model uploads, etc. We’ll create our Wallaroo pipeline that is used to deploy our arbitrary Python model.
workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
pipeline = get_pipeline(pipeline_name)
Upload Arbitrary Python Model
Arbitrary Python models are uploaded to Wallaroo through the Wallaroo Client upload_model
method.
Upload Arbitrary Python Model Parameters
The following parameters are required for Arbitrary Python models. Note that while some fields are considered as optional for the upload_model
method, they are required for proper uploading of a Arbitrary Python model to Wallaroo.
Parameter | Type | Description |
---|
name | string (Required) | The name of the model. Model names are unique per workspace. Models that are uploaded with the same name are assigned as a new version of the model. |
path | string (Required) | The path to the model file being uploaded. |
framework | string (Upload Method Optional, Arbitrary Python model Required) | Set as Framework.CUSTOM . |
input_schema | pyarrow.lib.Schema (Upload Method Optional, Arbitrary Python model Required) | The input schema in Apache Arrow schema format. |
output_schema | pyarrow.lib.Schema (Upload Method Optional, Arbitrary Python model Required) | The output schema in Apache Arrow schema format. |
convert_wait | bool (Upload Method Optional, Arbitrary Python model Optional) (Default: True) | - True: Waits in the script for the model conversion completion.
- False: Proceeds with the script without waiting for the model conversion process to display complete.
|
Once the upload process starts, the model is containerized by the Wallaroo instance. This process may take up to 10 minutes.
Upload Arbitrary Python Model Return
The following is returned with a successful model upload and conversion.
Field | Type | Description |
---|
name | string | The name of the model. |
version | string | The model version as a unique UUID. |
file_name | string | The file name of the model as stored in Wallaroo. |
image_path | string | The image used to deploy the model in the Wallaroo engine. |
last_update_time | DateTime | When the model was last updated. |
For our example, we’ll start with setting the input_schema
and output_schema
that is expected by our ImageClustering._predict()
method.
input_schema = pa.schema([
pa.field('images', pa.list_(
pa.list_(
pa.list_(
pa.int64(),
list_size=3
),
list_size=32
),
list_size=32
)),
])
output_schema = pa.schema([
pa.field('predictions', pa.int64()),
])
Upload Model
Now we’ll upload our model. The framework is Framework.CUSTOM
for arbitrary Python models, and we’ll specify the input and output schemas for the upload.
model = wl.upload_model(model_name,
model_file_name,
framework=Framework.CUSTOM,
input_schema=input_schema,
output_schema=output_schema,
convert_wait=True)
model
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime..........................successful
Ready
Name | vgg16-clustering |
Version | 49e8e007-27ae-49da-8a69-6ed5c4aec890 |
File Name | model-auto-conversion-BYOP-vgg16-clustering.zip |
SHA | 7bb3362b1768c92ea7e593451b2b8913d3b7616c19fd8d25b73fb6990f9283e0 |
Status | ready |
Image Path | proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/mlflow-deploy:v2023.4.0-main-4005 |
Architecture | None |
Updated At | 2023-20-Oct 15:40:13 |
'flight'
Deploy Pipeline
The model is uploaded and ready for use. We’ll add it as a step in our pipeline, then deploy the pipeline. For this example we’re allocated 0.25 cpu and 4 Gi RAM to the pipeline through the pipeline’s deployment configuration.
pipeline.add_model_step(model)
name | vgg16-clustering-pipeline |
---|
created | 2023-10-20 15:37:53.960266+00:00 |
---|
last_updated | 2023-10-20 15:37:53.960266+00:00 |
---|
deployed | (none) |
---|
tags | |
---|
versions | a7cc9967-3311-4540-93d6-9ea30fa48cd5 |
---|
steps | |
---|
published | False |
---|
deployment_config = DeploymentConfigBuilder() \
.cpus(0.25).memory('4Gi') \
.build()
pipeline.deploy(deployment_config=deployment_config)
pipeline.status()
{'status': 'Error',
'details': [],
'engines': [{'ip': '10.244.3.130',
'name': 'engine-6fcbccd45c-wrrfq',
'status': 'Running',
'reason': None,
'details': [],
'pipeline_statuses': {'pipelines': [{'id': 'vgg16-clustering-pipeline',
'status': 'Running'}]},
'model_statuses': {'models': [{'name': 'vgg16-clustering',
'version': '49e8e007-27ae-49da-8a69-6ed5c4aec890',
'sha': '7bb3362b1768c92ea7e593451b2b8913d3b7616c19fd8d25b73fb6990f9283e0',
'status': 'Running'}]}}],
'engine_lbs': [{'ip': '10.244.2.188',
'name': 'engine-lb-584f54c899-ssbjw',
'status': 'Running',
'reason': None,
'details': []}],
'sidekicks': [{'ip': '10.244.3.129',
'name': 'engine-sidekick-vgg16-clustering-55-7bcb767b67-ljl6k',
'status': 'Running',
'reason': None,
'details': [],
'statuses': None}]}
Run inference
Everything is in place - we’ll now run a sample inference with some toy data. In this case we’re randomly generating some values in the data shape the model expects, then submitting an inference request through our deployed pipeline.
input_data = {
"images": [np.random.randint(0, 256, (32, 32, 3), dtype=np.uint8)] * 2,
}
dataframe = pd.DataFrame(input_data)
dataframe
| images |
---|
0 | [[[181, 128, 112], [109, 189, 153], [127, 171,... |
---|
1 | [[[181, 128, 112], [109, 189, 153], [127, 171,... |
---|
pipeline.infer(dataframe, timeout=10000)
| time | in.images | out.predictions | check_failures |
---|
0 | 2023-10-20 15:41:48.657 | [181, 128, 112, 109, 189, 153, 127, 171, 206, ... | 1 | 0 |
---|
1 | 2023-10-20 15:41:48.657 | [181, 128, 112, 109, 189, 153, 127, 171, 206, ... | 1 | 0 |
---|
Undeploy Pipelines
The inference is successful, so we will undeploy the pipeline and return the resources back to the cluster.