Wallaroo SDK Upload Tutorials: Arbitrary Python

How to upload different Arbitrary Python models to Wallaroo.

The following tutorials cover how to upload sample arbitrary python models into a Wallaroo instance.

Parameter Description
Web Site https://www.python.org/
Supported Libraries python==3.8
Framework Framework.CUSTOM aka custom
Runtime Containerized aka mlflow

Arbitrary Python models, also known as Bring Your Own Predict (BYOP) allow for custom model deployments with supporting scripts and artifacts. These are used with pre-trained models (PyTorch, Tensorflow, etc) along with whatever supporting artifacts they require. Supporting artifacts can include other Python modules, model files, etc. These are zipped with all scripts, artifacts, and a requirements.txt file that indicates what other Python models need to be imported that are outside of the typical Wallaroo platform.

Contrast this with Wallaroo Python models - aka “Python steps”. These are standalone python scripts that use the python libraries natively supported by the Wallaroo platform. These are used for either simple model deployment (such as ARIMA Statsmodels), or data formatting such as the postprocessing steps. A Wallaroo Python model will be composed of one Python script that matches the Wallaroo requirements.

Arbitrary Python File Requirements

Arbitrary Python (BYOP) models are uploaded to Wallaroo via a ZIP file with the following components:

Artifact Type Description
Python scripts aka .py files with classes that extend mac.inference.Inference and mac.inference.creation.InferenceBuilder Python Script Extend the classes mac.inference.Inference and mac.inference.creation.InferenceBuilder. These are included with the Wallaroo SDK. Further details are in Arbitrary Python Script Requirements.
requirements.txt Python requirements file This sets the Python libraries used for the arbitrary python model. These libraries should be targeted for Python 3.8 compliance. These requirements and the versions of libraries should be exactly the same between creating the model and deploying it in Wallaroo. This insures that the script and methods will function exactly the same as during the model creation process.
Other artifacts Files Other models, files, and other artifacts used in support of this model.

For example, the if the arbitrary python model will be known as vgg_clustering, the contents may be in the following structure, with vgg_clustering as the storage directory:


Note the inclusion of the custom_inference.py file. This could have been vgg_custom_model.py or any other name as long as it includes the extension of the classes listed above.

The arbitrary python model file would be created with the command zip -r vgg_clustering.zip vgg_clustering/.

Wallaroo Arbitrary Python uses the Wallaroo SDK mac module, included in the Wallaroo SDK 2023.2.1 and above. See the Wallaroo SDK Install Guides for instructions on installing the Wallaroo SDK.

Arbitrary Python Script Requirements

The entry point of the arbitrary python model is any python script that extends the following classes. These are included with the Wallaroo SDK. The required methods that must be overridden are specified in each section below.

  • mac.inference.Inference interface serves model inferences based on submitted input some input. Its purpose is to serve inferences for any supported arbitrary model framework (e.g. scikit, keras etc.).

        class Inference {
            +model Optional[Any]
            +expected_model_types()* Set
            +predict(input_data: InferenceData)*  InferenceData
            -raise_error_if_model_is_not_assigned() None
            -raise_error_if_model_is_wrong_type() None
  • mac.inference.creation.InferenceBuilder builds a concrete Inference, i.e. instantiates an Inference object, loads the appropriate model and assigns the model to to the Inference object.

        class InferenceBuilder {
            +create(config InferenceConfig) * Inference
            -inference()* Any


mac.inference.Inference Objects
Object Type Description
model Optional[Any] An optional list of models that match the supported frameworks from wallaroo.framework.Framework included in the arbitrary python script.
mac.inference.Inference Methods
Method Returns Description
expected_model_types (Required) Set Returns a Set of models expected for the inference. The set of models must match the Wallaroo supported model frameworks. Typically this is a set of one. Wallaroo checks the expected model types to verify that the model submitted through the InferenceBuilder method matches what this Inference class expects.
_predict (input_data: mac.types.InferenceData) (Required) mac.types.InferenceData The entry point for Wallaroo to perform the inference. The input InferenceData is a dictionary of numpy arrays derived from the input_schema detailed when the model is uploaded - see Upload Arbitrary Python Model below, and _predict returns a dictionary of numpy arrays. The InferenceDataValidationError exception is raised when the input data does not match mac.types.InferenceData.
raise_error_if_model_is_not_assigned N/A Error when expected_model_types is not set.
raise_error_if_model_is_wrong_type N/A Error when the model does not match the expected_model_types.


InferenceBuilder builds a concrete Inference, i.e. instantiates an Inference object, loads the appropriate model and assigns the model to the Inference.

    class InferenceBuilder {
        +create(config InferenceConfig) * Inference
        -inference()* Any

Each model that is included requires its own InferenceBuilder. InferenceBuilder loads one model, then submits it to the Inference class when created. The Inference class checks this class against its expected_model_types() Set.

mac.inference.creation.InferenceBuilder Methods
Method Returns Description
create(config mac.config.inference.CustomInferenceConfig) (Required) The custom Inference instance. Creates an Inference subclass, then assigns a model and attributes. The CustomInferenceConfig is used to retrieve the config.model_path, which is a pathlib.Path object pointing to the folder where the model artifacts are saved. Every artifact loaded must be relative to config.model_path. This is set when the arbitrary python .zip file is uploaded and the environment for running it in Wallaroo is set. For example: loading the artifact vgg_clustering\feature_extractor.h5 would be set with config.model_path \ feature_extractor.h5. The model loaded must match an existing module. For our example, this is from sklearn.cluster import KMeans, and this must match the Inference expected_model_types.
inference custom Inference instance. Returns the instantiated custom Inference object created from the create method.

Arbitrary Python Runtime

Arbitrary Python always run in the containerized model runtime.

Wallaroo SDK Upload Arbitrary Python Tutorial: Generate VGG16 Model

How to generate a VGG166 model for arbitrary python model deployment in Wallaroo.

Wallaroo SDK Upload Arbitrary Python Tutorial: Deploy VGG16 Model

How to deploy a VGG166 model as a arbitrary python model in Wallaroo.