LLM (Large Language Models)

• 1: Large Language Models: Summarization
• 1.1: Sentiment Analysis: Upload and Deploy
• 1.2: Sentiment Analysis: Automate Data Connections
• 1.3: LLM Summarization: Deploy Pipeline to Edge Devices

1 - Large Language Models: Summarization

1.1 - Sentiment Analysis: Upload and Deploy

Workshop Notebook 1: Deploy a Model

For this workshop, let’s pretend that you work for a marketing firm. You have developed a model that wants to take large documents and summarize the most important points.

This demonstration will focus on deployment to the edge. The sample model is available at the following URL. This model should be downloaded and placed into the ./models folder before beginning this demonstration.

model-auto-conversion_hugging-face_complex-pipelines_hf-summarisation-bart-large-samsun.zip (1.4 GB)

In this set of exercises, you will used a pre-trained model and deploy it to Wallaroo. This will require understanding the following concepts:

• Wallaroo Workspaces: Workspaces are environments were users upload models, create pipelines and other artifacts. The workspace should be considered the fundamental area where work is done. Workspaces are shared with other users to give them access to the same models, pipelines, etc.
• Wallaroo Model Upload and Registration: ML Models are uploaded to Wallaroo through the SDK or the MLOps API to a workspace. ML models include default runtimes (ONNX, Python Step, and TensorFlow) that are run directly through the Wallaroo engine, and containerized runtimes (Hugging Face, PyTorch, etc) that are run through in a container through the Wallaroo engine.
• Wallaroo Pipelines: Pipelines are used to deploy models for inferencing. Each model is a pipeline step in a pipelines, where the inputs of the previous step are fed into the next. Pipeline steps can be ML models, Python scripts, or Arbitrary Python (these contain necessary models and artifacts for running a model).

For this tutorial, we will be providing pre-trained models in ONNX format. To see how to upload and deploy your particular model, see the Wallaroo Documentation site.

Before we start, let’s load some libraries that we will need for this notebook (note that this may not be a complete list).

• IMPORTANT NOTE: This tutorial is geared towards a Wallaroo 2023.2.1 environment.

# preload needed libraries 

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

from IPython.display import display

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

import pyarrow as pa

import json

import datetime
import time

# used for unique connection names

import string
import random

Get ready to work with Wallaroo

With the libraries loaded, you can log into Wallaroo. This will provide access to your workspaces, workspaces shared with you from other users, and all other aspects of the Wallaroo environment.

Logging into Wallaroo via the cluster’s integrated JupyterLab is quite straight forward:

# Login through local Wallaroo instance 
wl = wallaroo.Client()

See the documentation if you are logging into Wallaroo some other way such as from a remote location. This tutorial assumes you’re logging in through the Wallaroo JupyterHub service.

Notice that the Wallaroo client connection is stored into a variable called wl. This variable can be anything you want it to be - you can have client = wallaroo.Client() or myWallarooClient = wallaroo.Client().

This variable is about to become your best friend - a lot of the commands you’ll be running will be through this variable, like wl.list_workspaces() to get all of the workspaces available to you in your Wallaroo environment, or wl.list_models() to show all of the models in your current workspace. We’ll go into these commands and more - just make sure you saved that Wallaroo client to a variable so you can use it for the other commands.

When we log into the Wallaroo through the SDK, the Client will provide a url to verify your authentication. Either click it, or copy and past that URL, then authenticate into your Wallaroo instance with your email address and password.

Login to Wallaroo Exercise

Time to login to your Wallaroo instance. By now you should be logged into your Wallaroo JupyterHub service and looking at this notebook.

Copy the code below and place it into the code block and run it. When prompted, select the authentication URL by either clicking it, or copying and pasting it into a browser. Log into your Wallaroo instance, and then the client will be set.

# Login through local Wallaroo instance 
wl = wallaroo.Client()

# put your wallaroo Client login code here.

wl = wallaroo.Client()

Run Client Commands

We’re now logged into our Wallaroo instance, let’s run some commands to get used to working within the environment.

The following are going to be very useful as you work in Wallaroo.

List Workspaces

The command wallaroo.Client.list_workspaces gives a list of all of the workspaces you have access to in the Wallaroo environment. Here’s an example of running it. For this example, our Wallaroo client is stored in the wl variable, but this could have been named wallaroo_client or whatever you like.

wl.list_workspaces()

Listing the workspaces will show the following fields:

• name: The user created workspace name. Workspace names must be unique across the Wallaroo instance.
• created_at: The date and time the workspace was created.
• users: The users in the workspace. The user who created the workspace is always listed.
• models: The number of models in the workspace.
• pipelines: The number of pipelines in the workspace.

When you first login to Wallaroo, the workspace {your email address} - Default Workspace is created. This is assigned as your current workspace once you have logged into Wallaroo. We’ll cover workspaces in a moment - for now, just remember that every time you do wallaroo.Client(), that your default workspace is set as your current workspace, and any commands you issue will be directed to that workspace.

Notice that in that example above, you only see one default workspace - the one of the user who ran the list_workspaces command. Other users will have their own default workspaces. You can only see workspaces you have access to.

List Users

Users are listed with the wallaroo.Client.list_users command. This shows all of the user’s email addresses and names across the Wallaroo instance.

There’s more commands, but we’ll stop here until we’ve created our workspace and uploaded some models.

Client Commands Exercise

For this exercise, use your Wallaroo client variable and list the current workspaces and current users. You can do this with your Wallaroo client variable you saved in a previous step.

For example, if your Wallaroo client variable is wl, then the following would list the workspaces and then the users:

# list the workspaces available to you
workspaces = wl.list_workspaces()
print(workspaces)

# list the users
users = wl.list_users()
print(users)

# empty space to get workspaces and users

display(wl.list_workspaces())

display(wl.list_users())

[User({"id": "568f86b1-8b18-4db8-a40b-eca338fe371d", "email": "admin@keycloak", "username": "admin", "enabled": "True),
 User({"id": "b030ff9c-41eb-49b4-afdf-2ccbecb6be5d", "email": "john.hummel@wallaroo.ai", "username": "john.hummel@wallaroo.ai", "enabled": "True)]

Workspace Creation and Management

A Wallaroo workspace is place to organize the deployment artifacts for a project, and to collaborate with other team members. For more information, see the Wallaroo 101.

When you upload a ML model to Wallaroo, you upload it to a workspace. When a pipeline is created and models assigned to it, that pipeline is in side of a workspace. Wallaroo workload orchestrations? They’re assigned to a workspace.

When you first login, the SDK assigns you to your default workspace. Most of the commands issued through your Wallaroo client will target that workspace.

You can see what workspace you are currently in with the wallaroo.Client.get_current_workspace() method.

This shows you the following workspace fields:

• name: The user created workspace name. Workspace names must be unique across the Wallaroo instance.
• id: The numerical identifier of the workspace.
• archived: Whether the workspace was archived or not.
• created_by: The Keycloak ID of the user who created the workspace. This is in UUID format, and is used to identify specific users. Most of the time you’ll refer to users by their email address.

• created_at: The date and time the workspace was created.
• models: The models uploaded to the workspace and their details.
• pipelines: The number of pipelines in the workspace and their details.

Get Current Workspace Exercise

Get your current workspace with the wallaroo.Client.get_current_workspace(). For example, if your Wallaroo client was saved to the variable wl, this command would be:

wl.get_current_workspace()

When done,

## blank space to get your current workspace

wl.get_current_workspace()

{'name': 'john.hummel@wallaroo.ai - Default Workspace', 'id': 1, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-09-27T15:17:23.923705+00:00', 'models': [{'name': 'v5s6', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 9, 28, 19, 23, 3, 425821, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 9, 28, 18, 55, 24, 860357, tzinfo=tzutc())}], 'pipelines': [{'name': 'cv-arm-edge', 'create_time': datetime.datetime(2023, 9, 27, 15, 17, 45, 593032, tzinfo=tzutc()), 'definition': '[]'}, {'name': 'retailimage', 'create_time': datetime.datetime(2023, 9, 28, 18, 55, 14, 954338, tzinfo=tzutc()), 'definition': '[]'}]}

Create New Workspace

Workspaces are created with the wallaroo.Client.create_workspace(name), where name is the new name of the workspace. When a new workspace is created, the workspace user is assigned to the user that created it (in this case - you). For example, if the Wallaroo client is stored to the variable wl, then the following will create the new workspace ‘sparkly-bunnies, then store the workspace information into the variable workspace`:

workspace = wl.create_workspace('sparkly-bunnies')

Once this is created, this shows you the following workspace fields:

• name: The user created workspace name. Workspace names must be unique across the Wallaroo instance, and must be DNS compliant. So ‘my-cool-workspace’ is ok, but ‘?? workspace’ is not. - are ok, but _ is not.
• id: The numerical identifier of the workspace.
• archived: Whether the workspace was archived or not.
• created_by: The Keycloak ID of the user who created the workspace. This is in UUID format, and is used to identify specific users. Most of the time you’ll refer to users by their email address.

• created_at: The date and time the workspace was created.
• models: The models uploaded to the workspace and their details. For a new workspace, this will be empty [].
• pipelines: The number of pipelines in the workspace and their details. For a new workspace, this will be empty [].

Workspace names must be unique. So the following will fail:

wl.create_workspace('sparkly-bunnies')
wl.create_workspace('sparkly-bunnies')
Exception: Failed to create workspace.

Create New Workspace Exercise

Now it’s time for us to create our own workspace. To make this easy, we’ll call the workspace ‘workshop-workspace-{firstname}’. If someone else has the same first name and is in this workshop, each of you decide who should change their name for this to work. Or - if it’s easier - change the firstname to something else like john1.

For example, if your Wallaroo client was saved to the variable wl, then the command to create a new workspace workshop-workspace-sample is:

wl.create_workspace('workshop-workspace-sample')

When you’re done, list the workspaces. You did that in a previous step, so you can copy that here.

# blank space to create your workspace

print(wl.create_workspace('workshop-workspace-summarization'))

# list all the workspaces here

wl.list_workspaces()

{'name': 'workshop-workspace-summarization', 'id': 13, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-10-05T16:17:44.447738+00:00', 'models': [], 'pipelines': []}

Retrieve Workspace

In the above example, you saw this:

workspace = wl.create_workspace('sparkly-bunnies')

This creates the workspace sparkly-bunnies, then assigns it to the variable workspace. We can display that workspace and see what it looks like. For example:

workspace = wl.create_workspace('sparkly-bunnies')

print(workspace)
{'name': 'sparkly-bunnies', 'id': 9, 'archived': False, 'created_by': '66d3b2c4-9b22-4429-b16e-3bcdc1ac28e3', 'created_at': '2023-08-22T17:30:40.475785+00:00', 'models': [], 'pipelines': []}

If we had created a workspace earlier, and want to work with it, we will have to retrieve it. We do that with the wallaroo.Client.list_workspaces method, which returns a List. Then we can set the workspace we want to a variable.

Here’s an example. We start with list_workspaces:

wl.list_workspaces()

In our case, we want to use sparkly-bunnies. If we count down from the top, it’s the 2nd in the List. Since lists start at 0, it’s as position 1. For those not experienced with Lists, here’s how it looks.

And so on.

To store a specific workspace from list_workspaces, we assign it to a variable based on its position. If we want to store the workspace sparkly-bunnies to a variable workspace, we do it like so:

workspace = wl.list_workspaces()[1]
print(workspace)

{'name': 'sparkly-bunnies', 'id': 9, 'archived': False, 'created_by': '66d3b2c4-9b22-4429-b16e-3bcdc1ac28e3', 'created_at': '2023-08-22T17:30:40.475785+00:00', 'models': [], 'pipelines': []}

For those more familiar with lists, you can do things like filter by name like so:

def get_workspace(name, client):
    workspace = None
    for ws in client.list_workspaces():
        if ws.name() == name:
            workspace= ws

We’ll introduce helper functions like this later, but know for now if you know how to use a List, then you can retrieve a workspace.

Retrieve Workspace Exercise

Retrieve the workspace to a variable you created earlier through the following steps:

1. List the workspaces.
2. Determine the position of the workspace from the list - remember the list positions start at 0.
3. Assign the workspace to a variable that you’ll use later.

For example, if the workspace is position 1 in the list_workspaces list, then you would retrieve the workspace like so:

workspace = wl.list_workspaces()[1]

# blank space to retrieve your workspace

display(wl.list_workspaces())
workspace = wl.list_workspaces()[-1]
print(workspace)

{'name': 'workshop-workspace-summarization', 'id': 13, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-10-05T16:17:44.447738+00:00', 'models': [], 'pipelines': []}

Set the Current Workspace

We mentioned earlier that when you login to Wallaroo, the SDK assigns you to the default workspace - the one named {your email address} - Default Workspace - replacing your email address in the front.

Usually you’ll want to work in some other workspace - perhaps one that you’re a part of with other users, or one you set up yourself for test purposes. It is highly recommended that workspaces be divided by project or some specific goal where the same models are used for different purposes.

We’ve gone over how to create a workspace, and how to retrieve a workspace that was previously created. Now we’ll use that to set our current workspace with the wallaroo.Client.set_current_workspace(workspace).

The current workspace is where your SDK commands are routed to. When you give the upload models command - they are uploaded to your current workspace. Build a pipeline? Associated to the current workspace.

So we’re going to make sure that what we’re doing is done in the right workspace with two commands:

• wallaroo.Client.get_current_workspace(): Shows what the current workspace.
• wallaroo.Client.set_current_workspace(workspace): Sets the current workspace to the target workspace.

For example, if your workspace is saved to a variable as shown in the previous step, we can change from the default workspace and set our current workspace to the new one as follows:

# show the current workspace
print(wl.get_current_workspace())
{'name': 'john.hummel@wallaroo.ai - Default Workspace', 'id': 1, 'archived': False, 'created_by': '66d3b2c4-9b22-4429-b16e-3bcdc1ac28e3', 'created_at': '2023-08-21T19:06:07.404363+00:00', 'models': [{'name': 'm1', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 21, 19, 38, 36, 672465, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 21, 19, 38, 36, 672465, tzinfo=tzutc())}], 'pipelines': [{'name': 'p1', 'create_time': datetime.datetime(2023, 8, 21, 19, 38, 44, 314377, tzinfo=tzutc()), 'definition': '[]'}]}

# change to the new workspace
wl.set_current_workspace(workspace)

# show the new current workspace
print(wl.get_current_workspace())
{'name': 'workshop-workspace-john', 'id': 12, 'archived': False, 'created_by': '66d3b2c4-9b22-4429-b16e-3bcdc1ac28e3', 'created_at': '2023-08-22T17:38:13.612187+00:00', 'models': [], 'pipelines': []}

Setting your current workspace to the one you want to work in is an important step. We highly recommend that once a Wallaroo client connection is established, the next task should be setting whatever workspace is the proper one to work in as the current workspace, then proceeding with any other tasks.

Set the Current Workspace Exercise

Previously you created a workspace and retrieved it to a variable. Using the * wallaroo.Client.get_current_workspace() and wallaroo.Client.set_current_workspace(workspace) methods:

1. Get your current workspace.
2. Set your current workspace to the one created in the previous steps.
3. Get your current workspace again to verify that the change was made.

For example, if your Wallaroo client was stored as the variable wl, and your new workspace saved to the variable workspace, you can change your current workspace to the new one with the following:

wl.set_current_workspace(workspace)
wl.get_current_workspace()

print(wl.get_current_workspace())
wl.set_current_workspace(workspace)
print(wl.get_current_workspace())

{'name': 'john.hummel@wallaroo.ai - Default Workspace', 'id': 1, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-09-27T15:17:23.923705+00:00', 'models': [{'name': 'v5s6', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 9, 28, 19, 23, 3, 425821, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 9, 28, 18, 55, 24, 860357, tzinfo=tzutc())}], 'pipelines': [{'name': 'cv-arm-edge', 'create_time': datetime.datetime(2023, 9, 27, 15, 17, 45, 593032, tzinfo=tzutc()), 'definition': '[]'}, {'name': 'retailimage', 'create_time': datetime.datetime(2023, 9, 28, 18, 55, 14, 954338, tzinfo=tzutc()), 'definition': '[]'}]}
{'name': 'workshop-workspace-summarization', 'id': 13, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-10-05T16:17:44.447738+00:00', 'models': [], 'pipelines': []}

Upload a Model

Now that we have our current workspace set, it’s time to start uploading some models.

We already have some ONNX models available in the folder ./models. All three do the same thing: predict house price values based on some values. We have those values stored in the folder ./data.

ML Models are uploaded to the current Wallaroo workspace the wallaroo.Client.upload_model method. Wallaroo supports different model types, as well as Arbitrary Python and containerized models.

For full details, see Wallaroo SDK Essentials Guide: Model Uploads and Registrations.

Wallaroo supports ONNX models as part of the default runtime, so these will run in Wallaroo without additional configurations.

When uploading models, the following is needed:

• The name of the model. This needs to be unique across the workspace.
• The path to the ML model file. For example, ./models/xgb_model.onnx.
• The framework of the model. These are listed through the wallaroo.framework.Framework list. For these examples we will be using wallaroo.framework.Framework.ONNX to specify we are using ONNX models.
• The input_schema and output_schema. For ONNX models, we can skip this. For non-native runtime models, that has to be specified in Apache Arrow schema format. See the Wallaroo SDK Essentials Guide: Model Uploads and Registrations for full details.

Here’s an example of uploading a model to a Wallaroo workspace and assigning it the name ‘house-price-prime’, with the Wallaroo client assigned to the variable wl, then retrieving the model version from Wallaroo once the upload it complete:

house_price_model_version = wl.upload_model('house-price-prime',
                                            './models/xgb_model.onnx',
                                            framework=wallaroo.framework.Framework.ONNX)
house_price_model_version

We store this new version of a model to the variable house_price_model_version. This is used for later processes involving pipeline deployments, generating assays, and so on.

In Wallaroo, you have the model, which is based on the name parameter. Each model has one or more versions.

IMPORTANT NOTE: Models in Wallaroo are organized by name. If a model is uploaded with the same name, it will create a new version of the model with the same name. For example, the following will create a model named house-price-prime with two versions.

# set the model from the XGB model converted to ONNX
house_price_model_version = wl.upload_model('house-price-prime',
                                            './models/xgb_model.onnx',
                                            framework=wallaroo.framework.Framework.ONNX)
print(house_price_model_version)
{'name': 'house-price-prime', 'version': '83d89260-9aac-41ea-b2b4-79aae48b5a65', 'file_name': 'xgb_model.onnx', 'image_path': None, 'last_update_time': datetime.datetime(2023, 8, 22, 19, 59, 2, 26718, tzinfo=tzutc())}

# create the new model version to the model converted from an RF model
house_price_model_version = wl.upload_model('house-price-prime',
                                            './models/rf_model.onnx',
                                            framework=wallaroo.framework.Framework.ONNX)
print(house_price_model_version)
{'name': 'house-price-prime', 'version': 'c86fd309-7c28-4e95-9d3e-831fefa51a12', 'file_name': 'rf_model.onnx', 'image_path': None, 'last_update_time': datetime.datetime(2023, 8, 22, 19, 59, 3, 381581, tzinfo=tzutc())}

Notice that the model is the same - house-price-prime - but the model version changes each time we do an upload_model. This allows you to change the model version to a totally different flavor and framework if you desire.

Upload a Model Exercise

For this exercise, upload the model models/model-auto-conversion_hugging-face_complex-pipelines_hf-summarisation-bart-large-samsun.zip and assign it a name, with the framework =wallaroo.framework.Framework.HUGGING_FACE_SUMMARIZATION.

Because this is a Hugging Face model, we need to include the input and output schemas. They are listed in the example below.

For example, if the Wallaroo client is saved to the variable wl and we want to name out model house-price-prime, we would do the following:

wl.upload_model('house-price-prime',
                './models/xgb_model.onnx',
                framework=wallaroo.framework.Framework.ONNX)

input_schema = pa.schema([
    pa.field('inputs', pa.string()),
    pa.field('return_text', pa.bool_()),
    pa.field('return_tensors', pa.bool_()),
    pa.field('clean_up_tokenization_spaces', pa.bool_()),
    # pa.field('generate_kwargs', pa.map_(pa.string(), pa.null())), # dictionaries are not currently supported by the engine
])

output_schema = pa.schema([
    pa.field('summary_text', pa.string()),
])

model = wl.upload_model('hf-summarizer', 
                        '../models/model-auto-conversion_hugging-face_complex-pipelines_hf-summarisation-bart-large-samsun.zip', 
                        framework=wallaroo.framework.Framework.HUGGING_FACE_SUMMARIZATION, 
                        input_schema=input_schema, 
                        output_schema=output_schema
                        )
model

Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime...
Model is attempting loading to a container runtime.......................successful
Ready

Retrieve Model Version

Once a model is uploaded to Wallaroo, we can list the models in a workspace with the wallaroo.workspace.models() method. This returns a List of all of the models and how many versions are associated with that model.

Here’s an example:

workspace.models()
[{'name': 'house-price-prime', 'versions': 3, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 22, 19, 59, 3, 381581, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 22, 19, 55, 26, 603685, tzinfo=tzutc())}]

We can retrieve the model by specifying the position on the list. In our case, we only have one model, so we can retrieve it to a model by specifying position 0. For example:

my_model = workspace.models()[0]
my_model

And finally - we retrieve what we really need, the model version by using the wallaroo.model.version() method, which returns the list of versions of the model:

my_model = workspace.models()[0]
my_model.versions()

[{'name': 'house-price-prime', 'version': 'cc3ba784-ffdf-4a0f-982a-9a8ac4db8ba9', 'file_name': 'xgb_model.onnx', 'image_path': None, 'last_update_time': datetime.datetime(2023, 8, 22, 19, 55, 26, 603685, tzinfo=tzutc())},
 {'name': 'house-price-prime', 'version': '83d89260-9aac-41ea-b2b4-79aae48b5a65', 'file_name': 'xgb_model.onnx', 'image_path': None, 'last_update_time': datetime.datetime(2023, 8, 22, 19, 59, 2, 26718, tzinfo=tzutc())},
 {'name': 'house-price-prime', 'version': 'c86fd309-7c28-4e95-9d3e-831fefa51a12', 'file_name': 'rf_model.onnx', 'image_path': None, 'last_update_time': datetime.datetime(2023, 8, 22, 19, 59, 3, 381581, tzinfo=tzutc())}]

As you can see - the most recent version is the last element in the list, or the [-1] position. We can now retrieve the most recent version of the model version as follows:

my_model = workspace.models()[0]
my_model_version = my_model.versions()[-1]
my_model_version

The model version is an important concept - that is what is added to a pipeline for deployments and other uses.

Retrieve Model Version Exercise

This exercise will have you retrieving the model you uploaded earlier. For example, if the Wallaroo client was stored as wl, and the workspace saved to workspace, then the command to get the current model version would be:

my_model = workspace.models()[0]
my_model_version = my_model.versions()[-1]
my_model_version

# blank space to retrieve the model version and store it

my_model = workspace.models()[0]
my_model_version = my_model.versions()[-1]
my_model_version

Build a Pipeline

Pipelines are the method of taking submitting data and processing that data through the models. Each pipeline can have one or more steps that submit the data from the previous step to the next one. Information can be submitted to a pipeline as a file, or through the pipeline’s URL.

Each pipeline step is a model version. These can be ML models like we uploaded earlier, or they can be Python scripts that manipulate the data into a format needed for another model in the chain.

When an inference is performed, data is submitted to the pipeline. The pipeline then submits the data to the first step, receives the output, then transmits that data to the next step. When all steps are complete, the pipeline returns the final values to the requesting client.

For this workshop, we will focus one a very simple pipeline: one ML model.

Pipeline are created in the current workspace with the wallaroo.Client.build_pipeline(name) command, where the name is unique to the workspace. For example, to create the pipeline named houseprice-estimator and the Wallaroo client is saved to wl the command would be:

pipeline = wl.build_pipeline('houseprice-estimator')
pipeline

name | houseprice-estimator
created | 2023-08-22 20:39:35.853683+00:00
last_updated | 2023-08-22 20:39:35.853683+00:00
deployed | (none)
tags |
versions | f42c0457-e4f3-4370-b152-0a220347de11
steps |

Just like models, pipelines have version. Each time pipeline steps are changed, a new version is created.

See Wallaroo SDK Essentials Guide: Pipeline Management for full details.

Build a Pipeline

Build your own pipeline! Use the wallaroo.Client.build_pipeline(name) command and create a pipeline named houseprice-pipeline. Recall that this creates the pipeline in the current workspace, so verify that the current workspace is the one you want to create a pipeline in.

For example, if the Wallaroo client is saved to the variable wl, the command would be:

wl.build_pipeline('houseprice-estimator')

# blank space for you to create the pipeline

wl.build_pipeline('hf-summarizer')

Retrieve a Pipeline

Pipelines that are associated with a workspace are retrieved wallaroo.Client.list_pipelines method - this returns a List of pipelines. From there, a variable is assigned to the pipeline in the list we want to work with.

The list_pipelines method returns a list of pipelines and their details as follows:

wl.list_pipelines()

Just like with a model version, we can set a variable to the pipeline by assigning it from its position in the list. In my case, if we want to retrieve the pipeline houseprice-estimator at position 0, we do so as follows:

this_pipeline = wl.list_pipelines()[0]
this_pipeline

Retrieve a Pipeline Exercise

For this exercise, retrieve your the pipeline you built in the previous step and store it into the variable my_pipeline. You’ll need to use list_pipelines() to get the list, then see where the position of the pipeline is in the list and assign it to the variable. Here’s an example if the Wallaroo client is stored in the variable wl and our pipeline is at position 0 of the list_pipelines list, the command would be:

my_pipeline = wl.list_pipelines()[0]
my_pipeline

# empty space to retrieve your pipeline

my_pipeline = wl.list_pipelines()[0]
my_pipeline

Add Model Step

Models are added to a pipeline as pipeline steps. There are different kinds of pipeline steps that can host one or more models.

For this workshop, we will use the method wallaroo.pipeline.add_model_step(model_version). This adds a single step to a Pipeline. Pipeline steps start at 0 and increment from there. We can see the steps in our pipeline with the wallaroo.pipeline.steps() method.

# add modelA as a pipeline steps
pipeline.add_model_step(modelA)

# display the steps
pipeline.steps()

[{'ModelInference': {'models': [{'name': 'house-price-prime', 'version': 'c86fd309-7c28-4e95-9d3e-831fefa51a12', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6'}]}}]

For now, there’s three commands you should know:

• wallaroo.pipelineadd_model_step(model_version): Add a step to a pipeline from a model version.
• wallaroo.pipeline.steps(): Display the current steps in the pipeline.
• wallaroo.pipeline.clear(): Clear all pipeline steps.

Pipeline steps are not saved in the Wallaroo instance until the pipeline has been deployed - more on that shortly. So you can add steps, clear them, add new ones - they all stay in your local script until you issue the command to deploy the pipeline. More on that later.

The second thing to watch out for it every time add_model_step is performed on a pipeline, another step is created. For example, if we have a pipeline with two models, modelA and modelB, then the following creates two steps in the same pipelines:

# clear the steps
pipeline.clear()

# add modelA then modelB as pipeline steps
pipeline.add_model_step(modelA)
pipeline.add_model_step(modelB)

# display the steps
pipeline.steps()

[{'ModelInference': {'models': [{'name': 'house-price-prime', 'version': 'c86fd309-7c28-4e95-9d3e-831fefa51a12', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6'}]}},
 {'ModelInference': {'models': [{'name': 'house-price-gbr', 'version': '248a6eab-b159-4821-830b-22cc137a1ace', 'sha': 'ed6065a79d841f7e96307bb20d5ef22840f15da0b587efb51425c7ad60589d6a'}]}}]

This means the data from the inference is fed first to modelA, and that output is fed into modelB. The problem is if the input data from modelA doesn’t match what modelB expects, the process will fail.

Because of this, check your pipeline steps before you deploy a pipeline, clear them if you need to.

Add Pipeline Step Exercise

We have our model version uploaded from the previous steps, and we have our pipeline. Time to put them together and create a pipeline step with our model version.

Just for practice, do the following:

1. Clear the pipeline steps.
2. Add the sample model uploaded earlier. In our examples, that was my_model_version.
3. Show the current pipeline steps.

Here’s an example with the Wallaroo client stored to wl, with the pipeline my_pipeline and my_model_version:

my_pipeline.clear()
my_pipeline.add_model_step(my_model_version)
my_pipeline.steps()

my_pipeline.clear()
my_pipeline.add_model_step(my_model_version)
my_pipeline.steps()

[{'ModelInference': {'models': [{'name': 'hf-summarizer', 'version': 'afa11be6-e3f0-4f95-996e-617c37888e5c', 'sha': 'ee71d066a83708e7ca4a3c07caf33fdc528bb000039b6ca2ef77fa2428dc6268'}]}}]

Deploy a Pipeline

Now we reach what we’ve been aiming for: deploying a pipeline.

By now, you’ve seen how workspaces contain the models, pipelines, and other artifacts. You’ve uploaded a model and retrieved the latest version of the model. You’ve built a pipeline and added the model version as a pipeline step.

Now we will deploy the pipeline. Deploying a pipeline allocated resources from the cluster to that pipeline for it’s use. The amount of resources has a default value of 4 CPUs, but for this workshop we’ll be adjusting that to just 0.5 cpus and 1 GB RAM per pipeline.

• IMPORTANT NOTE: Please stick to these resource configurations when using a Wallaroo instance with other users. Otherwise, pipelines might not deploy if the available resources are used up.

To deploy a pipeline, we do two things:

• Create a deployment configuration: This is an optional step, but we will make it mandatory for this workshop to allow other users to work in the same Wallaroo instance without running out of resources.
• Deploy the pipeline with the deployment configuration: This is the active step that saves the pipeline steps, and allocates system resources to the pipeline for performing inferences.

Deployment configurations are made with the wallaroo.DeploymentConfigBuilder() class, and then we assign the resource settings from there. This is saved to a variable so we can apply it to our pipeline deployment.

Here’s an example of setting up a deployment with just 0.5 cpu and 1Gi RAM:

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

Notice the replica_count(1) configuration - this tells Wallaroo to only spin up one replica for this pipeline. In a production environment, we could spin multiple replicas either manually or automatically as more resources are needed to improve performance.

Now we deploy the pipeline with our deployment configuration with the wallaroo.pipeline.deploy(deploy_configuration) method. If our pipeline variable is my_pipeline, then we would deploy it as follows:

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()
my_pipeline.deploy(deployment_config=deploy_config)

We can check the status of the pipeline deployment with the wallaroo.pipeline.status() method:

my_pipeline.status()

{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.83',
   'name': 'engine-6d4fccf5cb-dmwfl',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'houseprice-estimator',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'house-price-prime',
      'version': 'c86fd309-7c28-4e95-9d3e-831fefa51a12',
      'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.100',
   'name': 'engine-lb-584f54c899-sswnw',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': []}

Deploy a Pipeline Exercise

This exercise will have you deploy your pipeline with the deployment settings we listed above. For example, if your pipeline was called my_pipeline, then your deployment will look like this:

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .cpus(0.25).memory('1Gi') \
    .sidekick_cpus(model, 1) \
    .sidekick_memory(model, "4Gi") \
    .build()
my_pipeline.deploy(deployment_config=deployment_config)
my_pipeline.status()

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .cpus(0.25).memory('1Gi') \
    .sidekick_cpus(model, 1) \
    .sidekick_memory(model, "4Gi") \
    .build()
my_pipeline.deploy(deployment_config=deployment_config)
my_pipeline.status()

{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.246',
   'name': 'engine-55bf4446cc-rchxx',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'hf-summarizer',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'hf-summarizer',
      'version': 'afa11be6-e3f0-4f95-996e-617c37888e5c',
      'sha': 'ee71d066a83708e7ca4a3c07caf33fdc528bb000039b6ca2ef77fa2428dc6268',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.20',
   'name': 'engine-lb-584f54c899-bd6pz',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': [{'ip': '10.244.3.247',
   'name': 'engine-sidekick-hf-summarizer-16-78bd5b9d9f-s2fkk',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'}]}

Pipeline Inference with Files

Wallaroo deployed pipelines accept three types of data:

• JSON
• pandas DataFrames
• Apache Arrow

We do this with one of two commands on a deployed pipeline.

• wallaroo.pipeline.infer(input): Submits either JSON, a DataFrame, or Apache Arrow to the pipeline for inferences.
• wallaroo.pipeline.infer_from_file(path): Submits either a JSON, a DataFrame in pandas Record format, or an Apache Arrow binary file inferences.

We’ll start with a single input file: ./data/singleton.df.json, which contains input data as a tensor:

[
    {
        "tensor": [
            4.0,
            3.0,
            3710.0,
            20000.0,
            2.0,
            0.0,
            2.0,
            5.0,
            10.0,
            2760.0,
            950.0,
            47.6696014404,
            -122.2610015869,
            3970.0,
            20000.0,
            79.0,
            0.0,
            0.0
        ]
    }
]

When we use infer_from_file, Wallaroo determines whether the file submitted is one of the three types above, then submits it to the pipeline to perform an inference request.

The data received through the SDK is always of the same type submitted: Submit a DataFrame, get a DataFrame with the data back. Submit an Arrow table file, get an Arrow table back. Here’s an example of submitting our sample file through a pipeline saved to the variable pipeline:

result = pipeline.infer_from_file('./data/singleton.df.json')
display(result)

Let’s break down each of these fields:

• Index (unnamed): This field doesn’t have a label, but in the example above this is the index. We only have one submission, so we have one result. If we had 20 inputs, we’d have 20 inference results, and each result would be aligned with each row we sent as an input.
• time: The date and time the inference request.
• in.{variable}: Every input to the inference request is listed as in.{variable_name}. For example, if our inputs were house_size_in_square_feet and year_house_built, then the inputs would be listed as in.house_size_in_square_feet and in.year_house_built.
• out.{variable}: Every output to the inference is listed as out.{variable_name}. Our sample model outputs just one output: variable, but others such might output estimated_house_price, initial_offer_price, and in the inference result those would be listed as out.estimated_house_price and out.initial_offer_price.
• check_failures: Indicates if any validation checks failed. This is covered in later sessions.

There’s additional data that an inference has that is retrieved by requesting it. For full details, see the Wallaroo SDK Essentials Guide: Inference Management.

Pipeline Inference with Files Exercise

Let’s do an inference with two files:

• ./data/test_summarization.df.json: Inference input in DataFrame records format with one input.

Use each and perform an inference request through your deployed pipeline with the infer_from_file method. For example, if your pipeline is my_pipeline, here’s the sample inference requests:

single_result = my_pipeline.infer_from_file('../data/test_summarization.df.json', timeout=60)
display(single_result.loc[0, ['out.summary_text']])

single_result = my_pipeline.infer_from_file('../data/test_summarization.df.json', timeout=60)
display(single_result.loc[0, ['out.summary_text']])

out.summary_text    LinkedIn is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships.
Name: 0, dtype: object

Batch Inferences through API

For Wallaroo instances that enable external endpoints connections to pipelines, each pipeline has it’s own URL that can be used to perform inferences through an API call.

Performing an inference through an API requires the following:

• The authentication token to authorize the connection to the pipeline.
• The pipeline’s inference URL.
• Inference data to sent to the pipeline - in JSON, DataFrame records format, or Apache Arrow.

Full details are available through the Wallaroo API Connection Guide on how retrieve an authorization token and perform inferences through the pipeline’s API.

For this demonstration, we’ll use some Wallaroo methods to retrieve those items. We’ll use the print command to create the curl command to perform an inference on the pipeline deployment URL.

deploy_url = my_pipeline._deployment._url()

headers = wl.auth.auth_header()

headers['Content-Type']='application/json; format=pandas-records'
headers['Accept']='application/json; format=pandas-records'

dataFile = '../data/test_data.df.json'

print(f'''
!curl -X POST {deploy_url} \\
    -H "Authorization:{headers['Authorization']}" \\
    -H "Content-Type:{headers['Content-Type']}" \\
    -H "Accept:{headers['Accept']}" \\
    --data @{dataFile} > curl_response.df
      ''')

You should have an output that looks similar to this:

!curl -X POST https://doc-test.api.wallarooexample.ai/v1/api/pipelines/infer/houseprice-estimator-89/houseprice-estimator \
    -H "Authorization:Bearer SOME TOKEN STUFF" \
    -H "Content-Type:application/json; format=pandas-records" \
    -H "Accept:application/json; format=pandas-records" \
    --data @../data/test_data.df.json > curl_response.df

Then you can just run the !curl... sample. The inference results will be saved to the file curl_response.df.

Batch Inferences through API Exercise

Perform an inferece request through the curl command. Here’s some sample code to run through:

deploy_url = my_pipeline._deployment._url()

headers = wl.auth.auth_header()

headers['Content-Type']='application/json; format=pandas-records'
headers['Accept']='application/json; format=pandas-records'

dataFile = '../data/test_data.df.json'

print(f'''
!curl -X POST {deploy_url} \\
    -H "Authorization:{headers['Authorization']}" \\
    -H "Content-Type:{headers['Content-Type']}" \\
    -H "Accept:{headers['Accept']}" \\
    --data @{dataFile} > curl_response.df
      ''')

deploy_url = my_pipeline._deployment._url()

headers = wl.auth.auth_header()

headers['Content-Type']='application/json; format=pandas-records'
headers['Accept']='application/json; format=pandas-records'

dataFile = '../data/test_summarization.df.json'

print(f'''
!curl -X POST {deploy_url} \\
    -H "Authorization:{headers['Authorization']}" \\
    -H "Content-Type:{headers['Content-Type']}" \\
    -H "Accept:{headers['Accept']}" \\
    --data @{dataFile} > curl_response.df
      ''')

!curl -X POST https://doc-test.api.wallarooexample.ai/v1/api/pipelines/infer/hf-summarizer-34/hf-summarizer \
    -H "Authorization:Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJURV9fTFgzRTZHZVdkRlBXaXVwaXN0OU0ySHR3dFVZbzk2NkJCUG1CUjJvIn0.eyJleHAiOjE2OTY1Mjg0NDgsImlhdCI6MTY5NjUyODM4OCwiYXV0aF90aW1lIjoxNjk2NTI3NjYxLCJqdGkiOiI0ZTAwOTUyYS1mNGRmLTQ2ODQtYTQ5NS0zMDJiYTVlNzNkNzUiLCJpc3MiOiJodHRwczovL2RvYy10ZXN0LmtleWNsb2FrLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphL2F1dGgvcmVhbG1zL21hc3RlciIsImF1ZCI6WyJtYXN0ZXItcmVhbG0iLCJhY2NvdW50Il0sInN1YiI6ImIwMzBmZjljLTQxZWItNDliNC1hZmRmLTJjY2JlY2I2YmU1ZCIsInR5cCI6IkJlYXJlciIsImF6cCI6InNkay1jbGllbnQiLCJzZXNzaW9uX3N0YXRlIjoiNmVmOTVkNTAtZjM1ZS00YzdkLTllYjAtOTg2ZDRjMThkM2QzIiwiYWNyIjoiMSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJkZWZhdWx0LXJvbGVzLW1hc3RlciIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJtYXN0ZXItcmVhbG0iOnsicm9sZXMiOlsibWFuYWdlLXVzZXJzIiwidmlldy11c2VycyIsInF1ZXJ5LWdyb3VwcyIsInF1ZXJ5LXVzZXJzIl19LCJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6InByb2ZpbGUgZW1haWwiLCJzaWQiOiI2ZWY5NWQ1MC1mMzVlLTRjN2QtOWViMC05ODZkNGMxOGQzZDMiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsImh0dHBzOi8vaGFzdXJhLmlvL2p3dC9jbGFpbXMiOnsieC1oYXN1cmEtdXNlci1pZCI6ImIwMzBmZjljLTQxZWItNDliNC1hZmRmLTJjY2JlY2I2YmU1ZCIsIngtaGFzdXJhLWRlZmF1bHQtcm9sZSI6InVzZXIiLCJ4LWhhc3VyYS1hbGxvd2VkLXJvbGVzIjpbInVzZXIiXSwieC1oYXN1cmEtdXNlci1ncm91cHMiOiJ7fSJ9LCJuYW1lIjoiSm9obiBIYW5zYXJpY2siLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJqb2huLmh1bW1lbEB3YWxsYXJvby5haSIsImdpdmVuX25hbWUiOiJKb2huIiwiZmFtaWx5X25hbWUiOiJIYW5zYXJpY2siLCJlbWFpbCI6ImpvaG4uaHVtbWVsQHdhbGxhcm9vLmFpIn0.WGDJ1OP7CnBO3h698hg2jxsVocdOIOIJh2HKVZELuYoIt-zfLISyxDR3zMmzVwDFY7wvTygd3EhlXtwMiuo7AQDhD-rZTs-p_OWPXtnDeJAV26haxT7pN956fgjO7xwy5GOXzm0mU2CYLndDf2C0vHD9eZQoCIb7EzN8qtPxWqo4OIxtfLC0r1Mull_UiiCR82ypAyI34-Cuom31Jb8IQ4LxojPvYEq2nl0GOREwHKYHRT8RE6mnuh5e9ocAYVEjjh4OGcG80-3XM4u-H6JNPsvB3TEfLMRncrFrdPqYQY8zP9m-mIZEb7ohMMtPbNWb1u9dkYIIc_WziAJsrWKmYw" \
    -H "Content-Type:application/json; format=pandas-records" \
    -H "Accept:application/json; format=pandas-records" \
    --data @../data/test_summarization.df.json > curl_response.df

!curl -X POST https://doc-test.api.wallarooexample.ai/v1/api/pipelines/infer/hf-summarizer-34/hf-summarizer \
    -H "Authorization:Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJURV9fTFgzRTZHZVdkRlBXaXVwaXN0OU0ySHR3dFVZbzk2NkJCUG1CUjJvIn0.eyJleHAiOjE2OTY1Mjg0NDgsImlhdCI6MTY5NjUyODM4OCwiYXV0aF90aW1lIjoxNjk2NTI3NjYxLCJqdGkiOiI0ZTAwOTUyYS1mNGRmLTQ2ODQtYTQ5NS0zMDJiYTVlNzNkNzUiLCJpc3MiOiJodHRwczovL2RvYy10ZXN0LmtleWNsb2FrLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphL2F1dGgvcmVhbG1zL21hc3RlciIsImF1ZCI6WyJtYXN0ZXItcmVhbG0iLCJhY2NvdW50Il0sInN1YiI6ImIwMzBmZjljLTQxZWItNDliNC1hZmRmLTJjY2JlY2I2YmU1ZCIsInR5cCI6IkJlYXJlciIsImF6cCI6InNkay1jbGllbnQiLCJzZXNzaW9uX3N0YXRlIjoiNmVmOTVkNTAtZjM1ZS00YzdkLTllYjAtOTg2ZDRjMThkM2QzIiwiYWNyIjoiMSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJkZWZhdWx0LXJvbGVzLW1hc3RlciIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJtYXN0ZXItcmVhbG0iOnsicm9sZXMiOlsibWFuYWdlLXVzZXJzIiwidmlldy11c2VycyIsInF1ZXJ5LWdyb3VwcyIsInF1ZXJ5LXVzZXJzIl19LCJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6InByb2ZpbGUgZW1haWwiLCJzaWQiOiI2ZWY5NWQ1MC1mMzVlLTRjN2QtOWViMC05ODZkNGMxOGQzZDMiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsImh0dHBzOi8vaGFzdXJhLmlvL2p3dC9jbGFpbXMiOnsieC1oYXN1cmEtdXNlci1pZCI6ImIwMzBmZjljLTQxZWItNDliNC1hZmRmLTJjY2JlY2I2YmU1ZCIsIngtaGFzdXJhLWRlZmF1bHQtcm9sZSI6InVzZXIiLCJ4LWhhc3VyYS1hbGxvd2VkLXJvbGVzIjpbInVzZXIiXSwieC1oYXN1cmEtdXNlci1ncm91cHMiOiJ7fSJ9LCJuYW1lIjoiSm9obiBIYW5zYXJpY2siLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJqb2huLmh1bW1lbEB3YWxsYXJvby5haSIsImdpdmVuX25hbWUiOiJKb2huIiwiZmFtaWx5X25hbWUiOiJIYW5zYXJpY2siLCJlbWFpbCI6ImpvaG4uaHVtbWVsQHdhbGxhcm9vLmFpIn0.WGDJ1OP7CnBO3h698hg2jxsVocdOIOIJh2HKVZELuYoIt-zfLISyxDR3zMmzVwDFY7wvTygd3EhlXtwMiuo7AQDhD-rZTs-p_OWPXtnDeJAV26haxT7pN956fgjO7xwy5GOXzm0mU2CYLndDf2C0vHD9eZQoCIb7EzN8qtPxWqo4OIxtfLC0r1Mull_UiiCR82ypAyI34-Cuom31Jb8IQ4LxojPvYEq2nl0GOREwHKYHRT8RE6mnuh5e9ocAYVEjjh4OGcG80-3XM4u-H6JNPsvB3TEfLMRncrFrdPqYQY8zP9m-mIZEb7ohMMtPbNWb1u9dkYIIc_WziAJsrWKmYw" \
    -H "Content-Type:application/json; format=pandas-records" \
    -H "Accept:application/json; format=pandas-records" \
    --data @../data/test_summarization.df.json > curl_response.df

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2997  100  1813  100  1184     75     49  0:00:24  0:00:24 --:--:--   386-:--   517

Undeploying Your Pipeline

You should always undeploy your pipelines when you are done with them, or don’t need them for a while. This releases the resources that the pipeline is using for other processes to use. You can always redeploy the pipeline when you need it again. As a reminder, here are the commands to deploy and undeploy a pipeline:

# when the pipeline is deployed, it's ready to receive data and infer
pipeline.deploy()

# "turn off" the pipeline and releaase its resources
pipeline.undeploy()

If you are continuing on to the next notebook now, you can leave the pipeline deployed to keep working; but if you are taking a break, then you should undeploy.

## blank space to undeploy the pipeline, if needed

my_pipeline.undeploy()

Congratulations!

You have now

• Created a workspace and set it as the current workspace.
• Uploaded an ONNX model.
• Created a Wallaroo pipeline, and set the most recent version of the uploaded model as a pipeline step.
• Successfully send data to your pipeline for inference through the SDK and through an API call.

1.2 - Sentiment Analysis: Automate Data Connections

Workshop Notebook 2: Automation with Wallaroo Connections

Wallaroo Connections are definitions set by MLOps engineers that are used by other Wallaroo users for connection information to a data source.

This provides MLOps engineers a method of creating and updating connection information for data stores: databases, Kafka topics, etc. Wallaroo Connections are composed of three main parts:

• Name: The unique name of the connection.
• Type: A user defined string that designates the type of connection. This is used to organize connections.
• Details: Details are a JSON object containing the information needed to make the connection. This can include data sources, authentication tokens, etc.

Wallaroo Connections are only used to store the connection information used by other processes to create and use external connections. The user still has to provide the libraries and other elements to actually make and use the conneciton.

The primary advantage is Wallaroo connections allow scripts and other code to retrieve the connection details directly from their Wallaroo instance, then refer to those connection details. They don’t need to know what those details actually - they can refer to them in their code to make their code more flexible.

For this step, we will use a Google BigQuery dataset to retrieve the inference information, predict the next month of sales, then store those predictions into another table. This will use the Wallaroo Connection feature to create a Connection, assign it to our workspace, then perform our inferences by using the Connection details to connect to the BigQuery dataset and tables.

Prerequisites

• A Wallaroo instance version 2023.2.1 or greater.

References

• Wallaroo SDK Essentials Guide: Model Uploads and Registrations: Python Models
• Wallaroo SDK Essentials Guide: Pipeline Management
• Wallaroo SDK Essentials Guide: Data Connections Management

Preliminaries

In the blocks below we will preload some required libraries.

For convenience, the following helper functions are defined to retrieve previously created workspaces, models, and pipelines:

• get_workspace(name, client): This takes in the name and the Wallaroo client being used in this session, and returns the workspace matching name. If no workspaces are found matching the name, raises a KeyError and returns None.
• get_model_version(model_name, workspace): Retrieves the most recent model version from the model matching the model_name within the provided workspace. If no model matches that name, raises a KeyError and returns None.
• get_pipeline(pipeline_name, workspace): Retrieves the most pipeline from the workspace matching the pipeline_name within the provided workspace. If no model matches that name, raises a KeyError and returns None.

import json
import os
import datetime

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
import numpy as np

pd.set_option('display.max_colwidth', None)

import time
import pyarrow as pa

## convenience functions from the previous notebooks

# return the workspace called <name> through the Wallaroo client.
def get_workspace(name, client):
    workspace = None
    for ws in client.list_workspaces():
        if ws.name() == name:
            workspace= ws
            return workspace
    # if no workspaces were found
    if workspace==None:
        raise KeyError(f"Workspace {name} was not found.")
    return workspace

# returns the most recent model version in a workspace for the matching `model_name`
def get_model_version(model_name, workspace):
    modellist = workspace.models()
    model_version = [m.versions()[-1] for m in modellist if m.name() == model_name]
    # if no models match, return None
    if len(modellist) <= 0:
        raise KeyError(f"Model {mname} not found in this workspace")
        return None
    return model_version[0]

# get a pipeline by name in the workspace
def get_pipeline(pipeline_name, workspace):
    plist = workspace.pipelines()
    pipeline = [p for p in plist if p.name() == pipeline_name]
    if len(pipeline) <= 0:
        raise KeyError(f"Pipeline {pipeline_name} not found in this workspace")
        return None
    return pipeline[0]

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

## blank space to log in 

wl = wallaroo.Client()

Set Configurations

Set the workspace, pipeline, and model used from Notebook 1. The helper functions will make this task easier.

Set Configurations References

• Wallaroo SDK Essentials Guide: Workspace Management
• Wallaroo SDK Essentials Guide: Pipeline Management

# retrieve the previous workspace, model, and pipeline version

workspace_name = "workshop-workspace-summarization"

workspace = get_workspace(workspace_name, wl)

# set your current workspace to the workspace that you just created
wl.set_current_workspace(workspace)

model_name = 'hf-summarizer'

prime_model_version = get_model_version(model_name, workspace)

pipeline_name = 'hf-summarizer'

pipeline = get_pipeline(pipeline_name, workspace)

# display workspace, model, and pipeline

display(wl.get_current_workspace())
display(prime_model_version)
display(pipeline)

{'name': 'workshop-workspace-summarization', 'id': 13, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-10-05T16:17:44.447738+00:00', 'models': [{'name': 'hf-summarizer', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 10, 5, 16, 19, 43, 718500, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 10, 5, 16, 19, 43, 718500, tzinfo=tzutc())}], 'pipelines': [{'name': 'hf-summarizer', 'create_time': datetime.datetime(2023, 10, 5, 16, 31, 44, 8667, tzinfo=tzutc()), 'definition': '[]'}]}

Deploy the Pipeline with the Model Version Step

As per the other workshops:

1. Clear the pipeline of all steps.
2. Add the model version as a pipeline step.
3. Deploy the pipeline with the following deployment configuration:

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

pipeline.clear()
pipeline.add_model_step(prime_model_version)

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .cpus(0.25).memory('1Gi') \
    .sidekick_cpus(prime_model_version, 1) \
    .sidekick_memory(prime_model_version, "4Gi") \
    .build()
pipeline.deploy(deployment_config=deployment_config)

Create the Connection

For this demonstration, the connection set to a specific file on a GitHub repository. The connection details can be anything that can be stored in JSON: connection URLs, tokens, etc.

This connection will set a URL to pull a file from GitHub, then use the file contents to perform an inference.

Wallaroo connections are created through the Wallaroo Client create_connection(name, type, details) method. See the Wallaroo SDK Essentials Guide: Data Connections Management guide for full details.

Note that connection names must be unique across the Wallaroo instance - if needed, use random characters at the end to make sure your connection doesn’t have the same name as a previously created connection.

Here’s an example connection used to retrieve the same CSV file used in ./data/test_data.df.json: https://raw.githubusercontent.com/WallarooLabs/Workshops/main/Linear%20Regression/Real%20Estate/data/test_data.df.json

Create the Connection Exercise

# set the connection information for other steps
# suffix is used to create a unique data connection

# set the connection information for other steps
# suffix is used to create a unique data connection
suffix=''
summary_connection_input_name = f'summary-sample-connection{suffix}'
summary_connection_input_type = "HTTP"
summary_connection_input_argument = { 
    "url": "https://raw.githubusercontent.com/WallarooLabs/Workshops/20230927-foundations-v2/LLM/Summarization/data/test_summarization.df.json"
    }

wl.create_connection(summary_connection_input_name, summary_connection_input_type, summary_connection_input_argument)

# set the connection information for other steps
# suffix is used to create a unique data connection
suffix=''
summary_connection_input_name = f'summary-sample-connection{suffix}'
summary_connection_input_type = "HTTP"
summary_connection_input_argument = { 
    "url": "https://raw.githubusercontent.com/WallarooLabs/Workshops/20230927-foundations-v2/LLM/Summarization/data/test_summarization.df.json"
    }

wl.create_connection(summary_connection_input_name, summary_connection_input_type, summary_connection_input_argument)

List Connections

Connections for the entire Wallaroo instance are listed with Wallaroo Client list_connections() method.

List Connections Exercise

Here’s an example of listing the connections when the Wallaroo client is wl.

wl.list_connections()

# list the connections here

wl.list_connections()

Get Connection by Name

To retrieve a previosly created conneciton, we can assign it to a variable with the method Wallaroo Client.get_connection(connection_name). Then we can display the connection itself. Notice that when displaying a connection, the details section will be hidden, but they are retrieved with connection.details(). Here’s an example:

myconnection = client.get_connection("My amazing connection")
display(myconnection)
display(myconnection.details()

Use that code to retrieve your new connection.

Get Connection by Name Example

Here’s an example based on the Wallaroo client saved as wl.

wl.get_connection(forecast_connection_input_name)

# get the connection by name

this_connection = wl.get_connection(summary_connection_input_name)
this_connection

Add Connection to Workspace

We’ll now add the connection to our workspace so it can be retrieved by other workspace users. The method Workspace add_connection(connection_name) adds a Data Connection to a workspace. The method Workspace list_connections() displays a list of connections attached to the workspace.

Add Connection to Workspace Exercise

Use the connection we just created, and add it to the sample workspace. Here’s a code example where the workspace is saved to the variable workspace and the connection is saved as forecast_connection_input_name.

workspace.add_connection(forecast_connection_input_name)

workspace.add_connection(summary_connection_input_name)
workspace.list_connections()

Retrieve Connection from Workspace

To simulate a data scientist’s procedural flow, we’ll now retrieve the connection from the workspace. Specific connections are retrieved by specifying their position in the returned list.

For example, if we have two connections in a workspace and we want the second one, we can assign it to a variable with list_connections[1].

Create a new variable and retrieve the connection we just assigned to the workspace.

Retrieve Connection from Workspace Exercise

Retrieve the connection that was just associated with the workspace. You’ll use the list_connections method, then assign a variable to the connection. Here’s an example if the connection is the most recently one added to the workspace workspace.

forecast_connection = workspace.list_connections()[-1]

forecast_connection = workspace.list_connections()[-1]
display(forecast_connection)

Run Inference with Connection

Connections can be used for different purposes: uploading new models, engine configurations - any place that data is needed. This exercise will use the data connection to perform an inference through our deployed pipeline.

Run Inference with Connection Exercise

We’ll now retrieve sample data through the Wallaroo connection, and perform a sample inference. The connection details are retrieved through the Connection details() method. Use them to retrieve the pandas record file and convert it to a DataFrame, and use it with our sample model.

Here’s a code example that uses the Python requests library to retrieve the file information, then turns it into a DataFrame for the inference request.

display(forecast_connection.details()['url'])

import requests

response = requests.get(
                    forecast_connection.details()['url']
                )

# display(response.json())

df = pd.DataFrame(response.json())

pipeline.infer(df)

display(forecast_connection.details()['url'])

import requests

response = requests.get(
                    forecast_connection.details()['url']
                )

# display(response.json())

df = pd.DataFrame(response.json())
display(df)

multiple_result = pipeline.infer(df, timeout=60)
display(multiple_result)

'https://raw.githubusercontent.com/WallarooLabs/Workshops/20230927-foundations-v2/LLM/Summarization/data/test_summarization.df.json'

Cleaning up.

Now that the workshop is complete, don’t forget to undeploy your pipeline to free up the resources.

pipeline.undeploy()

Congratulations!

In this workshop you have:

• Deployed a single step house price prediction pipeline and sent data to it.
• Create a new Wallaroo connection.
• Assigned the connection to a workspace.
• Retrieved the connection from the workspace.
• Used the data connection to retrieve information from outside of Wallaroo, and use it for an inference.

Great job!

1.3 - LLM Summarization: Deploy Pipeline to Edge Devices

Workshop Notebook 7: Deploy Pipeline to Edge Devices

For this workshop, we will take a Wallaroo pipeline and publish it to an Open Container (OCI) Registry. The registry details are stored in the Wallaroo instance as the Edge Registry.

In this set of exercises, you will:

1. Use a pre-trained model and deploy it to Wallaroo.
2. Perform sample inferences.
3. Publish the pipeline to the Edge Registry.
4. See the steps to deploy the published pipeline to an Edge device and perform inferences through it.

Deployment to the Edge allows data scientists to work in Wallaroo to test their models in Wallaroo, then once satisfied with the results publish those pipelines. DevOps engineers then take those published pipeline details from the Edge registry and deploy them into Docker and Kubernetes environments.

This workshop will demonstrate the following concepts:

• Wallaroo Workspaces: Workspaces are environments were users upload models, create pipelines and other artifacts. The workspace should be considered the fundamental area where work is done. Workspaces are shared with other users to give them access to the same models, pipelines, etc.
• Wallaroo Model Upload and Registration: ML Models are uploaded to Wallaroo through the SDK or the MLOps API to a workspace. ML models include default runtimes (ONNX, Python Step, and TensorFlow) that are run directly through the Wallaroo engine, and containerized runtimes (Hugging Face, PyTorch, etc) that are run through in a container through the Wallaroo engine.
• Wallaroo Pipelines: Pipelines are used to deploy models for inferencing. Each model is a pipeline step in a pipelines, where the inputs of the previous step are fed into the next. Pipeline steps can be ML models, Python scripts, or Arbitrary Python (these contain necessary models and artifacts for running a model).
• Pipeline Edge Publication: How to publish a Wallaroo pipeline to an OCI registry, then deploy that pipeline into other environments.

For this tutorial, we will be providing pre-trained models in ONNX format, and have connected a sample Edge Registry to our Wallaroo instance.

For more Wallaroo procedures, see the Wallaroo Documentation site.

Preliminaries

In the blocks below we will preload some required libraries.

For convenience, the following helper functions are defined to retrieve previously created workspaces, models, and pipelines:

• get_workspace(name, client): This takes in the name and the Wallaroo client being used in this session, and returns the workspace matching name. If no workspaces are found matching the name, raises a KeyError and returns None.
• get_model_version(model_name, workspace): Retrieves the most recent model version from the model matching the model_name within the provided workspace. If no model matches that name, raises a KeyError and returns None.
• get_pipeline(pipeline_name, workspace): Retrieves the most pipeline from the workspace matching the pipeline_name within the provided workspace. If no model matches that name, raises a KeyError and returns None.

import json
import os
import datetime

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
import numpy as np

pd.set_option('display.max_colwidth', None)

import time
import pyarrow as pa

## convenience functions from the previous notebooks

# return the workspace called <name> through the Wallaroo client.
def get_workspace(name, client):
    workspace = None
    for ws in client.list_workspaces():
        if ws.name() == name:
            workspace= ws
            return workspace
    # if no workspaces were found
    if workspace==None:
        raise KeyError(f"Workspace {name} was not found.")
    return workspace

# returns the most recent model version in a workspace for the matching `model_name`
def get_model_version(model_name, workspace):
    modellist = workspace.models()
    model_version = [m.versions()[-1] for m in modellist if m.name() == model_name]
    # if no models match, return None
    if len(modellist) <= 0:
        raise KeyError(f"Model {mname} not found in this workspace")
        return None
    return model_version[0]

# get a pipeline by name in the workspace
def get_pipeline(pipeline_name, workspace):
    plist = workspace.pipelines()
    pipeline = [p for p in plist if p.name() == pipeline_name]
    if len(pipeline) <= 0:
        raise KeyError(f"Pipeline {pipeline_name} not found in this workspace")
        return None
    return pipeline[0]

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

## blank space to log in 

wl = wallaroo.Client()

Set Configurations

Set the workspace, pipeline, and model used from Notebook 1. The helper functions will make this task easier.

Set Configurations References

• Wallaroo SDK Essentials Guide: Workspace Management
• Wallaroo SDK Essentials Guide: Pipeline Management

# retrieve the previous workspace, model, and pipeline version

workspace_name = "workshop-workspace-summarization"

workspace = get_workspace(workspace_name, wl)

# set your current workspace to the workspace that you just created
wl.set_current_workspace(workspace)

model_name = 'hf-summarizer'

prime_model_version = get_model_version(model_name, workspace)

pipeline_name = 'hf-summarizer'

pipeline = get_pipeline(pipeline_name, workspace)

# display workspace, model, and pipeline

display(wl.get_current_workspace())
display(prime_model_version)
display(pipeline)

{'name': 'workshop-workspace-summarization', 'id': 13, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-10-05T16:17:44.447738+00:00', 'models': [{'name': 'hf-summarizer', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 10, 5, 16, 19, 43, 718500, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 10, 5, 16, 19, 43, 718500, tzinfo=tzutc())}], 'pipelines': [{'name': 'hf-summarizer', 'create_time': datetime.datetime(2023, 10, 5, 16, 31, 44, 8667, tzinfo=tzutc()), 'definition': '[]'}]}

Deploy the Pipeline with the Model Version Step

As per the other workshops:

1. Clear the pipeline of all steps.
2. Add the model version as a pipeline step.
3. Deploy the pipeline with the following deployment configuration:

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

pipeline.clear()
pipeline.add_model_step(prime_model_version)

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .cpus(0.25).memory('1Gi') \
    .sidekick_cpus(prime_model_version, 1) \
    .sidekick_memory(prime_model_version, "4Gi") \
    .build()
pipeline.deploy(deployment_config=deployment_config)

Sample Inference

Verify the pipeline is deployed properly with a sample inference with the file ./data/test_data.df.json.

# sample inference from previous code here

single_result = pipeline.infer_from_file('../data/test_summarization.df.json', timeout=60)
display(single_result.loc[0, ['out.summary_text']])

out.summary_text    LinkedIn is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships.
Name: 0, dtype: object

Undeploying Your Pipeline

You should always undeploy your pipelines when you are done with them, or don’t need them for a while. This releases the resources that the pipeline is using for other processes to use. You can always redeploy the pipeline when you need it again. As a reminder, here are the commands to deploy and undeploy a pipeline:

# "turn off" the pipeline and releaase its resources
my_pipeline.undeploy()

# blank space to undeploy the pipeline
pipeline.undeploy()

Publish the Pipeline for Edge Deployment

It worked! For a demo, we’ll take working once as “tested”. So now that we’ve tested our pipeline, we are ready to publish it for edge deployment.

Publishing it means assembling all of the configuration files and model assets and pushing them to an Open Container Initiative (OCI) repository set in the Wallaroo instance as the Edge Registry service. DevOps engineers then retrieve that image and deploy it through Docker, Kubernetes, or similar deployments.

See Edge Deployment Registry Guide for details on adding an OCI Registry Service to Wallaroo as the Edge Deployment Registry.

This is done through the SDK command wallaroo.pipeline.publish(deployment_config) which has the following parameters and returns.

Publish a Pipeline Parameters

The publish method takes the following parameters. The containerized pipeline will be pushed to the Edge registry service with the model, pipeline configurations, and other artifacts needed to deploy the pipeline.

Publish a Pipeline Returns

Publish the Pipeline for Edge Deployment Exercise

We will now publish the pipeline to our Edge Deployment Registry with the pipeline.publish(deployment_config) command. deployment_config is an optional field that specifies the pipeline deployment. This can be overridden by the DevOps engineer during deployment.

In this example, assuming that the pipeline was saved to the variable my_pipeline, we would publish it to the Edge Registry already stored in the Wallaroo instance and store the pipeline publish to the variable my_pub with the following command:

my_pub=pipeline.publish(deploy_config)
# display the publish
my_pub

## blank space to publish the pipeline

my_pub=pipeline.publish(deployment_config)
# display the publish
my_pub

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is Publishing..........Published.

List Published Pipelines

The method wallaroo.client.list_pipelines() shows a list of all pipelines in the Wallaroo instance, and includes the published field that indicates whether the pipeline was published to the registry (True), or has not yet been published (False).

List Published Pipelines Exercise

List all pipelines and see which ones are published or not. For example, if your client was saved to the variable wl, then the following will list the pipelines and display which ones are published.

wl.list_pipelines()

# list the pipelines and view which are published

wl.list_pipelines()

List Publishes from a Pipeline

All publishes created from a pipeline are displayed with the wallaroo.pipeline.publishes method. The pipeline_version_id is used to know what version of the pipeline was used in that specific publish. This allows for pipelines to be updated over time, and newer versions to be sent and tracked to the Edge Deployment Registry service.

List Publishes Parameters

N/A

List Publishes Returns

A List of the following fields:

List Publishes from a Pipeline Exercise

List all of the publishes from our pipeline. For example, if our pipeline is my_pipeline, then we would list all publishes from the pipeline with the following:

my_pipeline.publishes()

pipeline.publishes()

Congratulations!

You have now

• Created a workspace and set it as the current workspace.
• Uploaded an ONNX model.
• Created a Wallaroo pipeline, and set the most recent version of the uploaded model as a pipeline step.
• Successfully send data to your pipeline for inference through the SDK and through an API call.

DevOps - Pipeline Edge Deployment

Once a pipeline is deployed to the Edge Registry service, it can be deployed in environments such as Docker, Kubernetes, or similar container running services by a DevOps engineer.

Docker Deployment

First, the DevOps engineer must authenticate to the same OCI Registry service used for the Wallaroo Edge Deployment registry.

For more details, check with the documentation on your artifact service. The following are provided for the three major cloud services:

• Set up authentication for Docker
• Authenticate with an Azure container registry
• Authenticating Amazon ECR Repositories for Docker CLI with Credential Helper

For the deployment, the engine URL is specified with the following environmental variables:

{published engine url}
-e DEBUG=true -e OCI_REGISTRY={your registry server} \
-e CONFIG_CPUS=4 \ # optional number of CPUs to use
-e OCI_USERNAME={registry username} \
-e OCI_PASSWORD={registry token here} \
-e PIPELINE_URL={published pipeline url}

Docker Deployment Example

Using our sample environment, here’s sample deployment using Docker with a computer vision ML model, the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

docker run -p 8080:8080 \
    -e DEBUG=true -e OCI_REGISTRY={your registry server} \
    -e CONFIG_CPUS=4 \
    -e OCI_USERNAME=oauth2accesstoken \
    -e OCI_PASSWORD={registry token here} \
    -e PIPELINE_URL={your registry server}/pipelines/edge-cv-retail:bf70eaf7-8c11-4b46-b751-916a43b1a555 \
    {your registry server}/engine:v2023.3.0-main-3707

Docker Compose Deployment

For users who prefer to use docker compose, the following sample compose.yaml file is used to launch the Wallaroo Edge pipeline. This is the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

services:
  engine:
    image: {Your Engine URL}
    ports:
      - 8080:8080
    environment:
      PIPELINE_URL: {Your Pipeline URL}
      OCI_REGISTRY: {Your Edge Registry URL}
      OCI_USERNAME:  {Your Registry Username}
      OCI_PASSWORD: {Your Token or Password}
      CONFIG_CPUS: 4

For example:

services:
  engine:
    image: sample-registry.com/engine:v2023.3.0-main-3707
    ports:
      - 8080:8080
    environment:
      PIPELINE_URL: sample-registry.com/pipelines/edge-cv-retail:bf70eaf7-8c11-4b46-b751-916a43b1a555
      OCI_REGISTRY: sample-registry.com
      OCI_USERNAME:  _json_key_base64
      OCI_PASSWORD: abc123
      CONFIG_CPUS: 4

Docker Compose Deployment Example

The deployment and undeployment is then just a simple docker compose up and docker compose down. The following shows an example of deploying the Wallaroo edge pipeline using docker compose.

docker compose up
[+] Running 1/1
 ✔ Container cv_data-engine-1  Recreated                                                                                                                                                                 0.5s
Attaching to cv_data-engine-1
cv_data-engine-1  | Wallaroo Engine - Standalone mode
cv_data-engine-1  | Login Succeeded
cv_data-engine-1  | Fetching manifest and config for pipeline: sample-registry.com/pipelines/edge-cv-retail:bf70eaf7-8c11-4b46-b751-916a43b1a555
cv_data-engine-1  | Fetching model layers
cv_data-engine-1  | digest: sha256:c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
cv_data-engine-1  |   filename: c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
cv_data-engine-1  |   name: resnet-50
cv_data-engine-1  |   type: model
cv_data-engine-1  |   runtime: onnx
cv_data-engine-1  |   version: 693e19b5-0dc7-4afb-9922-e3f7feefe66d
cv_data-engine-1  |
cv_data-engine-1  | Fetched
cv_data-engine-1  | Starting engine
cv_data-engine-1  | Looking for preexisting `yaml` files in //modelconfigs
cv_data-engine-1  | Looking for preexisting `yaml` files in //pipelines

Helm Deployment

Published pipelines can be deployed through the use of helm charts.

Helm deployments take up to two steps - the first step is in retrieving the required values.yaml and making updates to override.

1. Pull the helm charts from the published pipeline. The two fields are the Helm Chart URL and the Helm Chart version to specify the OCI . This typically takes the format of:

helm pull oci://{published.helm_chart_url} --version {published.helm_chart_version}

1. Extract the tgz file and copy the values.yaml and copy the values used to edit engine allocations, etc. The following are required for the deployment to run:

ociRegistry:
  registry: {your registry service}
  username:  {registry username here}
  password: {registry token here}

Store this into another file, suc as local-values.yaml.

1. Create the namespace to deploy the pipeline to. For example, the namespace wallaroo-edge-pipeline would be:

kubectl create -n wallaroo-edge-pipeline

1. Deploy the helm installation with helm install through one of the following options:

   1. Specify the tgz file that was downloaded and the local values file. For example:

      helm install --namespace {namespace} --values {local values file} {helm install name} {tgz path}
      

   2. Specify the expended directory from the downloaded tgz file.

      helm install --namespace {namespace} --values {local values file} {helm install name} {helm directory path}
      

   3. Specify the Helm Pipeline Helm Chart and the Pipeline Helm Version.

      helm install --namespace {namespace} --values {local values file} {helm install name} oci://{published.helm_chart_url} --version {published.helm_chart_version}
      

2. Once deployed, the DevOps engineer will have to forward the appropriate ports to the svc/engine-svc service in the specific pipeline. For example, using kubectl port-forward to the namespace ccfraud that would be:

   kubectl port-forward svc/engine-svc -n ccfraud01 8080 --address 0.0.0.0`
   

The following code segment generates a docker run template based on the previously published pipeline, assuming our publish was listed as my_pub.

docker_deploy = f'''
docker run -p 8080:8080 \\
    -e DEBUG=true -e OCI_REGISTRY=$REGISTRYURL \\
    -e CONFIG_CPUS=4 \\
    -e OCI_USERNAME=$REGISTRYUSERNAME \\
    -e OCI_PASSWORD=$REGISTRYPASSWORD \\
    -e PIPELINE_URL={my_pub.pipeline_url} \\
    {my_pub.engine_url}
'''

print(docker_deploy)

docker run -p 8080:8080 \
    -e DEBUG=true -e OCI_REGISTRY=$REGISTRYURL \
    -e CONFIG_CPUS=4 \
    -e OCI_USERNAME=$REGISTRYUSERNAME \
    -e OCI_PASSWORD=$REGISTRYPASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/hf-summarizer:60bb46b0-52b8-464a-a379-299db4ea26c0 \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.3.0-3854

Docker Compose Deployment Exercise

Use the docker compose up command on your own compose.yaml using the sample above, replacing the OCI_USERNAME and OCI_PASSWORD with the values provided by your instructor.

Edge Deployed Pipeline API Endpoints

Once deployed, we can check the pipelines and models available. We’ll use a curl command, but any HTTP based request will work the same way.

Pipelines Endpoints

The endpoint /pipelines returns:

• id (String): The name of the pipeline.
• status (String): The status as either Running, or Error if there are any issues.

For this example, the deployment is made on a machine called testboy.local. Replace this URL with the URL of you edge deployment.

Pipelines Endpoints Exercise

Use the following curl command to view the pipeline data. For example, if the pipeline was deployed on localhost, then the command would be:

!curl locahost:8080/pipelines

# blank space to run the command - replace testboy.local with the host

!curl testboy.local:8080/pipelines

{"pipelines":[{"id":"hf-summarizer","status":"Running"}]}

Models Endpoints

The endpoint /models returns a List of models with the following fields:

• name (String): The model name.
• sha (String): The sha hash value of the ML model.
• status (String): The status of either Running or Error if there are any issues.
• version (String): The model version. This matches the version designation used by Wallaroo to track model versions in UUID format.

Models Endpoints Exercise

Use the following curl command to view the models data. For example, if the pipeline was deployed on localhost, then the command would be:

!curl locahost:8080/models

# blank space to run the command - replace testboy.local with the host

!curl testboy.local:8080/models

{"models":[{"name":"hf-summarizer","sha":"ee71d066a83708e7ca4a3c07caf33fdc528bb000039b6ca2ef77fa2428dc6268","status":"Running","version":"afa11be6-e3f0-4f95-996e-617c37888e5c"}]}

Edge Deployed Inference

The inference endpoint takes the following pattern:

• /pipelines/{pipeline-name}: The pipeline-name is the same as returned from the /pipelines endpoint as id.

Wallaroo inference endpoint URLs accept the following data inputs through the Content-Type header:

• Content-Type: application/vnd.apache.arrow.file: For Apache Arrow tables.
• Content-Type: application/json; format=pandas-records: For pandas DataFrame in record format.

It returns a application/json; format=pandas-records - the same pandas record we’ve been working with.

Edge Deployed Inference Exercise

Perform an inference on the deployed pipeline using curl. This command will look like this:

!curl -X POST localhost:8080/pipelines/{YOUR PIPELINE NAME} -H "Content-Type: application/json; format=pandas-records" --data @../data/singleton.df.json

!curl -X POST testboy.local:8080/pipelines/hf-summarizer \
    -H "Content-Type: application/json; format=pandas-records" \
    --data @../data/test_summarization.df.json

[{"time":1696530356375,"in":{"clean_up_tokenization_spaces":false,"inputs":"LinkedIn (/lɪŋktˈɪn/) is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. It is now owned by Microsoft. The platform is primarily used for professional networking and career development, and allows jobseekers to post their CVs and employers to post jobs. From 2015 most of the company's revenue came from selling access to information about its members to recruiters and sales professionals. Since December 2016, it has been a wholly owned subsidiary of Microsoft. As of March 2023, LinkedIn has more than 900 million registered members from over 200 countries and territories. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships. Members can invite anyone (whether an existing member or not) to become a connection. LinkedIn can also be used to organize offline events, join groups, write articles, publish job postings, post photos and videos, and more","return_tensors":false,"return_text":true},"out":{"summary_text":"LinkedIn is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships."},"check_failures":[],"metadata":{"last_model":"{\"model_name\":\"hf-summarizer\",\"model_sha\":\"ee71d066a83708e7ca4a3c07caf33fdc528bb000039b6ca2ef77fa2428dc6268\"}","pipeline_version":"","elapsed":[209624,3002227645],"dropped":[]}}]