In-Line Model Updates at the Edge Tutorial

A demonstration performing inline model replacements on edge location deployments.
Features:
Models:

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

In-Line Model Updates at the Edge Tutorial

Wallaroo pipeline publishes are containerized versions of the pipeline, models and model steps, and the inference engine published to an Open Container Initiative (OCI) Registry. Once published, they are used to deploy models to edge devices and used for inference requests.

Edge locations added to a pipeline published allow deployments on edge devices to connect with the Wallaroo Ops instance and transmit their inference results as part of the pipeline logs.

Pipeline publishes can be replaced by new pipeline publishes from the following sources:

  • A new version of the same pipeline that creates a new pipeline publish. In these instances, the edge locations associated with the pipeline publish are preserved with the original pipeline along with their pipeline inference logs. Any input or output schema changes from the new models or models steps are reflected in the pipeline logs.
  • A separate pipeline with its own models and model steps. In this scenario, the edge locations associated with the original pipeline publish are assigned to the new pipeline. The inference logs are stored with the original pipeline, and new inference logs for the edge locations are stored as part of the new pipeline.

For more information on pipeline inference logs, see Inference Logs.

  • IMPORTANT NOTE: Assays and other log data is based on the pipeline and model name. If these values change, assays that are currently running will no longer generate analyses. These should be paused, and new assays based on the current pipeline and model name should be created. For more information, see Model Drift Detection with Model Insights.

Goal

  • Package a model in Wallaroo and perform sample inferences. Publish the pipeline and create an edge location, then deploy the model to an edge location.
  • Perform sample inferences on the edge location and verify the inference logs are added to the pipeline associated with the edge location.
  • Create a new pipeline version with a new model step and perform sample inferences. Update the publish with the new pipeline version, and verify that the model is deployed to the edge location deployment.
  • Perform sample inferences on the edge location to verify the new model is being used, and verify the inference logs are added to the pipeline associated with the edge location.
  • Create a new pipeline with a different model and perform sample inferences. Publish the pipeline, then replace the publish used for the edge location. Verify that the edge location is transferred to the new pipeline.
  • Perform sample inferences on the edge location to verify model is updated from the new pipeline is being used, and verify the inference logs are added to the new pipeline associated with the edge location.

Resources

This tutorial provides the following:

  • Models:
    • models/rf_model.onnx: The champion model that predicts house prices.
    • models/xgb_model.onnx: A new challenger model that takes the same inputs as models/rf_model.onnx and outputs house prices with a field of the same name.
    • models/gbr_model.onnx: A new challenger model that takes the same inputs as models/rf_model.onnx and outputs house prices with a field of the same name.

Prerequisites

  • A deployed Wallaroo instance with Edge Registry Services and Edge Observability enabled.
  • The following Python libraries installed:
    • wallaroo: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.
    • pandas: Pandas, mainly used for Pandas DataFrame
  • A X64 Docker deployment to deploy the model on an edge location.

Edge Deployment Preparation Steps

The following is used to set up the environment that is used for the pipeline publish replacement and in-line model updates on an edge tutorial.

In these steps, we will:

  • Deploying a sample ML model used to determine house prices based on a set of input parameters.
  • Publish the model deployment configuration to an OCI registry.
  • Use the publish and set edge locations.
  • Deploy the model in an edge location and perform sample inferences.

Import Libraries

The first step will be to import our libraries, and set variables used through this tutorial.

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

from IPython.display import display

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)

import json
import time

workspace_name = 'in-line-edge-replacement-demo'

# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()

Create Workspace

We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace.

Workspace names are unique across the Wallaroo instance. If a user requests a workspace they have not been granted access to, an error message is returned.

workspace = wl.get_workspace(name=workspace_name, create_if_not_exist=True)

wl.set_current_workspace(workspace)

{'name': 'in-line-edge-replacement-demo', 'id': 5, 'archived': False, 'created_by': '76b893ff-5c30-4f01-bd9e-9579a20fc4ea', 'created_at': '2024-04-22T15:27:33.76544+00:00', 'models': [], 'pipelines': []}

Upload The Champion Model

For our example, we will upload the champion model that has been trained to derive house prices from a variety of inputs. The model file is rf_model.onnx, and is uploaded with the name rf-house-price-estimator.

housing_model_control = (wl.upload_model("rf-house-price-estimator", 
                                        './models/rf_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

Build and Deploy the Pipeline

This pipeline is made to be an example of an existing situation where a model is deployed and being used for inferences in a production environment. We’ll call it edge-inline-replacement-demonstration, set housing_model_control as a pipeline step, then run a sample inference.

mainpipeline = wl.build_pipeline('edge-inline-replacement-demon')

# undeploy if already created earlier
mainpipeline.undeploy()
# clear the steps if used before
mainpipeline.clear()

mainpipeline.add_model_step(housing_model_control)
nameedge-inline-replacement-demon
created2024-04-22 15:27:36.828826+00:00
last_updated2024-04-22 15:27:36.828826+00:00
deployed(none)
archNone
accelNone
tags
versionsfbc4bf00-d97f-4be1-a47c-85c788dd90d5
steps
publishedFalse
#minimum deployment config
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

mainpipeline.deploy(deployment_config = deploy_config)
nameedge-inline-replacement-demon
created2024-04-22 15:27:36.828826+00:00
last_updated2024-04-22 15:27:39.074449+00:00
deployedTrue
archx86
accelnone
tags
versions2ef51c5c-bc58-49b3-9ecf-9aa4bb0a0bae, fbc4bf00-d97f-4be1-a47c-85c788dd90d5
stepsrf-house-price-estimator
publishedFalse

Inference Request Testing

We’ll use a single row inference request that should return a result of around $700k.

normal_input = pd.DataFrame.from_records({"tensor": [[4.0, 
                                                      2.5, 
                                                      2900.0, 
                                                      5505.0, 
                                                      2.0, 
                                                      0.0, 
                                                      0.0, 
                                                      3.0, 
                                                      8.0, 
                                                      2900.0, 
                                                      0.0, 
                                                      47.6063, 
                                                      -122.02, 
                                                      2970.0, 
                                                      5251.0, 
                                                      12.0, 
                                                      0.0, 
                                                      0.0]]})
result = mainpipeline.infer(normal_input)
display(result)
timein.tensorout.variableanomaly.count
02024-04-22 15:27:54.780[4.0, 2.5, 2900.0, 5505.0, 2.0, 0.0, 0.0, 3.0, 8.0, 2900.0, 0.0, 47.6063, -122.02, 2970.0, 5251.0, 12.0, 0.0, 0.0][718013.7]0

Edge Deployment

We can now deploy the original pipeline to an edge device. This will require the following steps:

  • Publish the pipeline: Publishes the pipeline to the OCI registry.
  • Add Edge: Add the edge location to the pipeline publish.
  • Deploy Edge: Deploy the edge device with the edge location settings.

Publish Pipeline

Publishing the pipeline uses the pipeline wallaroo.pipeline.publish() command. This requires that the Wallaroo Ops instance have Edge Registry Services enabled.

The following publishes the pipeline to the OCI registry and displays the container details. For more information, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.

pub = mainpipeline.publish()

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing....... Published.

Add Edge Location

The edge location is added with the wallaroo.pipeline_publish.add_edge(name) method. This returns the OCI registration information, and the EDGE_BUNDLE information. The EDGE_BUNDLE data is a base64 encoded set of parameters for the pipeline that the edge device is associated with, the workspace, and other data.

For full details, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication: Edge Observability.

For this example, we will add two locations:

  • houseprice-edge-inline-demo

These will be used in later steps for demonstrating inferences through different locations.

edge_name_01 = "houseprice-edge-inline-demonstration"
edge_publish_01 = pub.add_edge(edge_name_01)
display(edge_publish_01)
ID1
Pipeline Nameedge-inline-replacement-demon
Pipeline Version7c702eca-8acf-45e0-bdcd-bbb48a5102e5
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demon:7c702eca-8acf-45e0-bdcd-bbb48a5102e5
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/edge-inline-replacement-demon
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:56b14a042d69a3a59045c4a07a62326dc6cfde1a07ccb22e5d7f831425e414bc
Helm Chart Version0.0.1-7c702eca-8acf-45e0-bdcd-bbb48a5102e5
Engine Config{'engine': {'resources': {'limits': {'cpu': 4.0, 'memory': '3Gi'}, 'requests': {'cpu': 4.0, 'memory': '3Gi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': None}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2024-04-22 15:27:55.628456+00:00
Updated At2024-04-22 15:27:55.628456+00:00
Replaces
Docker Run Command
docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246N2M3MDJlY2EtOGFjZi00NWUwLWJkY2QtYmJiNDhhNTEwMmU1CmV4cG9ydCBKT0lOX1RPS0VOPTM4NjdmNjJlLWQyZTMtNDFjZC1iMGMyLTc2ZWM4NWUyZjBhMApleHBvcnQgT0NJX1JFR0lTVFJZPWdoY3IuaW8=\
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demon:7c702eca-8acf-45e0-bdcd-bbb48a5102e5 \
    -e CONFIG_CPUS=4 ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963

Note: Please set the PERSISTENT_VOLUME_DIR, EDGE_PORT, OCI_USERNAME, and OCI_PASSWORD environment variables.
Helm Install Command
helm install --atomic $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-inline-replacement-demon \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-7c702eca-8acf-45e0-bdcd-bbb48a5102e5 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246N2M3MDJlY2EtOGFjZi00NWUwLWJkY2QtYmJiNDhhNTEwMmU1CmV4cG9ydCBKT0lOX1RPS0VOPTM4NjdmNjJlLWQyZTMtNDFjZC1iMGMyLTc2ZWM4NWUyZjBhMApleHBvcnQgT0NJX1JFR0lTVFJZPWdoY3IuaW8=

Note: Please set the HELM_INSTALL_NAME, HELM_INSTALL_NAMESPACE, OCI_USERNAME, and OCI_PASSWORD environment variables.

DevOps Deployment

The following shows two methods of model deployments on edge devices. For more details on model edge deployments using Docker or helm with Wallaroo, see Model Operations: Run Anywhere.

Docker Run Based Deployment

For the model deployment to the edge location via docker run, use the Docker Run Command displayed with the pipeline publish. The following variables must be set first.

  • PERSISTENT_VOLUME_DIR: The file path for the persistent volume to store session data.
  • EDGE_PORT: The external port to connect to, typically 8080.
  • OCI_USERNAME: The username for the OCI registry service.
  • OCI_PASSWORD: The password or token for the OCI registry service.
  • `

Before deploying the pipeline, verify that Docker is able to connect to the OCI registry with the following command.

docker login -u $OCI_USERNAME --password $OCI_PASSWORD (Your registry URL)

Edge Inference Example

We will perform sample inferences on the current model edge deployment to demonstrate how the inferences change as the models are updated through the in-line edge deployments.

With the model deployed on the edge location, we will perform sample inferences on the edge deployment. For this example, the hostname is testboy.local. Adjust the hostname to fit your edge location deployment.

The following endpoints are available.

EndpointTypeDescription
/modelsGETReturns the models and model versions used in the edge deployment.
/pipelinesGETReturns the pipeline ID and status of the pipeline used in the edge deployment.
/inferPOSTThe inference request endpoint. This accepts either an Apache Arrow table, or a JSON in pandas record format.

For more details on model edge deployments with Wallaroo, see Model Operations: Run Anywhere.

First we’ll retrieve the model. We should see the same name we set when we uploaded it to the Wallaroo instance.

!curl testboy.local:8080/models

{"models":[{"sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6","name":"rf-house-price-estimator","version":"02f05eac-5b37-4784-84a4-5db7ab95b65b","status":"Running"}]}

Now we’ll retrieve the pipeline details, and verify the name is the same as set earlier and it is running.

!curl testboy.local:8080/pipelines

{"pipelines":[{"id":"edge-inline-replacement-demon","version":"7c702eca-8acf-45e0-bdcd-bbb48a5102e5","status":"Running"}]}

Now do the infer with the same DataFrame we used for our sample inference when the model was deployed in the Wallaroo instance.

We use the endpoint /infer; this endpoint remains the same even if the models and the pipeline changes, which is demonstrated in the later steps.

import json

df = pd.DataFrame.from_records({"tensor": [[4.0, 
                                            2.5, 
                                            2900.0, 
                                            5505.0, 
                                            2.0, 
                                            0.0, 
                                            0.0, 
                                            3.0, 
                                            8.0, 
                                            2900.0, 
                                            0.0, 
                                            47.6063, 
                                            -122.02, 
                                            2970.0, 
                                            5251.0, 
                                            12.0, 
                                            0.0, 
                                            0.0]]})

data = df.to_dict(orient="records")

!curl -X POST testboy.local:8080/infer \
    -H "Content-Type: Content-Type: application/json; format=pandas-records" \
    --data '{json.dumps(data)}' > ./inferenceoutput.df.json

# get the dataframe from what we just did

df_result = pd.read_json('./inferenceoutput.df.json', orient="records")
df_result.loc[:, ['time', 'out', 'metadata']]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   628  100   500  100   128   1811    463 --:--:-- --:--:-- --:--:--  2283
timeoutmetadata
01713800294502{'variable': [718013.7]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '7c702eca-8acf-45e0-bdcd-bbb48a5102e5', 'elapsed': [142461514, 55185536], 'dropped': [], 'partition': 'houseprice-edge-inline-demonstration'}

How to Replace a Pipeline Publish

Pipeline publishes are replaced with the wallaroo.pipeline.Pipeline.publish() method. This creates a new publish from the most recent pipeline version and takes the following parameters.

ParameterTypeDescription
deployment_configwallaroo.deployment_config.DeploymentConfig (Optional)The deployment configuration used for the edge deployment. By default, this is the same deployment configuration used for the pipeline.
replaces[List[wallaroo.pipeline_publish]] (Optional)The pipeline publish(es) to replace.

When a pipeline published is replaced with a new one, the edge locations are transferred to the pipeline that the publish came from. In this example, this is the same pipeline. Inference results from the edge location deployments are stored with the pipeline that generated the publish. Note that if the model or pipeline steps have changed from one pipeline version to the next, the pipeline log schema will change with it. For more information, see Wallaroo SDK Essentials Guide: Pipeline Log Management.

Verify Pipeline Publishes and Edge Locations

We verify that the pipeline is published and is assigned edge locations via the wallaroo.pipeline.Pipeline.publishes() method and wallaroo.pipeline.Pipeline.list_edges() method.

display('Publishes:')
display(mainpipeline.publishes())
display('Edge Locations')
display(mainpipeline.list_edges())

'Publishes:'
idpipeline_version_nameengine_urlpipeline_urlcreated_bycreated_atupdated_at
17c702eca-8acf-45e0-bdcd-bbb48a5102e5ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demon:7c702eca-8acf-45e0-bdcd-bbb48a5102e5john.hummel@wallaroo.ai2024-22-Apr 15:27:552024-22-Apr 15:27:55
'Edge Locations'
IDNameTagsSPIFFE ID
e3c7946b-24f9-4c20-b57a-46ad3c1d66e2houseprice-edge-inline-demonstration[]wallaroo.ai/ns/deployments/edge/e3c7946b-24f9-4c20-b57a-46ad3c1d66e2

How to Replace a Pipeline Publish from a New Pipeline Version

The following demonstrates replacing a pipeline publish with a new publish from a pipeline version. In this procedure, the edge location has an existing model edge deployment. The publish that is used for that deployment is replaced, and an in-line update of the model deployment is performed on the edge device.

Create New Pipeline Version

For the next demonstration, we will use the same pipeline and update the pipeline steps with a new model. This time with the model ./models/xgb_model.onnx, which takes the same input as ./models/rf_model.onnx. We will also set a different name for this model to distinguish it from the one previously uploaded and used as the pipeline step.

Once set, we will deploy the pipeline with the new model, and perform a new inference request. From this, we will see that this new model outputs a slightly different prediction that the previous one - closer to $650k.

housing_model_challenger01 = (wl.upload_model("xgb-house-price-estimator", 
                                        './models/xgb_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

# undeploy if already created earlier
mainpipeline.undeploy()
# clear the steps if used before
mainpipeline.clear()

mainpipeline.add_model_step(housing_model_challenger01)
nameedge-inline-replacement-demon
created2024-04-22 15:27:36.828826+00:00
last_updated2024-04-22 15:27:54.932609+00:00
deployedFalse
archx86
accelnone
tags
versions7c702eca-8acf-45e0-bdcd-bbb48a5102e5, 2ef51c5c-bc58-49b3-9ecf-9aa4bb0a0bae, fbc4bf00-d97f-4be1-a47c-85c788dd90d5
stepsrf-house-price-estimator
publishedFalse
#minimum deployment config
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

mainpipeline.deploy(deployment_config = deploy_config)
mainpipeline.steps()

[{'ModelInference': {'models': [{'name': 'xgb-house-price-estimator', 'version': 'a86668be-03ed-47f8-8f49-9e31fe502566', 'sha': '31e92d6ccb27b041a324a7ac22cf95d9d6cc3aa7e8263a229f7c4aec4938657c'}]}}]

normal_input = pd.DataFrame.from_records({"tensor": [[4.0, 
                                                      2.5, 
                                                      2900.0, 
                                                      5505.0, 
                                                      2.0, 
                                                      0.0, 
                                                      0.0, 
                                                      3.0, 
                                                      8.0, 
                                                      2900.0, 
                                                      0.0, 
                                                      47.6063, 
                                                      -122.02, 
                                                      2970.0, 
                                                      5251.0, 
                                                      12.0, 
                                                      0.0, 
                                                      0.0]]})
result = mainpipeline.infer(normal_input)
display(result)
timein.tensorout.variableanomaly.count
02024-04-22 15:39:30.567[4.0, 2.5, 2900.0, 5505.0, 2.0, 0.0, 0.0, 3.0, 8.0, 2900.0, 0.0, 47.6063, -122.02, 2970.0, 5251.0, 12.0, 0.0, 0.0][659806.0]0

Replace Pipeline Publish with new Pipeline Version

From our pipeline, we will publish the new version. The method wallaroo.pipeline.Pipeline.publish() creates a new publish from the most recent pipeline version and takes the following parameters.

We will replace our publish earlier, labeled pub, with the new publish generated from the pipeline labeled new_pub.

The details on deploying the replaced pipeline publish(es) is under the Replaces section.

new_pub = mainpipeline.publish(replaces=[pub])
new_pub

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing....... Published.
ID2
Pipeline Nameedge-inline-replacement-demon
Pipeline Versionc6a1e945-7de0-4f2c-addb-4f4746114a86
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demon:c6a1e945-7de0-4f2c-addb-4f4746114a86
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/edge-inline-replacement-demon
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:b5cbf51760fcf319c2c5610346fa9bc55047340509d94091eac36ec2beec9158
Helm Chart Version0.0.1-c6a1e945-7de0-4f2c-addb-4f4746114a86
Engine Config{'engine': {'resources': {'limits': {'cpu': 4.0, 'memory': '3Gi'}, 'requests': {'cpu': 4.0, 'memory': '3Gi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': None}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2024-04-22 15:40:51.411106+00:00
Updated At2024-04-22 15:40:51.411106+00:00
ReplacesPublish 1, Pipeline "edge-inline-replacement-demon", Version 3
Docker Run Command
EdgeCommand
houseprice-edge-inline-demonstration
docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demon:c6a1e945-7de0-4f2c-addb-4f4746114a86\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246YzZhMWU5NDUtN2RlMC00ZjJjLWFkZGItNGY0NzQ2MTE0YTg2CmV4cG9ydCBKT0lOX1RPS0VOPW9taXR0ZWRfZm9yX3JlcGxhY2UKZXhwb3J0IE9DSV9SRUdJU1RSWT1naGNyLmlv \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963

Note: Please set the PERSISTENT_VOLUME_DIR, EDGE_PORT, OCI_USERNAME, and OCI_PASSWORD environment variables.
Helm Install Command
EdgeCommand
houseprice-edge-inline-demonstration
helm upgrade $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-inline-replacement-demon \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-c6a1e945-7de0-4f2c-addb-4f4746114a86 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246YzZhMWU5NDUtN2RlMC00ZjJjLWFkZGItNGY0NzQ2MTE0YTg2CmV4cG9ydCBKT0lOX1RPS0VOPW9taXR0ZWRfZm9yX3JlcGxhY2UKZXhwb3J0IE9DSV9SRUdJU1RSWT1naGNyLmlv

Note: Please set the HELM_INSTALL_NAME, HELM_INSTALL_NAMESPACE, OCI_USERNAME, and OCI_PASSWORD environment variables.

Note that in the Replaces section, updates are given for each edge location. On the edge device, we deploy the new pipeline publish either by the provided Docker Run Command or Helm Install Command. Note that the Helm Install Command uses helm upgrade rather than helm install to create a new revision. The pipeline steps and models will be changed at the model edge location.

Before we do so, we will verify the edge locations associated for our pipelines. mainpipeline will still have its edges, since we only updated the pipeline version, while new_pipeline has no edges yet.

Inference Requests from Replaced Pipeline Version Deployment Examples

We deploy the in-line model update in our new publish above for the edge location, then perform the same checks of the models, pipeline, then a sample inference using the same inference input data as before.

We will see the new model name and model version used.

!curl testboy.local:8080/models

{"models":[{"sha":"31e92d6ccb27b041a324a7ac22cf95d9d6cc3aa7e8263a229f7c4aec4938657c","name":"xgb-house-price-estimator","version":"a86668be-03ed-47f8-8f49-9e31fe502566","status":"Running"}]}

!curl testboy.local:8080/pipelines

{"pipelines":[{"id":"edge-inline-replacement-demon","version":"c6a1e945-7de0-4f2c-addb-4f4746114a86","status":"Running"}]}

We now perform the same inference request on the edge location deployment, and show the results match that as the pipeline with the new model.

import json

df = pd.DataFrame.from_records({"tensor": [[4.0, 
                                            2.5, 
                                            2900.0, 
                                            5505.0, 
                                            2.0, 
                                            0.0, 
                                            0.0, 
                                            3.0, 
                                            8.0, 
                                            2900.0, 
                                            0.0, 
                                            47.6063, 
                                            -122.02, 
                                            2970.0, 
                                            5251.0, 
                                            12.0, 
                                            0.0, 
                                            0.0]]})

data = df.to_dict(orient="records")

!curl -X POST testboy.local:8080/infer \
    -H "Content-Type: Content-Type: application/json; format=pandas-records" \
    --data '{json.dumps(data)}' > inferenceoutput.df.json

# get the dataframe from what we just did

df_result = pd.read_json('./inferenceoutput.df.json', orient="records")
df_result.loc[:, ['time', 'out', 'metadata']]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   624  100   496  100   128  30468   7862 --:--:-- --:--:-- --:--:-- 39000
timeoutmetadata
01713800558148{'variable': [659806.0]}{'last_model': '{"model_name":"xgb-house-price-estimator","model_sha":"31e92d6ccb27b041a324a7ac22cf95d9d6cc3aa7e8263a229f7c4aec4938657c"}', 'pipeline_version': 'c6a1e945-7de0-4f2c-addb-4f4746114a86', 'elapsed': [359571, 613994], 'dropped': [], 'partition': 'houseprice-edge-inline-demonstration'}

How to Replace a Pipeline Publish from a New Pipeline

For this example, we will create a separate pipeline than the previous one with its own model. This model will still accept the same inputs and output fields as the models previously deployed. This is not a requirement - completely different models with different inputs and output schemas can be used; this example is used for simplicity.

For this set of examples, we will upload the third model, ./models/gbr_model.onnx and create a new pipeline named new-edge-inline-replacement. This model is added as a pipeline step and deployed.

housing_model_challenger02 = (wl.upload_model("gbr-house-price-estimator", 
                                        './models/gbr_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

new_pipeline = wl.build_pipeline("new-edge-inline-replacement")

# undeploy if used before
new_pipeline.undeploy()
# clear the steps if used before
new_pipeline.clear()

new_pipeline.add_model_step(housing_model_challenger02)

#minimum deployment config
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

new_pipeline.deploy(deployment_config = deploy_config)
new_pipeline.steps()

[{'ModelInference': {'models': [{'name': 'gbr-house-price-estimator', 'version': '18b322c6-7681-44c5-a607-d8ec1f021651', 'sha': 'ed6065a79d841f7e96307bb20d5ef22840f15da0b587efb51425c7ad60589d6a'}]}}]

normal_input = pd.DataFrame.from_records({"tensor": [[4.0, 
                                                      2.5, 
                                                      2900.0, 
                                                      5505.0, 
                                                      2.0, 
                                                      0.0, 
                                                      0.0, 
                                                      3.0, 
                                                      8.0, 
                                                      2900.0, 
                                                      0.0, 
                                                      47.6063, 
                                                      -122.02, 
                                                      2970.0, 
                                                      5251.0, 
                                                      12.0, 
                                                      0.0, 
                                                      0.0]]})
result = new_pipeline.infer(normal_input)
display(result)
timein.tensorout.variableanomaly.count
02024-04-22 15:43:20.716[4.0, 2.5, 2900.0, 5505.0, 2.0, 0.0, 0.0, 3.0, 8.0, 2900.0, 0.0, 47.6063, -122.02, 2970.0, 5251.0, 12.0, 0.0, 0.0][704901.9]0

Replace Pipeline Publish with Publish from Different Pipepline

We will now publish the new pipeline, and replace the previous edge location publish from our original pipeline. The last publish is labeled new_pub is labeled new_pipeline_pub.

new_pipeline_pub = new_pipeline.publish(replaces=[new_pub])
new_pipeline_pub

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing....... Published.
ID3
Pipeline Namenew-edge-inline-replacement
Pipeline Version455a7840-08c3-43bb-b6a6-7535894e6055
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/new-edge-inline-replacement:455a7840-08c3-43bb-b6a6-7535894e6055
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/new-edge-inline-replacement
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:e6d53f03950ffeb7ecf623a5ac43f6d5d65a4edf935b43d1b038f629bf9c0a0b
Helm Chart Version0.0.1-455a7840-08c3-43bb-b6a6-7535894e6055
Engine Config{'engine': {'resources': {'limits': {'cpu': 4.0, 'memory': '3Gi'}, 'requests': {'cpu': 4.0, 'memory': '3Gi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': None}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2024-04-22 15:44:47.149059+00:00
Updated At2024-04-22 15:44:47.149059+00:00
ReplacesPublish 2, Pipeline "edge-inline-replacement-demon", Version 5
Docker Run Command
EdgeCommand
houseprice-edge-inline-demonstration
docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/new-edge-inline-replacement:455a7840-08c3-43bb-b6a6-7535894e6055\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvbmV3LWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50OjQ1NWE3ODQwLTA4YzMtNDNiYi1iNmE2LTc1MzU4OTRlNjA1NQpleHBvcnQgSk9JTl9UT0tFTj1vbWl0dGVkX2Zvcl9yZXBsYWNlCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw== \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963

Note: Please set the PERSISTENT_VOLUME_DIR, EDGE_PORT, OCI_USERNAME, and OCI_PASSWORD environment variables.
Helm Install Command
EdgeCommand
houseprice-edge-inline-demonstration
helm upgrade $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/new-edge-inline-replacement \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-455a7840-08c3-43bb-b6a6-7535894e6055 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvbmV3LWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50OjQ1NWE3ODQwLTA4YzMtNDNiYi1iNmE2LTc1MzU4OTRlNjA1NQpleHBvcnQgSk9JTl9UT0tFTj1vbWl0dGVkX2Zvcl9yZXBsYWNlCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw==

Note: Please set the HELM_INSTALL_NAME, HELM_INSTALL_NAMESPACE, OCI_USERNAME, and OCI_PASSWORD environment variables.

Once complete, we update the edge deployment with the new replacement docker run or helm based deployment commands from the configuration.

With the edge location replacement complete with the new pipeline, we list the edge locations from the original pipeline and the new one to show the edge location is transferred to the new pipeline.

display(mainpipeline.list_edges())

(no pipelines)

display(new_pipeline.list_edges())
IDNameTagsSPIFFE ID
e3c7946b-24f9-4c20-b57a-46ad3c1d66e2houseprice-edge-inline-demonstration[]wallaroo.ai/ns/deployments/edge/e3c7946b-24f9-4c20-b57a-46ad3c1d66e2

Sample Inference Requests with New Pipeline Publish on Edge Location

We deploy the new publish on the edge device either with the Docker Run Command or the Helm Install Command, with whatever modifications are required on the edge device. Once complete, we run through the /models and /pipelines endpoints, then perform an inference request. We will see the name of the new model - and the new pipeline used for the same edge location.

!curl testboy.local:8080/models

{"models":[{"sha":"ed6065a79d841f7e96307bb20d5ef22840f15da0b587efb51425c7ad60589d6a","name":"gbr-house-price-estimator","version":"18b322c6-7681-44c5-a607-d8ec1f021651","status":"Running"}]}

!curl testboy.local:8080/pipelines

{"pipelines":[{"id":"new-edge-inline-replacement","version":"455a7840-08c3-43bb-b6a6-7535894e6055","status":"Running"}]}

# new inference

import json

df = pd.DataFrame.from_records({"tensor": [[4.0, 
                                            2.5, 
                                            2900.0, 
                                            5505.0, 
                                            2.0, 
                                            0.0, 
                                            0.0, 
                                            3.0, 
                                            8.0, 
                                            2900.0, 
                                            0.0, 
                                            47.6063, 
                                            -122.02, 
                                            2970.0, 
                                            5251.0, 
                                            12.0, 
                                            0.0, 
                                            0.0]
                                            ]
                                })

data = df.to_dict(orient="records")

!curl -X POST testboy.local:8080/infer \
    -H "Content-Type: Content-Type: application/json; format=pandas-records" \
    --data '{json.dumps(data)}' > inferenceoutput.df.json

# get the dataframe from what we just did

df_result = pd.read_json('./inferenceoutput.df.json', orient="records")
df_result.loc[:, ['time', 'out', 'metadata']]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   624  100   496  100   128  25180   6498 --:--:-- --:--:-- --:--:-- 32842
timeoutmetadata
01713800828863{'variable': [704901.9]}{'last_model': '{"model_name":"gbr-house-price-estimator","model_sha":"ed6065a79d841f7e96307bb20d5ef22840f15da0b587efb51425c7ad60589d6a"}', 'pipeline_version': '455a7840-08c3-43bb-b6a6-7535894e6055', 'elapsed': [337802, 597249], 'dropped': [], 'partition': 'houseprice-edge-inline-demonstration'}

With the tutorial complete, we undeploy the pipelines and return the resources back to the cluster.

mainpipeline.undeploy()
new_pipeline.undeploy()
namenew-edge-inline-replacement
created2024-04-22 15:43:04.574641+00:00
last_updated2024-04-22 15:44:46.403831+00:00
deployedFalse
archx86
accelnone
tags
versions455a7840-08c3-43bb-b6a6-7535894e6055, 5210e01e-6d0f-4cdc-92ea-d3499bcc42fc, d61fcf2d-95ad-41e7-9e53-50610d9e0419
stepsgbr-house-price-estimator
publishedFalse